Large mysql dump pipe to s3

John W. Nesk asked:

Is there any problem in doing this with lots of data?

mysqldump ... | gzip | s3cmd put - s3://bucket/file.sql.gz

The MySQL dump is about 100GB in size. What happens if gzip or s3cmd can’t process the dump fast enough? Will it overflow the pipe buffer?

My answer:


The writer will block waiting for the reader to become ready.

What you really should worry about is what happens if s3cmd fails. Then you have to start all over.


View the full question and answer on Server Fault.

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License.