Stormbackup

This is a list of storm errors and potential fixes for them.

Tune2fs Error's

Error #1

Alright first off if you run into this error:

2012-03-26 00:27:39 	Yes 	10.30.114.32 	0.0000 	Error running '/sbin/tune2fs': Command exit code [1] [/sbin/tune2fs -l /dev/mapper/1EHYR2-BACKUP-20120326T042354p2 >> /usr/local/lp/temp/1EHYR2.tune2fs.4600734.out] () [1]

Notice: THIS IS NOT A FATAL ERROR AND WILL NOT STOP THE BACKUP FROM RUNNING.

We need to stop and take a minute to look over the error. Now you will notice that there are three copies of this line in the provisioning log.

2012-03-26 00:27:39 	No 	10.30.114.32 	0.0000 	Executing: "/sbin/tune2fs -l /dev/mapper/1EHYR2-BACKUP-20120326T042354p1 >> /usr/local/lp/temp/1EHYR2.tune2fs.4600734.out"
2012-03-26 00:27:39 	No 	10.30.114.32 	0.0000 	Execution time: 0.266281
2012-03-26 00:27:39 	No 	10.30.114.32 	0.0000 	Executing: "/sbin/tune2fs -l /dev/mapper/1EHYR2-BACKUP-20120326T042354p3 >> /usr/local/lp/temp/1EHYR2.tune2fs.4600734.out"
2012-03-26 00:27:39 	Yes 	10.30.114.32 	0.0000 	Error running '/sbin/tune2fs': Command exit code [1] [/sbin/tune2fs -l /dev/mapper/1EHYR2-BACKUP-20120326T042354p2 >> /usr/local/lp/temp/1EHYR2.tune2fs.4600734.out] () [1]
2012-03-26 00:27:39 	No 	10.30.114.32 	0.0000 	Executing: "/sbin/tune2fs -l /dev/mapper/1EHYR2-BACKUP-20120326T042354p2 >> /usr/local/lp/temp/1EHYR2.tune2fs.4600734.out"

Now notice in each line there is a p1, p2, or a p3, these each have a defined meaning.

p1 is /boot
p2 is swap
p3 is /

Now we are getting this error because it is trying to run the tune2fs on the p2 mount which is the swap drive and it does not have a file system so this command fails to run. Ignore this backup it this is the only red line as this is not the error that you are looking for.

Rsync Error's

Now there is a few of these errors to go over.

Error #1

2012-03-24 05:34:48 	Yes 	10.30.119.242 	0.0000 	failed: Failed daily backup (Identity added: /usr/local/lp/etc/1EHYR2.4575357.backup (/usr/local/lp/etc/1EHYR2.4575357.backup) ; rsync: write failed on "/backup2/1EHYR2/20120324T042616-daily_IN_PROGRESS/usr/lib64/valgrind/exp-ptrcheck-amd64-linux": No space left on device (28) ; rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6] ; rsync: connection unexpectedly closed (934 bytes received so far) [generator] ; rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6] ; ... ; Identity added: /usr/local/lp/etc/1EHYR2.4575357.backup (/usr/local/lp/etc/1EHYR2.4575357.backup) ; rsync: write failed on "/backup2/1EHYR2/20120324T042616-daily_IN_PROGRESS/usr/lib64/valgrind/exp-ptrcheck-amd64-linux": No space left on device (28) ; rsync error: error in file IO (code 11) at receiver.c(301) [receiver=3.0.6] ; rsync: connection unexpectedly closed (934 bytes received so far) [generator] ; rsync error: error in rsync protocol data stream (code 12) at io.c(600) [generator=3.0.6] ) 
2012-03-24 05:34:45 	Yes 	10.30.119.242 	0.0000 	Exit code from backup wrapper [12] 
2012-03-24 03:47:36 	No 	10.30.119.242 	0.0000 	Started Daily backup (1EHYR2)

Notice: This error can be fatal to a workflow and stop a backup from running.

This rsync error occurs when the target disk runs out of space and forces the rsync to terminate.

Proposed fix: run another backup and it should choose a different server.