What to do, if the filesystem in your Ploop-Image is broken, the container still runs and a downtime is not an option?
I’ve migrated about 20 Containers to a new RAID on a new Host via vzmigrate. So far, so good. But the migration failed on one Container, because vzctl was reporting a filesystem error, when the container was ready to resume on the new host. Fortunately the migration process ended up with resuming the Container on the old Host.
After some investigation, I found some sporadic EXT4-Errors reported for the related ploop-Image of the container in the Logfile of the old host. The error was related to only 3 inodes. But to fix this with fsck, I would have to shut the container down for the time, the manual initiated fsck-process would last. This could easily take a couple of hours on a 700GB Image. No way!
So, vzmigrate creates a copy of the image, a snapshot and a merge on the new host. Because it is a dumb copy of the root.hdd file, it also moves the broken inodes to the new image. It’s like a dd from one, broken harddrive to another, new harddrive. If you do so, You’ll have exactly the same problems on the new drive, as you had on the old.
On the other hand, the Image was about 700 Gig, while the Container itself only used about 180 Gig. Doing a compact on a ploop-device with a corrupted file system inside? Better not!
So, how can I migrate the Container to the new Host without hours of downtime?
This is what I have done:
Let’s assume, we have the hosts OLDHOST and NEWHOST. The Containers ID is 123 and the Ploop-Image size shall be 400Gig. Let’s go:
On NEWHOST:
Prepare a new image
ploop init -s 400G -t ext4 /vz/private/123/root.hdd/root.hdd
Mount the Image
mkdir /mnt/plooptemp ploop mount -m /mnt/plooptemp/ /vz/private/123/root.hdd/DiskDescriptor.xml
Perform an rsync from the containers root to the new image
rsync -auv OLDHOST:/vz/root/123/ /mnt/plooptemp/ --exclude "/proc/*" --exclude "/sys/*" --exclude "/.cpt*" -h -stats
The Parameters -h and -stats will give us a nice, human readable summary after rsync has finished.
Then do the rsync again.
It will transfer only those files, that have changed since the last rsync. Expand the command with parameter –delete-after. This will remove vanished files from the new image, as well.
rsync -auv OLDHOST:/vz/root/123/ /mnt/plooptemp/ --exclude "/proc/*" --exclude "/sys/*" --exclude "/.cpt*" --delete-after -h -stats
If there are problems to read files in the old container, you’ll find them easily in the output of the second rsync. In my case, I was able to identify the broken files (related to the broken inodes) with the second rsync.
It reported three files as not readable and stopped the process of deletion, because of this error. So I extended the rsync line with the exclusion of the three files.
Then, do it again.
rsync -auv OLDHOST:/vz/root/123/ "/mnt/plooptemp/" --exclude "/proc/*" --exclude "/sys/*" --exclude "/.cpt*" --exclude "/path/to/broken/file1" --exclude "/path/to/broken/file2" --exclude "/path/to/broken/file3" --delete-after -h -stats
The rsync finished without complains.
This was a dry run, just to figure out, what to do.
Okay, Let’s get serious.
First of all, empty the new ploop.
Delete and create it from scratch, to have it empty or just do:
rm -Rf /mnt/plooptemp/*
Then I’ve created a tiny batch, to migrate the container with the least possible downtime.
At this time, the new ploop device still has to be mounted on /mnt/plooptemp/
#!/bin/bash
# copy the containers config to the NEWHOST
scp OLDHOST:/etc/sysconfig/vz-scripts/123.conf /etc/sysconfig/vz-scripts/
# do the initial sync and have some coffee or pizza
rsync -auv OLDHOST:/vz/root/123/ "/mnt/plooptemp/" \
--exclude "/proc/*" \
--exclude "/sys/*" \
--exclude "/.cpt*" \
--exclude "/path/to/broken/file1" \
--exclude "/path/to/broken/file2" \
--exclude "/path/to/broken/file3" \
--exclude "/some/other/paths/i/*" \
--exclude "/do/not/need/anymore/* \
--delete-after -h -stats
# after first run, do the diff by repeating the rsync
rsync -auv OLDHOST:/vz/root/123/ "/mnt/plooptemp/" \
--exclude "/proc/*" \
--exclude "/sys/*" \
--exclude "/.cpt*" \
--exclude "/path/to/broken/file1" \
--exclude "/path/to/broken/file2" \
--exclude "/path/to/broken/file3" \
--exclude "/some/other/paths/i/*" \
--exclude "/do/not/need/anymore/* \
--delete-after -h -stats
# now stop the container on the OLDHOST
ssh OLDHOST vzctl stop 123
# mount the old image to the containers default root path
ssh OLDHOST ploop mount -m /vz/root/123/ /vz/private/123/root.hdd/DiskDescriptor.xml
# do the rsync again to fetch the absolute final state
rsync -auv OLDHOST:/vz/root/123/ "/mnt/plooptemp/" \
--exclude "/proc/*" \
--exclude "/sys/*" \
--exclude "/.cpt*" \
--exclude "/path/to/broken/file1" \
--exclude "/path/to/broken/file2" \
--exclude "/path/to/broken/file3" \
--exclude "/some/other/paths/i/*" \
--exclude "/do/not/need/anymore/* \
--delete-after -h -stats
# before we start the container, unmount the new ploop-image.
ploop umount -m /mnt/plooptemp
# we are ready to run the container
vzctl start 123
# The container should start without any issues.
# finally unmount the source-image.
ssh OLDHOST ploop umount -m /vz/root/123/
# make totally sure, you will not run the old container accidentally
ssh OLDHOST move /vz/private/123 /vz/private/123-deleteme
ssh OLDHOST move /vz/root/123 /vz/root/123-deleteme
ssh OLDHOST move /etc/sysconfig/vz-scripts/123.conf /etc/sysconfig/vz-scripts/123.deleteme
Just to be sure, enter the fresh migrated container and check, if everything runs as it should.
In my case, the new image is now ‚just‘ about 230Gig. It’s just fresh!
Oh – the most important thing: The total downtime was less than a minute. Acceptable!
I hope, it’ll help others in some cases.