Version 7 (modified by Jamie McClelland, 5 years ago) (diff)


Moving guests from one server to another using bdsync

bdsync is a synchronizing tool designed to work on block devices, as opposed to rsync which is designed to work on file systems.

It works like this:

  • Generate a diff file that records the differences between two block devices (typically one is local and the other is on a remote machine). With a remote block device, you have to have ssh access to the remote server - but just to read the remote block device - bdsync doesn't write to it at this stage.
  • Copy the diff file to the remote machine.
  • Apply the diff file to the remove machine's block device so that they are in sync.
  • Repeat.

Using bdsync to move virtual guests on MF/PL infrastructure

bdsync is not (yet) in debian. So, it has to be compiled. Fortunately it is a small program that is easy to compile.

In puppet

Since you are sync'ing two block devices, both block devices need to exist. On the source machine, of course, it will exist. However, it won't exist on the target machine.

To ensure it is properly created on the target, you will need to add a guest stanza in the target machine's .pp file in puppet. You can copy, exactly, the stanza from the source machine. Then commit the change and push to the origin and push to the target machine. When you push to the target machine, the target block device and guest will be greated - but the guest won't be started or have anything installed.

On the source machine

These steps are only necessary the first time. If your target server is ken, you don't need to do anything. Otherwise, you can probably copy the binary bdsync program from and skip this step entirely.

  • As root, install dependencies:
    apt-get install make gcc libssl-dev libc6-dev
  • As root, Create a bdsync user to compile the program:
    useradd bdsync-builder
  • Become that user
    su - bdsync-builder
  • Check out the source and cd into the source directory:
    git clone && cd bdsync
  • Make:

You should end up with a bdsync binary in your current directory.

  • Since you will be creating giant diff files - you have to be sure you have a device that is as big as the biggest logical volume you are going to move.
    • Check available space:
    • Create a lvs:
      lvcreate --size SIZEGB --name bdsync NAME_OF_VG
  • Make a filesystem:
    mkfs --type ext4 PATH/TO/LV

On the target machine

  • If the source and target machines are running the same version of debian and have the same architecture, just copy the bdsync binary to the target machine. Otherwise, repeat this process on the target machine.
  • Unlike the source machine (where you will run as the root user), you will be running as the bdsync user on the target machine. So, you need a bdsync user created.
  • If you haven't already, create a bdsync user on the target machine:
    useradd bdsync
  • Add the source server's root user to the monkeysphere authorized user ids file. Replace $TARGETSERVER with the name of the target server, e.g. "chavez":
    echo "root@$" >> /home/bdsync/.monkeysphere/authorized_user_ids
  • Update monkeysphere credentials
    monkeysphere-authentication update-users bdsync
  • Ensure the bdsync user can write to the device. On the target machine, figure out which device is the one being used, e.g.:
    0 cleaver:~# ls -l /dev/mapper/vg_cleaver0-fuller 
    lrwxrwxrwx 1 root root 7 Dec 11 19:39 /dev/mapper/vg_cleaver0-fuller -> ../dm-3
    0 cleaver:~#
  • Then, chgrp it so that the bdsync user can read it:
    chgrp bdsync /dev/dm-3
  • You also need to make sure the target server has a big enough volume to receive the giant diff that is coming. Repeat the steps above for creating a logical volume and mounting it under the bdsync user's home directory.

Let's begin

On the source machine execute the following command as root. Replace $LOCALDEV (e.g. /dev/mapper/vg_ken0-debs) with the path to the block device on the local machine and $REMDEV with the path of the block device on the remote machine (e.g. /dev/mapper/vg_cleaver0-debs). Replace $TARGETSERVER with the name of the target server (e.g. "cleaver"):

Running via ionice ensures that it doesn't kill all disk i/o on the machine.

ionice -c 2 -n 7 bdsync "ssh bdsync@$ bdsync --server" $LOCDEV $REMDEV | gzip > DEV.bdsync.gz

If you have problems, check syslog on the target server.

Now, copy the diff to the target machine.

scp DEV.bdsync.gz bdsync@$

Apply first diff

On the target server, apply the diff:

gunzip -c DEV.bdsync.gz | bdsync --patch=$REMDEV

Final sync

After running once, shutdown the source machine and repeat to get the final diff. Then repeat the application of the diff.