wiki:bdsync

Version 11 (modified by https://id.mayfirst.org/jamie, 10 days ago) (diff)

--

Moving guests from one server to another using bdsync

bdsync is a synchronizing tool designed to work on block devices, as opposed to rsync which is designed to work on file systems.

Sometimes we want to move a KVM guest from one physical server to another physical server (if we are retiring a physical server or if we want to free up resources). This page explains how to do this procedure. NOTE: both physical servers must be in the same colocation for networking to work flawlessly. Otherwise, you might need to modify the network settings on the target after you complete the copy.

It works like this:

  • Generate a diff file that records the differences between two block devices (typically one is local and the other is on a remote machine). With a remote block device, you have to have ssh access to the remote server - but just to read the remote block device - bdsync doesn't write to it at this stage.
  • Copy the diff file to the remote machine.
  • Apply the diff file to the remove machine's block device so that they are in sync.

Normally, you will take this step once to copy over the vast majority of the data - and it will take a long time.

Then, at some point during off hours, you would stop the KVM guest on the source machine and re-run the same procedure. Since the volumes are amost the same, it should go more quickly and it should result in a much smaller diff.

When it is complete, you can simply start the guest on the target machine and delete it from the source machine.

Using bdsync to move virtual guests on MF/PL infrastructure

bdsync is not (yet) in debian. So, it has to be compiled. Fortunately it is a small program that is easy to compile.

In puppet

Since you are sync'ing two block devices, both block devices need to exist. On the source machine, of course, it will exist. However, it won't exist on the target machine.

To ensure it is properly created on the target, you will need to add a guest stanza in the target machine's .pp file in puppet. You can copy, exactly, the stanza from the source machine. Then commit the change and push to the origin and push to the target machine. When you push to the target machine, the target block device and guest will be greated - but the guest won't be started or have anything installed.

When you are finished, be sure to delete the stanza from the source machine.

On the source machine

Build bdsync

These steps are only necessary the first time. If your target server is ken, you don't need to do anything. Otherwise, you can probably copy the binary bdsync program from ken.mayfirst.org and skip this step entirely.

  • As root, install dependencies:
    apt-get install make gcc libssl-dev libc6-dev pandoc
    
  • As root, Create a bdsync user to compile the program:
    useradd bdsync-builder
    mkdir /home/bdsync-builder && chown bdsync-builder:bdsync-builder /home/bdsync-builder
    
  • Become that user
    su - bdsync-builder
    
  • Check out the source and cd into the source directory:
    git clone https://github.com/TargetHolding/bdsync && cd bdsync
    
  • Make:
    make
    

You should end up with a bdsync binary in your current directory.

  • Copy
    exit # to be come root
    cp /home/bdsync-builder/bdsyc/bdsync /usr/local/bin/
    

Create a logical volume to hold your diff file

  • Since you will be creating giant diff files - you have to be sure you have a device that is as big as the biggest logical volume you are going to move.
    • Check available space:
      vgs
      
    • Create a lvs:
      lvcreate --size SIZEGB --name bdsync NAME_OF_VG
      
  • Make a filesystem:
    mkfs --type ext4 PATH/TO/LV
    
  • Mount the filesystem somewhere convenient. On the source machine you will be running as root, so putting it in the /root directory makes sense:
    mkdir /root/bdsync
    mount /PATH/TO/LV /root/bdsync
    

On the target machine

Ensure you have the bdsync program available

  • If the source and target machines are running the same version of debian and have the same architecture, just copy the bdsync binary to the target machine. Otherwise, repeat this process on the target machine.

Create a bdsync user on the target machine

  • Unlike the source machine (where you will run as the root user), you will be running as the bdsync user on the target machine. So, you need a bdsync user created.
  • If you haven't already, create a bdsync user on the target machine. This only needs to happen once per physical server.
    useradd bdsync
    
  • Add the source server's root user to the monkeysphere authorized user ids file. Replace $TARGETSERVER with the name of the target server, e.g. "chavez":
    echo "root@$TARGETSERVER.mayfirst.org" >> /home/bdsync/.monkeysphere/authorized_user_ids
    
  • Update monkeysphere credentials
    monkeysphere-authentication update-users bdsync
    

Ensure bdsync user can write to device

  • This step has to be taken for each guest that is moved (since it will be a different device for each guest)
  • Ensure the bdsync user can write to the device. On the target machine, figure out which device is the one being used, e.g.:
    0 cleaver:~# ls -l /dev/mapper/vg_cleaver0-fuller 
    lrwxrwxrwx 1 root root 7 Dec 11 19:39 /dev/mapper/vg_cleaver0-fuller -> ../dm-3
    0 cleaver:~#
    
  • Then, chgrp it so that the bdsync user can read it:
    chgrp bdsync /dev/dm-3
    

Ensure you have a device big enough to receive the diff file

  • You also need to make sure the target server has a big enough volume to receive the giant diff that is coming. Repeat the steps above (Create a logical volume to hold your diff file) for creating a logical volume.
  • On the target server, you will be running as the bdsync user. So, instead of mounting it under /root/bdsync you should mount it under /home/bdsync/bdsync and ensure it is writable by the bdsync user.

Let's begin

On the source machine execute the following command as root. Replace $SOURCE_VG with the volume group on the source machine and $TARGET_VG with the volume group name on the target machine. Replace $GUEST with the logical volume name and $TARGET_SERVER with the name of the target server:

Running via ionice ensures that it doesn't kill all disk i/o on the machine.

cd /root/bdsync
ionice -c 2 -n 7 bdsync "ssh bdsync@$TARGET_SERVER.mayfirst.org bdsync --server" /dev/mapper/$SOURCE_VG-$GUEST /dev/mapper/$TARGET_VG-$GUEST | gzip > $GUEST.bdsync.gz

If you were moving the guest augusto from the source victoria to the target llosa, it would look like this:

# From victoria...
cd /root/bdsync
ionice -c 2 -n 7 bdsync "ssh bdsync@llosa.mayfirst.org bdsync --server" /dev/mapper/vg_victoria0-augusto /dev/mapper/llosa0-augusto | gzip > augusto.bdsync.gz

If you have problems, check syslog on the target server.

Now, copy the diff to the target machine.

scp $GEUST.bdsync.gz bdsync@$TARGET_SERVER.mayfirst.org:bdsync/

Apply first diff

On the target server, apply the diff:

gunzip -c $GUEST.bdsync.gz | bdsync --patch=/dev/mapper/$TARGET_VG-$GUEST

Final sync

After running once, shutdown the source machine and repeat to get the final diff. Then repeat the application of the diff.