How to reboot a physical server and guests

On occasion we need to reboot a physical server and all of it's guests (often due to a kernel upgrade). This is an intensive process and can take a long time as many servers need to run fsck on reboot which demands intensive disk activity. This page documents the currently recommended process for such reboots.

Make sure all guests on a host are updated

  • run package updates
    apt-get update
    apt-get -s dist-upgrade
    • if the server needs upgrades
      apt-get dist-upgrade

Check available disk space

  • Make sure there is available space on the logical volumes, and take appropriate steps if not, see extend-disk-on-kvm-guest
    • if you added disk space from the host, reboot the guest with
      shutdown -h now

Determine if the guest will fsck on reboot

  • On kvm guests or vserver hosts check when the next fsck is scheduled for all logical volumes with:
    for i in $(ls /dev/mapper/vg_* | grep -v swap); do tune2fs -l $i | grep Next; done
    • e.g.
      0 malcolm:~# for i in $(ls /dev/mapper/vg_* | grep -v swap); do tune2fs -l $i | grep Next; done
      Next check after:         Fri Oct 11 12:12:47 2013
      Next check after:         Fri Oct 11 12:12:28 2013
      Next check after:         Thu Aug 29 23:16:36 2013
      Next check after:         Fri Oct 11 13:07:33 2013
      Next check after:         Fri Oct 11 13:08:17 2013
      Next check after:         Fri Oct 11 13:08:40 2013
      0 malcolm:~# 
  • If a guest will fsck on reboot (i.e. the date from the above is earlier than the current date). Reboot a kvm guest from root@GUEST with. You can watch reboot progress with ssh -t guest@host screen -x
    shutdown -r now

Do this for all guests on the host.

Take down the host's guests

FIXME: Add better information for vservers, this is currently just about kvm

  • From root@host check running services:
    cd /etc/service
    for x in *; do sv status $x; done
  • Take down each guest's service:
    sv down GUESTNAME
    • Make sure the guest has gone down, this may take a few seconds.
      • If the guest has not gone down in a short time, you may need to manually restart from root@guest with:
        shutdown -h now
  • Check to make sure all guests are down, repeat the status command above. You should see a line that starts with down: GUESTNAME for all guests.

Reboot the host

  • Assuming all the guests are down, you should be able to restart the server.
  • Get access to the serial console
  • Shut down the system from root@host with
    shutdown -r now
  • Watch the reboot process from the serial console and enter the encrypted password when requested (accessed via keyringer)
Last modified 9 years ago Last modified on May 25, 2013, 1:00:40 AM