How to reboot a physical server and guests

On occasion we need to reboot a physical server and all of it's guests (often due to a kernel upgrade). This is an intensive process and can take a long time as many servers need to run fsck on reboot which demands intensive disk activity. This page documents the currently recommended process for such reboots.

Make sure all guests on a host are updated

• run package updates

  apt-get update
  apt-get -s dist-upgrade
  

  • if the server needs upgrades

    apt-get dist-upgrade
    

Check available disk space

• Make sure there is available space on the logical volumes, and take appropriate steps if not, see extend-disk-on-kvm-guest
  • if you added disk space from the host, reboot the guest with

    shutdown -h now
    

Determine if the guest will fsck on reboot

• On kvm guests or vserver hosts check when the next fsck is scheduled for all logical volumes with:

  for i in $(ls /dev/mapper/vg_* | grep -v swap); do tune2fs -l $i | grep Next; done
  

  • e.g.

    0 malcolm:~# for i in $(ls /dev/mapper/vg_* | grep -v swap); do tune2fs -l $i | grep Next; done
    Next check after:         Fri Oct 11 12:12:47 2013
    Next check after:         Fri Oct 11 12:12:28 2013
    Next check after:         Thu Aug 29 23:16:36 2013
    Next check after:         Fri Oct 11 13:07:33 2013
    Next check after:         Fri Oct 11 13:08:17 2013
    Next check after:         Fri Oct 11 13:08:40 2013
    0 malcolm:~# 
    

• If a guest will fsck on reboot (i.e. the date from the above is earlier than the current date). Reboot a kvm guest from root@GUEST with. You can watch reboot progress with ssh -t guest@host screen -x

  shutdown -r now
  

Do this for all guests on the host.

Take down the host's guests

FIXME: Add better information for vservers, this is currently just about kvm

• From root@host check running services:

  cd /etc/service
  for x in *; do sv status $x; done
  

• Take down each guest's service:

  sv down GUESTNAME
  

  • Make sure the guest has gone down, this may take a few seconds.
    • If the guest has not gone down in a short time, you may need to manually restart from root@guest with:

      shutdown -h now
      

• Check to make sure all guests are down, repeat the status command above. You should see a line that starts with down: GUESTNAME for all guests.

Reboot the host

• Assuming all the guests are down, you should be able to restart the server.
• Get access to the serial console
  • Telehouse use telehouse_serial_access
  • XO use xo_serial_access
• Shut down the system from root@host with

  shutdown -r now
  

• Watch the reboot process from the serial console and enter the encrypted password when requested (accessed via keyringer)