wiki:restart-failed-server

Version 3 (modified by Jamie McClelland, 10 years ago) (diff)

--

Restart Failed Server

From time to time our virtual servers exceed their allocated memory and fail with oom (out of memory) errors. It is usually necessary to perform a hard reset of the virtual servers.

How to reset a virtual server depends on the virtualization technology in use by the server.

The first step is to checkout our SVN repository of servers.

Then, examine the "available-servers" directory. You should see a directory for every server we are running.

Inside each directory, there's a file called "virtualization".

The virtualization file will say either: vserver, xen, or kvm.

In addition, there's a symlink called "host" which will point to the host server for this virtual server. For all servers, you will need to ssh into the host server to perform a reset.

Important Note If the host server is sontag or gramsci you may be prompted to enter an encryption pass phrase. You will need this passphrase to start the host because sontag and gramsci do not have their base disks encrypted. If the host server is fred - you may be prompted to enter a passphrase, but you can hit enter to continue (fred does have it's base disks encrypted). All other hosts have their base disks encrypted so no need to decrypt a virtual server to start it.

xen

  • Unplug the server:
    xm destroy <server-name>
    
  • Restart the server:
    xm create -c <server-name>
    

Important note: if you are restarting a xen server on sontag and you are prompted for a cryptsetup passphrase, you must lookup it up via keyringer. If you are restarting a xen server on fred and you are prompted for a cryptsetup passphrase, there is no passphrase - just keep hitting enter until the prompt stops appearing. The reason you get the prompt is because each virtual server is using the initramfs from the host server (fred) which does have a cryptsetup password. However, the virtual servers do not.

kvm

  • unplug the server:
    sv down <server-name>
    
  • restart:
    sv up <server-name>
    

vserver

  • shutdown
    vserver <server-name> stop
    
  • start
    vserver <server-name> start