This page documents the steps to take in decommissioning a kvm guest. '''Make sure any needed data is backed up.''' The steps below assume that nothing needs to be preserved from the server. If something does need to be preserved, copy the data to another location before following these steps. == Revoke all Known Published Host keys == === Key re-vocation should be two steps: === {{{ monkeysphere-host revoke-key }}} That revokes the server's ssh monkeysphere key. * '''E.g''' {{{ 0 attucks:~# monkeysphere-host revoke-key This will generate a revocation certificate for key A0E7C6828C00CDDFEE82652EBF235CB9287D59CF and dump the certificate to standard output. It can also directly publish the new revocation certificate to the public keyservers via keys.mayfirst.org if you want it to. Publishing this certificate will IMMEDIATELY and PERMANENTLY revoke your host key! Publish the certificate after generation? (y/n/Q) y sec 2048R/287D59CF 2011-12-08 ssh://attucks.mayfirst.org Create a revocation certificate for this key? (y/N) y Please select the reason for the revocation: 0 = No reason specified 1 = Key has been compromised 2 = Key is superseded 3 = Key is no longer used Q = Cancel (Probably you want to select 1 here) Your decision? 3 Enter an optional description; end it with an empty line: > attucks.mayfirst.org decomissioned > Reason for revocation: Key is no longer used attucks.mayfirst.org decomissioned Is this okay? (y/N) y NOTE: This key is not protected! Revocation certificate created. Please move it to a medium which you can hide away; if Mallory gets access to this certificate he can use it to make your key unusable. It is smart to print this certificate and store it away, just in case your media become unreadable. But have some caution: The print system of your machine might store the data and make it available to others! -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: A revocation certificate should follow iQFBBCABAgArBQJRjR3YJB0DYXR0dWNrcy5tYXlmaXJzdC5vcmcgZGVjb21pc3Np b25lZAAKCRC/I1y5KH1Zz01jCACK8d9actSgsdQ8R4iOcPFvyRS397WrzA2NxWvd +y9SJCFves68yMh+HmH3Xr2+IWm9wgwDPWVTWTbDiQYLKpb0Jx+wXwzeayZYHU/X rg3THvyRhHMM2ccBV2h1eKa+e+hyd8sA2r3SLow37dgjKbb1ELfFgwtz6maGxJrE okwl680iCvONxg6GnMy2PVVqgASFbeCABFTGd8MKnjWuVSkMV9O3nScjkokJJeQ5 r0ESu8reYJyQfGc/5xz8fpAK16TjIX8ZLhyb6rsB16xniN3lg/XJbQvrSY9utLMq viFBnbImJK/X9jXHKm59mCam1SjbCzOMni7nAHWo/hZO9tFv =87HD -----END PGP PUBLIC KEY BLOCK----- Really publish this cert to keys.mayfirst.org ? (Y/n) gpg: sending key 287D59CF to hkp server keys.mayfirst.org 0 attucks:~# }}} === Revoke the root@server.mayfirst.org's key: === * '''Get the secret key id for the root user.''' {{{ gpg --list-secret-key to get the secret key id for the root user. }}} * '''Edit the key''' {{{ gpg --edit-key }}} * '''Apply the revocation and save''' You will be asked for a passphrase in this process, which can be found in our puppet repository [source: puppet/manifests/globals.pp] defined by the variable `$mfpl_gpg_passphrase`. {{{ gpg> revkey gpg> save }}} * '''Send the revoked key to the key server''' {{{ gpg --send-key }}} == Shutting down the guest == We need the machine to be out of service so that we can wipe any sensitive data from the disks. In order to shutdown a guest so that it will not reboot, you'll need to be `root@HOSTNAME.mayfirst.org`. From the host, issue the command: {{{ # update-service --remove /etc/sv/kvm/GUESTNAME }}} This command will shutdown the virtual machine. We need the machine to be out of service so that we can wipe any sensitive data from the disks. == Removing the guest directory == To ensure that the guest will not come back online, ever, you should remove the guest kvm directory with the following command as `root@HOSTNAME.mayfirst.org`: {{{ # rm -rf /etc/sv/kvm/GUESTNAME }}} == Ensure all sensitive data is overwritten == For this step, login as `GUESTNAME@HOSTNAME.mayfirst.org` to ensure that you do not overwrite data for other guests. We want to zero out all bytes on the logic volume to avoid this. However, be careful: doing this at full-speed can cause guests on the hosts to stop responding. Use the pv command to limit the IO rate. You may need to use apt-get to install pv on the host. The command is: {{{ $ pv --rate-limit=1m < /dev/zero > /dev/mapper/VOLUMEGROUPNAME-LOGICALVOLUMENAME }}} A real world completed example would be: {{{ 0 maria@malaka:~$ pv --rate-limit=1m < /dev/zero > /dev/mapper/vg_malaka0-maria pv: write failed: No space left on device <=> ] 1 maria@malaka:~$ }}} Once this command finishes running, you can return the logical volume to the volume group. == Removing a Logical Volume == From `root@HOSTNAME.mayfirst.org` check the volume group with: {{{ # vgs }}} This should give output that looks like this: {{{ 0 ken:/dev# vgs VG #PV #LV #SN Attr VSize VFree vg_ken0 1 15 0 wz--n- 1.82t 1.10t 0 ken:/dev# }}} Now you can remove the logical volume with the following command: {{{ # lvremove VOLUMEGROUPNAME/LOGICALVOLUMENAME }}} Real world example: {{{ 0 ken:/dev# lvremove vg_ken0/bataille Do you really want to remove active logical volume bataille? [y/n]: y Logical volume "bataille" successfully removed 0 ken:/dev# }}} Now run 'vgs' again and make sure the new disk space has been added back to the volume group. You should see an increase in the 'VFree' column of the output of the 'vgs' command. == Removing the guest user == {{{ # deluser --remove-home GUESTNAME }}} == Clean up on helper servers == On jojobe.mayfirst.org (nagios server), check for and delete: {{{ /etc/nagios3/conf.d/nodes/GUESTNAME.cfg }}} On the designated backup servers (should be designated in puppet file if it exists): {{{ deluser --remove-home GUESTNAME-sync }}} On your own workstation in the MFPL puppet git repo: * Edit the host manifest and remove the guest server stanza. {{{ git rm manifests/nodes/production/GUESTNAME.pp git commit -m "decomissioning GUESTNAME" }}} Delete node from http://servers.mayfirst.org/ == Other system cleanup == On the host, you might also want to: * remove the guest user account * remove the associated `/etc/udev/rules.d/92-kvm_creator-GUEST.rules` You've now decommissioned a server!