= Kiyoshi recovery 2010-02 =

This page documents the planned steps for the Kiyoshi disk recovery (see #2828).

== Monday Night 9:00 pm ==

 * Booting into debirf image (which allows us full access to the underlying filesystems:
  * Configure debirf for networking (IP: 209.51.171.182/27, Gateway: 209.51.171.161, netmark: 255.255.255.224)
  * Prepare the disks so we can access them:
   * Initialize the RAID arrays
{{{
# incomplete!!
for foo in 0 1 2 ;do mknod /dev/md$foo b 9 $foo ; done
mdadm --assemble /dev/md1 /dev/sda2
mdadm --assemble /dev/md2 /dev/sdb2
}}}
   * Decrypt the RAID arrays
{{{
cryptsetup luksOpen /dev/md1 crypt_md1
cryptsetup luksOpen /dev/md2 crypt_md2
}}}
   * Scan for logical volumes
{{{
vgscan --mknodes
vgchange -aly
}}}
  * Move all logical volumes to sdc2
   * Find logical volumes with:
{{{
lvs
}}}
   * Examine to figure out which ones are on which physical volumes
{{{
lvdisplay -m vg_kiyoshi0/<logical-volume-name>
}}}
   * Move with:
{{{
pvmove --verbose --name vg_kiyoshi0/<logical-volume-name> <path/to/old/volume> /dev/sdc2
}}}
  * Ensure /dev/sdb is no longer in use
{{{
pvdisplay /dev/mapper/md2_crypt
}}}

== Evaluation point ==

If the transfer to sdc goes quickly and smoothly, then this is an acceptable stopping point. We can restart with all data coming from sdc and expect a stable (although not redundant) system for tuesday.

If the transfer is going really slowly or we have doubts it can finish by tomorrow, we should stop. The last thing we want to do is move all the data to sdc and when it finally completes at 7:00 am, realize the performance is even worse.

== Monday night or Tuesday during the day ==

  * Setup benchmarking to test performance on sdb prior to our change
   * Install postmark
{{{
aptitude install postmark
}}}
   * Move testy partition to sdb
{{{
pvmove --verbose vg_kiyoshi0/testy /dev/sdc2 /dev/mapper/md2_crypt
}}}
   * Create a file system
{{{
mkfs -t ext3 /dev/mapper/vg_kiyoshi0-testy
}}}
   * Mount it
{{{
mount /dev/mapper/vg_kiyoshi0-tesyt /mnt
}}}
   * Create a file called postmark.conf:
{{{
set location /mnt/
set seed 12345678
set read 1024
set write 1024
set buffering false
set transactions 4096
set size 512 2048
set number 51115
run
quit
}}}
   * Run postmark:
{{{
postmark postmark.conf
}}}
  * If you get Error: Cannot open /mnt/123 for writing then reduce the set number to a lower number
  * It should output something like:
{{{
guest@chicken:~$ postmark postmark.conf 
PostMark v1.51 : 8/14/01
Reading configuration from file 'postmark.conf'
Creating files...Done
Performing transactions...........Done
Deleting files...Done
Time:
	24 seconds total
	11 seconds of transactions (372 per second)

Files:
	53225 created (2217 per second)
		Creation alone: 51115 files (8519 per second)
		Mixed with transactions: 2110 files (191 per second)
	2061 read (187 per second)
	2035 appended (185 per second)
	53225 deleted (2217 per second)
		Deletion alone: 51239 files (7319 per second)
		Mixed with transactions: 1986 files (180 per second)

Data:
	2.54 megabytes read (108.33 kilobytes per second)
	65.72 megabytes written (2.74 megabytes per second)
guest@chicken:~$
}}}
  * De-commission /dev/sdb
   * Move testy partition back
{{{
umount /mnt
pvmove --verbose --name vg_kiyoshi0/testy /dev/mapper/md2_crypt /dev/sdc2
}}}
   * Remove as logical volume
{{{
pvremove /dev/mapper/md2_crypt
}}}
   * Unmap crypto layer:
{{{
cryptsetup luksClose md2_crypt 
}}}
   * Remove from RAID
{{{
mdadm --fail /dev/md2 /dev/sdb2 
### shouldn't this remove /dev/md2 entirely?
mdadm --fail /dev/md0 /dev/sdb1 
}}}
  * Properly re-partition
   * Setup partitions
{{{
# following http://article.gmane.org/gmane.linux.utilities.util-linux-ng/2955
parted /dev/sdb
# do we want gpt??
(parted) mklabel gpt
(parted) unit s
### dkg thinks we should not go all the way to the end; rather, we should leave a bit of free space
### this is because we don't know if the other disk is exactly the same size or not.
### so we should change -1 to something several sectors in from the end.
(parted) mkpart primary ext2 40 -1
# parted will complain about end location, ignore
(parted) quit
}}}
   * Re-add to raid arrays: this step re-adds it to the LVM raid array it was previously a part of and adds it to the boot partition RAID array
{{{
mdadm /dev/md2 --create --level=mirror -n 2 /dev/sdb2 missing
mdadm --add /dev/md0 /dev/sdb1
}}}
  * add crypto layer to md2
{{{
cryptsetup luksFormat /dev/md2
cryptsetup luksOpen /dev/md2 md2_crypt
}}}
  * Test to make sure this new partition means it really does write/read faster.
	 * Move test partition back and mount
{{{
pvmove --verbose --name vg_kiyoshi0/testy /dev/sdc2 /dev/mapper/md2_crypt
mount /dev/mapper/vg_kiyoshi0-test /mnt
}}}
   * Run postmark and compare with earlier test results
{{{
postmark postmark.conf
}}}
  * Move logical volumes back from sdc (see above)
  * Restart vservers

This is another stopping place 

== Tuesday/Wednesday night ==

 * Fail sda on all raids it is a part of.
 * Take down the machine
 * Replace sda disk with new disk
 * Start machine
 * Create partition table on sda matching sdb
 * Add sda partitions back to RAID