= Kiyoshi recovery 2010-02 =

This page documents the planned steps for the Kiyoshi disk recovery (see #2828).

== Monday Night ==

 * Install debirf image - this will allow us to boot into a debian installation loaded completely in memory giving us full access to all underlying disks
 * After booting into debirf:
  * Configure debirf for networking (IP: 209.51.171.182/27, Gateway: 209.51.171.161, netmark: 255.255.255.224)
  * Prepare the disks so we can access them:
   * Initialize the RAID arrays
{{{
# incomplete!!
mdadm --scan
madadm --assemble
}}}
   * Decrypt the RAID arrays
{{{
cryptsetup luksOpen /dev/md1 crypt_md1
cryptsetup luksOpen /dev/md2 crypt_md2
}}}
   * Scan for logical volumes
{{{
vgscan --mknodes
vgchange -aly
}}}
  * Move all logical volumes to sdc2
   * Find logical volumes with:
{{{
lvs
}}}
   * Examine to figure out which ones are on which physical volumes
{{{
lvdisplay -m vg_kiyoshi0/<logical-volume-name>
}}}
   * Move with:
{{{
pvmove --verbose --name vg_kiyoshi0/<logical-volume-name> <path/to/old/volume> /dev/sdc
}}}
  * Ensure /dev/sdb is no longer in use
{{{
pvdisplay /dev/dm-2
}}}
  * Setup benchmarking to test performance on sdb prior to our change
   * Install postmark
{{{
aptitude install postmark
}}}
   * Move testy partition to sdb
{{{
pvmove --verbose vg_kiyoshi0/testy /dev/sdc /dev/dm-2
}}}
   * Create a file system
{{{
mkfs -t ext3 /dev/mapper/vg_kiyoshi0-testy
}}}
   * Mount it
{{{
mount /dev/mapper/vg_kiyoshi0-test /mnt
}}}
   * Create a file called postmark.conf:
{{{
set location /mnt/
set seed 12345678
set read 1024
set write 1024
set buffering false
set transactions 4096
set size 512 2048
set number 51115
run
quit
}}}
   * Run postmark:
{{{
postmark postmark.conf
}}}
  * If you get Error: Cannot open /mnt/123 for writing then reduce the set number to a lower number
  * It should output something like:
{{{
guest@chicken:~$ postmark postmark.conf 
PostMark v1.51 : 8/14/01
Reading configuration from file 'postmark.conf'
Creating files...Done
Performing transactions...........Done
Deleting files...Done
Time:
	24 seconds total
	11 seconds of transactions (372 per second)

Files:
	53225 created (2217 per second)
		Creation alone: 51115 files (8519 per second)
		Mixed with transactions: 2110 files (191 per second)
	2061 read (187 per second)
	2035 appended (185 per second)
	53225 deleted (2217 per second)
		Deletion alone: 51239 files (7319 per second)
		Mixed with transactions: 1986 files (180 per second)

Data:
	2.54 megabytes read (108.33 kilobytes per second)
	65.72 megabytes written (2.74 megabytes per second)
guest@chicken:~$
}}}
  * De-commission /dev/sdb
   * Move testy partition back
{{{
umount /mnt
pvmove --verbose --name vg_kiyoshi0/testy /dev/dm-2 /dev/sdc
}}}
   * Remove as logical volume
{{{
pvremove /dev/dm-2
}}}
   * Remove from RAID
{{{
mdadm --fail /dev/md2 /dev/sdb2 
mdadm --fail /dev/md0 /dev/sdb1 
}}}
  * Properly re-partition
   * Setup partitions
{{{
# following http://article.gmane.org/gmane.linux.utilities.util-linux-ng/2955
parted /dev/sdb
# do we want gpt??
(parted) mklabel gpt
(parted) unit s
(parted) mkpart primary ext2 40 -1
# parted will complain about end location, ignore
(parted) quit
}}}
   * Re-add to raid arrays: this step re-adds it to the LVM raid array it was previously a part of and adds it to the boot partition RAID array
{{{
mdadm --add /dev/md2 /dev/sdb2
mdadm --add /dev/md0 /dev/sdb1
}}}
  * Test to make sure this new partition means it really does write/read faster.
	 * Move test partition back and mount
{{{
pvmove --verbose --name vg_kiyoshi0/testy /dev/sdc2 /dev/dm-2
mount /dev/mapper/vg_kiyoshi0-test /mnt
}}}
   * Run postmark and compare with earlier test results
{{{
postmark postmark.conf
}}}
  * Move logical volumes back from sdc (see above)
  * Restart vservers

This would be the ideal stopping place for Monday night

== Tuesday night ==

 * Fail sda on all raids it is a part of.
 * Take down the machine
 * Replace sda disk with new disk
 * Start machine
 * Create partition table on sda matching sdb
 * Add sda partitions back to RAID