Version 5 (modified by 14 years ago) ( diff ) | ,
---|
Kiyoshi recovery 2010-02
This page documents the planned steps for the Kiyoshi disk recovery (see #2828).
Monday Night
- Install debirf image - this will allow us to boot into a debian installation loaded completely in memory giving us full access to all underlying disks
- After booting into debirf:
- Configure debirf for networking (IP: 209.51.171.182/27, Gateway: 209.51.171.161, netmark: 255.255.255.224)
- Prepare the disks so we can access them:
- Initialize the RAID arrays
# incomplete!! mdadm --scan madadm --assemble
- Decrypt the RAID arrays
cryptsetup luksOpen /dev/md1 crypt_md1 cryptsetup luksOpen /dev/md2 crypt_md2
- Scan for logical volumes
vgscan --mknodes vgchange -aly
- Initialize the RAID arrays
- Move all logical volumes to sdc2
- Find logical volumes with:
lvs
- Examine to figure out which ones are on which physical volumes
lvdisplay -m vg_kiyoshi0/<logical-volume-name>
- Move with:
pvmove --verbose --name vg_kiyoshi0/<logical-volume-name> <path/to/old/volume> /dev/sdc
- Find logical volumes with:
- Ensure /dev/sdb is no longer in use
pvdisplay /dev/dm-2
- Setup benchmarking to test performance on sdb prior to our change
- Install postmark
aptitude install postmark
- Move testy partition to sdb
pvmove --verbose vg_kiyoshi0/testy /dev/sdc /dev/dm-2
- Move testy partition to sdb
- Create a file system
mkfs -t ext3 /dev/mapper/vg_kiyoshi0-testy
- Mount it
mount /dev/mapper/vg_kiyoshi0-test /mnt
- Create a file called postmark.conf:
set location /mnt/ set seed 12345678 set read 1024 set write 1024 set buffering false set transactions 4096 set size 512 2048 set number 51115 run quit
- Run postmark:
postmark postmark.conf
- Install postmark
- If you get Error: Cannot open /mnt/123 for writing then reduce the set number to a lower number
- It shoud output something like:
guest@chicken:~$ postmark postmark.conf PostMark v1.51 : 8/14/01 Reading configuration from file 'postmark.conf' Creating files...Done Performing transactions...........Done Deleting files...Done Time: 24 seconds total 11 seconds of transactions (372 per second) Files: 53225 created (2217 per second) Creation alone: 51115 files (8519 per second) Mixed with transactions: 2110 files (191 per second) 2061 read (187 per second) 2035 appended (185 per second) 53225 deleted (2217 per second) Deletion alone: 51239 files (7319 per second) Mixed with transactions: 1986 files (180 per second) Data: 2.54 megabytes read (108.33 kilobytes per second) 65.72 megabytes written (2.74 megabytes per second) guest@chicken:~$
- It shoud output something like:
- De-commission /dev/sdb
- Move testy partition back
umount /mnt pvmove --verbose --name vg_kiyoshi0/testy /dev/dm-2 /dev/sdc
- Move testy partition back
- Remove as logical volume
pvremove /dev/dm-2
- Remove from RAID
mdadm ...
- Properly re-partition
- Setup partitions
parted ...
- Setup partitions
- Re-add to raid arrays
mdadm --add /dev/md2 /dev/sdb2 mdadm --add /dev/md0 /dev/sdb1
- Test to make sure this new partition means it really does write/read faster.
- Move test partition back
pvmove --verbose --name vg_kiyoshi0/testy /dev/sdc2 /dev/dm-2
- Move test partition back
- Run postmark and compare with earlier test results
postmark postmark.conf
- Move logical volumes back from sdc (see above)
- Restart vservers
This would be the ideal stopping place for Monday night
Tuesday night
- Fail sda on all raids it is a part of.
- Take down the machine
- Replace sda disk with new disk
- Start machine
- Create partition table on sda matching sdb
- Add sda partitions back to RAID
Note:
See TracWiki
for help on using the wiki.