Changes between Version 3 and Version 4 of kiyoshi-recovery-2010-02
- Timestamp:
- Feb 8, 2010, 10:32:44 PM (14 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
kiyoshi-recovery-2010-02
v3 v4 3 3 This page documents the planned steps for the Kiyoshi disk recovery (see #2828). 4 4 5 == Monday Night == 6 5 7 * Install debirf image - this will allow us to boot into a debian installation loaded completely in memory giving us full access to all underlying disks 6 8 * After booting into debirf: 9 * Configure debirf for networking (IP: 209.51.171.182/27, Gateway: 209.51.171.161, netmark: 255.255.255.224) 7 10 * Prepare the disks so we can access them: 8 11 * Initialize the RAID arrays … … 31 34 * Move with: 32 35 {{{ 33 pvmove --verbose vg_kiyoshi0/<logical-volume-name> <path/to/old/> </dev/sdc>36 pvmove --verbose --name vg_kiyoshi0/<logical-volume-name> <path/to/old/volume> /dev/sdc 34 37 }}} 35 38 * Ensure /dev/sdb is no longer in use … … 37 40 pvdisplay /dev/dm-2 38 41 }}} 42 * Setup benchmarking to test performance on sdb prior to our change 43 * Install postmark 44 {{{ 45 aptitude install postmark 46 }}} 47 * Move testy partition to sdb 48 {{{ 49 pvmove --verbose vg_kiyoshi0/testy /dev/sdc /dev/dm-2 50 }}} 51 * Create a file system 52 {{{ 53 mkfs -t ext3 /dev/mapper/vg_kiyoshi0-testy 54 }}} 55 * Mount it 56 {{{ 57 mount /dev/mapper/vg_kiyoshi0-test /mnt 58 }}} 59 * Create a file called postmark.conf: 60 {{{ 61 set location /mnt/ 62 set seed 12345678 63 set read 1024 64 set write 1024 65 set buffering false 66 set transactions 4096 67 set size 512 2048 68 set number 51115 69 run 70 quit 71 }}} 72 * Run postmark: 73 {{{ 74 postmark postmark.conf 75 }}} 76 * If you get Error: Cannot open /mnt/123 for writing then reduce the set number to a lower number 77 * It shoud output something like: 78 {{{ 79 guest@chicken:~$ postmark postmark.conf 80 PostMark v1.51 : 8/14/01 81 Reading configuration from file 'postmark.conf' 82 Creating files...Done 83 Performing transactions...........Done 84 Deleting files...Done 85 Time: 86 24 seconds total 87 11 seconds of transactions (372 per second) 88 89 Files: 90 53225 created (2217 per second) 91 Creation alone: 51115 files (8519 per second) 92 Mixed with transactions: 2110 files (191 per second) 93 2061 read (187 per second) 94 2035 appended (185 per second) 95 53225 deleted (2217 per second) 96 Deletion alone: 51239 files (7319 per second) 97 Mixed with transactions: 1986 files (180 per second) 98 99 Data: 100 2.54 megabytes read (108.33 kilobytes per second) 101 65.72 megabytes written (2.74 megabytes per second) 102 guest@chicken:~$ 103 }}} 39 104 * De-commission /dev/sdb 105 * Move testy partition back 106 {{{ 107 umount /mnt 108 pvmove --verbose --name vg_kiyoshi0/testy /dev/dm-2 /dev/sdc 109 }}} 40 110 * Remove as logical volume 41 111 {{{ … … 46 116 mdadm ... 47 117 }}} 48 * Run benchmarks to test read/write speeds based on current partition table49 {{{50 dd ....51 }}}52 118 * Properly re-partition 53 119 * Setup partitions 54 120 {{{ 55 fdisk ....121 parted ... 56 122 }}} 57 * Test to make sure this new partition means it really does write/read faster 58 {{{ 59 dd .... 60 }}} 123 61 124 * Re-add to raid arrays 62 125 {{{ 63 mdadm --add /dev/md 1/dev/sdb2126 mdadm --add /dev/md2 /dev/sdb2 64 127 mdadm --add /dev/md0 /dev/sdb1 65 128 }}} 129 * Test to make sure this new partition means it really does write/read faster. 130 * Move test partition back 131 {{{ 132 pvmove --verbose --name vg_kiyoshi0/testy /dev/sdc2 /dev/dm-2 133 }}} 134 * Run postmark and compare with earlier test results 135 {{{ 136 postmark postmark.conf 137 }}} 66 138 * Move logical volumes back from sdc (see above) 139 * Restart vservers 140 141 This would be the ideal stopping place for Monday night 142 143 == Tuesday night == 144 145 * Fail sda on all raids it is a part of. 146 * Take down the machine 147 * Replace sda disk with new disk 148 * Start machine 149 * Create partition table on sda matching sdb 150 * Add sda partitions back to RAID