Changes between Version 3 and Version 4 of kiyoshi-recovery-2010-02


Ignore:
Timestamp:
Feb 8, 2010, 10:32:44 PM (16 years ago)
Author:
Jamie McClelland
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • kiyoshi-recovery-2010-02

    v3 v4  
    33This page documents the planned steps for the Kiyoshi disk recovery (see #2828).
    44
     5== Monday Night ==
     6
    57 * Install debirf image - this will allow us to boot into a debian installation loaded completely in memory giving us full access to all underlying disks
    68 * After booting into debirf:
     9  * Configure debirf for networking (IP: 209.51.171.182/27, Gateway: 209.51.171.161, netmark: 255.255.255.224)
    710  * Prepare the disks so we can access them:
    811   * Initialize the RAID arrays
    … …  
    3134   * Move with:
    3235{{{
    33 pvmove --verbose vg_kiyoshi0/<logical-volume-name> <path/to/old/> </dev/sdc>
     36pvmove --verbose --name vg_kiyoshi0/<logical-volume-name> <path/to/old/volume> /dev/sdc
    3437}}}
    3538  * Ensure /dev/sdb is no longer in use
    … …  
    3740pvdisplay /dev/dm-2
    3841}}}
     42  * Setup benchmarking to test performance on sdb prior to our change
     43   * Install postmark
     44{{{
     45aptitude install postmark
     46}}}
     47         * Move testy partition to sdb
     48{{{
     49pvmove --verbose vg_kiyoshi0/testy /dev/sdc /dev/dm-2
     50}}}
     51   * Create a file system
     52{{{
     53mkfs -t ext3 /dev/mapper/vg_kiyoshi0-testy
     54}}}
     55   * Mount it
     56{{{
     57mount /dev/mapper/vg_kiyoshi0-test /mnt
     58}}}
     59   * Create a file called postmark.conf:
     60{{{
     61set location /mnt/
     62set seed 12345678
     63set read 1024
     64set write 1024
     65set buffering false
     66set transactions 4096
     67set size 512 2048
     68set number 51115
     69run
     70quit
     71}}}
     72   * Run postmark:
     73{{{
     74postmark postmark.conf
     75}}}
     76  * If you get Error: Cannot open /mnt/123 for writing then reduce the set number to a lower number
     77        * It shoud output something like:
     78{{{
     79guest@chicken:~$ postmark postmark.conf
     80PostMark v1.51 : 8/14/01
     81Reading configuration from file 'postmark.conf'
     82Creating files...Done
     83Performing transactions...........Done
     84Deleting files...Done
     85Time:
     86        24 seconds total
     87        11 seconds of transactions (372 per second)
     88
     89Files:
     90        53225 created (2217 per second)
     91                Creation alone: 51115 files (8519 per second)
     92                Mixed with transactions: 2110 files (191 per second)
     93        2061 read (187 per second)
     94        2035 appended (185 per second)
     95        53225 deleted (2217 per second)
     96                Deletion alone: 51239 files (7319 per second)
     97                Mixed with transactions: 1986 files (180 per second)
     98
     99Data:
     100        2.54 megabytes read (108.33 kilobytes per second)
     101        65.72 megabytes written (2.74 megabytes per second)
     102guest@chicken:~$
     103}}}
    39104  * De-commission /dev/sdb
     105         * Move testy partition back
     106{{{
     107umount /mnt
     108pvmove --verbose --name vg_kiyoshi0/testy /dev/dm-2 /dev/sdc
     109}}}
    40110   * Remove as logical volume
    41111{{{
    … …  
    46116mdadm ...
    47117}}}
    48   * Run benchmarks to test read/write speeds based on current partition table
    49 {{{
    50 dd ....
    51 }}}
    52118  * Properly re-partition
    53119   * Setup partitions
    54120{{{
    55 fdisk ....
     121parted ...
    56122}}}
    57    * Test to make sure this new partition means it really does write/read faster
    58 {{{
    59 dd ....
    60 }}}
     123   
    61124  * Re-add to raid arrays
    62125{{{
    63 mdadm --add /dev/md1 /dev/sdb2
     126mdadm --add /dev/md2 /dev/sdb2
    64127mdadm --add /dev/md0 /dev/sdb1
    65128}}}
     129  * Test to make sure this new partition means it really does write/read faster.
     130         * Move test partition back
     131{{{
     132pvmove --verbose --name vg_kiyoshi0/testy /dev/sdc2 /dev/dm-2
     133}}}
     134   * Run postmark and compare with earlier test results
     135{{{
     136postmark postmark.conf
     137}}}
    66138  * Move logical volumes back from sdc (see above)
     139  * Restart vservers
     140
     141This would be the ideal stopping place for Monday night
     142
     143== Tuesday night ==
     144
     145 * Fail sda on all raids it is a part of.
     146 * Take down the machine
     147 * Replace sda disk with new disk
     148 * Start machine
     149 * Create partition table on sda matching sdb
     150 * Add sda partitions back to RAID