Kiyoshi recovery 2010-02

This page documents the planned steps for the Kiyoshi disk recovery (see #2828).

• Install debirf image - this will allow us to boot into a debian installation loaded completely in memory giving us full access to all underlying disks
• After booting into debirf:
  • Prepare the disks so we can access them:
    • Initialize the RAID arrays

      mdadm ....
      

    • Decrypt the RAID arrays

      cryptsetup luksOpen /dev/md1 crypt_md1
      cryptsetup luksOpen /dev/md2 crypt_md2
      

    • Scan for logical volumes

      vgscan --mknodes
      vgchange -aly
      

  • Move all logical volumes to sdc2
    • Find logical volumes with:

      lvs
      

    • Examine to figure out which ones are on which physical volumes

      lvdisplay -m vg_kiyoshi0/<logical-volume-name>
      

    • Move with:

      pvmove --verbose vg_kiyoshi0/<logical-volume-name> <path/to/old/> </dev/sdc>
      

  • Ensure /dev/sdb is no longer in use

    pvdisplay /dev/dm-2
    

  • De-commission /dev/sdb
    • Remove as logical volume

      pvremove /dev/dm-2
      

    • Remove from RAID

      mdadm ...
      

  • Run benchmarks to test read/write speeds based on current partition table

    dd ....
    

  • Properly re-partition
    • Setup partitions

      fdisk ....
      

    • Test to make sure this new partition means it really does write/read faster

      dd ....
      

  • Re-add to raid arrays

    mdadm --add /dev/md1 /dev/sdb2
    mdadm --add /dev/md0 /dev/sdb1
    

  • Move logical volumes back from sdc (see above)