= Solid State Drives = To help alleviate slowness from a lot of disk reads and writes, we are in the processing of adding solid state drives to our existing servers. Solid state drives (SSDs) are significantly faster than the traditional spinning SATA drives we currently have installed. A major source of the disk read/write contention comes from access to MySQL databases. This contention slows everything down and it is one of the significantly bottle necks when a web site that is driven by a MySQL database feels slow to load. Therefore, our strategy is to move most MySQL database partitions to the newly installed solid state drives. That should allow MySQL powered web sites to load much faster, while also reducing the disk contention on the other partitions. == Technical Overview == Like all disks, solid state drives are installed in a RAID configuration for redundancy and are also encrypted. They are added to their own dedicated volume group on the host server (typically named: vg_HOST1 - as opposed to vg_HOST0 for the volume group with SATA disks). Just as each guest is allocated a logical volume named after the guest, each guest is additionally allowed a logical volume from the SSD volume group (also named after the guest). So, a guest may have both: * vg_wiwa0-jacobs * vg_wiwa1-jacobs In the guest, the initial disk shows up as /dev/sda and the second disk shows up as /dev/sdb. For the normal SATA disks - they are added to a logical volume group on the guest and then further allocated. However, with the SSD disks - a single partition is created from the block device and a filesystem is installed directly on it. Due to limitations of SSD, the operating system must communicate to the disk which sectors are free for writing (trim/discards), we have to configure all the disk layers specifically to pass these "disard" messages. == Technical Details == Following are the steps for adding a solid state drive to both a host and a guest === Adding solid state drives to a host === First, order the disks (see [ticket:11870#comment:7 previous SSDs we have purchased] and order the cables and cages needed to install them (see [ticket:11870#comment:10 previous cables we have ordered]). Once installed, the should show up via `cat /proc/partitions` as something like /dev/sda and /dev/sdb. Next, modify the physical servers .pp file in puppet, changing the call to m_physical server to be: {{{ class { "mayfirst::m_physical": ssd => true } }}} Once you push, it should change /etc/lvm/lvm.conf to have: `issue_discards = 1` Next, create a single partition on each device, e.g.: {{{ 0 wiwa:/etc/lvm# parted /dev/sdb GNU Parted 3.2 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt (parted) unit s mkpart main 8192 -196608 (parted) p Model: ATA SAMSUNG MZ7KM480 (scsi) Disk /dev/sdb: 937703088s Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 8192s 937506480s 937498289s main (parted) }}} Then a RAID 1 array: {{{ mdadm --create --raid-devices=2 --level=1 --metadata=1.0 --verbose /dev/md2 /dev/sda1 /dev/sdb1 }}} Scan your RAID devices: {{{ mdadm --examine --scan }}} And based on the output, edit /etc/mdadm/mdam.conf to add a line for your new RAID (something like: `ARRAY /dev/md/2 metadata=1.0 UUID=70345fc6:661e67ce:715261cf:84112f10 name=wiwa:2` Now, setup encryption and be sure to enable `--allow-discards`: {{{ 0 wiwa:/etc/lvm# cryptsetup luksFormat /dev/md2 WARNING! ======== This will overwrite data on /dev/md2 irrevocably. Are you sure? (Type uppercase yes): YES Enter passphrase: Verify passphrase: 0 wiwa:/etc/lvm# cryptsetup --allow-discards luksOpen /dev/md2 md2_crypt Enter passphrase for /dev/md2: 0 wiwa:/etc/lvm# blkid /dev/md2 /dev/md2: UUID="b360ef4f-84ea-457c-8f86-a7f94b1f9277" TYPE="crypto_LUKS" 0 wiwa:/etc/lvm# echo md2_crypt UUID=b360ef4f-84ea-457c-8f86-a7f94b1f9277 none luks,discard >> /etc/crypttab 0 wiwa:/etc/lvm# cat /etc/crypttab # md1_crypt UUID=ae7a55a5-cc91-4064-8e5f-06eb293188a2 none luks md2_crypt UUID=b360ef4f-84ea-457c-8f86-a7f94b1f9277 none luks,discard 0 wiwa:/etc/lvm# pvcreate /dev/mapper/md2_crypt Physical volume "/dev/mapper/md2_crypt" successfully created 0 wiwa:/etc/lvm# vgcreate vg_wiwa1 /dev/mapper/md2_crypt Volume group "vg_wiwa1" successfully created 0 wiwa:/etc/lvm# }}} === Adding solid state drives to a guest === On the host, create a logical volume for your given host, e.g.: {{{ 0 wiwa:/etc/lvm# lvcreate --size 10GB --name jacobs vg_wiwa1 Logical volume "jacobs" created 0 wiwa:/etc/lvm# }}} Change the group ownership to match the user the guests runs as: {{{ 1 wiwa:/etc/sv/kvm/jacobs# ls -l /dev/mapper/vg_wiwa1-jacobs lrwxrwxrwx 1 root root 8 Sep 23 15:16 /dev/mapper/vg_wiwa1-jacobs -> ../dm-28 0 wiwa:/etc/sv/kvm/jacobs# chgrp jacobs /dev/dm-28 0 wiwa:/etc/sv/kvm/jacobs# }}} Reboot the guest. From the guest... Look for the new block device (/dev/sdb below): {{{ 0 jacobs:~# cat /proc/partitions major minor #blocks name 8 16 10485760 sdb 8 0 524288000 sda 8 1 248832 sda1 8 2 524037120 sda2 254 0 3997696 dm-0 254 1 499712 dm-1 254 2 999424 dm-2 254 3 4997120 dm-3 254 4 4997120 dm-4 254 5 20971520 dm-5 0 jacobs:~# }}} Create a single partition: {{{ 0 jacobs:~# parted /dev/sdb GNU Parted 3.2 Using /dev/sdb Welcome to GNU Parted! Type 'help' to view a list of commands. (parted) mklabel gpt (parted) unit s mkpart main 8192 -196608 (parted) p Model: QEMU QEMU HARDDISK (scsi) Disk /dev/sdb: 20971520s Sector size (logical/physical): 512B/512B Partition Table: gpt Disk Flags: Number Start End Size File system Name Flags 1 8192s 20774912s 20766721s main (parted) quit Information: You may need to update /etc/fstab. 0 jacobs:~# }}} Create a filesystem: {{{ mkfs -t ext4 /dev/sdb1 }}} ==== Replacing MySQL partition ==== If you are using this partition to replace MySQL... Stop mysql: {{{ systemctl stop mysql }}} Unmount /var/lib/mysql: {{{ umount /var/lib/mysql }}} Edit /etc/fstab, change the line for mounting /var/lib/mysql to match /dev/sdb1 instead of the existing logical volume. Mount the old logical volume on /mnt: {{{ mount /dev/mapper/vg_jacobs0-var+lib+mysql /mnt }}} Mount the new partition: {{{ mount /var/lib/mysql }}} Move the data: {{{ mv /mnt/* /var/lib/mysql }}} Umount /mnt and remove the logical volume: {{{ umount /mnt lvremove /dev/mapper/vg_jacobs0-var+lib+mysql }}} Retart mysql: {{{ systemctl start mysql }}}