Opened 7 years ago

Closed 7 years ago

#5529 closed Bug/Something is broken (fixed)

Increased /var on stone - Temporary server outage

Reported by: Ross Owned by: Ross
Priority: Medium Component: Tech
Keywords: stone.mayfirst.org mysql disk-full Cc:
Sensitive: no

Description

This morning the var directory on stone filled up it's 4.5G size. While attempting to extend the logical volume, we encountered errors that caused mysql to fail to start, giving the following errors.

0 stone:/var/lib/mysql/saharadev2_drup# mysl
-bash: mysl: command not found
127 stone:/var/lib/mysql/saharadev2_drup# mysql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
1 stone:/var/lib/mysql/saharadev2_drup# which mysqld
/usr/sbin/mysqld
0 stone:/var/lib/mysql/saharadev2_drup# /usr/bin/mysqld --skip-grant
-bash: /usr/bin/mysqld: No such file or directory
127 stone:/var/lib/mysql/saharadev2_drup# /usr/sbin/mysqld --skip-grant
120404  2:06:30 [Warning] Can't create test file /var/lib/mysql/stone.lower-test
120404  2:06:30 [Warning] Can't create test file /var/lib/mysql/stone.lower-test
120404  2:06:30 [Note] Plugin 'FEDERATED' is disabled.
120404  2:06:30  InnoDB: Started; log sequence number 99 1867204271
120404  2:06:30 [ERROR] Can't start server : Bind on unix socket: Invalid argument
120404  2:06:30 [ERROR] Do you already have another mysqld server running on socket: /var/run/mysqld/mysqld.sock ?
120404  2:06:30 [ERROR] Aborting

120404  2:06:30  InnoDB: Starting shutdown...
120404  2:06:35  InnoDB: Shutdown completed; log sequence number 99 1867204271
120404  2:06:35 [Note] /usr/sbin/mysqld: Shutdown complete

1 stone:/var/lib/mysql/saharadev2_drup#

After failing to unmount the var volume, I rebooted stone and forced an fsck which resolved the issues. Scary half an hour, but all seems to be resolved now. The var lv is now at 7G.

~/ross

Change History (2)

comment:1 Changed 7 years ago by Ross

Owner: set to Ross
Status: newassigned

comment:2 Changed 7 years ago by Ross

Resolution: fixed
Status: assignedclosed

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.