Opened 11 days ago

Last modified 8 days ago

#13693 assigned Bug/Something is broken

Database errors in claudette

Reported by: https://id.mayfirst.org/jaimev Owned by: https://id.mayfirst.org/jaimev
Priority: High Component: Tech
Keywords: mysql Cc: https://id.mayfirst.org/jamie
Sensitive: no

Description

mysql crashed on claudette earlier. I was able to start again but I am seeing some errors in the logs. Reading the suggested help files it is not immediately clear to me what should be done.

InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Error: space header page consists of zero bytes in file ./dtw_wordpress/wp_wfVulnScanners.ibd
InnoDB: Error: space header page consists of zero bytes in file ./lecrapaud/wp_wfVulnScanners.ibd
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 88451236401
180509 12:19:26  InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
InnoDB: Apply batch completed
180509 12:19:27  InnoDB: Error: table 'dtw_wordpress/wp_wfVulnScanners'
InnoDB: in InnoDB data dictionary has tablespace id 116716,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
180509 12:19:27  InnoDB: Error: table 'lecrapaud/wp_wfVulnScanners'
InnoDB: in InnoDB data dictionary has tablespace id 115202,
InnoDB: but tablespace with that id or name does not exist. Have
InnoDB: you deleted or moved .ibd files?
InnoDB: This may also be a table created with CREATE TEMPORARY TABLE
InnoDB: whose .ibd and .frm files MySQL automatically removed, but the
InnoDB: table still exists in the InnoDB internal data dictionary.
InnoDB: Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html
InnoDB: for how to resolve the issue.
180509 12:19:27  InnoDB: Waiting for the background threads to start
180509 12:19:28 InnoDB: 5.5.60 started; log sequence number 88451236401
180509 12:19:28 [Note] Server hostname (bind-address): '127.0.0.1'; port: 3306
180509 12:19:28 [Note]   - '127.0.0.1' resolves to '127.0.0.1';
180509 12:19:28 [Note] Server socket created on IP: '127.0.0.1'.
180509 12:19:28 [Warning] Found an entry in the 'db' table with empty database name; Skipped
180509 12:19:28 [Warning] Found an entry in the 'db' table with empty database name; Skipped
180509 12:19:28 [Warning] Found an entry in the 'db' table with empty database name; Skipped
180509 12:19:28 [Warning] Found an entry in the 'db' table with empty database name; Skipped
180509 12:19:28 [Warning] Found an entry in the 'db' table with empty database name; Skipped
180509 12:19:28 [Note] Event Scheduler: Loaded 0 events
180509 12:19:28 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.5.60-0+deb8u1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Debian)

Change History (2)

comment:1 Changed 11 days ago by https://id.mayfirst.org/jaimev

This seems directly related to the issue you saw in #12366 There the only solution was to drop those databases and restore from backup.

comment:2 Changed 8 days ago by https://id.mayfirst.org/jaimev

  • Owner set to https://id.mayfirst.org/jaimev
  • Status changed from new to assigned

I created backups of databases dtw_wordpress and lecrapaud. mysql did complain about not being able to lock tables. I assumed it was related to the errant tables. I saw that the file was created and was roughly the same size as the automated backups so I didn't stop to investigate further. That was a mistake.

After dropping the databases and restarting claudette to load the extra ram when I attempted to restore the databases from backup I realized the dumps were not complete. I was able to find a usable backup for dtw_wordpress from May 4th, but none of the backups for lecrapaud going 10 days back were valid. I am contacting the member to inquire if they have their own backups.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.