Opened 4 months ago

Last modified 3 months ago

#13793 assigned Bug/Something is broken

Left Forum Website down

Reported by: https://id.mayfirst.org/leftforum Owned by: https://id.mayfirst.org/jaimev
Priority: Urgent Component: Tech
Keywords: Cc: gabriel_morales_96@…, dazdndcunfusd@…, marcus@…, matt@…, ashley@…
Sensitive: no

Description

this error message shows up:

Error messagePDOException: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in lock_may_be_available() (line 167 of /home/members/leftforum/sites/conference.leftforum.org/web/includes/lock.inc).

Change History (15)

comment:1 Changed 4 months ago by https://id.mayfirst.org/leftforum

  • Priority changed from Medium to Urgent

comment:2 Changed 4 months ago by https://id.mayfirst.org/jamie

  • Resolution set to fixed
  • Status changed from new to closed

Sorry for the delay - due to travel schedules this slipped through the cracks. I've restarted mysql and the site is up and running again. The failure was caused by the server running out of memory. We ran out of memory because we increased the number of PHP processes allowed to try to cover the increase in site usage during the conference. I've just reduced that back down to a reasonable level.

comment:3 Changed 4 months ago by https://id.mayfirst.org/leftforum

  • Cc gabriel_morales_96@… dazdndcunfusd@… marcus@… added
  • Resolution fixed deleted
  • Status changed from closed to assigned

Hello,

the left forum website is down again. Same error as the one above....

=== The website encountered an unexpected error. Please try again later. Error messagePDOException: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in lock_may_be_available() (line 167 of /home/members/leftforum/sites/conference.leftforum.org/web/includes/lock.inc). ==

comment:4 Changed 4 months ago by https://id.mayfirst.org/leftforum

Can we prevent this from happening?

Marcus

comment:5 Changed 4 months ago by https://id.mayfirst.org/jamie

Hi Marcus - It should be up again now.

I've made an additional re-adjustment to the amount of memory we were allowing the database to consume (another adjustment we made prior to the conference in an effort to improve performance).

I think the database was allowed to consume too much memory causing a crash.

Jaime and I have spent considerable time trying to figure out how to best support the left forum web site. Just this week, Jaime transferred the entire virtual guest to a brand new machine - Left Forum is literally the only site on this brand new server.

Yet still page loads are remarkably slow.

So, I did some more experiments on the version I keep locally on my laptop.

I measured the most basic page load time at about 2 seconds (that doesn't including any of the images - just the time it takes to render the page).

Then, I switched from your theme to the stock theme that comes with Drupal, and the page load time was 50 milliseconds.

In other words, a speed boost of 95.7% by switching themes.

I'm not sure if that's the Drupal bootstrap theme that is the dog here - or whether there is something to do with your customization of the theme, but I do think we need to get you a new theme before the next confererence.

comment:6 Changed 4 months ago by https://id.mayfirst.org/leftforum

Hello,

The site is down again....

Marcus

comment:7 Changed 4 months ago by https://id.mayfirst.org/jaimev

  • Owner set to https://id.mayfirst.org/jaimev

I am seeing the site up from here, at least the home page and a few other pages. Was the drupal login not working? I am restarting the php process manager and database now. Let us know if there is any change please.

comment:8 Changed 4 months ago by https://id.mayfirst.org/leftforum

  • Cc matt@… ashley@… added

The website is down again...on various computers.

Is there anything we can do to restart it?

Marcus

comment:9 Changed 4 months ago by https://id.mayfirst.org/jaimev

Hi, sorry for the delay it looks like the mysql server crashed on stokely. I'm going to go ahead and move mysql to an SSD partition which should hopefully alleviate these problems. I should have it back up and running in a few more minutes.

comment:10 Changed 4 months ago by https://id.mayfirst.org/jaimev

  • Resolution set to fixed
  • Status changed from assigned to feedback

This is done now.

comment:11 Changed 4 months ago by https://id.mayfirst.org/jamie

  • Status changed from feedback to closed

I'm not sure why - but mysql didn't seem to propery start after you made this change:

0 stokely:/home/members/leftforum/sites/conference.leftforum.org/logs# systemctl status mysql
● mysql.service - LSB: Start and stop the mysql database server daemon
   Loaded: loaded (/etc/init.d/mysql)
   Active: active (exited) since Sat 2018-07-07 17:48:30 EDT; 2h 40min ago
  Process: 7972 ExecStop=/etc/init.d/mysql stop (code=exited, status=0/SUCCESS)
  Process: 8032 ExecStart=/etc/init.d/mysql start (code=exited, status=0/SUCCESS)

Jul 07 17:48:30 stokely mysql[8032]: Starting MySQL database server: mysqld ..
Jul 07 17:48:30 stokely mysql[8032]: Checking for tables which need an upgrade, are corrupt or were
Jul 07 17:48:30 stokely mysql[8032]: not closed cleanly..
Jul 07 17:48:30 stokely systemd[1]: Started LSB: Start and stop the mysql database server daemon.
Jul 07 17:48:30 stokely /etc/mysql/debian-start[8547]: Upgrading MySQL tables if necessary.
Jul 07 17:48:30 stokely /etc/mysql/debian-start[8568]: Checking for insecure root accounts.
Jul 07 17:48:30 stokely /etc/mysql/debian-start[8574]: Triggering myisam-recover for all MyISAM tables
Jul 07 20:29:07 stokely systemd[1]: Started LSB: Start and stop the mysql database server daemon.
0 stokely:/home/members/leftforum/sites/conference.leftforum.org/logs# ps -eFH | grep mysql
root      9787  9529  0  3182  2112   0 20:29 pts/1    00:00:00               grep mysql
0 stokely:/home/members/leftforum/sites/conference.leftforum.org/logs#

I just started it again now.

comment:12 Changed 3 months ago by https://id.mayfirst.org/leftforum

  • Resolution fixed deleted
  • Status changed from closed to assigned

The website is down again....

can we get a more stable fix? The Conference in LA is coming up in August 25th and our website needs to be used for that one.

marcus

comment:13 Changed 3 months ago by https://id.mayfirst.org/jamie

Ugg. This is really frustrating. It's back again. there seems to be some problem with the database and memory - getting: InnoDB: Fatal error: cannot allocate memory for the buffer pool

I'm investigating now. This has to be fixed ASAP.

comment:14 Changed 3 months ago by https://id.mayfirst.org/jamie

Despite having 4GB of RAM, the server is running out of ram when mysql is started. I'm re-booting with 8GB of ram to try to diagnose what is consuming all the memory

comment:15 Changed 3 months ago by https://id.mayfirst.org/jamie

This post suggests the problem is not enough memory, so I suspect doubling the RAM should resolve the issue for now.

I'm still investigating why the guest would need twice as much RAM as any other member we have (including high traffic sites like BLM).

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.