Opened 2 years ago

Closed 2 years ago

#12078 closed Bug/Something is broken (fixed)

alp.org is down, apache not running on viewsic

Reported by: https://id.mayfirst.org/jackaponte Owned by:
Priority: High Component: Tech
Keywords: viewsic.mayfirst.org apache2 slaapbeen.mayfirst.org rodolpho.mayfirst.org daza.mayfirst.org Cc:
Sensitive: no

Description

We received an Icinga alert about the ALP website being down. I was able to SSH into viewsic.mayfirst.org with no problems, but don't see any apache processes running as I do on other shared servers. Unless I'm looking for the wrong thing, seems that apache isn't running!

Change History (7)

comment:1 Changed 2 years ago by https://id.mayfirst.org/dkg

  • Keywords viewsic.mayfirst.org apache2 added; viewsic removed
  • Resolution set to fixed
  • Status changed from new to closed

Should be back up now. for some reason after i did some upgrades systemd didn't believe that apache was in a bad state even though no processes were running:

0 viewsic:~# systemctl status apache2
● apache2.service - LSB: Apache2 web server
   Loaded: loaded (/etc/init.d/apache2)
   Active: active (exited) since Thu 2016-09-01 21:52:32 EDT; 41min ago
  Process: 6802 ExecReload=/etc/init.d/apache2 reload (code=exited, status=0/SUCCESS)

Sep 01 21:52:32 viewsic apache2[30504]: Starting web server: apache2.
Sep 01 21:54:48 viewsic apache2[6745]: Reloading web server: apache2.
Sep 01 21:54:49 viewsic apache2[6802]: Reloading web server: apache2.
0 viewsic:~# 

i did systemctl restart apache2 and it started back up.

weird :(

comment:2 Changed 2 years ago by https://id.mayfirst.org/dkg

  • Keywords slaapbeen.mayfirst.org rodolpho.mayfirst.org daza.mayfirst.org added

Looking in /var/log/apache2/error.log i see:

[Thu Sep 01 21:54:48.793168 2016] [mpm_worker:notice] [pid 30520:tid 139901466236800] AH00297: SIGUSR1 received.  Doing graceful restart
[Thu Sep 01 21:54:49.209294 2016] [core:notice] [pid 30520] AH00060: seg fault or similar nasty error detected in the parent process

This happened during the upgrade i just did, and it looks like systemd didn't properly restart apache as a result. I'm not sure what to make of this, but it also happened on slaapbeen and rodolpho and daza during their upgrades :/

comment:3 Changed 2 years ago by https://id.mayfirst.org/jackaponte

  • Resolution fixed deleted
  • Status changed from closed to assigned

Sorry to be the bearer of more bad news, but now mysql isn't running on viewsic! Saw an Icinga recovery notification come in at 22:44:44 EDT, followed by another problem alert for alp.org at 23:46:54 EDT. Apache is running now, but Drupal's showing the "Site off-line" screen and no mysql processes are running.

comment:4 Changed 2 years ago by https://id.mayfirst.org/jamie

I've started mysql on viewsic. I see the following in the journalctl logs:

-- Logs begin at Thu 2016-09-01 09:59:02 EDT, end at Fri 2016-09-02 08:35:49 EDT. --
Sep 01 21:49:08 viewsic mysql[19691]: Stopping MySQL database server: mysqld.
Sep 01 21:53:17 viewsic mysql[31508]: Starting MySQL database server: mysqld . ..
Sep 01 21:53:17 viewsic mysql[31508]: Checking for tables which need an upgrade, are corrupt or were
Sep 01 21:53:17 viewsic mysql[31508]: not closed cleanly..
}}

And...

{{{
Sep 01 23:37:39 viewsic mysql[17235]: Stopping MySQL database server: mysqld.
Sep 02 08:33:49 viewsic mysql[31024]: Starting MySQL database server: mysqld . . . . . . . . . ..
Sep 02 08:33:49 viewsic mysql[31024]: Checking for tables which need an upgrade, are corrupt or were
Sep 02 08:33:49 viewsic mysql[31024]: not closed cleanly..
Sep 02 08:33:50 viewsic /etc/mysql/debian-start[31676]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
}}}

The starting at 8:33 am was me. I'm not sure what triggered mysql to stop at 23:37.

comment:5 Changed 2 years ago by https://id.mayfirst.org/jackaponte

Thanks for the updates and the help, Jamie and dkg!

I'll let you know if we get more notifications about viewsic. Things have been fine since Jamie's last update; fingers crossed!

comment:6 Changed 2 years ago by https://id.mayfirst.org/jaimev

  • Resolution set to fixed
  • Status changed from assigned to feedback

comment:7 Changed 2 years ago by automatic

  • Status changed from feedback to closed

No news is good news (we hope)! Given the lack of feedback, we think this ticket can be closed.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.