Opened 13 months ago

Last modified 13 months ago

#12721 assigned Bug/Something is broken

"Error establishing a database connection" for a site on colin

Reported by: https://id.mayfirst.org/ninareyes Owned by: https://id.mayfirst.org/jaimev
Priority: High Component: Tech
Keywords: Cc:
Sensitive: no

Description

Hi MF/PL,

One of our client sites, https://healthcare-now.org/, is hosted on colin. As of this morning, the site is giving an "Error establishing a database connection." Would it be possible to do a couple of things to try to fix this, like restart MySQL? Let me know if you need more information.

Thanks,

  • Nina

Change History (8)

comment:1 Changed 13 months ago by https://id.mayfirst.org/jaimev

  • Owner set to https://id.mayfirst.org/jaimev
  • Status changed from new to assigned

I've just restarted mysql and the web server. Before that I saw 48 php processes had accumulated and the following long running queries in the mysql processlist.

0 colin:~# mysqladmin processlist
+-------+----------------+-----------+------------------+---------+------+--------------------------------+------------------------------------------------------------------------------------------------------+
| Id    | User           | Host      | db               | Command | Time | State                          | Info                                                                                                 |
+-------+----------------+-----------+------------------+---------+------+--------------------------------+------------------------------------------------------------------------------------------------------+
| 29290 | root           | localhost | healthcaren_civi | Query   | 2356 | Writing to net                 | SELECT /*!40001 SQL_NO_CACHE */ * FROM `log_civicrm_activity_contact`                                |
| 32192 | healthcaren_wp | localhost | healthcaren_civi | Query   | 7041 | Waiting for table level lock   | UPDATE civicrm_group g
SET    refresh_date = '20170508085645'
WHERE  g.cache_date < '20170508084645' |
| 40518 | healthcaren_wp | localhost | healthcaren_civi | Query   | 7012 | Waiting for table level lock   | UPDATE  civicrm_cache  SET group_name = 'CiviCRM Session' , path = '_CRM_Campaign_Form_Petition_Sign |
| 40519 | healthcaren_wp | localhost | healthcaren_wp   | Sleep   | 7072 |                                |                                                                                                      |
| 40559 | healthcaren_wp | localhost | healthcaren_wp   | Sleep   | 7002 |                                |                                                                                                      |
| 40566 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6997 | Waiting for table level lock   | INSERT INTO civicrm_job_log (domain_id , description ) VALUES ( 9 , 'Starting scheduled jobs executi |
| 40575 | healthcaren_wp | localhost | healthcaren_wp   | Sleep   | 6987 |                                |                                                                                                      |
| 40576 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6984 | Waiting for table level lock   | INSERT INTO civicrm_cache (group_name , path , data , created_date ) VALUES ('CiviCRM Session' , '_C |
| 40585 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6982 | Waiting for table level lock   | INSERT INTO civicrm_mailing_event_opened (event_queue_id , time_stamp ) VALUES ( 5615897 ,  20170508 |
| 40586 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6982 | Waiting for table level lock   | INSERT INTO civicrm_mailing_event_opened (event_queue_id , time_stamp ) VALUES ( 5615897 ,  20170508 |
| 40588 | healthcaren_wp | localhost | healthcaren_wp   | Sleep   | 6976 |                                |                                                                                                      |
| 40590 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6967 | Waiting for table level lock   | INSERT INTO civicrm_cache (group_name , path , data , created_date ) VALUES ('CiviCRM Session' , '_C |
| 40606 | healthcaren_wp | localhost | healthcaren_wp   | Sleep   | 6958 |                                |                                                                                                      |
| 40608 | healthcaren_wp | localhost | healthcaren_wp   | Sleep   | 6946 |                                |                                                                                                      |
| 40614 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6943 | Waiting for table level lock   | INSERT INTO civicrm_cache (group_name , path , data , created_date ) VALUES ('CiviCRM Session' , '_C |
| 40620 | healthcaren_wp | localhost | healthcaren_wp   | Sleep   | 6938 |                                |                                                                                                      |
| 40625 | healthcaren_wp | localhost | healthcaren_wp   | Sleep   | 6940 |                                |                                                                                                      |
| 40629 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6934 | Waiting for table level lock   | INSERT INTO civicrm_job_log (domain_id , description ) VALUES ( 10 , 'Starting scheduled jobs execut |
| 40654 | healthcaren_wp | localhost | healthcaren_wp   | Sleep   | 6880 |                                |                                                                                                      |
| 40658 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6876 | Waiting for table level lock   | INSERT INTO civicrm_job_log (domain_id , description ) VALUES ( 11 , 'Starting scheduled jobs execut |
| 40677 | healthcaren_wp | localhost | healthcaren_wp   | Sleep   | 6818 |                                |                                                                                                      |
| 40683 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6813 | Waiting for table level lock   | INSERT INTO civicrm_job_log (domain_id , description ) VALUES ( 12 , 'Starting scheduled jobs execut |
| 40703 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6770 | Waiting for table level lock   | INSERT INTO civicrm_mailing_event_opened (event_queue_id , time_stamp ) VALUES ( 6571713 ,  20170508 |
| 40704 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6770 | Waiting for table level lock   | INSERT INTO civicrm_mailing_event_opened (event_queue_id , time_stamp ) VALUES ( 6524924 ,  20170508 |
| 40744 | healthcaren_wp | localhost | healthcaren_civi | Query   | 6712 | Waiting for table level lock   | INSERT INTO civicrm_mailing_event_trackable_url_open (event_queue_id , trackable_url_id , time_stamp |
| 42518 | healthcaren_wp | localhost | healthcaren_civi | Query   | 4675 | Waiting for table level lock   | INSERT INTO civicrm_mailing_event_opened (event_queue_id , time_stamp ) VALUES ( 6434414 ,  20170508 |
| 42918 | wikidb         | localhost | mediawiki        | Sleep   | 2    |                                |                                                                                                      |
| 43124 | wikidb         | localhost | mediawiki        | Query   | 0    | Writing to net                 | SELECT /* JobQueueDB::claimRandom  */  job_id,job_cmd,job_namespace,job_title,job_timestamp,job_para |
| 43205 | wikidb         | localhost | mediawiki        | Query   | 0    | Writing to net                 | SELECT /* JobQueueDB::claimRandom 199.241.186.154 */  job_id,job_cmd,job_namespace,job_title,job_tim |
| 43898 | wikidb         | localhost | mediawiki        | Query   | 0    | Sending data                   | SELECT /* JobQueueDB::claimRandom 191.101.73.155 */  job_id,job_cmd,job_namespace,job_title,job_time |
| 44635 | wikidb         | localhost | mediawiki        | Query   | 0    | Writing to net                 | SELECT /* JobQueueDB::claimRandom FranziskaMcdade */  job_id,job_cmd,job_namespace,job_title,job_tim |
| 44983 | wikidb         | localhost | mediawiki        | Sleep   | 0    |                                |                                                                                                      |
| 45797 | wikidb         | localhost | mediawiki        | Query   | 0    | Sending data                   | SELECT /* JobQueueDB::claimRandom KindraPartin259 */  job_id,job_cmd,job_namespace,job_title,job_tim |
| 46001 | wikidb         | localhost | mediawiki        | Query   | 0    | Sending data                   | SELECT /* JobQueueDB::claimRandom  */  job_id,job_cmd,job_namespace,job_title,job_timestamp,job_para |
| 46405 | wikidb         | localhost | mediawiki        | Sleep   | 0    |                                |                                                                                                      |
| 46446 | wikidb         | localhost | mediawiki        | Query   | 0    | Writing to net                 | SELECT /* JobQueueDB::claimRandom 83.161.67.152 */  job_id,job_cmd,job_namespace,job_title,job_times |
| 46574 | wikidb         | localhost | mediawiki        | Query   | 0    | Sending data                   | SELECT /* JobQueueDB::claimRandom 191.101.125.171 */  job_id,job_cmd,job_namespace,job_title,job_tim |
| 46681 | wikidb         | localhost | mediawiki        | Query   | 0    | checking query cache for query | SELECT /* JobQueueDB::claimRandom 46.166.145.224 */  job_id,job_cmd,job_namespace,job_title,job_time |
| 46884 | wikidb         | localhost | mediawiki        | Sleep   | 0    |                                |                                                                                                      |
| 46948 | wikidb         | localhost | mediawiki        | Query   | 0    | Sending data                   | SELECT /* JobQueueDB::claimRandom 37.153.168.48 */  job_id,job_cmd,job_namespace,job_title,job_times |
| 47088 | wikidb         | localhost | mediawiki        | Query   | 0    | Sending data                   | SELECT /* JobQueueDB::claimRandom  */  job_id,job_cmd,job_namespace,job_title,job_timestamp,job_para |
| 47379 | wikidb         | localhost | mediawiki        | Query   | 0    | Writing to net                 | UPDATE /* JobQueueDB::claimRandom 154.16.39.84 */  `hcnwikijob` SET job_token = '8930c3855d2c6456a75 |
| 47460 | wikidb         | localhost | mediawiki        | Sleep   | 0    |                                |                                                                                                      |
| 47486 | wikidb         | localhost | mediawiki        | Query   | 0    |                                | SELECT /* JobQueueDB::claimRandom 51.15.8.53 */  job_id,job_cmd,job_namespace,job_title,job_timestam |
| 47569 | wikidb         | localhost | mediawiki        | Sleep   | 0    |                                |                                                                                                      |
| 47738 | wikidb         | localhost | mediawiki        | Query   | 21   | Updating                       | UPDATE /* JobQueueDB::recycleAndDeleteStaleJobs ChauNavarrete */  `hcnwikijob` SET job_token = '',jo |
| 47782 | wikidb         | localhost | mediawiki        | Query   | 0    | Sending data                   | SELECT /* JobQueueDB::claimRandom PatAdey6481807 */  job_id,job_cmd,job_namespace,job_title,job_time |
| 47784 | wikidb         | localhost | mediawiki        | Query   | 19   | Updating                       | UPDATE /* JobQueueDB::recycleAndDeleteStaleJobs 88.198.23.179 */  `hcnwikijob` SET job_token = '',jo |
| 47794 | wikidb         | localhost | mediawiki        | Query   | 1    | User lock                      | SELECT /* JobQueueDB::recycleAndDeleteStaleJobs 148.251.111.129 */ GET_LOCK('jobqueue-recycle-refres |
| 47795 | wikidb         | localhost | mediawiki        | Query   | 1    | User lock                      | SELECT /* JobQueueDB::recycleAndDeleteStaleJobs 154.16.39.36 */ GET_LOCK('jobqueue-recycle-refreshLi |
| 47796 | wikidb         | localhost | mediawiki        | Sleep   | 1    |                                |                                                                                                      |
| 47797 | root           | localhost |                  | Query   | 0    |                                | show processlist                                                                                     |
+-------+----------------+-----------+------------------+---------+------+--------------------------------+------------------------------------------------------------------------------------------------------+

comment:2 Changed 13 months ago by https://id.mayfirst.org/ninareyes

Thanks for jumping on this, Jaime. We'll take a look at what happened here on our end.

comment:3 Changed 13 months ago by https://id.mayfirst.org/jaimev

It look slike the mediawiki site set up at wiki.healthcare-now.org has been receiving a lot of false user registrations and is being populated with spam content. Just a small sample of the URL's that look like they've been created this way.

https://wiki.healthcare-now.org/User:VitoNolen184
https://wiki.healthcare-now.org/User:Ashleigh92V
https://wiki.healthcare-now.org/User:RichLesina50858
https://wiki.healthcare-now.org/User:BessieNewsom
https://wiki.healthcare-now.org/User:MollieBenavidez
https://wiki.healthcare-now.org/User:ElsaKidwell02
https://wiki.healthcare-now.org/User:BrigidaJowett4
https://wiki.healthcare-now.org/User:KentCoulston0
https://wiki.healthcare-now.org/User:OQATheron551831
https://wiki.healthcare-now.org/User:Winnie26K61
https://wiki.healthcare-now.org/User:LouisaPino1305
https://wiki.healthcare-now.org/User:BryonSteward620
https://wiki.healthcare-now.org/User:RaleighThomas93
https://wiki.healthcare-now.org/User:LashawndaMulga
https://wiki.healthcare-now.org/User:Roberta00Q
https://wiki.healthcare-now.org/User:JereRoot8049695
https://wiki.healthcare-now.org/User:FranciscoGuest9
https://wiki.healthcare-now.org/User:LieselotteMcGirr
https://wiki.healthcare-now.org/User:MinnaPettway774

Is this site still in use? We should consider disabling it.

comment:4 Changed 13 months ago by https://id.mayfirst.org/healthcarenow

Hi Jaime - Thanks for catching this, and thanks Nina for flagging the crash. We do actively use wiki.healthcare-now.org for a number of things, and plan on using it for even more.

It looks like the whole subdomain is down right now, but if we get it back up we can temporarily freeze new user registrations, delete the spam accounts, and figure out a better policy for new user registrations. I think we just used whatever new user setup comes out of the box from mediawiki - I assumed that would have some basic verification steps since we set this up very recently - but I'll take a look at the docs to see what more we can do.

In what might be a related issue, I'm noting that we haven't been getting any incoming mail at any healthcare-now.org emails since about 10PM eastern time last night, and test emails do not arrive. I double checked using webmail, so it's not an email client thing. Outgoing mail seems to work fine.

Ben

comment:5 Changed 13 months ago by https://id.mayfirst.org/jaimev

Thanks for the heads up. It looks like the clamav-daemon was down. I've just restarted that for you.

comment:6 Changed 13 months ago by https://id.mayfirst.org/healthcarenow

Ok - I temporarily changed the LocalSettings.php on the wiki to block any new user creation or page creation. There are a lot of spam users and spam pages in there, so we have some cleaning to do before adding additional security extensions, but shouldn't cause any additional problems on the server end for now.

We're still not receiving emails, but I'll hold tight on that.

Ben

comment:7 Changed 13 months ago by https://id.mayfirst.org/stephaniehcn

just letting ya'll know that it's almost the end of the day and still neither of us are receiving any mails!

Thanks,

Stephanie

comment:8 Changed 13 months ago by https://id.mayfirst.org/jaimev

Very sorry about that. I've just found that spampd the server that does the spam filtering had also failed silently and was stopping mail from being delivered.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.