Opened 5 years ago

Last modified 5 years ago

#8093 assigned Bug/Something is broken


Reported by: Owned by:
Priority: Urgent Component: Tech
Keywords: civicrm drupal Cc: pbennett@…
Sensitive: no


Hi - something seems to be up with the server. won't load and I keep getting this message:

The connection to the server was reset while the page was loading.

Please help.



Attachments (1)

civi.jpg (65.4 KB) - added by 5 years ago.
screenshot of civi page after logging in

Download all attachments as: .zip

Change History (46)

comment:1 Changed 5 years ago by

I'm looking into this now.

comment:2 Changed 5 years ago by

  • Owner set to
  • Status changed from new to assigned

Thanks Ross

comment:3 Changed 5 years ago by

Thanks again Ross.

Not sure if this will help but maybe you can disable the secure ssl module. That's a legacy module that we needed to use on our old server, but I don't think we need it now and it might be causing some conflicts. Not sure though.

comment:4 Changed 5 years ago by

Hi Carrette,

There was some problem with the filesystem, that we're looking into. As part of diagnosing this, I made a disastrous mistake and rebooted the physical server hosting stoney. We're currently waiting for everything to come back up and will work on stoney's filesystem errors as soon is it does.


comment:5 Changed 5 years ago by

Eeek, totally understand and thanks very much.


comment:6 Changed 5 years ago by

Hi Carrette,

Okay, you're back up and running now. Apparently, the root file system got corrupted so we needed to run a disk repair (fsck) on the root file system, which seemed to have resolved the problem.

Looking through the logs, it appears that at some point early this morning stoney's RAM exceeded it's capacity. I'm not sure if this caused the file system corruption or not. However, just to be on the safe side, I allocated stoney with an additional gigabyte of ram and also added a CPU core.

I also tweaked your mysql settings based on this command:

mysql> SELECT CEILING(SUM(data_length+index_length)/POWER(1024,2)) RIBPS  FROM information_schema.tables WHERE engine='InnoDB';
|   887 |
1 row in set (12.33 sec)


This command suggests you could use about 900 megabytes of ram in your innodb_buffer_pool_size variable. So I added the line innodb_buffer_pool_size = 900M to your /etc/mysql/my.cnf file. This should reduce some disk reads and also improve performance.

In practice though, I still find the site relatively slow to load for an anonymous user. I'm wondering if you might consider installing the boost module, which would dramatically increase the page load times for anonymous users and would likely also reduce the load on the apache server.


comment:7 Changed 5 years ago by

  • Resolution set to fixed
  • Status changed from assigned to closed

Hi Ross - thanks so much for fixing this issue. Do you know why the RAM exceeded capacity? I did see you recommend the boost module to someone else and thought it was an excellent suggestion and was prepared to install it so I definitely will.

All the best and have a great holiday.


comment:8 Changed 5 years ago by

  • Resolution fixed deleted
  • Status changed from closed to assigned

Hi Ross - something is wrong. Just tried to connect again and got this error:

PDOException: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in lock_may_be_available() (line 167 of /home/members/nyccah/sites/

I refreshed the page and the site came up but thought you should be aware of this.



comment:9 Changed 5 years ago by

Hi Carrette,

As far as exceeding RAM goes, I believe (though I cannot be 100% sure) that when the nightly backup process begins, which includes dumping the database and syncing all new files on the server, that there just wasn't enough extra RAM for the server to allocate. When that happened, all the RAM on the server got allocated and started pulling from the swap disk (the hard drive portion of memory used as RAM in an overflow situation, 500M extra). I believe that this too filled up at which point the system started killing off processes.

Upon further inspection, I discovered that your theme was causing most of the load time issues on the site. I also discovered that you did not have any javascript or css caching turned on. I manually set these variables to on with drush vset preprocess_css 1 and drush vset preprocess_js 1. This should assist in theme load times.

Finally, I also discovered that you did not have a regular cron job being run on the site, so I ran drush cron (which could have caused the PDO Exception error), and I set up a regular cron by adding the following line to your crontab:

10 * * * * /usr/bin/drush --root=/home/members/nyccah/sites/ --quiet cron

During the cron run I noticed numerous errors like these:

array_key_exists() expects parameter 2 to be array, null given ConfigSetting.php:190                                        [warning]
array_key_exists() expects parameter 2 to be array, null given ConfigSetting.php:190                                        [warning]

It appears that you're currently running civicrm version 4.3.7, but the current version is 4.4.2. I suspect that upgrading civicrm might resolve these error messages.

Finally, I'm also seeing these errors in your watchdog log:

Warning: array_flip(): Can only flip STRING and INTEGER values! in                   DrupalDefaultEntityController->cacheGet() (line 350 /home/members/nyccah/sites/

There are other errors in the watchdog log, that I suspect are related to the older version of civicrm. I also note that there are a few modules with security updates (including drupal core). So it could be good to make those updates as well.

My guess is that at least some of the performance issues we're encountering are very much related to the old version of civicrm so if I were you I'd focus first on the drupal core update and then the civicrm upgrade.

Finally, let me know if you continue to see that PDO Exception error. If that continues we may need to reduce the amount of RAM we're giving to mysql innodb tables.

Sorry to list so many issues, but I very much would like to help you get the site running at it's optimal level.

All the best,


P.S. I'm going to re-open this ticket as we try to navigate all the issues listed herein.

comment:10 Changed 5 years ago by

Whew! That's a lot of stuff.

I will do the Drupal and Civi updates this weekend. Hopefully, that will resolve some of these issues.

And I would just like to say that I absolutely appreciate all your help tackling these issues and helping us to get our site operating at its best.

You guys are the best hosting providers ever.



comment:11 Changed 5 years ago by

Hi Ross - I just visited the site to start making the upgrades and got this error:


The website encountered an unexpected error. Please try again later.

Error message

PDOException: SQLSTATE[42000]: Syntax error or access violation: 1286 Unknown table engine 'InnoDB': SELECT expire, value FROM {semaphore} WHERE name = :name; Array ( [:name] => variable_init ) in lock_may_be_available() (line 167 of /home/members/nyccah/sites/ .

Uncaught exception thrown in shutdown function.

PDOException: SQLSTATE[42000]: Syntax error or access violation: 1286 Unknown table engine 'InnoDB': DELETE FROM {semaphore} WHERE (value = :db_condition_placeholder_0) ; Array ( [:db_condition_placeholder_0] => 688446605529286b616e4f8.90468818 ) in lock_release_all() (line 269 of /home/members/nyccah/sites/

So as of now I can't log into the site.


comment:12 Changed 5 years ago by

Just tried to get in again and the site came up. I will go ahead and log on, complete the upgrades and post to this ticket when I'm all done.



comment:13 Changed 5 years ago by

Hi Carrette,

Uggg...So I reduced the amount of RAM allocated to innodb_buffer_pool_size to 400M and restart mysql. Restarting mysql did not fix the problem, so I restarted apache which did fix the problem.

I'm not exactly sure why this error occurred, but hopefully reducing innodb_buffer_pool_size will keep this from happening again until you get the site upgraded. And if we're lucky this upgrade will help improve performance so that it doesn't happen again.

Eventually we'll find the best configuration for the site,


comment:14 Changed 5 years ago by

Hi Ross - I completed the Drupal core update as well as module updates. Two of the module updates can't complete because they require changes being made to the database, however, every time I run the database update script it's unsuccessful. There's no real error message Basically, when it completes it automatically goes to the front page and there's a no active batch error on the front page. I don't really know what that means.

Also, the site theme seems to be a little bit off and the admin theme very much so in the back end of Civi. I am holding off on the civi upgrade until these issues are resolved. If you can take a look and see if there's some issue with the database, I'd appreciate it much.

Gotta make sure the site is up and functional. This week is actually a key week for us, so I'd rather wait to do Civi and just make sure that the Drupal issues are resolved.



comment:15 Changed 5 years ago by

OK, so now the site is totally shot and i need it to be up as soon as humanly possible. If you can do a restore of the database as soon as you are able, that will be great.



comment:16 Changed 5 years ago by

FYI - I ultimately reverted drupal core back to 7.23. Things however are stll not right with the theme, and is not rendering properly in firefox and is off in ie as well. Appreciate any help in getting it back in operational order for today.



comment:17 Changed 5 years ago by

View in both ie and firefox for an example of the theme not rendering properly. This also occurs in the backend of civi:

comment:18 Changed 5 years ago by

Can't get in the site:

See error below:

Error The website encountered an unexpected error. Please try again later. Error message PDOException: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in lock_may_be_available() (line 167 of /home/members/nyccah/sites/

comment:19 Changed 5 years ago by

Hi Ross - was able to log back into the site and I removed the aggregate css file setting and things are rendering much better now. I think I will remove the javascript setting as well to see if things get better.

comment:20 Changed 5 years ago by

OK so those 2 performance settings really affected civi in a bad way. I will attempt the upgrade again sometime next week but the drupal core update had caused some issues as well that seemed to be database related. But will tackle that as I go through the upgrade process.

Thanks and sorry for the many postings but I wanted to keep you informed so you had a sense of the issues that were coming up as I did the upgrade.

All the best,


comment:21 Changed 5 years ago by

Finally - visitors to the site still keep getting these errors periodically:

Skip to main content

Error The website encountered an unexpected error. Please try again later. Error message PDOException: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in lock_may_be_available() (line 167 of /home/members/nyccah/sites/

comment:22 Changed 5 years ago by

Here's another error that I've gotten as well. What does it mean?

PDOException: SQLSTATE[42000] [1226] User 'nyccah' has exceeded the 'max_user_connections' resource (current value: 25) in lock_may_be_available() (line 167 of /home/members/nyccah/sites/

Exceeded max user_connections? Is there a max user settings someplace?

comment:23 Changed 5 years ago by

Hi Carrette,

In the control panel under mysql users, you can set the max number of connections for the mysql user. The old setting was 25, I just increased it to 50. My guess about what is happening is that because the civi and the drupal database user is the same, that user was making both civicrm and drupal connections and simply needed a larger value to handle those connections.

As far as the drupal upgrade goes, it should not impact your site negatively. There are no database updates, only security updates in the newer version of drupal.

As far as css and javascript caching, those can and apparently did have an effect on the site. I believe that's because not all of the css and javascript was getting cached. Though I'm not sure why this would be.

Let's see if the higher mysql connection value improves the errors, and try to get drupal and civi upgraded as soon as possible.


comment:24 Changed 5 years ago by

Hi Ross - after doing a quick check on Drupal, it does seem as if the security update have broken a number of sites and many of those were theme related. It seems the changes made to the .htaccess files totally broke some sites. It certainly did in my case.

Also, once I had worked through those issues and errors with the htaccess files and tried completing the 7.24 upgrade and module upgrades, I kept getting a "no active batch" error when I tried running the database update script. This made it impossible to complete the updates for the back up and location modules since they both needed to write changes to the database tables. Do you think the changes you just made will resolve that particular error?



comment:25 Changed 5 years ago by

Hi Carrette,

I think we're dealing with multiple things simultaneously. One of the things we were dealing with was mysql connection errors, which I actually think caused the database update problems you mentioned.

The .htaccess files would not have caused problems other than leaving security holes open.

However, it is quite probable that other things were going on that I cannot address b/c you managed to revert the changes to a more stable system. Nice going, by the way, getting things back in functional order!!!

At this point, it seems that the only modules remaining to upgrade are (Drupal core, location, and civicrm). I would suggest that you attempt to upgrade the location module again first and then report back.

How does that sound?


comment:26 Changed 5 years ago by

Yes, that sounds like a great plan. I'll also reinstall the back up and migrate module, which I had removed.

Also, about the drupal and civi databases being combined, I think it's an awful idea. When I first installed civi on our drupal installation a few years back (civi 2.2) I made sure the databases were separate, however, a couple of years ago, a consultant did some work on the site and unfortunately, I was not around to stop the madness of them combining the databases and apparently the staff member who was managing the project was clueless, even though I had sent her 2 separate database dumps, one for drupal and one for civi. Anyhow, when I moved the site over to the mayfirst servers I thought about trying to separate the databases again but I am no database expert and was not confident enough to attempt it. So,if you know of a quick, somewhat easy and foolproof way of accomplishing this, I would appreciate your feed back. This is something I would tackle a few months out though.

Thanks again and will report back once I've upgraded the modules,


comment:27 Changed 5 years ago by

Website down again - see error message:

Error The website encountered an unexpected error. Please try again later. Error message PDOException: SQLSTATE[42000]: Syntax error or access violation: 1286 Unknown table engine 'InnoDB': SELECT expire, value FROM {semaphore} WHERE name = :name; Array ( [:name] => variable_init ) in lock_may_be_available() (line 167 of /home/members/nyccah/sites/ Uncaught exception thrown in shutdown function.

PDOException: SQLSTATE[42000]: Syntax error or access violation: 1286 Unknown table engine 'InnoDB': DELETE FROM {semaphore} WHERE (value = :db_condition_placeholder_0) ; Array ( [:db_condition_placeholder_0] => 8849457395293b02979ccc8.19921209 ) in lock_release_all() (line 269 of /home/members/nyccah/sites/

comment:28 Changed 5 years ago by

Hi Ross - just so you are aware, another member of our staff, Paul Bennett will be following up on the website issues with you.

His email is pbennett@…

comment:29 Changed 5 years ago by

Hi Ross. The website is back and working. Thank you and/or Carrette for doig this. Can we talk sometime? I want to understand more what happened. thanks Paul 646.395.9122

comment:30 Changed 5 years ago by

hi Ross

I talked to our ED Joel and Carrette (a little- she is sick). They wonder if you could take down our website and have it working by 6am tomorrow?

Also - when it goes down can we put up either some kind of simple shell site or redirect people to

comment:31 Changed 5 years ago by

hi Ross

I talked to our ED Joel and Carrette (a little- she is sick). They wonder if you could take down our website and have it working by 6am tomorrow?

Also - when it goes down can we put up either some kind of simple shell site or redirect people to

comment:32 Changed 5 years ago by

Hi Ross

I am not a technical expert so I'll quote an email from Carrette where she adds that you all may be able to "restore it to a previously known good state"

Is that possible?

comment:33 Changed 5 years ago by

Hi Paul,

I could do some work on the site for you tonight. Would it be okay if we just had a maintenance message that says

"The site is under maintenance in the meantime please see We will be back up shortly."

Does this sound acceptable?


comment:34 Changed 5 years ago by

Hi Ross - Carrette here - I think that's fine. I also wanted to ask if you think the site might be causing the instability issues with

Let me know what you think.



comment:35 Changed 5 years ago by

Hi Carrette,

Well the site is up and running. I think we still may see this mysql error occasionally:

Error message
PDOException: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) in lock_may_be_available() (line 167 of /home/members/nyccah/sites/

I'm pretty sure I know what the biggest problem is on the site, the main theme. Currently, it appears that you're using a drupal 6 theme, modified to work in drupal 7. Unfortunately, not everything works correctly with the them, which I believe eats up some resources.

I've made significant changes to the site infrastructure, I'll give the highlights below, but you can read my notes file stored in the nyccah home directory, the file name is

First, I did not revert the site to an old state, but instead opted to upgrade everything. I took the following steps:

  1. Moved the civicrm database out of the drupal database. They are now independent nyccah_drup and nyccah_civi. They are controlled by different users.
  2. Upgraded drupal and civicrm. The civicrm upgrade also required upgrading webforms and webforms_civicrm, which in turn required installing options_element module.
    • I moved drupal core into the control panel, so it's now a web app, installed here. This should make it easier to manage core upgrades in the future.
  3. As part of this process, I had to rebuild the civicrm configuration. I suggest you go to these links and make sure all of the settings are correct:
  1. I also enabled memcached on the server and configured civicrm to use it. This should help civi run a bit faster.
  2. Fixed some of the obvious problems with the theme, which was calling css and js files directly from the template, thus making it impossible for drupal to do adequate caching.
  3. Fixed an errant <div> in /node/9 which was causing the theme to break on that page.

That's about it, though I did make sure civi was configured to use ssl. I remain concerned that we're going to continue seeing some mysql lock errors. If you notice those recur, please post them to this ticket. I'm not certain, but I believe we may end up needing to increase the RAM on stoney yet again.

Let me know how it goes,


comment:36 Changed 5 years ago by

Hi Ross

(Paul here).

I am not a highly technical person so I don't understand big chunks of this message but it seems like you redid a lot on the site to give it a strong foundation. Thank you.

It also sounds like we have some work to do on our end as far as fixing things.

I am just trying to get my feet under me to understand what the situation is with Mayfirst and us, could I talk to someone to learn about how we work with Mayfirst and what the ideal relationship is going forward?

Thanks Paul

comment:37 Changed 5 years ago by

Hi Ross

(Paul here)

Thanks for all your help.

Our Civi database is having some problems now. Ivonne, our development director, can log into Civi but can't access anything beyond logging in.

We did get a contribution today so we (happily) know we can recieve donations stil.

What should we do here?

I think Carrette is out sick today.

Question: if we used Salesforce instead of Civi for our fundraising/ contact management database, would we have these problems?


PS- when you update this ticket I don't get any notifications. my email is pbennett@… and my phone is 646.395.9122

Can this be fixed

comment:38 Changed 5 years ago by

  • Cc pbennett@… added
  • Keywords civicrm drupal added

Hi Paul,

The reason you don't get notifications is you are logging in with the nyccah account which has an email address associated with it. You could add a new user for yourself to the control panel here:

And then login to the support system with that user, add your email to the support system user preferences and then any tickets you open will automatically send emails to you.

You can also add your email address to the cc list at the bottom of any ticket. I just did that for this ticket.

As far as the civicrm problems, could you tell me what the username Ivonne is logging in with?

And just so you know, pretty much any piece of software you use for this purpose will come with it's own set of problems. Civicrm is quite stable if you keep it up to date, and get it configured properly.


comment:39 Changed 5 years ago by

Hi Ross - I already fixed the problem, had to remove those 2 performance settings (css and javascript) again for the seven admin theme to behave normally.

Everything else seems great, will have a more in-depth look tomorrow when I am in the office.

Thanks again, you've really been awesome, civi is already behaving with more stability with the databases separated.


comment:40 Changed 5 years ago by

Hi Carrette,

So it would be really helpful if we can figure out what's going on with the cached pages and why you think that's causing these problems. The impact on the server is huge when javascript and css aren't cached. You really need those configuration options.

Would it be possible for you to describe the problem you're encountering when these files are cached, so that we can attempt to solve the underlying problem?

Also, I wonder if, instead of disabling both css and javascript caching, we could try leaving one of them (probably css) enabled to see if we can more accurately diagnose where the problem is and at least take advantage of some caching.


Changed 5 years ago by

screenshot of civi page after logging in

comment:41 Changed 5 years ago by Hope I attached this correctly.

Hi Ross - here's a screenshot of what Ivonne was seeing, same thing I saw in Civi after updating to 7.24. I think another staff member saw the same thing yesterday. When i turned off the css setting, the theme started working again. Then I went a head and turned off the javascript settings as well and it improved even more. I believe I read somewhere that there is a conflict between the way Drupal and Civi caches pages and therein lies the problem. But this issue may have been fixed in newer versions, however, we seem to be experiencing this caching conflict.

comment:42 Changed 5 years ago by

Ah yes...this display isn't related to js and css caching. What has been happening, and I haven't figured out why is that civicrm's Resource URL setting is not always correct. In fact, it appears that you cannot save the correct resource url setting in the UI. The setting should be I have configured it in the civicrm.settings.php, however, if you go here and try to change CiviCRM Resource URL value it will not save a new value.

I just did a very dirty hack to fix this by copying the civicrm/packages and civicrm/i directories into webroot. That way when civi looks in the wrong location at least it will still find what it's looking for.

I'm looking into why this setting doesn't want to save, but I haven't found anything obvious yet.

I would still recommend trying to enable css caching, just to see if we experience any problems. But maybe do it on off hours, just to test.


comment:43 Changed 5 years ago by

Yes, I'll do it later tonight. About the not saving settings issue. That started happening after a consultant worked on the system, attempting to upgrade it from 6 to 7. He of course broke the system. Basically, he did a new installation instead of an upgrade and he created a whole new admin/superuser for the drupal installation. This caused a lot of problems in drupal and broke a few functions in civicrm, for instance, I could not enable any new components beyond what we were already using, we got that function back today. Over the last 2 years I've fixed what I could. The next thing was to tackle separating the databases, which you've done (again, you're awesome!).

I wonder if that problem is not a permissions issue. A few months ago I looked at the association table (not sure if I am remembering the table name correctly) but it's the table that associates drupal users with civi users, and the user associations were not correct. To tell you the truth it was a bit messed up but that was from all these people accessing the system and doing things at will without understand exactly what they are doing. Anyway, I tried fixing the association but it changed nothing and am not sure if using the sync user feature in civi would be useful.

Anyway, that's a very condensed version of some insight into how that particular problem came about.

comment:44 Changed 5 years ago by

Hi Ross - Happy Thanksgiving :-).

The site is down again, am getting the SQL exception below:

PDOException: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) in lock_may_be_available() (line 167 of /usr/local/share/drupal-7.24/includes/

Until we get the template situation fixed, is there anything I can do when this happens to get the site back up?

Also, Jamie has indicated that you do web development work, is upgrading our template to work with drupal 7 or customizing a drupal 7 template something you could take on for us as a consultant? Email me directly to talk about the template project.



comment:45 Changed 5 years ago by

It seems the entire server is down because both and is down as well. Is there a way that I can restart the vs? Looking at the support documentation, it seems like that's something only Mayfirst staff can do. Is it possible for someone at Mayfirst to restart the server?

It's extremely important that we get both sites back up as soon as possible. People who are volunteering today can't access the site.

I appreciate any help.



Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.