Opened 3 months ago

Last modified 3 months ago

#14283 assigned Question/How do I...?

Long wait times with

Reported by: Owned by:
Priority: Medium Component: Tech
Keywords: php apache Cc:,
Sensitive: no


Hey y'all,

I know we've had a variety of issues recently (slow backups, backup bloat, reverse proxy, etc.) but we're now having issues with load times on the website. The server doesn't appear to be under any load, but WP page wait times are upwards of 4-5 seconds and Civi page wait times are 20+ seconds. I'm trying to isolate what may have changed in the recent weeks that could be causing this. My initial debugging hasn't revealed anything, so I'm trying to gather more information. Have y'all changed anything about PHP or Apache on colin recently? Relatedly, is it possible to add Apache monitoring to Munin?


Change History (8)

comment:1 Changed 3 months ago by

  • Owner set to
  • Status changed from new to assigned

Joseph I don't we think we've changed anything and looking at munin and dircetly colin I don't see any resource contention or disk io issues at the moment. Maybe this is something we can try and correlate to whatever requests you were processing on the site when you noticed the slowdowns?

comment:2 Changed 3 months ago by

  • Cc added

comment:3 Changed 3 months ago by

I think this was the cause of the slow down:

[Mon Dec 03 16:33:55.226835 2018] [mpm_worker:error] [pid 25559:tid 139948708037824] AH00287: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting

Since each web page load can include dozens of connections to the server, when you reach the max, you might get some of your connections made, but then your page has to wait until those connections complete before additional connections are available to complete the page load, so it feels slow.

The server is well-resourced, so we could run out of apache workers without showing any sign of strain on the server.

I just boosted colin from the default of 150 clients to 250.

If we go too high, then when we have too many apache clients, the system will start to suffer. If we go too low, then we may be stopping connections when we have the resources to handle them.

So... let's try 250 and see how it goes. If we see that we've reached that max again, we can boost it again.

comment:4 Changed 3 months ago by

  • Resolution set to fixed
  • Status changed from assigned to feedback

I'm going to code this ticket as pending feedback... if this solves the problem then we're done! If it comes back, we can re-open.

comment:5 follow-up: Changed 3 months ago by

  • Resolution fixed deleted
  • Status changed from feedback to assigned

Thanks for the help y'all. I've also been digging into the application side of this problem and found a few others things as well.

  • WP's core caching wasn't enabled, so I've done that by adding define('ENABLE_CACHE', TRUE); to wp-config.php.
  • WP's default cron seems poorly optimized, especially for multisite and the amount of traffic HCN gets, so I've disabled the core cron, again with define('DISABLE_WP_CRON', true); in wp-config.php. I've created a replacement as a job with the users crontab.
  • Caldera Forms was contacting for some reason (still looking into why). I've commented out the line in the code that does this.

I'm going to keep this ticket open and assign it to myself for now.

comment:6 Changed 3 months ago by

  • Owner changed from to

comment:7 Changed 3 months ago by

Thanks Joseph!

comment:8 in reply to: ↑ 5 Changed 3 months ago by

Kevin from Tadpole confirms the same problem with this weird callback. He's going to ask one of the Caldera Forms developers what it's for.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.