Opened 11 days ago

Last modified 10 days ago

#14283 assigned Question/How do I...?

Long wait times with healthcare-now.org

Reported by: https://id.mayfirst.org/joseph Owned by: https://id.mayfirst.org/joseph
Priority: Medium Component: Tech
Keywords: colin.mayfirst.org php apache Cc: https://id.mayfirst.org/healthcarenow, https://id.mayfirst.org/jamie
Sensitive: no

Description

Hey y'all,

I know we've had a variety of issues recently (slow backups, backup bloat, reverse proxy, etc.) but we're now having issues with load times on the website. The server doesn't appear to be under any load, but WP page wait times are upwards of 4-5 seconds and Civi page wait times are 20+ seconds. I'm trying to isolate what may have changed in the recent weeks that could be causing this. My initial debugging hasn't revealed anything, so I'm trying to gather more information. Have y'all changed anything about PHP or Apache on colin recently? Relatedly, is it possible to add Apache monitoring to Munin?

Joseph

Change History (8)

comment:1 Changed 11 days ago by https://id.mayfirst.org/jaimev

  • Owner set to https://id.mayfirst.org/jaimev
  • Status changed from new to assigned

Joseph I don't we think we've changed anything and looking at munin and dircetly colin I don't see any resource contention or disk io issues at the moment. Maybe this is something we can try and correlate to whatever requests you were processing on the site when you noticed the slowdowns?

comment:2 Changed 11 days ago by https://id.mayfirst.org/jaimev

  • Cc https://id.mayfirst.org/jamie added

comment:3 Changed 10 days ago by https://id.mayfirst.org/jamie

I think this was the cause of the slow down:

[Mon Dec 03 16:33:55.226835 2018] [mpm_worker:error] [pid 25559:tid 139948708037824] AH00287: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting

Since each web page load can include dozens of connections to the server, when you reach the max, you might get some of your connections made, but then your page has to wait until those connections complete before additional connections are available to complete the page load, so it feels slow.

The server is well-resourced, so we could run out of apache workers without showing any sign of strain on the server.

I just boosted colin from the default of 150 clients to 250.

If we go too high, then when we have too many apache clients, the system will start to suffer. If we go too low, then we may be stopping connections when we have the resources to handle them.

So... let's try 250 and see how it goes. If we see that we've reached that max again, we can boost it again.

comment:4 Changed 10 days ago by https://id.mayfirst.org/jamie

  • Resolution set to fixed
  • Status changed from assigned to feedback

I'm going to code this ticket as pending feedback... if this solves the problem then we're done! If it comes back, we can re-open.

comment:5 follow-up: Changed 10 days ago by https://id.mayfirst.org/joseph

  • Resolution fixed deleted
  • Status changed from feedback to assigned

Thanks for the help y'all. I've also been digging into the application side of this problem and found a few others things as well.

  • WP's core caching wasn't enabled, so I've done that by adding define('ENABLE_CACHE', TRUE); to wp-config.php.
  • WP's default cron seems poorly optimized, especially for multisite and the amount of traffic HCN gets, so I've disabled the core cron, again with define('DISABLE_WP_CRON', true); in wp-config.php. I've created a replacement as a job with the users crontab.
  • Caldera Forms was contacting https://logger.calderaformspro.com for some reason (still looking into why). I've commented out the line in the code that does this.

I'm going to keep this ticket open and assign it to myself for now.

comment:6 Changed 10 days ago by https://id.mayfirst.org/joseph

  • Owner changed from https://id.mayfirst.org/jaimev to https://id.mayfirst.org/joseph

comment:7 Changed 10 days ago by https://id.mayfirst.org/jamie

Thanks Joseph!

comment:8 in reply to: ↑ 5 Changed 10 days ago by https://id.mayfirst.org/joseph

Kevin from Tadpole confirms the same problem with this weird callback. He's going to ask one of the Caldera Forms developers what it's for.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.