Opened 3 months ago

Closed 2 months ago

#13720 closed Bug/Something is broken (fixed)

Slow www.leftforum.org website

Reported by: https://id.mayfirst.org/leftforum Owned by: https://id.mayfirst.org/jaimev
Priority: High Component: Tech
Keywords: Cc: ashley@…, marcus@…, https://id.mayfirst.org/jamie
Sensitive: no

Description

Hello Mayfirst,

we are experiencing a very slow website on www.leftforum.org now.

It could be our new tech or it could be the mayfirst server.

Can you please check if there is something you can do on your end?

When you need a new computer let us know.

Marcus

Change History (12)

comment:1 Changed 3 months ago by https://id.mayfirst.org/jaimev

  • Cc https://id.mayfirst.org/jamie added
  • Owner set to https://id.mayfirst.org/jaimev
  • Status changed from new to assigned

I am seeing that stokely does manage to use 100% of both cpu cores allocated to it momentarily but it doesn't appear to stay in the red constantly. If it the slowness continues to be a problem we might consider moving the site behind our nginx caching server.

I will let jamie respond about our hardware needs.

comment:2 Changed 3 months ago by https://id.mayfirst.org/jamie

The left forum really needs extra resources for about 2 weeks every year and these are the two weeks :).

I just boosted your RAM and also doubled your CPU (pending a restart) which I will manage tonight during off hours.

Jaime - I would suggest we put them behind the nginx cache as soon as possible as well. I expect to see a huge spike in traffic to the site over the coming two weeks as people prepare for the conference.

comment:3 Changed 3 months ago by https://id.mayfirst.org/jamie

I've just doubled the CPU and increased the memory by 50%. The site should be moving much faster. I also enabled aggregation of CSS and Javascript files which will also speed things up. We had just a few minutes of downtime for these changes.

Given that we should expect lots of users to access the same pages over and over again in the coming months... I think the next step to speed things up is to use our caching services.

Jaime: can you add leftforum to dolores?

comment:4 Changed 3 months ago by https://id.mayfirst.org/jaimev

I can start getting this set up but I have some questions about how to correctly avoid caching anything sensitive with a drupal civicrm setup.

comment:5 Changed 3 months ago by https://id.mayfirst.org/jaimev

Well I've gone ahead and moved the leftforum site behind our caching server on dolores. It seems that drupal includes custom query strings along with status resources like images which effectively negates the cache for those urls. Stilll investigating if we should/how to get around that.

comment:6 Changed 3 months ago by https://id.mayfirst.org/jamie

Thanks Jaime.

The site is not running CiviCRM - so that's one less thing!

I noticed connection errors this morning - and got around them by increasing the max number of PHP connections from 12 to 36.

I also increased the APC cache from 256 to 512MB.

I'm still seeing about 10 seconds per page load - which I think can still be improved. It seems like the page is not being cached via nginx - not sure if some tweaking is needed.

comment:7 Changed 3 months ago by https://id.mayfirst.org/jaimev

Yeah, still researching this one. I added add_header X-Cache-Status $upstream_cache_status; to the config so that we cna use curl to check the headers.

I can confirm that the home page and other locations were not hitting cache.

I've added proxy_ignore_headers "Cache-Control" "Expires"; to default location block and that seems to be improving results.

0 jaimev@mayday:/tmp$ curl -I https://www.leftforum.org
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 26 May 2018 18:25:24 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Vary: Accept-Encoding
X-Drupal-Cache: MISS
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: no-cache, must-revalidate
X-Content-Type-Options: nosniff
Content-Language: en
X-Frame-Options: SAMEORIGIN
X-Generator: Drupal 7 (https://www.drupal.org)
Link: <https://www.leftforum.org/sites/default/files/logo/LF_red.png>; rel="image_src",<https://www.leftforum.org/>; rel="canonical",<https://www.leftforum.org/>; rel="shortlink"
Vary: Accept-Encoding
X-Cache-Status: HIT

Also in the static file location block I added proxy_cache_key $proxy_host$uri; which I hope forces nginx to use the cache for the url without args. a better solution might be for drupal not to send content with the cache busting random string query.

comment:8 Changed 3 months ago by https://id.mayfirst.org/jamie

Thanks for putting in this work Jaime - just got report from Marcus that some functions were not working so for now I changed the DNS to go directly to the site until we can sort it out.

The report is:

Images for speakers, panels etc. can still be uploaded, but the actual image is not displaying anymore. Although he file is on the server.

Second the page logs me (and others) out although I/we just logged in....

On chrome browser the login window creates an ajax error and let's people not log in at all.

Can this be because of the nigix cache or the other stuff you did last night?

See #13744

comment:9 Changed 3 months ago by https://id.mayfirst.org/jaimev

Oh no. The login problem likely has to do with a change I made while trying to find the issue with the home page caching. I will change that back now. Not sure about the issues with the images but I will undo the changes I made to the static image file caching mechanism and see if that helps. I'll make those changes and switch the DNS back to the cache. If there are any problems let us know and we'll switch it back right away.

comment:10 Changed 3 months ago by https://id.mayfirst.org/leftforum

Thanks for speeding up a little.

We experience downtime very often and we aren't at high pitch yet. Can you increase, memory max user capacity and all what can be maxed?

What do you need for more speed and more reliability to keep site online and functional even in high usage?

Is this the max already we can get?

If not please set the wheels to maximum at least til Sunday....

Thank you, Marcus

comment:11 Changed 3 months ago by https://id.mayfirst.org/jamie

Hi Marcus - I just boosted your allowed PHP processes again to 72 - that should be more than the server can handle. We've maxed out the CPUs and memory as much as we can without affecting other members.

The last thing we can do - with your permission - is to experiment with more aggressive caching. This *may* cause problems for logged in users who are making changes to the content - (or it might not). What do you think? Would you like us to do a bit more tinkering to see if we can speed things up?

comment:12 Changed 2 months ago by https://id.mayfirst.org/leftforum

  • Resolution set to fixed
  • Status changed from assigned to closed

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.