Opened 5 months ago

Last modified 4 weeks ago

#14068 assigned Bug/Something is broken

Website not working

Reported by: Owned by:
Priority: Urgent Component: Tech
Keywords: Cc:
Sensitive: no


Hi, our blog is not loading. What is the problem? This is like a daily occurrence. Something is obviously wrong. Please help us track down the issue.

Change History (39)

comment:1 Changed 5 months ago by

  • Owner set to
  • Status changed from new to assigned

Hi the problems seems to be intermittent which has made it hard to track down. I am looking for any source of resource contention in your shared server or its physical host and looking at your blogs wordpress files for any anomalies.

comment:2 Changed 5 months ago by

Thanks. It is still down now so a good hour.

comment:3 Changed 5 months ago by

Actually since the moment I first responded to the ticket I have been able to load without problems. We are however making plans to move your host chavez to a new physical server to alleviate some of the load. Would you like me to activate a TLS certificate for your site so it can be reached securely over https and browsers won't complain about it?

Last edited 5 months ago by (previous) (diff)

comment:4 Changed 5 months ago by

Yes, thanks.

comment:5 Changed 5 months ago by

As you may have seen in our service advisory we have been experiencing general slowdowns on several servers , especially on chavez where your site is. We were consistently seeing your wordpress blog site reach the limit of php processes on chavez and tracked that back to a cron job that was recurring every few seconds. - - [24/Sep/2018:13:28:42 -0400] "POST /blog/wp-cron.php?doing_wp_cron=1537810122.5123519897460937500000 HTTP/1.1" 200 166 "" "WordPress/4.9.8;" - - [24/Sep/2018:13:28:43 -0400] "POST /blog/wp-cron.php?doing_wp_cron=1537810123.8951709270477294921875 HTTP/1.1" 200 166 "" "WordPress/4.9.8;" - - [24/Sep/2018:13:28:44 -0400] "POST /blog/wp-cron.php?doing_wp_cron=1537810124.8592278957366943359375 HTTP/1.1" 200 166 "" "WordPress/4.9.8;" - - [24/Sep/2018:13:28:56 -0400] "POST /blog/wp-cron.php?doing_wp_cron=1537810136.6724200248718261718750 HTTP/1.1" 200 166 "" "WordPress/4.9.8;" - - [24/Sep/2018:13:29:01 -0400] "POST /blog/wp-cron.php?doing_wp_cron=1537810141.9011449813842773437500 HTTP/1.1" 200 166 "" "WordPress/4.9.8;"

That cron job is named "inpsyde_phone-home_checkin" and is apparently triggered by the "backwpup" wordpress plugin. I've found mention of it in several forum posts on wordpress support site. It is only supposed to run every 14 days but like other users have reported it is out of control on this instance. I have disabled the plugin for your site.

comment:6 Changed 4 months ago by

Thanks. We're still having major issues with our site and blog crashing every day. This has become a major issue for us. Its a drag on staff productivity and it makes us look like crap to people trying to access our page. What else can we do here?

comment:7 Changed 4 months ago by

Hi, we are sorry about this. I can see chavez under heavy load right now and I am working on resolving the issue.

comment:8 Changed 4 months ago by

While it has been difficult to pinpoint the exact cause of problems on chavez we think it may be related to sudden spikes in memory consumption that subsequently provoke a flood of disk io slowing down all processes. We have taken some steps to limit memory consumption and will be monitoring for changes.

comment:9 Changed 3 months ago by

Is there anything we can do? Our blog seems to go down for 5-10 mins at a time several times every day.

comment:10 Changed 3 months ago by

  • Cc added

There were some issues with chavez this morning still running backups from overnight.

Now that we've moved the chavez vm to a new physical server we can add an SSD backed virtual drive for mysql. I can get that setup tonight and I think this should provide better performance for all sites on chavez.

comment:11 Changed 3 months ago by

Thanks. Another issue that just popped up is on the web configuration page we are getting the following error message: You cannot modify a record that with the status set to pending-update.

comment:12 Changed 3 months ago by

I've just restored the web configuration.

comment:13 Changed 3 months ago by

Thanks, working now.

comment:14 Changed 3 months ago by

Our blog is down right now. Again, this is very critical to us. What can we do to improve this? This issue is not affecting our general website as often. Just the blog. Please advice.

comment:15 Changed 3 months ago by

Is it still down for you? Is this the address:

That site appears to be working at the moment. But I see that your site exceeded the limit on the allowed number of processes at around 11:38 am which would have resulted in it refusing additional connections.

comment:16 Changed 3 months ago by

Its back up now but it is regularly going down for 5 or so mins at a time. Is there anything we can do about this processes issue?

comment:17 Changed 3 months ago by

It seems you are running two wordpress sites and a static html site under the same hosting order. If we could split up the three sites into their own hosting orders that would help a lot. It would mean you would have 12 proceses per site instead of having to split the 12 processes.

It also will help us understand which of the three sites is causing the problem.

If you need help with splitting them up let us know.

It would mean setting up new domain names, e.g. and for each site. However, we can automatically redirect from the old addresses to the new ones.

comment:18 Changed 3 months ago by

Yes, we should do this. Please advise. I will speak to our IT guy about it.

comment:19 Changed 3 months ago by

Hi let me know if you need any help getting this done , I think it is something we've talked about before.

comment:20 Changed 3 months ago by

Thanks, we are working on this. Should take a couple weeks. Our blogs are down again now for the past 10 minutes or so. Anything that can be done now?

comment:21 Changed 3 months ago by

working on it now...

comment:22 Changed 3 months ago by

Things should be better now on chavez.

comment:23 Changed 3 months ago by

It isn't clear to us excatly why yet, but the wordpress instance is attracting several thousands of POST requests from the google robots. This is unusual because most legitimate bots send only GET requests. POST requests generally create more load on the server. Google does say however that they do use POST requests for pages that "are missing information and/or look completely broken without the resources returned from POST". The URL attracting these requests is created by the wordpress-popular-posts plugin on your site. I have disabled this temporarily. We'd like to leave this off a few days to see if there is any change in behavior.

comment:24 Changed 3 months ago by

Thanks. I don't know if that's the case because /blog and /building blocks are both down again now.

comment:25 Changed 3 months ago by

It seems like it is getting worse. We have long periods of time when these sites are down.

comment:26 Changed 3 months ago by

Working on it...

comment:27 Changed 3 months ago by

chavez is back to normal.

Would you be willing to give us persmission to break up your sites into separate hosting orders? We can do it fairly easily for you.

That would allow us to better pinpoint the activity generated by your username and figure out why it is so resource hungry.

comment:28 Changed 3 months ago by

Yes, we have our IT guy Roger working on it. I will ask him to comment here to check in on that status. Our sites are currently all down, both blogs and regular now.

comment:29 Changed 3 months ago by

Yes, please proceed to break up your sites into separate hosting orders for us and we will follow up on this ticket.

comment:30 Changed 3 months ago by

But our sites are still down now so anything you can do would be great. Thanks.

comment:31 Changed 3 months ago by

Hi, sorry I've just restarted apache on chavez.

This time there wasn't any clear resource contention but searching the apache logs I see this.

[Tue Dec 04 12:46:25.435813 2018] [mpm_worker:error] [pid 19097:tid 140146692650176] AH00286: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

comment:32 Changed 3 months ago by

I've increased the number of apache worker processes on chavez following jamie's example to resolve similar issues for colin in ticket #

comment:33 Changed 2 months ago by

Thanks, do you know the timeline on splitting the hosting orders? I want to have our IT guy aware so he is available to go fix any issues that may arise because of it. Thanks.

comment:34 Changed 2 months ago by

I can start working on this tomorrow afternoon so you can monitor for problems over the weekend.

comment:35 Changed 2 months ago by


comment:36 Changed 2 months ago by

I wasn't able to start this yesterday but spent several hours today trying to separate the wordpress folder into it's own site at

I was able to move the folder and restore wordpress there but could not get subpages like guided-tours and building-blocks working again and other elements working again.

Honestly all of and subsites appears to be an interweaving tangled mess. In my humble opinion no website should be created this way. I want to tone down that last comment. You may have a perfectly good reason for using this meshing strategy between multiple wp installs. I do think finding a way to separate to the subsites cleanly will make maintenance easier. Unfortunately I don't think this is something I can do for you without more knowledge of how the underlying structure is intended to work.

I've commented out my redirects and the site continues as it was before.

Last edited 2 months ago by (previous) (diff)

comment:37 Changed 6 weeks ago by

Thanks, yes we know its convoluted and are just trying to make things work like this for the next few months as we develop a new site. Our developer will be working on fixing these issues next week. Meanwhile, our sites are in and out up and down pretty consistently so anything you could do to prevent these outages would be greatly appreciated. Thanks.

comment:38 Changed 4 weeks ago by

Hi our and are down. Anything you can do? Thanks.

comment:39 Changed 4 weeks ago by

Hi, sorry for the delay. The problem appears to have been temporary or the server processes were restarted. I see both sites online now.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.