Opened 3 months ago

Closed 3 months ago

Last modified 3 months ago

#14796 closed Bug/Something is broken (fixed)

POST Request exceedingly slow

Reported by: Ross Owned by: Jamie McClelland
Priority: Urgent Component: Tech
Keywords: acholonu.mayfirst.org slow-post Cc: jamie@…
Sensitive: no

Description

Does anyone have an idea of why a POST request for a drupal 7 site would be ridiculously slow, but all other operations would function normally?.

This has to do with the secure part of saharareporters. It's not simply that the process is slow, but it takes in the neighborhood of 2 minutes for the connection to be established. The logs show zero activity for the entire two minutes after the browser has attempted to take an action to post a story. Eventually, the content does get posted. The same behavior occurs when attempting to delete a story. Normally I would expect mysql being overloaded when something like this occurred, but that is not the case in this situation.

I have tried everything I can think of and have ruled out traffic or server load being the culprit. I ran tcpdump on my local machine and saw similar behavior. Initially, a few connections were made with the server hosting saharareporters, but then for the next two minutes a bunch of connections were made with some third parties like fastly.com and amazon.

Finally, I used iptables to only allow my machine to communicate with the server. This improved nothing. I'm completely mystified by this behavior.

Any suggestions welcome.

Change History (5)

comment:1 Changed 3 months ago by JaimeV

Owner: set to Jamie McClelland
Status: newassigned

Which server/site is this happening on? It sounds like you've checked the same things I would but let's get jamie's input.

comment:2 Changed 3 months ago by Ross

Oh sorry this is on acholonu, secure.sahararreporters.com.

comment:3 Changed 3 months ago by Jamie McClelland

Wow, this is mysterious. What about third party spam protection services (like askimet? Maybe when you post content it is sent out for verification? - although that wouldn't happen on deletions...). If you download the site and database to your local machine does it happen also? Anything change prior to the problem starting?

If the site is perfectly responsive for other actions, then it sounds like some kind of security protection gone awry. The flood table lists IPs that are blocked - maybe something there?

How about checking watchdog for any clues? And the apache error log might shed some light.

And of course restarting apache, php fpm and mysql when you get desperate to just make it work. And drush cc all if you haven't already.

comment:4 Changed 3 months ago by Ross

Resolution: fixed
Status: assignedclosed

Thanks Jamie, your suggestions helped me find the issue. The big change that happened over the weekend was that ngugi.mayfirst.org was moved to another colo in Tokyo, causing a change in it's ip address. While I changed the ip for cloudflare, I did not change it's dns entry.

The drupal site is configured to purge any page on update using the domain name ngugi.mayfirst.org, so the purging had to wait to time out before allowing the post process to continue.

Anyway, thanks for your help.

~/ross

comment:5 Changed 3 months ago by Jamie McClelland

Nice sleuthing! Glad you figured it out. What a obscure one!

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.