Opened 2 years ago

Last modified 2 years ago

#11969 assigned Bug/Something is broken is Down

Reported by: Owned by:
Priority: Urgent Component: Tech
Keywords: Cc:,
Sensitive: no


Hi - IO is down. At least since last night. Jackie

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Apache/2.4.10 (Debian) Server at Port 80

Change History (11)

comment:1 Changed 2 years ago by

  • Owner set to
  • Status changed from new to assigned

The varnish server is down on molina and I am unable to get it to restart yet. Still working on it.

comment:2 Changed 2 years ago by

  • Cc added

Ok i was unable to get teh varnish server working by replacing references to the old "io" interface ip number with the new one in both.




The above file seems to be a customized systemd unit file. As I understand unit files shouldn't be customized there, default distribution files should be overridden in /etc/systemd/system instead

Even after getting varnish up again the continues to return an error. After closer inspection I see the that apache is actually listening for input on the principal interface and requests to are actually going straight to apache.

The logs do not reveal the cause of the error.

It is still not clear to me what exactly what is going on in molina or how it is intended to work. Cc'ing ross and jamie.

comment:3 Changed 2 years ago by

I was able to find the error in the apache error logs:

0 molina:~# tail -n1 /home/members/interoccupynet/sites/
[Wed Jul 27 15:29:14.574315 2016] [core:alert] [pid 28460] [client] /home/members/interoccupynet/sites/ RewriteRule: cannot compile regular expression '^([0-9]+)\\/([^\\d\\/]+)([0-9]+)&.*=[0-9]+%[0-9]+.*%[0-9]+\\([0-9]+,[0-9]+\\)=\\(.**.*\\(.*%[0-9]+.*_.*\\(([0-9]+)H(.*)R\\([0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+\\),[0-9]+\\),.*_.*\\(([0-9]+)H(.*)R\\([0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+\\),[0-9]+\\)\\).*\\)%[0-9]+--%[0-9]+.*%[0-9]+%[0-9]+([0-9]+)[0-9]+$'
0 molina:~# 

comment:4 Changed 2 years ago by

It looks like the .htaccess file was recently modified (judging from the content of the file it appears to have been compromised or some automated .htaccess update script has gone nuts), which seems to be the source of the error:

0 molina:/home/members/interoccupynet/sites/ ls -l .htaccess
-r--r--r-- 1 interoccupynet interoccupynet 88955 Jul 24 23:55 .htaccess
0 molina:/home/members/interoccupynet/sites/

I moved the .htaccess file into ~/compromised-files-2016-july and copied the file called .htaccess-bak into place and now the site is loading... however, I would strongly encourage you to fix the compromise if you can or debug why the .htaccess file has so many rewrite rules in it. We can help if you need it.

comment:5 Changed 2 years ago by

Ah, thanks jamie, I'd seen that error but didn't trace it back to the .htaccess file.

Do you know if the varnish server in molina should still be in use? The only site profile it has is for so it must have been used at some point. It seems plausible that the domain was directed back to molina's main ip at some point because of the configuration issues with varnish there.

comment:6 Changed 2 years ago by

I'd suggest purging varnish from molina. I don't think it's been in use for a while. We can always put it back if we need it - or better yet, re-configure to use dorothy.

comment:7 Changed 2 years ago by

Done. varnish has been purged from molina and references to varnish removed from molina.pp.

comment:8 Changed 2 years ago by

It looks like something funky is going on with the IO site. It's taking over a minute to resolve. Does that have something to do with what you just did with the varnish server?

Last edited 2 years ago by (previous) (diff)

comment:9 Changed 2 years ago by

Wow - 150 apache processes running with a load over 100. I can't quite tell which site on the server is doing the damage since the server is configured to run mod_php (so all sites run as the same user).

I've restarted apache and will try to figure it out.

comment:10 Changed 2 years ago by

Weird, now it is back to normal. I suggest we switch the server to use php5-fpm - which I think will be more stable and at least help us debug more effectively. Do you know how many of the sites on this server are still active?

comment:11 Changed 2 years ago by

I do not know which sites are being used. I only know about the site.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.