Opened 2 years ago

Last modified 2 years ago

#11969 assigned Bug/Something is broken

Interoccupy.net is Down

Reported by: https://id.mayfirst.org/jackie Owned by: https://id.mayfirst.org/jaimev
Priority: Urgent Component: Tech
Keywords: Cc: https://id.mayfirst.org/jamie, https://id.mayfirst.org/ross
Sensitive: no

Description

Hi - IO is down. At least since last night. Jackie

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Apache/2.4.10 (Debian) Server at interoccupy.net Port 80

Change History (11)

comment:1 Changed 2 years ago by https://id.mayfirst.org/jaimev

  • Owner set to https://id.mayfirst.org/jaimev
  • Status changed from new to assigned

The varnish server is down on molina and I am unable to get it to restart yet. Still working on it.

comment:2 Changed 2 years ago by https://id.mayfirst.org/jaimev

  • Cc https://id.mayfirst.org/jamie https://id.mayfirst.org/ross added

Ok i was unable to get teh varnish server working by replacing references to the old "io" interface ip number with the new one in both.

/etc/varnish/backends/molina

and

/lib/systemd/system/varnish.service

The above file seems to be a customized systemd unit file. As I understand unit files shouldn't be customized there, default distribution files should be overridden in /etc/systemd/system instead

Even after getting varnish up again the interoccupy.net continues to return an error. After closer inspection I see the that apache is actually listening for input on the principal interface and requests to interoccupy.net are actually going straight to apache.

The logs do not reveal the cause of the error.

It is still not clear to me what exactly what is going on in molina or how it is intended to work. Cc'ing ross and jamie.

comment:3 Changed 2 years ago by https://id.mayfirst.org/jamie

I was able to find the error in the apache error logs:

0 molina:~# tail -n1 /home/members/interoccupynet/sites/interoccupy.net/logs/error.log
[Wed Jul 27 15:29:14.574315 2016] [core:alert] [pid 28460] [client 54.165.230.219:50581] /home/members/interoccupynet/sites/interoccupy.net/web/.htaccess: RewriteRule: cannot compile regular expression '^([0-9]+)\\/([^\\d\\/]+)([0-9]+)&.*=[0-9]+%[0-9]+.*%[0-9]+\\([0-9]+,[0-9]+\\)=\\(.**.*\\(.*%[0-9]+.*_.*\\(([0-9]+)H(.*)R\\([0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+\\),[0-9]+\\),.*_.*\\(([0-9]+)H(.*)R\\([0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+,[0-9]+\\),[0-9]+\\)\\).*\\)%[0-9]+--%[0-9]+.*%[0-9]+%[0-9]+([0-9]+)[0-9]+$'
0 molina:~# 

comment:4 Changed 2 years ago by https://id.mayfirst.org/jamie

It looks like the .htaccess file was recently modified (judging from the content of the file it appears to have been compromised or some automated .htaccess update script has gone nuts), which seems to be the source of the error:

0 molina:/home/members/interoccupynet/sites/interoccupy.net/web# ls -l .htaccess
-r--r--r-- 1 interoccupynet interoccupynet 88955 Jul 24 23:55 .htaccess
0 molina:/home/members/interoccupynet/sites/interoccupy.net/web#

I moved the .htaccess file into ~/compromised-files-2016-july and copied the file called .htaccess-bak into place and now the site is loading... however, I would strongly encourage you to fix the compromise if you can or debug why the .htaccess file has so many rewrite rules in it. We can help if you need it.

comment:5 Changed 2 years ago by https://id.mayfirst.org/jaimev

Ah, thanks jamie, I'd seen that error but didn't trace it back to the .htaccess file.

Do you know if the varnish server in molina should still be in use? The only site profile it has is for interoccupy.net so it must have been used at some point. It seems plausible that the domain was directed back to molina's main ip at some point because of the configuration issues with varnish there.

comment:6 Changed 2 years ago by https://id.mayfirst.org/jamie

I'd suggest purging varnish from molina. I don't think it's been in use for a while. We can always put it back if we need it - or better yet, re-configure to use dorothy.

comment:7 Changed 2 years ago by https://id.mayfirst.org/jaimev

Done. varnish has been purged from molina and references to varnish removed from molina.pp.

comment:8 Changed 2 years ago by https://id.mayfirst.org/jackrabbit

It looks like something funky is going on with the IO site. It's taking over a minute to resolve. Does that have something to do with what you just did with the varnish server?

Last edited 2 years ago by https://id.mayfirst.org/jackrabbit (previous) (diff)

comment:9 Changed 2 years ago by https://id.mayfirst.org/jamie

Wow - 150 apache processes running with a load over 100. I can't quite tell which site on the server is doing the damage since the server is configured to run mod_php (so all sites run as the same user).

I've restarted apache and will try to figure it out.

comment:10 Changed 2 years ago by https://id.mayfirst.org/jamie

Weird, now it is back to normal. I suggest we switch the server to use php5-fpm - which I think will be more stable and at least help us debug more effectively. Do you know how many of the sites on this server are still active?

comment:11 Changed 2 years ago by https://id.mayfirst.org/jackrabbit

I do not know which sites are being used. I only know about the IO.net site.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.