Opened 11 years ago

Closed 11 years ago

#475 closed Bug/Something is broken (fixed)

Malcolm is experiencing excessively high loads

Reported by: Jamie McClelland Owned by: Jamie McClelland
Priority: Urgent Component: Tech
Keywords: malcolm.mayfirst.org Cc:
Sensitive: no

Description

While investigating the workers liberty ticket I noticed loads between 20 and 25 on malcolm (vmstat reported 20 - 25 processes trying to run, with 0 blocked).

Top revealed that a php process owned by workersliberty was consistently using a lot of resources. Tailing the workers liberty web log showed that the IP address 88.2.213.173 was hitting the site heavily and appeared to be accessing the user register page frequently (indicating a spammer).

Change History (10)

comment:1 Changed 11 years ago by Jamie McClelland

I just banned 88.2.213.173 using our /usr/local/sbin/ban-ip-address.ssh script.

Loads are still high.

comment:2 Changed 11 years ago by Jamie McClelland

I just repeated the process with 90.209.118.108.

comment:3 Changed 11 years ago by Jamie McClelland

Resolution: fixed
Status: newclosed

Load has returned to normal. The banned IP addresses are still banned (but the next time we reboot the server the ban will not be saved). I'm inclined to leave the ban (but not make it permanent across a reboot) unless we get a complaint from a user not being able to connect.

comment:4 Changed 11 years ago by Daniel Kahn Gillmor

Keywords: malcolm.mayfirst.org added

comment:5 Changed 11 years ago by Jamie McClelland

Resolution: fixed
Status: closedreopened

I just banned two more IP addresses:

77.204.128.72 and 88.25.223.183

I'm running the following command to look for offenders:

tail -f web.log | grep comment-form

Which produces results like:

88.25.223.183 - - [22/Jan/2008:12:13:06 -0500] "GET \
/user/register?destination=node/1906%23comment-form HTTP/1.1" \
200 21804 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; \
.NET CLR 1"

When I see more than 2 - 3 in 5 second period I suspect that it should be banned.

comment:6 Changed 11 years ago by Jamie McClelland

More blocked IP addresses:

79.179.179.59 and 88.117.118.232

comment:7 Changed 11 years ago by Daniel Kahn Gillmor

Is /usr/local/sbin/ban-ip-address.sh committed to svn? If it is, i couldn't find it. If it's not, maybe it should be?

Looking at it on malcolm, i don't see a comparable "unban" step -- are these addresses ever going to be able to be unbanned? If so, how will that be done?

A quick way to see what HTTP connections are currently open to different IP addresses is:

lsof -i -n | grep :www | grep -v LISTEN | awk '{print $8}' | sed 's/.*>\(.*\):.*/\1/' | sort | uniq -c | sort -n

This lists all open, active connections using port 80 (www) and counts up the number of connections made from any particular IP address.

How are you proposing to distinguish between legitimate access and illegitimate access, though? What if an IP address is simply NAT'ing a whole subnet of people who should be legitimately accessing the web site? That could easily account for 2-3 in a 5-second period, as could a recursive wget, a prefetching browser, or any number of other situations.

comment:8 in reply to:  7 ; Changed 11 years ago by Jamie McClelland

Replying to https://id.mayfirst.org/dkg:

Is /usr/local/sbin/ban-ip-address.sh committed to svn? If it is, i couldn't find it. If it's not, maybe it should be?

Not it's not. I think adding it to svn along with a suite of similar tools would be really useful (such as the lsof command you have below). I think in situations like this it is much better to have a one line bash script than to have to remember the syntax. I think this is related to [tickiet:70 developing a DOS protocol].

Looking at it on malcolm, i don't see a comparable "unban" step -- are these addresses ever going to be able to be unbanned? If so, how will that be done?

Unban would be a good script for the suite as well :).

How are you proposing to distinguish between legitimate access and illegitimate access, though? What if an IP address is simply NAT'ing a whole subnet of people who should be legitimately accessing the web site? That could easily account for 2-3 in a 5-second period,

Yes - but wouldn't account for the URL that was being accessed. All of these IP addresses are accessing the register for a new user account page (which is captcha protected) and then redirect to the comment field of a particular node.

as could a recursive wget, a prefetching browser, or any number of other situations.

I think that ultimately we will need a better method. I seem to recall an Apache module that helps with this situation - but can't seem to google it now. I think this is pretty important - given what happened today with Workers Liberty and yesterday with the WSF2008. It's not just a threat from malicious users, it's also a threat from buggy RSS readers. I'm less interested in differentiating between a user that is purposefully being bad from a user that is accidentally being bad, and more interested in differentiating between a user that is accessing the site in a way that is significantly affecting server load, whether it's on purpose or not.

comment:9 in reply to:  8 Changed 11 years ago by Daniel Kahn Gillmor

Replying to https://id.mayfirst.org/jamie:

I'm less interested in differentiating between a user that is purposefully being bad from a user that is accidentally being bad, and more interested in differentiating between a user that is accessing the site in a way that is significantly affecting server load, whether it's on purpose or not.

This is a good observation, but to be clear: if the outcome is banning by IP, it's not about users, it's about IP addresses. For example, if a member plans a workshop about how to use a particular MFPL-hosted service, and the workshop is held at a single location behind a NAT it would be pretty likely to significantly affect server load. It would be bad for the workshop, the member, and MFPL itself if we were to automatically cut the workshop off from the webapp because there was high load based on that IP address.

comment:10 Changed 11 years ago by Jamie McClelland

Resolution: fixed
Status: reopenedclosed

I opened ticket #726 to address the broader issue of protecting ourselves against DOS like attacks (and posted a response to dkg's last comment). I'm closing this ticket because the immediate problem it describes is not a problem any more.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.