Opened 3 years ago

Last modified 7 months ago

#11675 assigned Bug/Something is broken

Account being spammed by "New subscription requests"

Reported by: https://id.mayfirst.org/blederer Owned by: https://id.mayfirst.org/jaimev
Priority: Urgent Component: Tech
Keywords: https://id.mayfirst.org/jamie Cc:
Sensitive: no

Description

Since April 10, one of our lists: resistanceinbrooklyn@… has been getting up to 25 messages of a day purporting to be New subscription requests. But they are clearly spam, as each day several appear with the same first part of the addressed followed by "+" and a different suffix.

See sample below.

How can we stop this time-consuming spam?

Thanks, Bob Lederer


Subject: New subscription request to list Resistanceinbrooklyn from killerkeemstar+xkts@… Date: Sat, 16 Apr 2016 15:44:51 -0400 From: resistanceinbrooklyn-owner@… To: resistanceinbrooklyn-owner@…

Your authorization is required for a mailing list subscription request approval:

For: killerkeemstar+xkts@… List: resistanceinbrooklyn@…

At your convenience, visit:

https://lists.mayfirst.org/mailman/admindb/resistanceinbrooklyn

to process the request.

Change History (49)

comment:1 Changed 3 years ago by https://id.mayfirst.org/jaimev

  • Keywords https://id.mayfirst.org/jamie added
  • Owner set to https://id.mayfirst.org/jaimev
  • Priority changed from Medium to Urgent
  • Status changed from new to assigned

Thank you for reporting this. I can see that not only your list but all of our lists are being spammed by subscriptions. We have a fail2ban filter meant to limit this kind of list subscription spamming but these subscription are using a pattern that is not triggering the fail2ban rules all of the time. I'm copying to jamie here as I continue searching for a solution.

comment:2 Changed 3 years ago by https://id.mayfirst.org/jaimev

So again, thank you for bringing this to our attention. It seems that bots were auto subscribing a large number of addresses to all of our lists using a gmail alias patterns.

We were finally able to stop bots from auto subscribing using the technique described here: https://nigelb.me/2015-08-26-mailman-attacks.html

This is actually almost identical to one of the administrative scripts shared by mailman's lead developer. https://www.msapiro.net/scripts/add_banned.py

We successfully banned new subscriptions matching the appropriate regex like this:

/usr/lib/mailman/bin/withlist -a -r ban "^[^+]+\+[^+]+@gmail\.com"

This stopped the attack but there were already a few thousand pending subscription requests spread out across all lists and some lists processed subscriptions automatically. I made conservative use of another two of Mark Shapiro's scripts https://www.msapiro.net/scripts/erase and https://www.msapiro.net/scripts/list_requests to delete the pending subscription requests what appeared to be false subscriptions.

The same process was repeated on leslie and assata.

Last edited 3 years ago by https://id.mayfirst.org/jaimev (previous) (diff)

comment:3 Changed 3 years ago by https://id.mayfirst.org/jaimev

  • Resolution set to fixed
  • Status changed from assigned to closed

comment:4 Changed 15 months ago by https://id.mayfirst.org/jamie

  • Resolution fixed deleted
  • Status changed from closed to assigned

More subscriptions in the same spirit - here are examples:

the_real_green_giant+taetmck@hotmail.com
therealgreengiant+bhsxaf@hotmail.co.uk
squoon+vatua@live.com
sqoonart+xkku@outlook.com

comment:5 Changed 15 months ago by https://id.mayfirst.org/jamie

I banning new subscriptions with:

/usr/lib/mailman/bin/withlist -a -r ban "^(the_real_green_giant|therealgreengiant|squoon|sqoonart)\+[^+]+@(hotmail\.com|hotmail\.co\.uk|live\.com|outlook\.com)"

comment:6 Changed 15 months ago by https://id.mayfirst.org/jamie

I then looked for all requests using this pattern with:

/usr/lib/mailman/bin/list_requests  | egrep -o "(the_real_green_giant|therealgreengiant|squoon|sqoonart)\+[^+]+@(hotmail\.com|hotmail\.co\.uk|live\.com|outlook\.com)" | tee delete.these.requests

And then deleted them with:

for email in $(sort delete.these.requests | sort | uniq); do /var/lib/mailman/bin/erase "$email"; done

I'm pretty sure we are not done with this issue... not sure how we can address this is in a better way to detect and stop this kind of abuse.

comment:7 Changed 15 months ago by https://id.mayfirst.org/jamie

This takes a really long time.

I sped things up for resistanceinbrooklyn by getting a list of the addresses with pending subscribes for them:

/usr/lib/mailman/bin/list_requests -l resistanceinbrooklyn  | egrep -o "(the_real_green_giant|therealgreengiant|squoon|sqoonart)\+[^+]+@(hotmail\.com|hotmail\.co\.uk|live\.com|outlook\.com)" | tee res.deletes

And then removing just those from just this list:

for email in $(cat res.deletes); do echo /var/lib/mailman/bin/withlist -r discard_address resistanceinbrooklyn "$email"; done

comment:8 Changed 15 months ago by https://id.mayfirst.org/jamie

I've re-started running the process to delete them all - but this time running under nice. It seems to take about 45 seconds for each address (and there are thousands of them).

comment:9 Changed 15 months ago by https://id.mayfirst.org/jaimev

I found a few references to this problem.

There was an old patch that was created. https://bugs.launchpad.net/mailman/+bug/1082746

And this whole thread is good. https://mail.python.org/pipermail/mailman-users/2016-December/081659.html

I also just updated #11683 with a proposal for doing some preventative maintenance on inactive lists.

comment:10 Changed 12 months ago by https://id.mayfirst.org/jamie

More problems. Now resistanceinbrooklyn is getting spammed with email in the form:

  • ryan+zlqqcw@…
  • ryan+zkjomz@…
  • ryan+ytlavae@…

It seems like someone is picking a fight with http://ryanestrada.com/

I just added that pattern to the ban list:

/usr/lib/mailman/bin/withlist -a -r ban "^ryan\+[^+]+@ryanestrada.com"

comment:11 Changed 12 months ago by https://id.mayfirst.org/jamie

I also wanted to add that we have a cron job that clears pending messages after 60 days, so manually clearing these is not necessary.

comment:12 Changed 11 months ago by https://id.mayfirst.org/jamie

Bob reported today more fake requests.

comment:13 Changed 11 months ago by https://id.mayfirst.org/jamie

I found them with:

./bin/list_requests  -l resistanceinbrooklyn
17 Resistanceinbrooklyn moderator request(s) waiting

Pending subscriptions:
    vetus+twinu@mail.gr (twinu) Sun Jan  7 14:58:13 2018
    vetus+qtpwh@mail.gr (qtpwh) Sun Jan  7 17:17:38 2018
    vetus+pzvhvi@mail.gr (pzvhvi) Sun Jan  7 17:20:51 2018
    vetus+vjr@mail.gr (vjr) Sun Jan  7 17:47:13 2018
    vetus+fxqb@mail.gr (fxqb) Sun Jan  7 17:54:06 2018
    vetus+ydvi@mail.gr (ydvi) Sun Jan  7 17:57:04 2018
    vetus+dfyyg@mail.gr (dfyyg) Sun Jan  7 19:35:49 2018
    vetus+skctsb@mail.gr (skctsb) Sun Jan  7 20:33:36 2018
    vetus+usbjcay@mail.gr (usbjcay) Sun Jan  7 20:51:23 2018
    vetus+extdd@mail.gr (extdd) Sun Jan  7 21:49:50 2018
    vetus+ziuiskh@mail.gr (ziuiskh) Sun Jan  7 21:55:13 2018
    vetus+xidqxm@mail.gr (xidqxm) Sun Jan  7 22:01:25 2018
    vetus+joh@mail.gr (joh) Sun Jan  7 22:21:15 2018
    vetus+fyjt@mail.gr (fyjt) Mon Jan  8 12:23:51 2018
    vetus+myjcfliy@mail.gr (myjcfliy) Mon Jan  8 13:24:41 2018
    vetus+snijaovl@mail.gr (snijaovl) Mon Jan  8 13:35:12 2018
    vetus+year@mail.gr (year) Mon Jan  8 13:55:56 2018

And banned with:

/usr/lib/mailman/bin/withlist -a -r ban "^vetus\+[^+]+@mail\.gr"

comment:14 Changed 11 months ago by https://id.mayfirst.org/jamie

I've been giving this some thought and wonder if there is a better way to protect ourselves against this kind of attack - specifically using postfwd at the postfix level.

The approach we are taking has some problems:

  • It is only applied against current lists (new lists created after today don't have the protection)
  • It has to be constantly updated when a slight variation in the email is used
  • It is hard to fix mistakes

comment:15 Changed 8 months ago by https://id.mayfirst.org/jaimev

I've just initiated a new ban with the following.

/usr/lib/mailman/bin/withlist -a -r ban "^neil\+[^+]+@neildeer\.com"

I'm now deleting pending subscriptions.

Also looking at this https://wiki.list.org/SEC/Controlling%20spam

comment:16 Changed 8 months ago by https://id.mayfirst.org/jaimev

So I think removing or rate limiting somehow the postfix aliases -admin-, -confirm-, -join-, -leave-, -request-, -subscribe-, -unsubscribe- and other commands somehow would be an ideal solution. Would that mean editing and maintaning our own /var/lib/mailman/bin/postfix-to-mailman.py ?

There is also the option of setting generic_nonmember_action to discard messages by default. I am not sure if that works for subscription requests as well. And unless I'm mistaken it would also mean admins get no notice when non members attempt to send any type of post or request.

comment:17 Changed 8 months ago by https://id.mayfirst.org/jamie

I think you are on the right track.

I also think postfwd might be the appropriate way to manage this.

It's not installed on leslie/mailman but we should add it. (see /etc/postfix/postfwd.cf on a mosh to get an idea of how this works).

The nice part with postfwd is that I think we could differential between the use of those email addresses from people mailing from the outside in and email messages generated internally by mailman.

comment:18 follow-up: Changed 8 months ago by https://id.mayfirst.org/jaimev

I spent some time trying to look at patterns for messages to the mailman aliases in the logs. The owner and unsubscribe aliases are the most abused. I've started off with a general rule rate limiting recipient addresses. It is by no means complete but should allow for legitimate use and might begin to provide some immediate relief to list owners. I think we will have to create separate rules for some aliases.

id=RULE001
  recipient=~.+-(admin|confirm|join|leave|request|subscribe|unsubscribe|owner)@.*
  client_address!=127.0.0.1
  action=rate(recipient/4/28800/REJECT only 4 msgs allowed every 8 hours allowed for $$recipient)

comment:19 Changed 8 months ago by https://id.mayfirst.org/jaimev

And based on some of the patterns of attacks we've seen above rate_limiting to mailman aliases based on sender_domain seems useful.

id=RULE002                                                                             
  recipient_localpart=~.+-(admin|confirm|join|leave|request|subscribe|unsubscribe|owner)
  client_address!=127.0.0.1                                                            
  action=rate(sender_domain/6/3600/REJECT only 6 msgs an hour allowed from $$sender_domain to mailman aliases)

comment:20 Changed 8 months ago by https://id.mayfirst.org/jaimev

I've pushed out to both leslie and assata. postfwd logs any actions it takes so we should be able to scan for any possible false positives.

comment:21 Changed 8 months ago by https://id.mayfirst.org/jamie

Great work Jaime! I think we are on the right track and the rules you are starting with seem the most likely to work.

My only other thought is restricting by sender IP address (also limited to just the special email addresses). We should never get more than a few requests to the special mailman addresses from the same IP in the same hour.

Based on these results, we also might try experimenting with daily limits (if we find the spammers keep coming back every hour to send more).

comment:22 Changed 8 months ago by https://id.mayfirst.org/jaimev

I can add something like below to filter the repeat ip addresses.

id=RULE003
  recipient=~.+-(admin|confirm|join|leave|request|subscribe|unsubscribe|owner)@.*
  client_address!=127.0.0.1
  action=rate(client_address/6/3600/REJECT only 6 msgs allowed every hour allowed to mailman aliases from $$client_address)

However based on what I'm seeing in the logs this particular attack almost never repeats ip addresses. It is actually kind of astounding the sheer volume of ip numbers they have at their disposal. Over 5000 in the last week.

One pattern that is worrisome is that they are also flooding the bounces aliases. Mailman needs to get feedback about the msgs to list subscribers that are really bouncing. We may need a more thought out rule to filter these correctly.

comment:23 Changed 8 months ago by https://id.mayfirst.org/jaimev

Wait i think I see what is happening now. The below message for example:

May 01 20:56:19 leslie postfix/smtpd[19869]: 180361DB52: client=cleveland.smtp.mayfirst.org[162.247.75.112]
May 01 20:56:19 leslie postfix/cleanup[19846]: 180361DB52: message-id=<20180502005618.E34072865@cleveland.smtp.mayfirst.org>
May 01 20:56:19 leslie postfix/qmgr[26297]: 180361DB52: from=<>, size=18123, nrcpt=1 (queue active)
May 01 20:56:19 leslie postfix/smtpd[19208]: 3410E1DB5D: client=localhost[127.0.0.1], orig_queue_id=180361DB52, orig_client=cleveland.smtp.mayfirst.org[162.247.75.112]
May 01 20:56:19 leslie postfix/cleanup[19846]: 3410E1DB5D: message-id=<20180502005618.E34072865@cleveland.smtp.mayfirst.org>
May 01 20:56:19 leslie postfix/qmgr[26297]: 3410E1DB5D: from=<>, size=18381, nrcpt=1 (queue active)
May 01 20:56:19 leslie postfix/smtp[19499]: 180361DB52: to=<friends-bounces+2728229563=qq.com@lists.mayfirst.org>, relay=127.0.0.1[127.0.0.1]:10026, delay=0.28, delays=0.07/0/0.05/0.16, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 3410E1DB5D)
May 01 20:56:19 leslie postfix/qmgr[26297]: 180361DB52: removed
May 01 20:56:19 leslie postfix/pipe[19226]: 3410E1DB5D: to=<friends-bounces+2728229563=qq.com@lists.mayfirst.org>, relay=mailman, delay=0.35, delays=0.15/0/0/0.2, dsn=2.0.0, status=sent (delivered via mailman service)
May 01 20:56:19 leslie postfix/qmgr[26297]: 3410E1DB5D: removed

This is actually a valid bounce response, to a message that was sent a week ago

1 leslie:~/tickets/11675# zgrep '8820E1DB52:' /var/log/mail.log*
/var/log/mail.log.5.gz:Apr 26 20:39:34 leslie postfix/smtpd[21836]: 8820E1DB52: client=localhost[127.0.0.1]
/var/log/mail.log.5.gz:Apr 26 20:39:34 leslie postfix/cleanup[22033]: 8820E1DB52: message-id=<mailman.6720.1524789573.20002.friends@lists.mayfirst.org>
/var/log/mail.log.5.gz:Apr 26 20:39:34 leslie postfix/qmgr[29367]: 8820E1DB52: from=<friends-bounces+2728229563=qq.com@lists.mayfirst.org>, size=15572, nrcpt=1 (queue active)
/var/log/mail.log.5.gz:Apr 26 20:39:35 leslie postfix/smtp[21820]: 8820E1DB52: to=<2728229563@qq.com>, relay=therest.smtp.mayfirst.org[162.247.75.112]:25, delay=0.94, delays=0.06/0/0.31/0.56, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 02A3D3E47)
/var/log/mail.log.5.gz:Apr 26 20:39:35 leslie postfix/qmgr[29367]: 8820E1DB52: removed

That message above was triggered by this original fake unsubscribe mail.

0 leslie:~/tickets/11675# zgrep '2728229563@qq.com' /var/log/mail.log*
/var/log/mail.log.5.gz:Apr 26 20:39:29 leslie postgrey[832]: action=pass, reason=triplet found, client_name=unknown, client_address=118.113.2.9, sender=2728229563@qq.com, recipient=Friends-unsubscribe@lists.mayfirst.org
/var/log/mail.log.5.gz:Apr 26 20:39:29 leslie postgrey[832]: action=pass, reason=triplet found, client_name=unknown, client_address=118.113.2.9, sender=2728229563@qq.com, recipient=Friends-unsubscribe@lists.mayfirst.org
/var/log/mail.log.5.gz:Apr 26 20:39:32 leslie postfix/qmgr[29367]: ED7161DB52: from=<2728229563@qq.com>, size=13769, nrcpt=1 (queue active)
/var/log/mail.log.5.gz:Apr 26 20:39:32 leslie postfix/qmgr[29367]: AB2F81DB5D: from=<2728229563@qq.com>, size=14013, nrcpt=1 (queue active)
/var/log/mail.log.5.gz:Apr 26 20:39:32 leslie clamsmtpd: 13F5A1: from=2728229563@qq.com, to=Friends-unsubscribe@lists.mayfirst.org, status=CLEAN
/var/log/mail.log.5.gz:Apr 26 20:39:35 leslie postfix/smtp[21820]: 8820E1DB52: to=<2728229563@qq.com>, relay=therest.smtp.mayfirst.org[162.247.75.112]:25, delay=0.94, delays=0.06/0/0.31/0.56, dsn=2.0.0, status=sent (250 2.0.0 Ok: queued as 02A3D3E47)

Removing all those backed up spurious outgoing mails from the queues on cleveland will probably stop these from coming in.

comment:24 in reply to: ↑ 18 Changed 8 months ago by https://id.mayfirst.org/dkg

Thanks for working on this, Jaime!

Replying to https://id.mayfirst.org/jaimev:

id=RULE001
  recipient=~.+-(admin|confirm|join|leave|request|subscribe|unsubscribe|owner)@.*
  client_address!=127.0.0.1
  action=rate(recipient/4/28800/REJECT only 4 msgs allowed every 8 hours allowed for $$recipient)

I'm a bit concerned that this rule is punishing the target of the flooding attack instead of punishing the attacker. Does this mean that a new list that people subscribe to by mail can only gain 4 new members every 8 hours, for example?

comment:25 Changed 8 months ago by https://id.mayfirst.org/jamie

Technically yes, but in practice I think most mailman subscriptions are added by the administrator, followed by people filling out the mailman generated form. I think very few people subscribe to a list by sending directly to the subscribe address itself.

For people who subscribe by filling out a form - they have to confirm their address, which would get caught in this filter. So, if any exceptions would be made - I think it would be for the confirm style addresses (in other words, leave confirm out of this rule and then create a second rule with just confirm that is more lax - like 50 per hour). I suspect that a confirm email sent by a spammer without a corresponding subscription request would simply get dropped without bugging any one - but not sure about that (I suppose mailman might try to alert the list admin about an handled confirmation?).

comment:26 Changed 8 months ago by https://id.mayfirst.org/dkg

hm, the subscription confirmation message has a reply-to address that looks like: foo-confirm+666e1c30cf05e31f08cd56eb74e17b03989947df@lists.mayfirst.org -- i don't know postfwd, but i don't think those match the regex described above, so the filter should be ok for the messages in that stage of the subscription process. i'm not sure why you think we should leave confirm out of this -- am i missing something? or does it trigger after the +suffix bit is stripped?

comment:27 Changed 8 months ago by https://id.mayfirst.org/jamie

Woops - my mistake. I assumed the responses all followed the same convention of listname-$COMMAND@. However, I just tested by sending 8 emails to support-team-confirm+123 and it was eventually blocked (it was blocked by rule003 - because that matched on 6 messages, but it shows that it triggers after the +suffix is stripped).

Also, as list owner I didn't get any notifications about those spurious email messages, and mailman reports:

vette:May 02 10:31:28 2018 (20002) No command, message discarded, msgid: <c85f0fca-82c8-0db2-b848-e12d070b0e38@mayfirst.org>

Which leads me to believe that we can safely remove confirm from all the rules since it doesn't cause back scatter, doesn't generate email messages to list owners, and could possibly interrupt legit subscriptions.

Also, as an FYI, Here's a command to get addresses that have been blocked:

grep postfwd /var/log/mail.log | grep RULE | grep REJECT| grep -Eo "sender=<[^>]*>, recipient=<[^>]*>" | sort | uniq

Geez. 1341 rejects since 6 AM this morning! And I don't see any obvious false positives. This is friggin' awesome :).

comment:28 Changed 8 months ago by https://id.mayfirst.org/jaimev

I just added the client address rule and removed "confirm" from the regex.

comment:29 Changed 7 months ago by https://id.mayfirst.org/blederer

I appreciate all your efforts, but the spamming started again on 5/11. There have been dozens of fake "new subscription requests." When you have time, can you see what you can do to stop this? Thanks.

comment:30 Changed 7 months ago by https://id.mayfirst.org/jamie

Oh damn. It's a million requests for the same email address: s7g7j7@yahoo.com

Which is causing us some problems delivering to yahoo

comment:31 Changed 7 months ago by https://id.mayfirst.org/jamie

I just put a quick emergency rule in place to block requests from that address.

Jaime: can you take a closer look to figure out if this is working and what a better long term strategy is?

comment:32 Changed 7 months ago by https://id.mayfirst.org/jamie

Also running all yahoo variations of this to clear the queue on cleveland:

for id in $(mailq -C /etc/postfix-yahoo-a | grep s7g7j7= | awk '{print $1}' | tr -d '*'); do postsuper -c /etc/postfix-yahoo-a -d $id; done

comment:33 Changed 7 months ago by https://id.mayfirst.org/jaimev

The rule you've added to limit that e-mail is fine however it is not being triggered. We are not receiving any e-mail directly from that address in the logs. Ican't find any evidence that it was used with the above subscribe, unsubscribe commands we filter.

All of the activity in the logs is from bounce messages. Mailman is sending out bounce messages to that address in mass as well as receiving them.

Postfwd is actually REJECTing some of those bounce messages based on rule #2 which doesn't seem right because that rule doesn't attempt to filter the "bounces" keyword at all. This is not what we want, mailman will probably properly discard the bounce messages if they are invalid but if they get sent back they'll probably just be sent again and cause problems for us with that server.

I'm not sure how to get ahead of this. At this point if we could just drop all incoming and outgoing bounce messages related to this address I think it would help. I'm looking at Postfwd docs.

comment:34 Changed 7 months ago by https://id.mayfirst.org/jamie

Hi Jaime - any more luck with this? I think dropping all incoming and outgoing messages for this address is better than nothing. We can work on more fine-tuned rules to stop this attack in the future but for now stopping it seems to be the priority. Thanks!!

comment:35 Changed 7 months ago by https://id.mayfirst.org/jaimev

Sorry , hadn't updated.

I attempted to do some thing via postfwd but I suspect postfwd isn't actually processing mail coming from localhost. Still investigating that.

For now I've added this line to /etc/postfix/transport and that does seem to stop the outgoing mail to that address.

s7g7j7@yahoo.com discard:

Since making that change and and after deleting mail in other yahoo postfix queues on cleveland, I don't see any more related mail waiting in cleveland's queues to go out.

I'm working on deleting all of the pending requests with ./bin/erase -p s7g7j7@yahoo.com

There is still mail coming in to a pattern that includes the word owner which is triggering postfwd rule. These are coming from cleveland, maybe bounced from yahoo? Still trying to work out what to do with that.

May 14 16:43:25 leslie postfix/smtpd[24782]: NOQUEUE: reject: RCPT from cleveland.smtp.mayfirst.org[216.66.15.30]: 554 5.7.1 <hugs-bounces+s7g7j7-owner=yahoo.com@lists.mayfirst.org>: Recipient address rejected: only 6 msgs an hour allowed from  to mailman aliases; from=<> to=<hugs-bounces+s7g7j7-owner=yahoo.com@lists.mayfirst.org> proto=ESMTP helo=<cleveland.smtp.mayfirst.org>

comment:36 Changed 7 months ago by https://id.mayfirst.org/jamie

Thanks Jaime for staying on this!

Does it seem that these subscriptions were coming in via a bot submitting the subscribe form rather than via email?

comment:37 Changed 7 months ago by https://id.mayfirst.org/jaimev

Just looked at the apache the logs andit looks like that is exactly what is happening. I don't think fail2ban rules based on ip's are enough to stop bot activity. There have been over 2500 unique ips attempting to subscribe since this morning.

comment:38 Changed 7 months ago by https://id.mayfirst.org/jaimev

I don't think we want to use recaptcha but it might be possible. https://www.dragonsreach.it/2018/02/26/adding-recaptcha-v2-support-mailman/

Adding a hidden form field or something like that would be a great alternative. I'm not completely sure how that would work though.

comment:39 Changed 7 months ago by https://id.mayfirst.org/jamie

I was just reading this too... I was looking at /var/log/mailman/subscribe. I wish there was a way to configure fail2ban to search for email addresses the way it searches for IP addresses and then have a ban action that adds the email address to the global ban list.

If an email address shows up more than X times in that log in a 1 hour period it should be globally banned.

Unfortunately, fail2ban seems to be hard-wired to work with IP addresses only. We could try to fashion our own version of fail2ban for this purpose, but that is not a great long term solution either.

comment:40 Changed 7 months ago by https://id.mayfirst.org/jaimev

My worry is the above strategy would only stop this particular variation of automated subscription. Other versions use different addresses same domain but even more advanced attacks use unique fake addresses. That will be hard to filter against.

How can we take advantage of these qualities?

  • Real user subscriptions are infrequent,
  • Real users can read and follow some instructions (capatcha, special value)
  • Real users might be willing to wait for a small delay

comment:41 Changed 7 months ago by https://id.mayfirst.org/jamie

Just as a proof of concept I wrote /root/ban-mailman-abusers. I just ran it and it outputs (for today's log):

0 leslie:~# bash ban-mailman-abusers 
Found bottled.snow@gmail.com with 160 matches.
/usr/lib/mailman/bin/withlist -a -r ban ^bottled.snow(\+[^+]+)?@gmail.com
0 leslie:~# 

However... I agree with you about its limitations (not to mention fragility) and am not convinced it's the right solution.

I think you are right about all of those qualities but outside of captcha, not sure how to take advantage of them.

comment:42 Changed 7 months ago by https://id.mayfirst.org/jaimev

What about using an nginx cache to rate limit? Any real person has to "GET /mailman/listinfo/listame" before they can "POST /mailman/subscribe/". Even the bots are smart enough to fake this. But they are fast, 10-15 seconds. We could deny any subsequent request to a regex from the same ip that is faster than X seconds.

comment:43 follow-up: Changed 7 months ago by https://id.mayfirst.org/jamie

I'm not sure the bots GET the page first - I just ran:

grep '/mailman/subscribe/hugs' /var/log/apache2/ssl_access.log

And I only see a bunch of POSTs.

comment:44 follow-up: Changed 7 months ago by https://id.mayfirst.org/jamie

Woops - just found a problem with our postfwd rule. I sent a message to the list mx-admin@lists.mayfirst.org and it was rejected because that list has recieved more than 4 messages in last two hours.

We didn't take into account lists that have -admin in the name!

Not sure best way to avoid this :(.

comment:45 in reply to: ↑ 43 Changed 7 months ago by https://id.mayfirst.org/jaimev

Replying to https://id.mayfirst.org/jamie:

I'm not sure the bots GET the page first - I just ran:

grep '/mailman/subscribe/hugs' /var/log/apache2/ssl_access.log

And I only see a bunch of POSTs.

The subscribe url is only triggered by the subscribe button POST action or sending directly I guess but grepping for the subscribe url in the logs and then back grepping for the the ip again shows a consistent pattern of the same ip visiting the listinfo url just before the POST request. That's pretty smart, a real human couldn't get to the subscribe button without first visiting the listinfo page.

while read -r line
do
  ip=$(echo "$line" | awk ' /POST \/mailman\/subscribe\// {print $1}');
  grep -B10 -F "${line}" /var/log/apache2/ssl_access.log | tac | grep -m2 "$ip" | tac
done < <(grep 'POST /mailman/subscribe/' /var/log/apache2/ssl_access.log | tail -n30)
186.215.137.234 - - [16/May/2018:18:54:55 -0400] "GET /mailman/listinfo/teaac HTTP/1.1" 200 12991 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
186.215.137.234 - - [16/May/2018:18:55:06 -0400] "POST /mailman/subscribe/teaac HTTP/1.1" 200 7271 "https://lists.mayfirst.org/mailman/listinfo/teaac" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0"
77.222.158.135 - - [16/May/2018:18:55:03 -0400] "GET /mailman/listinfo/justice4youth HTTP/1.1" 200 13086 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
77.222.158.135 - - [16/May/2018:18:55:12 -0400] "POST /mailman/subscribe/justice4youth HTTP/1.1" 200 7324 "https://lists.mayfirst.org/mailman/listinfo/justice4youth" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
167.114.211.122 - - [16/May/2018:18:55:19 -0400] "GET /mailman/listinfo/actuplist HTTP/1.1" 200 12980 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
167.114.211.122 - - [16/May/2018:18:55:28 -0400] "POST /mailman/subscribe/actuplist HTTP/1.1" 200 7331 "https://lists.mayfirst.org/mailman/listinfo/actuplist" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
217.77.221.209 - - [16/May/2018:18:55:22 -0400] "GET /mailman/listinfo/updates HTTP/1.1" 200 12645 "-" "Mozilla/5.0 (Windows NT 6.1; rv:54.0) Gecko/20100101 Firefox/54.0"
217.77.221.209 - - [16/May/2018:18:55:30 -0400] "POST /mailman/subscribe/updates HTTP/1.1" 200 7268 "https://lists.mayfirst.org/mailman/listinfo/updates" "Mozilla/5.0 (Windows NT 6.1; rv:54.0) Gecko/20100101 Firefox/54.0"

It is a slow process to compare all of these but over the past day's logs the average time between these visits is 10 seconds with only a few outliers above 20 seconds. I tried to time myself doing it manually, if I stop to read where to put the right info in the fields I take over 30 seconds. I think we could filter out anything under 20 seconds without affecting real human subscriptions.

I'd like to try the rate limiting approach overnight.

comment:46 in reply to: ↑ 44 Changed 7 months ago by https://id.mayfirst.org/jaimev

Replying to https://id.mayfirst.org/jamie:

Woops - just found a problem with our postfwd rule. I sent a message to the list mx-admin@lists.mayfirst.org and it was rejected because that list has recieved more than 4 messages in last two hours.

We didn't take into account lists that have -admin in the name!

Not sure best way to avoid this :(.

I think we can safely remove 'admin' from the regex, it is only there for compatibility with old versions of mailman.

http://www.list.org/mailman-member/node41.html

comment:47 Changed 7 months ago by https://id.mayfirst.org/jamie

Excellent - thanks for clarifying the GET/POST situation - sounds like a good plan!

comment:48 Changed 7 months ago by https://id.mayfirst.org/jaimev

Ok, the rate limiting is in place now. You can see the config file on dolores. It will only allow POST requests to the subscribe URL and they must com from the correct referral URL "listinfo" We can't be sure of an exact sequence with this method but if you try to request either URL too quickly you get sent to a javascript countdown timer that will send you back again.

comment:49 Changed 7 months ago by https://id.mayfirst.org/jaimev

Made some more adjustments to the nginx rate limiting which seems to being doing a good job of blocking fake web subscriptions now. I've been looking at postfwd again today to see what we can improve.

Please login to add comments to this ticket.

Note: See TracTickets for help on using tickets.