Changes between Initial Version and Version 1 of clearing-spam-backscatter-from-mailq

Timestamp:

Sep 18, 2012, 10:32:26 PM (12 years ago)

Author:

Nat Meysenburg

Comment:

--

clearing-spam-backscatter-from-mailq

@@ +1 @@
+[[PageOutline]]
+There have been a few tickets lately (#6166, #6199), where an email account has gotten hacked and used to send spam. What results is a ton of bounces from other mail servers that are denying a message. This is commonly referred to as [https://en.wikipedia.org/wiki/Backscatter_%28e-mail%29 backscatter]. 
+
+This is a breakdown of the process used to resolve #6166 and #6199.
+
+= How it starts =
+We generally don't find out about these events until they are well underway, and members are complaining about the slowness of the mail server.
+
+The tool for checking the mail queue is `mailq`. Run without arguments, it will list a little about all the email that is currently in the queue. You can count the number of emails roughly with `mailq | wc -l`. The gotcha here is that each email takes three lines, meaning the number you're looking for is a third of that value. Under normal conditions (as I have observed them) there should only be about 40-100 emails in the queue. 
+
+{{{
+0 chavez:~/ticket6199# mailq |wc -l
+125
+}}}
+
+When we looked for #6166, that number was around 985,000; meaning about 300,000 spam messages. #6199 was only about 30,000 messages.
+
+= Figuring out the culprit =
+Generally, the beginning of the mailq will contain a lot more of the backscatter (though its mostly backscatter). The first thing to think about is finding out what account is receiving all of the backscatter. To get an idea for that, have a look at the first 20 or so emails.
+
+{{{
+mailq |head -60 |more
+}}}
+
+Look for a recurring email address. If there's a backscatter problem, there's a greater chance that the mailbox is being delivered to a user on the server (rather than forwarding it), the string to look for is something `USERNAME@chavez.mayfirst.org` (assuming `chavez` is the server that you're looking at).
+
+For this example lets call the user `spam-account`, with an email of `spam-account@example.com`.
+
+= Figuring out where all the spam lives =
+Depending on the nature of the backscatter, the actual messages might be in several places within the spool.
+
+Run this to get see which dirs might be full.
+
+{{{
+SPOOL=/var/spool/postfix/; for dir in $(ls $SPOOL);do echo "$dir: $(ls $SPOOL/$dir |wc -l)"; done
+}}}
+
+It looks something like this when run:
+{{{
+0 chavez:~# SPOOL=/var/spool/postfix/; for dir in $(ls $SPOOL);do echo "$dir: $(ls $SPOOL/$dir |wc -l)"; done
+active: 2
+bounce: 0
+corrupt: 0
+defer: 16
+deferred: 16
+dev: 1
+etc: 6
+flush: 0
+hold: 0
+incoming: 0
+lib: 12
+maildrop: 0
+pid: 17
+private: 26
+public: 5
+saved: 0
+trace: 0
+usr: 1
+}}}
+
+If the server were dealing with queue issues, some of those numbers would be in the thousands.
+
+'''Note:''' There's a gotcha here. the "defer" and "deferred" directories actually have sub directories. So if the counts don't look promising in the incoming or active dirs, try those.
+
+Make a note of which directories are full of spam.
+
+= Getting the mail flow started again =
+Since our members seem to actually want their emails delivered in a timely fashion, we need to non-destructively clean out the queue.
+
+Follow each of these steps!
+
+== Step 0: Disable the account ==
+Log into the control panel, and find  `spam-account@example.com` and disable it.
+
+== Step 1: Stop postfix ==
+{{{
+service postfix stop
+}}}
+
+== Step 2: Create clean spool dirs ==
+Now that our postfix spool directories aren't being written to, we can clean out directories. Since we don't want to destroy any real emails that are in the queue, we want to move the spool directories without deleting the mail itself.
+
+Lets say the problem dirs in this case are `incoming` and `active`. First move them to something with a different name that postfix won't write to. For the scripts that I've written I use two variant suffixes. `.spamfull` and `.name-collisions`. I'll get to `.name-collisions` later, for now, we're just moving the dir.
+
+'''Note:''' Remember to change these if you're not dealing with incoming or active.
+
+{{{
+mv /var/spool/postfix/incoming /var/spool/postfix/incoming.spamfull
+mv /var/spool/postfix/active /var/spool/postfix/incoming/active.spamfull
+}}}
+
+Now we need to recreate the original dirs.
+
+{{{
+mkdir /var/spool/postfix/incoming
+mkdir /var/spool/postfix/active
+}}}
+
+And make sure the permissions are correct.
+
+{{{
+chmod 700 /var/spool/postfix/incoming
+chmod 700 /var/spool/postfix/active
+chown postfix:postfix /var/spool/postfix/incoming
+chown postfix:postfix /var/spool/postfix/active
+}}}
+
+Now we have empty queue directories that postfix can write to. When it restarts all new mail should start getting handled as normal.
+
+== Step 3: Start Postfix ==
+{{{
+service postfix start
+}}}
+
+Email should once again be flowing at reasonable speeds... and now we can worry about separating out backscatter spam from the ham.
+
+= Reinserting the good messages back into the mail queue =
+So now that we have the new mail getting delivered, we need to get the real email messages delivered to their recipients.
+
+This step takes a little bit of art. What we're trying to do is come up with a couple of grep expressions that will hopefully match all of the spam messages. Thankfully, spammers usually have similar stuff in all of the emails they send out. 
+
+Have a look at the first few emails in the `.spamfull` directory, particularly looking for mails to `spam-account@example.com`. That's your first grep expression. In #6166 the second pattern was that all the messages were pointing to various domains with a `media.php` file. In #6199 all of them had an email address in the body (`swiftvalue@yahoo.cn`).
+
+Figuring this out takes a little work because the emails themselves are in a binary file format, so we need to run them through strings. I tend to work on a subset of mails.
+
+{{{
+for foo in $(ls incoming.spamful/ |head -20);do strings incoming.spamful/$foo > ~/ticket6199/testmails/$foo-strings-spamful;done
+}}}
+
+That will give you twenty messages, hopefully containing spam and ham. '''Be sure to delete this dir when you are done, since there are some actual emails there!'''
+
+Test your greps. For example, lets say our two criteria are `spam-account@example.com` and `media.php`. Any mail that contains both of those is spam.
+
+This is an edited example of how to look for such things. Assuming that you are in the test mail dir
+
+{{{
+0 chavez:~/ticket6199/testmails# for foo in $(ls); do echo "$foo: $(grep -c -e spam-account@example.com -e swiftvalue@yahoo.cn $foo)"; done
+000338CE68-strings-spamful: 3
+0003A711C9-strings-spamful: 3
+001127DED8-strings-spamful: 3
+001387199E-strings-spamful: 3
+0014B8C579-strings-spamful: 0
+001FD8D839-strings-spamful: 2
+002138C8FE-strings-spamful: 3
+002278D8B0-strings-spamful: 2
+0023D4EF87-strings-spamful: 3
+00285C0EC-strings-spamful: 2
+0029970F32-strings-spamful: 0
+002A38C95D-strings-spamful: 3
+002CA8DCD9-strings-spamful: 2
+002D7713F4-strings-spamful: 3
+002FF4E7D8-strings-spamful: 2
+003568D3ED-strings-spamful: 0
+003FF8CB54-strings-spamful: 2
+0048B8DBD1-strings-spamful: 2
+004F37DCD8-strings-spamful: 0
+004F87D469-strings-spamful: 0
+}}}
+
+The lines with 0 at the end are ham, and everything else spam. Probably in this case, some results with a 1 would have counted as ham (since there could perhaps be legit emails to the spam-account address; however since it is disabled it probably doesn't matter.
+
+So assuming we're satisfied with our grep patterns, you now need to create a handful of scripts to do the actual move of ham back into the mail queue for delivery to their recipients.
+
+Remember the `.name-collisions` directories. These are there out of an abundance of caution. None of us were sure how postfix assigns names to the email files, so the script checks to make sure it doesn't exist before moving something overtop of it. If there is a name collision we move the message into that directory. With both tickets, this wasn't an issue.
+
+Before running the script(s) we first need to create those dirs. As per our example:
+
+{{{
+mkdir /var/spool/postfix/incoming.name-collisions
+mkdir /var/spool/postfix/active.name-collisions
+}}}
+
+This script will need to be modified to meet the various criteria of you spam search. It will only work on the dirs that don't have sub directories (eg: not defer and deferred)
+
+{{{
+#!/bin/bash
+
+# simple script by nat to cleanup the spam backscatter in mailq that was killing the
+# server on 9/18/12
+
+SPAM_DIR=/var/spool/postfix/incoming.spamfull
+HAM_DIR=/var/spool/postfix/incoming
+COLLISION_DIR=/var/spool/postfix/incoming.name-collisions
+
+for message in $(ls -U $SPAM_DIR/);do
+  if [[ $(strings $SPAM_DIR/$message |grep -c -e spam-account@example.com -e media.php) -eq 0 ]];then 
+    if [[ ! -f $HAM_DIR/$message ]]; then 
+      mv -n $SPAM_DIR/$message $HAM_DIR/;
+    else
+      mv -n $SPAM_DIR/$message $COLLISION_DIR/
+    fi
+  fi
+done
+exit 0
+}}}
+
+If you have to traverse sub directories, something like this should work. Again, remember to change the grep patterns.
+
+{{{
+#!/bin/bash
+
+# simple script by nat to cleanup the spam backscatter in mailq that was killing the
+# server on 9/5/12
+
+SPAM_DIR=/var/spool/postfix/defer.spamfull
+HAM_DIR=/var/spool/postfix/defer
+COLLISION_DIR=/var/spool/postfix/defer.name-collisions
+for dir in $(ls $SPAM_DIR);do 
+  for message in $(ls -U $SPAM_DIR/$dir);do
+    if [[ $(strings $SPAM_DIR/$dir/$message |grep -c media.php) -eq 0
+    || $(strings $SPAM_DIR/$dir/$message |grep -c spam-user) -eq 0 ]];then 
+      if [[ -d $HAM_DIR/$dir && ! -f $HAM_DIR/$dir/$message ]]; then 
+        mv -n $SPAM_DIR/$dir/$message $HAM_DIR/$dir/
+      else
+        if [[ ! -d $COLLISION_DIR/$dir ]]; then
+          mkdir $COLLISION_DIR/$dir
+        fi
+        mv -n $SPAM_DIR/$dir/$message $COLLISION_DIR/$dir/
+      fi
+    fi
+  done
+done
+exit 0
+}}}
+
+Once these are done, all of the real emails should be delivered, and you now have a directory full of spam.
+
+These scripts are non-destructive. They move things, not delete them. So if you screwed up your greps, you could always refine or repeat.
+
+I have been using one script per directory that I'm cleaning. Future versions could probably do more than one at a time. Future versions may also benefit from turning the grep patterns into variables as well.