Bulk Mail Relay

See #8424 for implementation details.

See the email deliverability page for context.

See individual mail relay for relaying individual email.

See filter-check - a script for detecting if our email is going to spam boxes.

Big Picture

The over all concept is: most corporate providers allow mail through based on the ratio of delivered mail without complaints to bounced mail and complaints (complaints means people who click the "this is spam" link).

So, our strategy is to configure all of our bulk email servers to use the same dedicated servers to relay email to specific big providers, so we increase the volume of legit mail to help tip the balance when a member (inevitably) sends a bunch of bad addresses or has users click the "this is spam" button.

In other words, all email destined for AOL (for example), regardless of which server the email originates on, gets relayed to AOL via aol.smtp.mayfirst.org. Meanwhile, aol.smtp.mayfirst.org can point to one IP address or, ideally, two or three (so if one IP address gets blocked, we simply turn it off and rely on the remaining two).

The benefits of this strategy are:

• build a better reputation for any given email provider by combining all sent messages from all MF/PL servers so they originate from one, or a few, IP address
• provide flexibility so we can easily redirect messages being blocked on one server to be redirect to a different relay server
• permanently or temporarily rate-limit sending messages to certain providers, without affecting other providers
• take special action on messages just before being relayed (see, for example, our technique to avoid strict DMARC rules). By maintaining servers that only relay email out, we can safely apply rules that will not affect incoming messages.

Details

The following email list servers are currently using this system:

• assata.mayfirst.org (provides both bulk.mayfirst.org and lists.people-link.net)
• leslie.mayfirst.org (provides lists.mayfirst.org and lists.critpath.org)
• brown.mayfirst.org (used by Rochester IMC)
• morales.mayfirst.org (used by portside)
• marti.mayfirst.org (used by PTP)

This strategy should also be adopted by mx3.mayfirst.org and mx25.mayfirst.org.

We currently have two servers setup as dedicated mail relay servers (cleveland on malaka and rustin on florence). These servers run multi-postfix instances (see #6662 for more information on the setup). That means, each server has multiple instances of postfix running on different IP addresses.

We have setup special domain names for each of the main mail providers:

• All hotmail/outlook.com/etc is relayed via: msn.smtp.mayfirst.org
• Earthlink/Mindspring/IGC, etc is relayed via: earthlink.smtp.mayfirst.org
• AOL email is relayed via: aol.smtp.mayfirst.org
• Yahoo email is relayed via: yahoo.smtp.mayfirst.org
• Gmail: gmail.smtp.mayfirst.org
• Comcast: comcast.smtp.mayfirst.org
• Everything else goes through: therest.smtp.mayfirst.org

Future plans include adding:

• att.net (bellsouth, etc)

Each of our bulk mail servers is configured to relay email to the appropriate relay by using the transport_maps Postfix configuration parameter.

Each domain name has a 600 second time-to-live and points to one or more of the IP addresses assigned to the relay servers.

Special relay

In addition to the relays listed above which are designed for bulk email, we also have relays called "nospam." The server a different purpose. The nospam relays are for relaying email messages on behalf of our MOSH'es. MOSH'es relay the following email:

• Forwarding email addresses. An MF/PL user can configure their email address to forward to another address, onsite or offsite.
• Auto responder. An MF/PL user can configure an auto-response every time they receive an email message
• Web generated email. A web site (e.g. Drupal or WordPress) often sends email, such as password reminders (or massive amounts of spam if it is compromised). NOTE: CiviCRM or web sites designed to send mass email should be specially configured to relay via bulk.mayfirst.org)

All three kinds of email have a very large spam ratio and often cause the sending server to be blocked. As a result, the nospam relay is designed to drop all email with a spamassassin score of 5 or above. Note: it should only be used to relay less important email. Our typically policy is never to discard spam and let the user have the option of reviewing it. The nospam relay is a compromise of this principle to ensure consistent deliverability.

How Do I…

Review how well email is being delivered?

Run mf-mail-stats-print on rustin and/or cleveland. Here's sample output:

0 rustin:~# mf-mail-stats-print
postfix-aol-a:total:642
postfix-aol-a:bounced:62
postfix-aol-a:deferred:7
postfix-aol-a:sent:573

postfix-earthlink-a:total:452
postfix-earthlink-a:bounced:70
postfix-earthlink-a:deferred:0
postfix-earthlink-a:sent:382

postfix-msn-a:total:707
postfix-msn-a:bounced:87
postfix-msn-a:deferred:0
postfix-msn-a:sent:620

postfix-msn-b:total:649
postfix-msn-b:bounced:72
postfix-msn-b:deferred:9
postfix-msn-b:sent:568

postfix-therest-a:total:87281
postfix-therest-a:bounced:22242
postfix-therest-a:deferred:5404
postfix-therest-a:sent:59635

0 rustin:~#

Get a breakdown of which instance is filling the mailq

Run:

mf-mailq-distribution

View the mailq for a given instance

Run:

mf-postfix-queue-report /etc/{queue-name}

Flush deferred email from one queue to another queue

If deferred email is building up in one instance (referred to below as the clogged instance), you can move it to another instance (referred to below as the target instance). For example if aol-b is deferring email (clogged), but aol-a is delivering (target), you can move all deferred email from msn-a to msn-b by following these steps. NOTE: This only works if you move the messages between instances on the same server.

• Figure out which instance is running with which IP address:

  0 cleveland:~# ip addr | grep aol
      inet 216.66.23.44/27 brd 216.66.23.63 scope global eth0:aola
      inet 162.247.75.211/24 brd 162.247.75.255 scope global secondary eth0:aolb
  0 cleveland:~#
  

• Pull all deferred messages on hold on the clogged instance:

  0 cleveland:~# postsuper -c /etc/postfix-aol-b/ -h ALL deferred
  postsuper: Placed on hold: 127 messages
  0 cleveland:~#
  

• Move all held messages to the target maildrop folder

  0 cleveland:~# mv /var/spool/postfix-aol-b/hold/* /var/spool/postfix-aol-a/maildrop/
  0 cleveland:~#
  

• Clean up. Deferred messages have a corresponding file in the defer queue with the reason why they were deferred. Delete these files.

  for file in $(find /var/spool/postfix-aol-b/defer -type f); do name=$(echo $file | cut -d/ -f6,7) && [  ! -f "/var/spool/postfix-aol-b/deferred/$name" ] && rm -f "$file"; done
  

Warm up a new IP address for a given relay instance

For example, to add a new IP address for therest.smtp.mayfirst.org:

• Allocate new IP address
• Create a Host record that ties the IP address to the host name (e.g. rustin.smtp.mayfirst.org or cleveland.smtp.mayfirst.org). Set the time to live to 86400 (it's important to have along ttl since some providers do a reverse DNS lookup of the sending IP address and will punish you if the ttl is too short).
• Add this instance to either rustin.pp or cleveland.pp. Be sure to use rate limiting when warming up a new IP address. See the yahoo relays (which are permanently rate limited) for an example of how to do it.
• When you are sure it's configured properly, create an A record from the $instance.smtp.mayfirst.org to the IP address. When adding a new IP address, you may not want a lot of email to go to it initially. Instead, you can "warm it up" by controlling the ratio of IP addresses in use for the given domain. For example, you can add give A records for the existing IP address and one A record for the new one. After month, remove one of the old A records and add a new one for the new IP address.

Configure a bulk email server to use this setup

• Add the IP address of the bulk email server to the puppet file manifests/global.pp, to the variable $mfpl_relaying_ips.
• Be sure to push to both cleveland and rustin and restart postfix on both servers for good measure
• Add the following to the new server's /etc/postfix/main.cf file:

  transport_maps = hash:/etc/postfix/transport.relay
  

If main.cf already has a line for transport_maps, you can add a new value following a comma, e.g.:

transport_maps = hash:/etc/postfix/transport,hash:/etc/postfix/transport.relay

• Add the following to the server's pp file:

  m_postfix::map { "/etc/postfix/transport.relay":
   source => "puppet:///modules/mayfirst/postfix/transport.relay",
   replace => "yes"
  }