Changes between Version 5 and Version 6 of bulk-mail-relay


Ignore:
Timestamp:
Apr 21, 2014, 1:14:53 PM (5 years ago)
Author:
Jamie McClelland
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • bulk-mail-relay

    v5 v6  
     1[[PageOutline]]
     2
    13= Bulk Mail Relay =
    24
     
    911The over all concept is: most corporate providers allow mail through based on the ratio of delivered mail without complaints to bounced mail and complaints (complaints means people who click the "this is spam" link).
    1012
    11 So, our strategy is to use dedicated servers to send email to specific big providers, so we increase the volume of legit mail to help tip the balance when a member (inevitably) sends a bunch of bad addresses or has users click the "this is spam" button.
     13So, our strategy is to configure all of our bulk email servers to use the same dedicated servers to relay email to specific big providers, so we increase the volume of legit mail to help tip the balance when a member (inevitably) sends a bunch of bad addresses or has users click the "this is spam" button.
     14
     15In other words, all email destined for AOL (for example), regardless of which server the email originates on, gets relayed to AOL via aol.smtp.mayfirst.org. Meanwhile, aol.smtp.mayfirst.org can point to one IP address or, ideally, two or three (so if one IP address gets blocked, we simply turn it off and rely on the remaining two).
    1216
    1317== Details ==
     18
    1419The following email list servers are currently using this system: assata.mayfirst.org (provides both bulk.mayfirst.org and lists.people-link.net) and leslie.mayfirst.org (provides lists.mayfirst.org and lists.critpath.org). This strategy should also be adopted by morales.mayfirst.org (provides lists.portside.org), mx3.mayfirst.org and mx25.mayfirst.org.
    1520
    1621We currently have two servers setup as dedicated mail relay servers (cleveland in XO and rustin in Telehouse). These servers run multi-postfix instances (see #6662 for more information on the setup). That means, each server has multiple instances of postfix running on different IP addresses. NOTE: only rustin is currently in play.
    1722
    18 Each of our main email list servers can relay their email to any one of these instances on a domain-by-domain basis by putting a line in the email list server's /etc/postfix/transport.relay file that indicates the domain of the destination email address followed by the domain of the relay server that should send the email.
    19 
    2023We have setup special domain names for each of the main mail providers:
    2124
    22  * msn.smtp.mayfirst.org
    23  * earthlink.smtp.mayfirst.org
    24  * therest.smtp.mayfirst.org
     25 * All hotmail/outlook.com/etc is relayed via: msn.smtp.mayfirst.org
     26 * Earthlink/Mindspring/IGC, etc is relayed via: earthlink.smtp.mayfirst.org
     27 * AOL email is relayed via: aol.smtp.mayfirst.org
     28 * Everything else goes through: therest.smtp.mayfirst.org
    2529
    2630Future plans include adding:
    2731 * yahoo.smtp.mayfirst.org
    28  * aol.smtp.mayfirst.org
    29  * graveyard.smtp.mayfirst.org (see below)
     32 * gmail.smtp.mayfirst.org
    3033
    31 Each of our email list servers is configured to relay mail by domain to the appropriate provider, with all deferred email going to the graveyard. Similarly, deferred email on each of the relay instances also goes to the graveyard. This avoids deferred email build up, which than exerbates the problems on our good IP addresses.
     34Each of our bulk mail servers is configured to relay email to the appropriate relay by using the transport_maps Postfix configuration parameter.
    3235
    33 Each domain name will have a 600 second time-to-live and will point to one or more of the IP addresses assigned to the relay servers.
    34 
    35 We started by assigning each of these domains to IP addresses that are "warmed up" and currently able to send lots of email.
    36 
    37 Over time, we can warm up additional IP addresses by using round robin DNS. In other words, we assign a domain to a single, good IP address 4 times, and to a new IP address once. That means only 20% of the traffic goes to the new IP address so it can build it's reputation slowly over time. We then change the ratio of DNS assignments until it can handle more email.
     36Each domain name has a 600 second time-to-live and points to one or more of the IP addresses assigned to the relay servers.
    3837
    3938== How Do I... ==
    4039
    41 Add a new IP?
     40=== Review how well email is being delivered? ===
    4241
     42Run mf-mail-stats-print on `rustin` and/or `cleveland`. Here's sample output:
     43
     44{{{
     450 rustin:~# mf-mail-stats-print
     46postfix-aol-a:total:642
     47postfix-aol-a:bounced:62
     48postfix-aol-a:deferred:7
     49postfix-aol-a:sent:573
     50
     51postfix-earthlink-a:total:452
     52postfix-earthlink-a:bounced:70
     53postfix-earthlink-a:deferred:0
     54postfix-earthlink-a:sent:382
     55
     56postfix-msn-a:total:707
     57postfix-msn-a:bounced:87
     58postfix-msn-a:deferred:0
     59postfix-msn-a:sent:620
     60
     61postfix-msn-b:total:649
     62postfix-msn-b:bounced:72
     63postfix-msn-b:deferred:9
     64postfix-msn-b:sent:568
     65
     66postfix-therest-a:total:87281
     67postfix-therest-a:bounced:22242
     68postfix-therest-a:deferred:5404
     69postfix-therest-a:sent:59635
     70
     710 rustin:~#
     72}}}
     73
     74=== View the mailq for a given instance ===
     75
     76Each instance has a bash alias for mailq. Try mailq [tab] [tab] to see what is available.
     77
     78=== Flush deferred email from one queue to another queue ===
     79
     80If deferred email is building up in one instance, you can move it to another instance. For example if msn-a is deferring email, but msn-b is delivering, you can move all deferred email from msn-a to msn-b by following these steps.
     81 
     82 * Figure out which instance is running with which IP address:
     83{{{
     840 rustin:~# grep bind /etc/postfix-msn-{a,b}/master.cf
     85/etc/postfix-msn-a/master.cf:smtp      unix  -       -       -       -       -       smtp -o smtp_bind_address=209.51.172.9
     86/etc/postfix-msn-b/master.cf:smtp      unix  -       -       -       -       -       smtp -o smtp_bind_address=216.66.15.4
     870 rustin:~#
     88}}}
     89 * Create the file /etc/postfix-msn-a/transport.relay with the contents that instruct the instance to flush all email to the server running on the IP address 216.66.15.4.
     90{{{
     91* smtp:216.66.15.4
     92}}}
     93Note: you can replace * with a domain name if you only wanted to flush messages going to hotmail.com but not outlook.com.
     94 * Run `postmap /etc/postfix-msn-a/transport.relay`
     95 * Edit /etc/postfix-msn-a/main.cf and uncomment the transport_maps line at the end of the file.
     96 * Reload postfix: /etc/init.d/postfix reload
     97 * Flush:
     98{{{
     99postfix -c /etc/postfix-msn-a flush
     100}}}
     101
     102=== Warm up a new IP address for a given relay instance ===
     103
     104For example, to add a new IP address for therest.smtp.mayfirst.org:
     105
     106 * [wiki:ip_allocation Allocate new IP address]
    43107 * Create a Host record that ties the IP address to the host name (e.g. rustin.mayfirst.org or cleveland.mayfirst.org). Set the time to live to 86400 (it's important to have along ttl since some providers do a reverse DNS lookup of the sending IP address and will punish you if the ttl is too short).
    44  * Create an A record from the $instance.smtp.mayfirst.org to the IP address.
    45  * Add the IP address to the servers /etc/network/interfaces file and bring it up
     108 * Add the IP address to the server's (rustin or cleveland) /etc/network/interfaces file and bring it up
     109 * Create an A record from the $instance.smtp.mayfirst.org to the IP address. When adding a new IP address, you may not want a lot of email to go to it initially. Instead, you can "warm it up" by controlling the ratio of IP addresses in use for the given domain. For example, you can add give A records for the existing IP address and one A record for the new one. After month, remove one of the old A records and add a new one for the new IP address.
    46110
    47 Enable a new server to relay through this system?
     111=== Configure a bulk email server to use this setup ===
    48112
    49  * Add the IP address to the [wiki:puppet puppet file] manifests/global.pp, to the variable $mfpl_relaying_ips.
    50  * Be sure to push to both cleveland and rustin
     113 * Add the IP address of the bulk email server to the [wiki:puppet puppet file] manifests/global.pp, to the variable $mfpl_relaying_ips.
     114 * Be sure to push to both cleveland and rustin and restart postfix on both servers for good measure
    51115 * Add the following to the new server's /etc/postfix/main.cf file:
    52116{{{
    53117transport_maps = hash:/etc/postfix/transport.relay
    54118}}}
    55 
    56119If main.cf already has a line for transport_maps, you can add a new value following a comma, e.g.:
    57120{{{
    58121transport_maps = hash:/etc/postfix/transport,hash:/etc/postfix/transport.relay
    59122}}}
    60 
    61 Copy the file /etc/postfix/transport.relay from leslie or assata.
     123 * Copy the file /etc/postfix/transport.relay from leslie or assata.
     124 * Run `postmap /etc/postfix/transport.relay`
     125 * Reload postfix: /etc/init.d/postfix reload
    62126
    63127
     128