[[PageOutline]] = Bulk Mail Relay = See #8424 for implementation details. See [wiki:email-deliverability the email deliverability page] for context. == Big Picture == The over all concept is: most corporate providers allow mail through based on the ratio of delivered mail without complaints to bounced mail and complaints (complaints means people who click the "this is spam" link). So, our strategy is to configure all of our bulk email servers to use the same dedicated servers to relay email to specific big providers, so we increase the volume of legit mail to help tip the balance when a member (inevitably) sends a bunch of bad addresses or has users click the "this is spam" button. In other words, all email destined for AOL (for example), regardless of which server the email originates on, gets relayed to AOL via aol.smtp.mayfirst.org. Meanwhile, aol.smtp.mayfirst.org can point to one IP address or, ideally, two or three (so if one IP address gets blocked, we simply turn it off and rely on the remaining two). == Details == The following email list servers are currently using this system: assata.mayfirst.org (provides both bulk.mayfirst.org and lists.people-link.net) and leslie.mayfirst.org (provides lists.mayfirst.org and lists.critpath.org). This strategy should also be adopted by morales.mayfirst.org (provides lists.portside.org), mx3.mayfirst.org and mx25.mayfirst.org. We currently have two servers setup as dedicated mail relay servers (cleveland in XO and rustin in Telehouse). These servers run multi-postfix instances (see #6662 for more information on the setup). That means, each server has multiple instances of postfix running on different IP addresses. NOTE: only rustin is currently in play. We have setup special domain names for each of the main mail providers: * All hotmail/outlook.com/etc is relayed via: msn.smtp.mayfirst.org * Earthlink/Mindspring/IGC, etc is relayed via: earthlink.smtp.mayfirst.org * AOL email is relayed via: aol.smtp.mayfirst.org * Everything else goes through: therest.smtp.mayfirst.org Future plans include adding: * yahoo.smtp.mayfirst.org * gmail.smtp.mayfirst.org Each of our bulk mail servers is configured to relay email to the appropriate relay by using the transport_maps Postfix configuration parameter. Each domain name has a 600 second time-to-live and points to one or more of the IP addresses assigned to the relay servers. == How Do I... == === Review how well email is being delivered? === Run mf-mail-stats-print on `rustin` and/or `cleveland`. Here's sample output: {{{ 0 rustin:~# mf-mail-stats-print postfix-aol-a:total:642 postfix-aol-a:bounced:62 postfix-aol-a:deferred:7 postfix-aol-a:sent:573 postfix-earthlink-a:total:452 postfix-earthlink-a:bounced:70 postfix-earthlink-a:deferred:0 postfix-earthlink-a:sent:382 postfix-msn-a:total:707 postfix-msn-a:bounced:87 postfix-msn-a:deferred:0 postfix-msn-a:sent:620 postfix-msn-b:total:649 postfix-msn-b:bounced:72 postfix-msn-b:deferred:9 postfix-msn-b:sent:568 postfix-therest-a:total:87281 postfix-therest-a:bounced:22242 postfix-therest-a:deferred:5404 postfix-therest-a:sent:59635 0 rustin:~# }}} === View the mailq for a given instance === Each instance has a bash alias for mailq. Try mailq [tab] [tab] to see what is available. === Flush deferred email from one queue to another queue === If deferred email is building up in one instance, you can move it to another instance. For example if msn-a is deferring email, but msn-b is delivering, you can move all deferred email from msn-a to msn-b by following these steps. * Figure out which instance is running with which IP address: {{{ 0 rustin:~# grep bind /etc/postfix-msn-{a,b}/master.cf /etc/postfix-msn-a/master.cf:smtp unix - - - - - smtp -o smtp_bind_address=209.51.172.9 /etc/postfix-msn-b/master.cf:smtp unix - - - - - smtp -o smtp_bind_address=216.66.15.4 0 rustin:~# }}} * Create the file /etc/postfix-msn-a/transport.relay with the contents that instruct the instance to flush all email to the server running on the IP address 216.66.15.4. {{{ * smtp:216.66.15.4 }}} Note: you can replace * with a domain name if you only wanted to flush messages going to hotmail.com but not outlook.com. * Run `postmap /etc/postfix-msn-a/transport.relay` * Edit /etc/postfix-msn-a/main.cf and uncomment the transport_maps line at the end of the file. * Reload postfix: /etc/init.d/postfix reload * Flush: {{{ postfix -c /etc/postfix-msn-a flush }}} === Warm up a new IP address for a given relay instance === For example, to add a new IP address for therest.smtp.mayfirst.org: * [wiki:ip_allocation Allocate new IP address] * Create a Host record that ties the IP address to the host name (e.g. rustin.mayfirst.org or cleveland.mayfirst.org). Set the time to live to 86400 (it's important to have along ttl since some providers do a reverse DNS lookup of the sending IP address and will punish you if the ttl is too short). * Add the IP address to the server's (rustin or cleveland) /etc/network/interfaces file and bring it up * Create an A record from the $instance.smtp.mayfirst.org to the IP address. When adding a new IP address, you may not want a lot of email to go to it initially. Instead, you can "warm it up" by controlling the ratio of IP addresses in use for the given domain. For example, you can add give A records for the existing IP address and one A record for the new one. After month, remove one of the old A records and add a new one for the new IP address. === Configure a bulk email server to use this setup === * Add the IP address of the bulk email server to the [wiki:puppet puppet file] manifests/global.pp, to the variable $mfpl_relaying_ips. * Be sure to push to both cleveland and rustin and restart postfix on both servers for good measure * Add the following to the new server's /etc/postfix/main.cf file: {{{ transport_maps = hash:/etc/postfix/transport.relay }}} If main.cf already has a line for transport_maps, you can add a new value following a comma, e.g.: {{{ transport_maps = hash:/etc/postfix/transport,hash:/etc/postfix/transport.relay }}} * Copy the file /etc/postfix/transport.relay from leslie or assata. * Run `postmap /etc/postfix/transport.relay` * Reload postfix: /etc/init.d/postfix reload