Standard Server Split

Overview

Over the last several years, many May First/People Link members who are hosted on our shared, standard servers, have begun running web sites that require more resources than we are able to easily provide or that we are able to predict in advance. As a result, we have frequent periods in which any given standard server runs slowly or even crashes.

There are three problems with our infrastructure that hamper our ability to address these situations:

• While a series of bash scripts has eased the deployment of new standard servers, the process is still cumbersome, taking several hours.
• Our standard servers provide both web and mail services, preventing us from optimizing servers for one or the other task
• The mail services require members to customize their desktop email clients based on the particular standard server they are using (e.g. chavez.mayfirst.org or viewsic.mayfirst.org). Therefore, moving mail services from one server to another to re-distribute load requires a level of member coordination that makes it infeasible in most cases

Proposals

The following steps are a proposal for addressing these problems:

• Complete the transition to Puppet to ease the work involved with deploying new standard servers
• Abstract the use of all mail configuration so that all members use the same settings
• Move all mail services to dedicated mail servers, leaving current standard servers as dedicated web / mysql / shell servers

The middle task is the most complicated, so I will explain more here.

There are three mail settings in use by MFPL that require a server's domain name:

• Relay: all members who relay mail through May First/People Link must specify their primary server in their desktop email client settings (the "outgoing" server)
• MX records: all member domains that use MFPL for email, have a DNS MX record set to their primary server.
• POP/IMAP: all members who access their email via POP or IMAP must specify either in the desktop client or when they visit ​https://members.mayfirst.org/ to check their email via the web the name of their primary server

Rather than forcing our users to specify their primary servers for these settings, each one should have a organization-wide address used by everyone.

• relay.mayfirst.org: Currently, we have authenticated mail relay on each of our standard servers. If we setup one or several servers that had a login for every MFPL user, we could easily transition to the use of a single domain name for all mail relaying.

• mx.mayfirst.org: We could setup one or several mx servers with minimal postfix customizations, that could accept all mail intended for MFPL and then route mail to the particular server that houses the account. By keeping a list of all valid email accounts on all mx servers, we would also be able to reject mail for unknown mailboxes and we even deal with viruses, etc. before email reaches the server housing the actual mailbox

• incoming.mayfirst.org: Using ​nginx or ​perdition, we could setup one or several IMAP/POP proxies that similarly redirect requests to the appropriate server based on a database / table lookup.

Implementation

The transition would need to be implemented so that there is a long overlap period whereby the current approach to mail settings work as do the new approach.

All three changes would require changes to the Members Control panel, so that the creation of an email account triggers action not only on the user's primary server (to preserve backward compatibility), but also on the newly added servers.

Implementing relay.mayfirst.org is the easiest of the three, since we can use the same software and configuration we are currently using - with the exception of implementing it on a dedicated server. Once in place and tested, we can begin using it immediately. It will, however, require users to change their own desktop email settings.

mx.mayfirst.org requires us to explore a new postfix configuration. However, again, not significantly complicated or new. Furthermore, once this change is in place, we can update DNS entries without relying on users to make the change.

incoming.mayfirst.org requires us to explore new software, making it the most complex step. Furthermore, it will also require users to change their local settings.

Probably best to make all three changes first, and then request that users update their settings. Through our mail logs we can accurately assess exactly who has made the changes and who hasn't.

Only after everyone has made the changes can we really put this new setup to use by moving users around and splitting off our standard servers into web and mail servers.