2015-09-05 Support team meeting


  • Finance
  • reportDOS report: where are we now?
  • Status of dovecot migration
  • Jessie upgrade: What do we need to do?
  • Possibility of developing infrastructure in Mexico, Pablo reports there is a possibility of obtaining resources for this

Attending: Jamie, Jaime, Pablo, Kendra, Nat, Steve

DDOS Status Report

The DDOS attacks started by targeting fundabortion now, then our DNS servers, then MFPL infrastructure. We moved our main DNS servers to a provider with DDOS mitigation. We also put portions of MF/PL infrastructure behind deflect. We paid telehouse for filtering and more bandwidth. The filters block UDP packets, except to a small handful of addresses. Things have been quiet since these changes were put into place, until highwinds was attacked about 20 minutes ago. Highwinds provided (temporary) packet filtering, until last Thursday. Apparently attackers are testing us periodically. Called highwinds today, asked them to null route targeted IP, and restore filtering. The targed IP was

As of 13:39, not sure if latest highwinds attack is DNS amplification. Highwinds did confirm the removal of packet filters.

Finances & Infrastructure

Extra expenses for bandwidth/filtering. We haven't raised enough money to fully cover infrastructure expenses (bandwidth and filtering). Cost is around $16k/year, and we've raised about $7k so far. We're putting out grant proposals for additional funding. One proposal was rejected, but two are still pending. It's one thing to raise money for one year, but we have to get to a place where the additional costs are sustainable.

Before this attack started, the LC identified a financial problem in the organization; namely that our expenses exceed income. There are several reasons for this; it's a combination of lower income and higher expenses. We had to renegotiate our highwinds contract about a year ago, which raised expenses by $1k/month. We also lost a $10k/year grant that we'd gotten for the past three years. We've had to cut back on Jaime and Alfredo's hours due to funding losses, and we're working hard to bring them back to full time. We're doing more fundraising for POC techie project. If we get grants for that, they'll significantly offset Alfredo's salary. Jamie would like to see us find ways to reduce our hosting cost, and find ways to better defend ourselves.

Current infrastructure. Our servers are located in Telehouse(isp Hurricane electric) and XO (isp highwinds). If electicity goes out in one center, we have the second center. Telehouse costs $2k/month. XO costs $3K/month for cabinet.

Nick Merill, from calyx institute, offers to possiblly share cabinets in Telehouse. Advantage, Nick has his own ip blocks and could possibly assign us a class c block. Disadvantage, concentrating all servers in one colo center, although we can use two different isp's for internet connection.

Telehouse and XO are located in Manhattan. We've prioritized putting hosting on servers that we own and have physical access to. Because we have them in NY, we pay a premium. We might be able to save a lot of money by hosting servers elsewhere, but we'd lose physical security. Detroit might be an option for server location.

MX cooperative can apply for a grant for equipment financing; a special kind of grant that is only for cooperatives, so they can build infrastructure. They can receive up to $10k for equipment, but these funds can't be used for services or rent. We could use that money for buying servers, and locate them at Mayfirst office in Mexico. Part of problem in MX is connectivity. Internet access is very expensive, electricty is relatively expsive, and not always dependable. Reliable bandwidth is our biggest challenge. We have to do tests, to see which parts of our infrastructure could reliably be hosted in Mexico. MX co-location is very expensive. Pablo is willing to go through administrative steps of applying for the grant, if support team is interested.

Again, this grant couldn't be applied to internet access; has to be equipment, but a small amount could be spent on training.

Data center hosting is important for reliable electricity. In a normal commercial building, electricity can go out for periods of time. That tends not to be a problem in data centers.

It seems like there is a need for more people trained on maintaining these servers and if there are other possible locations, that is an option.

Remote hosting could be advangagous. As long as we have console access and can do full disk encryption. Hosting parts of infrastructure in MX seems like good idea. We'll have to test, and be conscious of high bandwidth costs.

Sharing a telehouse cabinet with Nick could be an advantage. Multiple cabinets and multiple upstreams in one data center with an IP block gives us flexibility to move IPs around (via BGP).

Mexico co-location seems like a great idea, and something we should pursue. DOS attacks mean that it's useful for us to have a lot of different locations. A few servers could offset an attack. Remote backup is also useful. Remote colocation in MX is really expensive. There is high speed residential access in MX city; not the most reliable, but is high speed. Could we set up low-power machines with UPS to do offsite backup, and content caching for DDOS mitigation?

At the hackerspace we rent (in MX), we can get a 200mbit (advertised speed) connection, that is fairly reliable. We can get two different service providers for redundancy. Jaime doing this for a few other projects. Small servers, and even a little generator for electricity. This arrangement has worked for backup services, but hasn't been tested with mission-critical services. There are at least 2--3 different bandwidth providers availablity.

Trying to build our own data center closet in MX. This could reduce cost for everyone involved. Highest cost is internet access. Bandwidth is around $70/month. Two uplinks would be $140/month. This would be located in a co-working space, so there are people in the facility, and it is monitored.

As OTI tries transition to becoming members, they might be able to provide some physical space and bandwidth, and fairly stable power.

Should we put a proposal on the wiki, w/a list of services that we want to set up remotely? Consider both Nat's offer with OTI and the hackerspace in MX. That seems like a reasonable next step. Nat and Jamie will work on first draft.

If we got a grant, equipment will have to be physically located at cooperative's registered address in Mexico City. We could move the equipment after some time, if we came to an agreement on sharing resources. There is a proposal deadline -- in about a week -- for submitting the application. The deadline is 18 de Septiembre, preferably 14th. Earlier is better, in case something comes up. We also need to build in translation time (Jamie and Nat will likely be drafting in english).

Exciting to be able to host in MX, but big question is what can be hosted here, and what requires too much bandwidth to host there. Initial (wiki) proposal has to answer this question (what services to host in MX), otherwise it's not worth doing. Might be offsite backups and varnish. Also have to consider that a caching service might draw attention to MX facilities, leading them to be attacked.

Dovecot migration

Jaime did a lot of dovecot migrations this morning. Got most the remaining moshes done before the attack started. There are about four moshes left to transition/finish. Only moshes need the transition. Very exciting to see all this happen.

Jessie Upgrade

Latest version of debian was released last spring, and we have around 200 servers to upgrade. Puppet scripts have to be tweaked for the new version of Debian. Most non-moshes should function properly with the upgrade. For moshes, about 90% of puppet fixes are in place. The new version of apache (in Jessie) requires configuration files to end in .conf. Apache deprecated the SSLCertificateChainFile directive; you're supposed to put certificates and chains in a single file. Once puppet changes are finished, next mosh should be set up with Jessie. We'll run that for a while to test. Then, upgrade a few moshes and test. Then schedule a few weekends to upgrade the rest.

For non-moshes, parents should examine their servers, and figure out what it would take to upgrade, and coordinate time for a jessie upgrade. Perhaps test upgrading non-mosh servers on Monday (labor day in the US).

When do wheezy security update stop? Debian generally provides security updates for a year after a new version is released; we should be okay until springtime.

Highwinds seems to be back, but the attackers are hitting our DNS servers (awknet and reliable).

Last modified 6 years ago Last modified on Sep 5, 2015, 5:33:42 PM