During the last two weeks, MF/PL was challenged by three separate situations each of which could have led to a major crisis and each of which was handled quickly and effectively by our techie members led by Jamie.

Handling this stuff is a daily part of our work and I wouldn't usually write a LowDown about but this situation was unique and the work was so conspicuous that I wanted to share it with all of you so that we can all be proud of this organization and, at the same time, conscious of the challenges we'll be facing in the future.

The first occurred two weeks ago when we noticed a dictionary attack being made against one of our servers. This is an attack in which a hacker attempts to guess passwords by trying a literal dictionary of possibilities. One score and in he or she gets.

We have ways of fending off those attacks so this wasn't going to succeed but the amount of attempts every second zapped the server's resources and we were experiencing what is called a DOS or "denial of service" attack. That means the person is using all the resources and nobody else can log in or do much with the sites on the server.

Jamie and Daniel Kahn Gilmor conferred and decided to reconfigure the servers to detect apparent dictionary attacks and block their IP for a period of time as soon as they start. This pre-emptive step closed those attacks down and saved the performance on all servers. Impact was hardly felt! :-)

But they led to another discovery.

While we were researching what to do, the Debian developers sent out an advisory about a horrible exploit. A couple of definitions are in order.

GNU/Debian is a free operating system that uses linux; it's the operating system we use on our servers.

An exploit is a piece of code that can be used by a clever attacker to get access to a server and do damage, steal data or illicitly use resources.

One other definition: a Secure Sockets Layer key. An SSL key is a long string of characters that is unique to every server that provides the basis for the server to establish a secure, encrypted connection with computers that connect to it.

These software packages are developed by large numbers of people all over the world who merely contribute to them; that's what Free and Open Source Software is. They work through collaborative work systems and they communicate on-line. Most have never met.

In this case, one person filed a report and the community's leaders checked and approved it. It seems that, with the Debian installed throughout the world, you can guess certain parts of the key that are supposed to be generated radomly. These means that, eventually, you may be able to guess the key itself.

They released a "patch" to fix this and we installed it on all servers but, of course, this meant that all of our servers identified themselves differently to the world. As a result, everyone who connects to our server using secure shell or Secure FTP started getting warnings. Since the warnings were being generated on members' personal computers, we could not fix them for the members.

We put together an approach to making the necessary changes on all three platforms: Mac, Window and Linux. And then we did how-tos on all three, published those and sent out advisories explaining the situation. We did all that in about a day! I've never seen anything like this in all my years on the Internet.

​https://support.mayfirst.org/wiki/ssl_host_key_changed

Most members went through the tasks and made the connections but we haven't heard from everyone and would like to know if you had problems and what they were. This isn't the last time we're going to go through this and the feedback is important.

One thing to remember when we talk about doing something on "all our servers". MF/PL has 36 servers so this is no small task.

Finally, Health-Care Now...

After all the above had been resolved, we noticed that email on one server was clogging and this was affected the performance of the entire server. We investigate and noticed that somebody was sending messages to fictitious users on the domain healthcare-now.org -- the Health Care Now campaign. The messages were coming in at 65,000 per hour and this lasted for nearly four days.

There was no question that this was a deliberate, denial of service attack by some person or organization who opposed the Campaign's work and was trying to bring it to a halt. It was, by all standards, a vicious, prolonger and intense attack.

The usual response would be closing off the server, and denying service to so many other members, or trying to guess the IP of the sender and blocking it. Closing off service like that is giving it to Internet terror tactics and there's no way we can set that precedent. Trying to guess the IP when this hacker was a huge amount of them (all made up, btw) would have wagted valuable time and probably not worke din the end.

So we rerouted email to another empty server ready to handle the load. The campaign's email service continued to be affected during the four day attack but all other members' functionality was returned to normal. After four days of failure, this hacker stopped the attack and the domain was routed by to its normal server.

This kind of creactive thinking and flexibility under fire is reminiscent of what oru people did in saving the on-line registration at the U.S. Social Forum.

These two weeks were hardly routine but they were perhaps a hint of the kinds of challenges we are going to be facing in the future. Every challenge is a challenge, of course but, based on these two weeks, I think we're ready.