Changes between Version 110 and Version 111 of WikiStart


Ignore:
Timestamp:
May 1, 2012, 9:34:58 AM (8 years ago)
Author:
Jamie McClelland
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • WikiStart

    v110 v111  
    11[[PageOutline]]
    22= Welcome to May First/People Link's Support Network =
    3 
    4 '''IMPORTANT NOTICE''' April 29, 2012 - Due to a power outage at our provider at Telehouse, we have lost power to our cabinet affecting about half of all May First/People Link members. This power outage is not within our control.
    5 
    6 All time America/New_York.
    7 
    8  * 1:32 am: - albizu is up and running. At this point all servers directly in use by members should be fully functional. The only remaining server not online is blanco, one of our two onsite backup servers. blanco is still running a file system integrity check. We expect it to come online in a few more hours.
    9  * 11:18 pm - most servers are running and operational, but responding very slowly as they deal with incoming mail that was queued while the servers were down (we do not expect any lost mail). albizu is the only primary/mosh server that is not yet up. It is running a slow file system check and is expected to be up in about 1 hour.
    10  * 10:45 pm: Power has been restored. All servers should be online (or soon to be online).
    11  * 10:06 pm: The UPS technician has confirmed that we are by-passing the UPS, and are being powered by the backup generator. The technician expects to switch back to UPS, possibly in the next 15 minutes, and advised that it might cause another power outage.
    12  * 9:49 pm: The power has stayed up for 40 minutes now, however, the Telehouse technicians have requested 30 minutes before confirming that the power will stay on. To avoid corrupted data, we are waiting on confirmation that it will stay up before we start the rest of the servers.
    13  * 9:10 pm: power is restored again. We have turned on robideau (providing serial console access) and negri (providing hay, malcolm, proudhon, ranciere, algernon,  bouazizi,  clyde,  dorothy) in order to have a functional OpenID provider. We are waiting before starting the other servers to ensure that the power will stay on.
    14  * 9:03 pm: power was just lost again. All power is down at the cabinet
    15  * 8:59 pm: power is restored, all servers coming back online now. It may be another 30 minutes before service is completely restored and another hour before traffic/resource usage is back to normal
    16  * 8:30 pm: Additional technicians have still not arrived
    17  * 7:30 pm: Additional Telehouse technicians scheduled to arrive at 8:15 pm America/New_York time.
    18  * 7:00 pm: still no update on ETA
    19  * 6:29 pm: we are onsite, however, still have no power to our cabinet. We have not been given an ETA on the return of power. At least two of the facility's five UPS systems are down, so it may be hours before power is restored. We will report as soon as we get an ETA.
    20  *  5:58 pm: Due to a power outage at our provider at Telehouse, we have lost power to our cabinet affecting about half of all May First/People Link members. This power outage is not within our control. We are pressuring Telehouse to bring power back up as soon as they can and we are positioned to bring our servers back online as soon as they do.
    21 
    22 More details: #5672
    233
    244== Do you need help? ==