Update 11:29 am ET (UTC -4):
The generator tank on the 17th floor has been filled (thanks to the bucket brigade!). Fuel usage is holding at the level previously observed (see 9:28 am ET (UTC -4) update below). The diesel truck is able to remain outside for the foreseeable future. We continue to project no downtime due to lack of fuel.
The building is still working on getting the pump in the basement operational to provide a sustainable solution (not depending on the labor of dozens of people). Our engineers continue to develop a contingency plan to move servers to a new location. Stay tuned for updates. You will be notified in advance of any potential move.
Update 9:57 am ET (UTC -4):
This is what is powering out internets right now:
And here is what is preventing pumping diesel from the ground level up to the generator on the 17th floor. They are working on clearing the water now, but it's not easy work.
Update 9:28 am ET (UTC -4):
Word from our troops on the ground suggests that we should not see an outage due to lack of fuel before Con Ed can restore power to the building. This is NOT guaranteed, but things are looking good.
The generator tank has about 2 hrs of fuel in it on the 17th floor. The bucket brigade has 8 barrels of fuel to carry up, or enough for ESTIMATED 15 hours of fuel. Since the diesel truck is tentatively staying available for the duration, our fuel concerns may be addressed. Provided a continuing supply of bucket-brigade labor, no outage is expected before power is restored.
However: note that the data center has two major challenges, getting power back from Con Ed, and getting the basement pumps operational.
Update 8:58 am ET (UTC -4):
The fuel truck has arrived! Barrels are being filled now. The bucket brigade continues to keep the generator supplied.
Original post:
Update on the status of FogBugz, Kiln and Copilot after flood damage from Hurricane Sandy:
The building housing our data center is still flooded in the sub-basement where the pumps live to supply the generator on the 17th floor with diesel fuel. The building is working on getting pumps clearing out the water, but we don't know when that will be complete. The situation downtown post-Hurricane Sandy is pretty bad, but we're all chipping in along with squarespace to keep fuel moving upstairs, one bucket at time. If we expect downtime for our services, we will let you know with as much warning as we can.
Here is an example of how it looks downtown:
You can follow the efforts of our neighboring tenant, squarespace here. Stay tuned here and on twitter for more information.
