Update 4:42 PM ET (UTC -4):
Look for future updates on the main blog page here: http://fogcreekstatus.typepad.com.
Update 2:43 PM ET (UTC -4):
Kiln On Demand and Trello are now online in addition to FogBugz On Demand as stated in the earlier post.
Our previous statements about the data center and possibility to bring down our services again remains current:
Our data center has told us that we have several hours worth of fuel to power our services. They are actively working on fuel delivery into lower Manhattan, and on getting that fuel to the high floor where the generators are. If they can get the fuel to the generator tanks for long enough to have power restored to the building, we will not have to bring our services down again.
If we have to bring our services down again, it will be with some notice (though not as much as we'd like, obviously). Please follow @fogcreekstatus on Twitter for timely updates, or check back to this page.
Update 2:00 PM ET (UTC -4):
You can see the timeline of this morning's unplanned downtime here:
http://status.fogcreek.com/2012/10/fog-creek-services-updates.html
Here's the current situation:
FogBugz On Demand appears to be running normally. Kiln On Demand will be coming up very soon.
Once we got the go-ahead to bring our services back online, FogBugz On Demand took 17 minutes to be brought back up. Kiln On Demand is expected come online quickly as well. Both services appear to be working normally. Trello is still in the process of being brought up. We hope to have it up soon.
Our data center has told us that we have several hours worth of fuel to power our services. They are actively working on fuel delivery into lower Manhattan, and on getting that fuel to the high floor where the generators are. If they can get the fuel to the generator tanks for long enough to have power restored to the building, we will not have to bring our services down again.
If we have to bring our services down again, it will be with some notice (though not as much as we'd like, obviously). Please follow @fogcreekstatus on Twitter for timely updates, or check back to this page.
As always, we're committed to a full post-mortem of this event. We'll turn our attention to that as soon as it's not better directed to more immediate concerns.
Update 1:30 PM ET (UTC -4):
We've received word that the generators have enough fuel in them to power our services for a few hours.
Our eyewitness at the data center indicates that there is more fuel available than they originally thought. The fuel is burning at a slower rate than anticipated.
We're in the process of bringing our services online.
This is a temporary reprieve from downtime, but it does give us enough time to assess our options. The entire team is actively working on next steps. More to follow shortly.
On another note, our phone lines are down. There's nothing we can do about that for the moment. We are actively answering email, though, to customer-service@fogcreek.com.
As we did with the first downtime, we will provide notice via the status blog of further downtime.
Update 1:00 PM ET (UTC -4):
From our eyewitness at our data center, the data center folks believe that there is enough fuel remaining to last until 2PM ET. There is hope that within this timeframe that they can get more fuel up to the generator before 2PM ET.
We are starting to bring up our basic services in preparation of starting FogBugz, Kiln, Trello, and CoPilot. Once we receive final word from the data center that they are sufficiently fueled up we'll bring up all our services.
We will keep you informed as our data center eyewitness keeps us updated.
Update 12:30 PM ET (UTC -4):
From our eyewitness at our data center, there is still power, but the big question is how much diesel fuel is left. We are determining whether we can safely power on our servers again, but it's not something to be done lightly.
If we experience a loss of power without bringing down our services (as we've done), unrecoverable data corruption could result, meaning we'd lose customer data. We want to avoid that at all costs.
So right now, our services are in a sort of induced coma. We don't want to bring them out until we are sure it's safe to do so.
We've been advised by our data center that we should not power on our servers.
Here's the physical situation:
The generators are on a high floor in the building and the pumps supplying the generators with fuel are submerged. The best option at this point is for people to physically lug diesel up over a dozen floors, or make other arrangements for pumping fuel to that high floor.
More updates as they become available.
Update 11:58 AM ET (UTC -4):
The Fog Creek On Demand services are down, due to massive flooding from Hurricane Sandy which has knocked out not the diesel generators, but the pumps supplying fuel to the generators at our data center. Thanks for bearing with us during this emergency. We are actively making arrangements for getting our services back up. But rest assured that your data is safe and that we will make this right.
Update 11:22 AM ET (UTC -4):
All Fog Creek services are now down temporarily. One of our Co-Founders was able to walk to the data center and it appears that they'll get the data center up and running in a few hours if they can refuel the generators. We're making plans for alternative service restoration depending on the data center's status.
Original Post:
From our previous shutdown post, FogBugz, Kiln, and Trello are now shut down. Stay tuned for updates as we hear more from our data center, or make additional plans.
