Fog Creek System Status

Outlook promising for all Fog Creek services

Trello is running happily on AWS and all other Fog Creek services continue to run in the data center without problems.

The fuel supply at ground level is sufficient for the foreseeable future. Additional labor has been added to the bucket brigade and the building is making progress on securing a fuel pump to replace the brigade. This all suggests the generator is at little to no risk of running out of fuel.

In a pleasant development, after previous ineffective efforts at pumping water from the basement, we are now removing one foot an hour, with about 15 feet left. See squarespace's status post for similar optimism and details.

Our sysadmin and engineering teams are hard at work putting the finer details on an emergency plan to remove the servers from the Manhattan data center and install them at one in New Jersey. This will only be done in the unusual circumstance of the current data center completely failing. This is highly unlikely due to the fuel supply and prospects of a new pump. If we do need to move, it will require approximately 24 hours of downtime for FogBugz, Kiln and Copilot.

We are highly optimistic about tonight. Stay tuned for updates here and on twitter.

5:15 PM - Additional Options Turning up, Diesel Fuel Aplenty

We are starting to see some the fruits of our labor in terms of persuing additional options. Once we have more details that we can share, please look for them here!

Peer1, the data center, has people driving in to New York City tonight from Virginia to lend additional helping hands. They also were able to successfully coordinate with the building and Internap to secure more diesel fuel from the additional diesel trucks nearby. We will have sufficient fuel for the forseeable future.

Peer1 is also actively making progress with the building about installing a pump system to the 17th floor.

We've heard about other buildings nearby having electricity returned. We do not yet have a status for the data center building, unfortunately.

Stay tuned for updates here and on twitter @FogCreekStatus. 

 

Photo amazement:

Yes, this was a tunnel in lower manhattan

 

5am bucket filling (not intentionally blurry)

4:00 PM - Changing of the Guard, Still on Backup Power

We're replenishing some of our staff, especially those at the data center, as we aggressively continue working on other options. Backup power is still going strong!

All services are up and running! We are actively responding to emails at customer-service@fogcreek.com.

We have another late night crew getting some rest in the likely event that we will need to swap shifts at the data center.

 

Oh yea, and here are some "fun" photos:

All is up and running!

 

Servers are looking good!

 

Server access is not a problem

 

Buckets and barrels lined up and ready

 

Passing and moving buckets of diesel

1:42PM ET - Still on Backup Power, Stop Gap Employed

All services up and running!

We are vigorously iterating on other options to improve the situation.

The building is making efforts to improve the situation as well. We will share information as it is available.

Peer1 is working on getting an alternate fuel pump and power source in order to pump the fuel to the generators on the 17th floor, but both pumps and mobile generators are difficult to acquire at the moment. They have some leads on acquiring a fuel-pump, so hopefully this will pan out and relieve the Bucket Brigade.

Our partners in effort at the data center, Squarespace, have also been supplying ongoing status updates in the event you're tired of reading ours: http://status.squarespace.com/.

Stay tuned to the latest updates here and on twitter https://twitter.com/FogCreekStatus.

Services still on backup power, diesel bucket brigade continues

Update 11:29 am ET (UTC -4):

The generator tank on the 17th floor has been filled (thanks to the bucket brigade!). Fuel usage is holding at the level previously observed (see 9:28 am ET (UTC -4) update below). The diesel truck is able to remain outside for the foreseeable future. We continue to project no downtime due to lack of fuel.

The building is still working on getting the pump in the basement operational to provide a sustainable solution (not depending on the labor of dozens of people). Our engineers continue to develop a contingency plan to move servers to a new location. Stay tuned for updates. You will be notified in advance of any potential move.

Update 9:57 am ET (UTC -4):

This is what is powering out internets right now:

And here is what is preventing pumping diesel from the ground level up to the generator on the 17th floor. They are working on clearing the water now, but it's not easy work.

Update 9:28 am ET (UTC -4):

Word from our troops on the ground suggests that we should not see an outage due to lack of fuel before Con Ed can restore power to the building. This is NOT guaranteed, but things are looking good.

The generator tank has about 2 hrs of fuel in it on the 17th floor. The bucket brigade has 8 barrels of fuel to carry up, or enough for ESTIMATED 15 hours of fuel. Since the diesel truck is tentatively staying available for the duration, our fuel concerns may be addressed. Provided a continuing supply of bucket-brigade labor, no outage is expected before power is restored.

However: note that the data center has two major challenges, getting power back from Con Ed, and getting the basement pumps operational.

Update 8:58 am ET (UTC -4):

The fuel truck has arrived! Barrels are being filled now. The bucket brigade continues to keep the generator supplied.

Original post:

Update on the status of FogBugz, Kiln and Copilot after flood damage from Hurricane Sandy:

The building housing our data center is still flooded in the sub-basement where the pumps live to supply the generator on the 17th floor with diesel fuel. The building is working on getting pumps clearing out the water, but we don't know when that will be complete. The situation downtown post-Hurricane Sandy is pretty bad, but we're all chipping in along with squarespace to keep fuel moving upstairs, one bucket at time. If we expect downtime for our services, we will let you know with as much warning as we can.

Here is an example of how it looks downtown:

You can follow the efforts of our neighboring tenant, squarespace here. Stay tuned here and on twitter for more information.

Diesel bucket brigade maintains services

All services are currently still up. Trello is safely moved to Amazon AWS and Kiln, FogBugz and Copilot are humming due to feats of endurance by the people at the data center. We continue to prepare for a potential early-morning move of our servers to a new data center.

As tweeted by Joel:

After a day of hard work, Trello is almost ready to move permanently out of our flooded data center onto Amazon Web Services - October 31, 2012 at 2:00 AM

Meanwhile, a bucket brigade has been humping diesel up 17 floors to the generator, 5 gallons at a time; the tank is already half full - October 31, 2012 at 2:00 AM

And finally, we are making last minute plans to physically yank servers across the river to New Jersey (Datapipe) if Peer1 NY facility goes - October 31, 2012 at 2:00 AM

The data center managed to get a delivery of diesel fuel for their generators in the recently-flooded Manhattan financial district:

Please don't tell the Fog Creek fish that we used their bucket to get diesel up 17 flights of stairs! Many thanks to the people of squarespace for lugging those buckets!

Stay tuned for updates here and from @fogcreekstatus on twitter.

FogBugz, Kiln and Copilot continue to run on backup power, Trello moving to Amazon AWS

Update 11:29 pm (UTC -4):

Trello is back up and running on it's new home in the cloud!

Update 10:52 pm (UTC -4):

Trello is currently experiencing planned downtime while we move it out of the flooded data center. It should be back up shortly.

Original Post:

Following up from our previous updates:

All our services are currently still up.

The folks at the data center continue to keep the supply of diesel fuel flowing to the generators. We don't, however, have an estimate from them as to how long they can continue to run in this fashion. As such, we are coordinating plans to move our equipment to another data center. Trello is in the process of being moved to Amazon AWS where it will not be affected by further data center issues.

We will update you here and via @fogcreekstatus on twitter with all new information.

7:30 PM Update - Systems Still Running On Diesel

All services are up now and are still running on backup power. However, our data center is having trouble getting enough diesel to the generator, so we don't know how long we'll be up. We are still working on backup plans to move our physical servers to another data center if the current data center turns out to be unusable. Unfortunately, this might result in a day or so outage while we physically move. We're truly hoping this won't be necessary.

As usual, stay tuned here and at twitter.com/fogcreekstatus and we'll keep you up to date.

5pm and All Systems Are Running on Diesel

Following up from our previous updates: 

All our services are up.

The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)

If you're a Trello user, you have probably gotten an email already about checking here for updates. We are actively working on contacting the admins for On Demand accounts, but our commitment to customer privacy and security actually makes it more difficult to get this information. We're also actively working on multiple contingency plans should electricity fail again, or should there be other incidents at the data center.

Fog Creek Services Updates

Update 4:42 PM ET (UTC -4):

Look for future updates on the main blog page here: http://fogcreekstatus.typepad.com.

 

Update 2:43 PM ET (UTC -4):

Kiln On Demand and Trello are now online in addition to FogBugz On Demand as stated in the earlier post. 

Our previous statements about the data center and possibility to bring down our services again remains current:

Our data center has told us that we have several hours worth of fuel to power our services. They are actively working on fuel delivery into lower Manhattan, and on getting that fuel to the high floor where the generators are. If they can get the fuel to the generator tanks for long enough to have power restored to the building, we will not have to bring our services down again.

If we have to bring our services down again, it will be with some notice (though not as much as we'd like, obviously). Please follow @fogcreekstatus on Twitter for timely updates, or check back to this page.

 

Update 2:00 PM ET (UTC -4):

You can see the timeline of this morning's unplanned downtime here:

http://status.fogcreek.com/2012/10/fog-creek-services-updates.html

Here's the current situation:

FogBugz On Demand appears to be running normally. Kiln On Demand will be coming up very soon.

Once we got the go-ahead to bring our services back online, FogBugz On Demand took 17 minutes to be brought back up. Kiln On Demand is expected come online quickly as well. Both services appear to be working normally. Trello is still in the process of being brought up. We hope to have it up soon.

Our data center has told us that we have several hours worth of fuel to power our services. They are actively working on fuel delivery into lower Manhattan, and on getting that fuel to the high floor where the generators are. If they can get the fuel to the generator tanks for long enough to have power restored to the building, we will not have to bring our services down again.

If we have to bring our services down again, it will be with some notice (though not as much as we'd like, obviously). Please follow @fogcreekstatus on Twitter for timely updates, or check back to this page.

As always, we're committed to a full post-mortem of this event. We'll turn our attention to that as soon as it's not better directed to more immediate concerns.

 

Update 1:30 PM ET (UTC -4):

We've received word that the generators have enough fuel in them to power our services for a few hours.

Our eyewitness at the data center indicates that there is more fuel available than they originally thought. The fuel is burning at a slower rate than anticipated.

We're in the process of bringing our services online.

This is a temporary reprieve from downtime, but it does give us enough time to assess our options. The entire team is actively working on next steps. More to follow shortly.

On another note, our phone lines are down. There's nothing we can do about that for the moment. We are actively answering email, though, to customer-service@fogcreek.com.

As we did with the first downtime, we will provide notice via the status blog of further downtime.

 

Update 1:00 PM ET (UTC -4):

From our eyewitness at our data center, the data center folks believe that there is enough fuel remaining to last until 2PM ET. There is hope that within this timeframe that they can get more fuel up to the generator before 2PM ET.

We are starting to bring up our basic services in preparation of starting FogBugz, Kiln, Trello, and CoPilot. Once we receive final word from the data center that they are sufficiently fueled up we'll bring up all our services.

We will keep you informed as our data center eyewitness keeps us updated.

 

Update 12:30 PM ET (UTC -4):

From our eyewitness at our data center, there is still power, but the big question is how much diesel fuel is left. We are determining whether we can safely power on our servers again, but it's not something to be done lightly.

If we experience a loss of power without bringing down our services (as we've done), unrecoverable data corruption could result, meaning we'd lose customer data. We want to avoid that at all costs.

So right now, our services are in a sort of induced coma. We don't want to bring them out until we are sure it's safe to do so.

We've been advised by our data center that we should not power on our servers.

Here's the physical situation:

The generators are on a high floor in the building and the pumps supplying the generators with fuel are submerged. The best option at this point is for people to physically lug diesel up over a dozen floors, or make other arrangements for pumping fuel to that high floor.

More updates as they become available.

 

Update 11:58 AM ET (UTC -4):

The Fog Creek On Demand services are down, due to massive flooding from Hurricane Sandy which has knocked out not the diesel generators, but the pumps supplying fuel to the generators at our data center. Thanks for bearing with us during this emergency. We are actively making arrangements for getting our services back up. But rest assured that your data is safe and that we will make this right. 

 

Update 11:22 AM ET (UTC -4):

All Fog Creek services are now down temporarily. One of our Co-Founders was able to walk to the data center and it appears that they'll get the data center up and running in a few hours if they can refuel the generators. We're making plans for alternative service restoration depending on the data center's status. 

 

Original Post:

From our previous shutdown post, FogBugz, Kiln, and Trello are now shut down. Stay tuned for updates as we hear more from our data center, or make additional plans.