Fog Creek Network Status

FogBugz On Demand Maintenance tonight, May 12, at 04:00 GMT

We will be upgrading all FogBugz On Demand accounts tonight. If you receive the "Upgrading" message when accessing your site, please give it 3-4 minutes and reload the page.

Thanks for your patience as we make FogBugz better and better.

All Services Up; Previous Service Outage Postmortem

This outage hit FogBugz and Kiln On Demand customers and lasted from about 4:05PM GMT to 4:40PM GMT. If this outage affected you, please contact us so we can make it right.

This was a step-by-step repeat of Saturday's outage on a different rack. One PDU inexplicably failed and the second tripped a circuit breaker as it took over the load.

In light of this common component failure, additional steps are necessary to prevent a repeat of this event.

Short term, we are stripping our cabinets down to the bare minimum of redundancy. Should another PDU fail, the secondary will be able to handle the entire cabinet's load without exceeding the circuit's capacity. Certain components (like switches) that have only a single power input will need to be manually swapped over, but such an event would incur a much briefer outage. This will be completed by this evening.

Longer term, we are replacing half of our PDUs with APC models. Each rack will have one Tripplite and one APC PDU to eliminate the possibility of a single bad batch of components wiping out a cabinet. This timeframe is dependent upon our vendors, but it will be completed as soon as possible.

Thank you again for your patience.

FogBugz and Kiln On Demand Partial Outage

update: services have been restored.  a postmortem will follow.

We have detected a switch failure at our data center and are working to replace it right now.  At this time, some FogBugz On Demand and Kiln customers are experiencing connectivity issues. 

More updates will soon follow.

All Services Up; Maintenance Complete

This maintenance was scrapped in favor of non-service-impacting work after last night's unexpected outage. You may have noticed some intermittent FogBugz and Kiln On Demand slowdowns during this window, but everything should be back to normal now.

All Services Up; Previous Service Outage Postmortem

As previously noted, we experienced a large outage that brought down all of our services.   

The outage lasted from 5 AM GMT to 8:30 AM GMT.  If this got in your way, contact us so we can make it right for you.

Here is what happened:

1. all of our racks are configured with dual PDUs, each one connecting to a separate power circuit.  The general idea being that servers are split between them, and can automatically fail over if one were to fail
2. At 5 AM GMT, a PDU in one of our cabinets failed, causing all devices that were leaning on that PDU to move to the second one
3. when this occurred, the extra load on the secondary PDU was too much and caused the circuit powering it to trip
4. this cabinet housed both of our primary and secondary routers, each one connected to a different PDU
5. I worked with a technician to restore power and then replaced the failed PDU.  This took a considerable amount of time because the technician is obligated to make sure that the root cause is understood and that further issues won't occur

As you can see from this, we have some to make some changes to our design.  For starters, our primary and secondary routers will be split not only between separate power circuits, but between completely different racks.  With that in place, this sort of cascading failure would not be able to bring service to a full stop like it did today.  I am committed to putting this in place within the next 14 days.  

Our second objective is to increase the spare power capacity in each rack so that one failing PDU will not lead to another.  This will take more time, but it is considered to be high priority.

As always, I appreciate your patience in these matters.  Again, if this outage prevented you from doing your job, let us know.

All Service Outage

update @ 3:00 Eastern: the outage was caused by a power failure in one of our main cabinets. I am working with a technician to restore.

update @ 02:01 Eastern: we have someone heading onsite to take a look in coordination with the datacenter's NOC

Our monitoring systems have detected a network failure at our data center.  We are looking into it.