Fog Creek System Status

Brief Kiln On Demand Outage

We deployed a new upgrade service to the Kiln On Demand infrastructure today.  Despite initially working well, the service had a small logic error that briefly disabled the Kiln On Demand storage cluster.  The bug has been fixed and no data has been lost. The total time that Kiln On Demand was affected was less than ten minutes.We're sorry for the interruption in service.

If you're still experiencing issues, please contact us at customer-service@fogcreek.com.

Hurricane Irene

Hurricane Irene is projected to hit New York City this weekend. Both our office and datacenter are located in lower Manhattan, so we're buckling in for some rough weather. Peer 1, our datacenter provider, has assured us they're stocked up on generator fuel and are ready to handle whatever comes their way. Likewise, our office building is prepped and ready to deal with the storm.

We've taken the precautions we can and are hoping for the best, but there's obviously no way to predict the exact effects of the hurricane. As always, we'll update this blog if necessary. 

Have a good weekend and stay safe.

Maintenance Complete

Tonight's maintenance is complete. Our PDUs have been replaced, all racks have had their power completely rewired, and we are now on our ISP's high-capacity network segment. Things are looking good. We are now more resilient than ever to power and networking issues both inside and outside our network.

We did end up going significantly beyond our original maintenance window during the rewiring phase. Fortunately, impact was limited to a couple networking blips as we failed over between our gateways and load balancers. 

Since this maintenance was relatively limited in scope (though long), we will not be posting a formal retrospective. Have a great rest of your weekend.

Maintenance has begun, part 2

Tonight's maintenance continues.  We are proceeding carefully with our power recabling and are extending the maintenance to 11:00 GMT.  As before, we are doing our best to avoid taking down any services.

Thank you for your patience as we make our On Demand services faster and more reliable.

Maintenance has begun

Tonight's maintenance has begun. You may see intermittent outages over the maintenance window, but impact should be relatively low tonight. We'll post updates as necessary.

All services maintenance on Sunday, 2011-08-14, at 04:00 GMT

We'll be performing switching and power maintenance this Sunday, from 04:00 GMT to 08:00 GMT. This is the last component of our switching and power refactor and will include:

• Replacing all PDUs older than a year with new models
• Completely recabling power in our cabinets to simplify maintenance and future equipment replacements
• Installing redundant power supplies for our switches

Pending confirmation from our ISP, we'll also be moving our uplinks to our provider's newer, higher-capacity, and more fault-tolerant switching fabric. This will remove various congestion issues and boost our burstability from 100Mbps to 1Gbps. 

Note that we're going to do our best to avoid taking down any services. All of our racks have redundant power circuits and most of our servers have redundant power supplies, so we're hopeful that we can do this in stages to keep the interruption to a minimum. That said, we will take servers down if necessary to complete this maintenance on time.

As always, thanks for your patience as we make our On Demand services faster and more reliable.

Maintenance Retrospective

Sunday's maintenance went well, but applying the SQL Server updates still took longer than we had hoped. Though we were able to significantly reduce our servers' restart time in the last week or so, the hotfix required two shutdown and startup cycles that pushed us about an hour over our intended window.

The only other hiccup was a delivery mistake that meant we were unable to install our switches' redundant power supplies. We've already received the missing units and will complete this during next weekend's maintenance window.

On the positive side, all affected SQL servers now have the relevant hotfix installed and our switching fabric is much more fault tolerant than ever before.

Thanks again for your patience. We've got one more major maintenance coming up next weekend and then our power and switching refactor is complete. 

Maintenance complete

Tonight's maintenance has been completed and all services are up. The two laggard SQL servers are still spooling up their databases -- if you see an error that FogBugz "Could not open your database," give it some time and it should come up normally. We'll post more details on Monday, after we've had some sleep.

Maintenance extended

Two of our SQL servers are still installing the update in question. The maintenance window will end as soon as these servers are finished. We expect them to be done soon but, unfortunately, have no way to tell exactly how much time is left.

Maintenance has begun

Tonight's maintenance has begun. All FogBugz and Kiln On Demand requests are redirecting to this blog. We will post updates as necessary.