Fog Creek Network Status

Maintenance Complete

Tonight's maintenance was completed without incident.

Kiln Outage Postmortem

During Saturday's regularly scheduled upgrade of Kiln On Demand and FogBugz On Demand, Kiln had a longer than expected amount of downtime.

As part of the upgrade process, Kiln was scheduled to rebuild some of its repository search indices. The resulting system load was orders of magnitude larger than initially anticipated, leading to extremely slow access to Mercurial repositories and 500 errors on some Kiln pages. To remedy the problem, the Kiln team throttled the reindexing task, restoring normal Kiln On Demand access by 2 AM EDT Sunday morning.

Long-term, we're working to better simulate production load in our pre-deployment tests so we can predict issues like this. Short-term, we'll continue to deploy in the wee hours. 

FogBugz On Demand maintenance on Sunday, 2011-10-30, at 04:00 GMT

We'll continue playing musical servers this weekend, migrating several larger accounts to ensure everybody can continue to grow at their own speed. As usual, these moves will be done between 04:00 and 08:00 GMT and service impact will be limited to one account at a time. 

Kiln Issues Resolved

Kiln performance has been restored. We've tracked down the root issue and bandaided it. The only side effect of the bandaid is a delay in search indexing; a permanent fix is forthcoming.

We'll post a detailed postmortem on Monday.

Kiln On Demand Outage

During our regularly scheduled upgrade window, we've run into a problem that is causing Kiln On Demand to run very slow when interacting with repositories in any way. We're tracking down the root cause right now and will post updates as soon as we have them.

Maintenance Complete

Tonight's maintenance was completed without incident.

FogBugz On Demand Upgrade on Sunday, 2011-10-16, at 02:00 GMT

At 02:00 Sunday morning (GMT) we will upgrade all accounts to FogBugz 8.7.7. This is a bugfix release that:

• Fixes an issue that could cause Administrators to lose administrative privileges
• Fixes an issue that could cause accounts to erroneously display an upgrading message
• Alerts users on deprecated or soon-to-be-deprecated browsers

Service impact should be no more than five minutes.

FogBugz and Kiln On Demand maintenance on Sunday, 2011-10-16, at 04:00 GMT

As part of our campaign to redistribute load more equitably, we'll be moving a handful of FogBugz and Kiln On Demand accounts across servers this Sunday, between 04:00 and 08:00 GMT. Service impact will be limited to a few minutes for each migrated account and may happen at any time during the maintenance window.

Maintenance Complete

Tonight's maintenance has been completed. Due to time constraints we didn't move all the accounts we intended to, so look for more announcements in the coming weeks. 

Kiln On Demand maintenance on Sunday, 2011-10-09, at 04:00 GMT

Our current Kiln database servers are a bit overloaded right now so, at 04:00 GMT this Sunday, we'll be redistributing about half of our existing Kiln On Demand accounts onto a shiny new pair of boxes. This should make things faster and more reliable for everybody. Each migrated account will see a couple minutes of service interruption sometime between 04:00 and 08:00 GMT.

Thanks for your patience as we make Kiln On Demand even better.