Fog Creek System Status

Kiln On Demand Outage Resolved

Kiln On Demand suffered an outage lasting approximately 33 minutes (1202 - 1235 EDT). The outage has been resolved and is not likely to recur. Customers would have experienced a combination of slow requests and 503 HTTP errors as a result. This would have impacted the website and Mercurial/Git interactions equally. Customers still on Kiln 2.9.52 or utilizing developers.kilnhg.com were up approximately 10 minutes earlier than customers on Kiln 3.0.27.

The cause was a caching server delaying cache hits due to an unexpected surge in memory usage. This started backing up the Kiln website as the web servers waited longer and longer for page loads. Once it was detected, we rebooted the cache server. To respond to the queued requests which were still causing problems but had no chance of completing, the Kiln application pools were recycled. The cache server came online successfully, but the flood of failed requests from the recycle caused the application pool to be disabled.

We manually started the application pools and adjusted their settings to more sanely handle this situation should it arise again. At that point, service was restored to all customers.

Engineers are monitoring the situation and are seeing no early warning signs of a recurrence.

Kiln On Demand Upgrade Completed

Our previously announced upgrade has been completed successfully. The service interruption for the upgrade was limited to 11 minutes, but there was a side-effect for some customers.

Customers with public repositories may have noticed some timeouts resulting in the "Kiln has overheated" message. This was the result of a pathological query plan getting generated after the old one was blown away. We've fixed the query for the next deployment, but the only recourse tonight was to clear the cache and let our database servers rebuild it. All customers should be recovered by now.

Kiln On Demand Upgrade: March 31, 0200 UTC (March 30, 2200 EDT)

We will be upgrading all Kiln On Demand accounts to version 3.0.27 this weekend. The upgrade will start promptly at on March 31 at 0200 UTC (March 30 at 2200 EDT), with a window lasting one hour. This upgrade will be service impacting. We expect customer accounts will be unavailable for approximately 30 minutes. This downtime may occur at any time during the window, but all services will be restored by 0300 UTC (2300 EDT).

Below you'll find the release notes for Kiln 3.0.27. If you have any questions about them, please feel free to contact us.

• Added a Sprint.ly web hook
• Fixed bugs in the Harmony backend
• Fix a bug where multiple "Processing changesets" events could appear for a single Git push
• Several performance enhancements
• Minor bug fixes

Brief Network Outage in On Demand

We've been notified by our monitoring system that there were network issues immediatly after midnight EDT. At this time, it appears this resulted in a 15 - 20 minute network outage of our On Demand infrastructure. Service has been restored and we will post a more complete post mortem shortly.

Kiln On Demand Upgrade and Network Maintenance - COMPLETE

The previously announced maintenance has been completed successfully.

Kiln On Demand Upgrade and Network Maintenance: March 28, 0200 - 0400 UTC (March 27, 2200 - March 28, 0000 EDT)

Update (March 27, 1130 EDT): The FogBugz team was jealous that everyone else was doing work, so I'm letting them deploy tonight, also. Like the others, there is no expected service interruption and no customer-facing changes. They will be deploying some behind-the-scenes code for the recently announced FogBugz Ocelot Beta!

We will be upgrading the Kiln On Demand test accounts tonight. This is a re-run of the failed portion of the deploy from Saturday. There will be no customer-facing changes and there will be no service interruption.

Additionally, we are going to perform some clean-up from the network maintenance over the weekend to increase our resiliency and service availability. There is no expected service interruption for this maintenance, either.

Weekend Maintenance Completed

We've completed all of our maintenance for the weekend.

The Kiln deploy went successfully, but we had to abort the upgrade due to some problems on one of the servers. There was no customer impact, but we will need to revisit that upgrade at a later date.

The network maintenance didn't go perfectly, but the customer impact was low. There was approximately 20 minutes in which emails (inbound and outbound) were delayed. Mail should have been queued by the respective servers, however. Additionally, some customers may have noticed some trouble with Mercurial or Git actions (push, pull, clone, etc.) during the same time period.

All services have been restored at this point and we're closing up shop for tonight.

Maintenance Changes for this Weekend!

Due to the network issues we’ve encountered this week, we are delaying the relocation of our equipment. That maintenance is now tentatively scheduled for March 30th. A new announcement will be sent out once a final schedule is determined. Instead of the original maintenance, we will be replacing some of our network gear and taking the opportunity to do some Kiln On Demand maintenance. The following is the schedule for Saturday night, but the brief version is that there will still be a service interruption, but should be no longer than one hour and should be significantly less.

Kiln On Demand Upgrade: March 24, 0000 - 0200 UTC (March 23, 2000 - 2200 EDT)

We are deploying some bug fixes to the backend infrastructure for Kiln On Demand. There will be no customer-facing changes with this upgrade. There is no expected downtime.

Network Maintenance: March 24, 0200 - 0500 UTC (March 23, 2200 - 0300 EDT)

In order to correct problems on the border of our network, we are replacing the old equipment with newer, more powerful equipment. Our expectation is that we will be able to fail over to the new equipment with a service interruption of approximately 5 minutes. However, in the event of any problems during the fail over process, customers should plan for one hour of downtime while our engineers work the kinks out. This interruption may occur at any time during the maintenance window.

FogBugz On Demand Upgrade Canceled

At maintenance time, we decided to abort our previously announced FogBugz upgrade and will perform it at a later date.

FogBugz On Demand Upgrade: March 22, 0200 UTC (March 21, 2200 EDT)

We will be deploying to FogBugz On Demand to provide bug fixes for our beta customers; there are no changes for standard accounts.

The deploy will begin promptly at 0200 UTC (2200 EDT). No customer impact is expected.