« October 2011 | Main | December 2011 »
The bug that caused the downtime last night also broke the signup process for new Kiln accounts. To get that working again, we'll be deploying a fix for all accounts at 2:00 am GMT on Thursday. No database upgrade will be required, and downtime should be minimal.
Posted by Rock Hymas at 08:11 PM | Permalink
On Saturday night and Monday night, we upgraded 40% of Kiln and FogBugz customers to new versions of both products. Last night we upgraded the remaining 60% of our users. Unfortunately, the upgrade took down Kiln for a short time. In an effort to understand what went wrong and how we can make sure it doesn't happen again, here's our post-mortem of the issue.
As mentioned in the post last night, the change that caused this problem was due to OPTIONS HTTP requests failing when sent with an empty Host: header. Normal Kiln requests always have a host header that indicates the account that the request is for. But our load balancer sends these without a host header just to verify that IIS is up and running. If it sees failures, it stops passing requests on to that web server. That way when an individual web server goes down, the load balancer automatically moves requests to servers that are up. Of course, since the OPTIONS request failed on both machines, the load balancer stopped sending requests to either of them, and began returning HTTP 503 errors.
The reason the bug didn't bite until we had upgraded all accounts is because, by default, requests for unknown accounts go to the default version of Kiln, which wasn't changed until last night. We rolled this version out over a few days because there were changes to the way Kiln's backend processing worked and we wanted to make sure it scaled. Leaking it out to progressively large numbers of accounts was an attempt to monitor and catch any issues with that. One change we did in this release was to upgrade all of the accounts based in time zones that would have been affected by a 10pm upgrade over the weekend. Unfortunately, due to the nature of the bug, all Kiln accounts were still taken down, not just those that were being upgraded.
Going forward, we'll be doing a few things to make sure this doesn't happen again. First, we'll go back to doing upgrades over the weekend, when they cause the least disruption if an outage occurs. Second, we're working on a staging environment for Kiln (and FogBugz) that closely mirrors our production system. If this had been in place we would have caught this issue well before releasing to our customers. Also, we'll make sure that we do testing of requests to invalid accounts before releasing.
Posted by Rock Hymas at 01:09 PM | Permalink
Tonight's Kiln outage was resolved shortly after we posted.
An in-depth post-mortem will be posted tomorrow, but the short version is that a change to our default Kiln generation caused our web servers to begin returning HTTP 500 errors on an empty Host: header. Our load balancers use simple OPTIONS HTTP 1.0 checks to monitor our servers' health. When this change was effected, our load balancers declared the servers dead and users began receiving HTTP 503 errors. We've modified the health checks to pass along a Host: header and will analyze tomorrow how this was overlooked in our testing.
We apologize for the interruption. If this got in your way, please contact us and we'll make it right.
Posted by Shawn Hargan at 12:06 AM | Permalink
Kiln On Demand is currently unavailable. We're on it and will post updates as soon as we have them.
Posted by Shawn Hargan at 11:17 PM | Permalink
At 02:00 Sunday morning (GMT) we will upgrade all accounts to FogBugz 8.7.17 and Kiln 2.7.7.
The FogBugz upgrade includes:
The Kiln upgrade includes:
The upgrade is expected to be quick but does include a minor database upgrade. If you run into the "We are upgrading your account" message tonight, give it just a minute or two and reload. The vast majority of you shouldn't even notice!
Posted by Shawn Hargan at 03:02 PM | Permalink
This weekend we released an update to FogBugz that affected how we store search indexes. The change went out as part of a group of changes that were intended to improve FogBugz performance. We tested each change individually at a smaller scale and were confident they would improve performance. Under the load of the production environment, however, we saw mail processing and search indexing delays that eventually led to corrupted search indexes for some customers.
Fortunately, FogBugz can automatically rebuild search indexes by resetting the index. We've triggered this rebuild for affected customers.
Rebuilding the index takes time, so it may take a while before all your cases are fully indexed again. During the rebuilding period, you will be notified that search results are incomplete when you perform a search. Note that using search axes like "editedby:me" does not usually rely on the index, so you can still find cases that way. You can also check to see search indexing progress on your Site Configuration page in the Search tab.
Note that this only affects a subset of our customers. If you haven't seen search errors or a notice that your index is being rebuilt, you are totally unaffected by this issue.
Posted by Shawn Hargan at 04:22 PM | Permalink
The following is a list of changes being made for this weekends planned maintenance:
FogBugz Changes:
Posted by Tim Stewart at 04:32 PM | Permalink
We'll continue our account-rebalancing saga this weekend, migrating accounts between servers to ensure growth can continue unhindered. As usual, these service interruptions are on a per-account basis and may occur at any time between 04:00 and 08:00 GMT on Sunday.
We're also taking this opportunity to make a few minor configuration changes to our web servers and load balancers. These will be made at the beginning of the maintenance window and service impact should be minimal.
Posted by Shawn Hargan at 01:48 PM | Permalink
