As part of the upgrade process, Kiln was scheduled to rebuild some of its repository search indices. The resulting system load was orders of magnitude larger than initially anticipated, leading to extremely slow access to Mercurial repositories and 500 errors on some Kiln pages. To remedy the problem, the Kiln team throttled the reindexing task, restoring normal Kiln On Demand access by 2 AM EDT Sunday morning.
Long-term, we're working to better simulate production load in our pre-deployment tests so we can predict issues like this. Short-term, we'll continue to deploy in the wee hours.
We'll continue playing musical servers this weekend, migrating several larger accounts to ensure everybody can continue to grow at their own speed. As usual, these moves will be done between 04:00 and 08:00 GMT and service impact will be limited to one account at a time.
During our regularly scheduled upgrade window, we've run into a problem that is causing Kiln On Demand to run very slow when interacting with repositories in any way. We're tracking down the root cause right now and will post updates as soon as we have them.
As part of our campaign to redistribute load more equitably, we'll be moving a handful of FogBugz and Kiln On Demand accounts across servers this Sunday, between 04:00 and 08:00 GMT. Service impact will be limited to a few minutes for each migrated account and may happen at any time during the maintenance window.
Our current Kiln database servers are a bit overloaded right now so, at 04:00 GMT this Sunday, we'll be redistributing about half of our existing Kiln On Demand accounts onto a shiny new pair of boxes. This should make things faster and more reliable for everybody. Each migrated account will see a couple minutes of service interruption sometime between 04:00 and 08:00 GMT.
Thanks for your patience as we make Kiln On Demand even better.