Starting Tuesday 8 November we started seeing issues with our On Demand service, which we have now resolved. These issues fell into three major categories (all dates/times Eastern):
- Network congestion starting Tuesday 8 November and continuing through a configuration change made during emergency maintenance the morning of Monday 14 November. This caused Kiln slowness and repository outages, and general slowness and intermittent failures of On Demand services.
- Failing hardware due to a series of power issues on Sunday 13 November. This caused two outages of On Demand services as we worked with our vendors to replace hardware and resolve the issue the afternoon of Monday 14 November.
Instability in FogBugz and Kiln On Demand continued as a complication of our previous issues even after their resolution. This caused dramatic slowdowns and intermittent errors throughout FogBugz and Kiln. We performed emergency maintenance on our Elasticsearch cluster 8 PM Eastern on Tuesday, 16 November to fully restore our On Demand services.
In the coming days we will post a post-mortem to document our resolution of these incidents. In the mean time, our services are fully restored and you should expect normal behavior and performance.
If you have any questions, or notice any errors or poor performance, please contact us