We came, we saw, we once again upgraded. FogBugz On Demand is now on version 8.9.81
A pleseant good evening to all.
Our monitoring systems alerted us to increased response time for some FogBugz On Demand customers. At this point, the symptoms appear to be similar to the issue we experienced yesterday.
We have already made adjustments to compensate and response times are returning to normal at this point. The incident lasted from approximately 17:25 to 18:00 (UTC -4).
We will be upgrading FogBugz On Demand to version 8.9.80. The upgrade window will begin at 0200 UTC (2200 EDT) and last for 90 minutes. Customers should expect a brief interruption during the upgrade. The interruption may occur at anytime during the windows.
As an update from our earlier announcement, the issue is currently no longer customer facing as of 5:15 PM EDT (21:15 UTC). You should no longer see any performance degradation, timeouts, or issues editing cases.
We're continuing to investigate this issue to get it fully resolved.
We have been alerted by our monitoring system to performance degradation of FogBugz On Demand. The issue appears to only impact the FogBugz Performance Upgrade, ironically enough.
Some customers will see general slowness and timeouts in the case and list pages. This includes editing cases.
If you have the toggle under My Settings > Options, you can revert to the non-performance upgrade version as a workaround.
Our FogBugz engineering team and Sys Admins are working on resolving this issue exclusively at this time.
Starting at 1900 UTC (1500 EDT) we will begin a rolling upgrade of Kiln On Demand to version 3.0.94. As a reminder, this means that every account will upgrade at the account's local midnight. That is, there will be no day-time interruptions.
However, customers should expect up to two minutes of service interruption when their account is upgraded. This may occur at any time between 0000 and 0100 of the account's local timezone.
The process will be fully completed by Saturday, October 26th at 2000 UTC (1600 EDT).
You can see release notes for Kiln On Demand 3.0.94 on our customer support portal: http://help.fogcreek.com. If you have any questions or concerns about the deploy process, please don't hesitate to contact us!
We have reports of users getting "Error in Kiln" messages. This is affecting non-admin users viewing or editing cases that have changesets/checkins. The FogBugz Team has identified the issue and is working on getting a fix out as soon as possible.
New Kiln accounts will not be able to log-in until this issue has been fixed.
Please contact us if this materially affected your ability to do business.
* Resolved (11:10 AM EDT) *
The FogBugz Team and System Administrators have fixed the issues. The "Error in Kiln" message should no longer show on any pages, and new Kiln accounts should now be able to log-in as expected. We'll post a full post-mortem as soon as we've reviewed our procedures and added extra checks to make sure something like this doesn't happen again.
** Post Mortem (3:15 PM EDT) **
An incorrect version of the Kiln Plugin was deployed along with last night's FogBugz upgrade. Unfortunately, our standard post-deploy checks did not catch this particular issue. We will be updating our deployment procedures to ensure this type of problem doesn't happen in the future.
We have completed our work on FogBugz and Kiln On Demand. The night went well.
The Kiln On Demand database maintenance went far smoother than we thought; there was no service interruption. Go team!
The FogBugz On Demand upgrade went as expected, but there was an incident toward the end which may have delayed search indexing and email (sending and receiving) by up to 20 minutes. Most customers would not have experienced any problems. We were further delayed by an unexpected, but non-impacting, network event which caused their "fearless" leader to halt the proceedings: We lost a management port in the FogBugz storage network and it struck fear into my heart. I've been instructed to take a "chill pill" before bed.
Both products are online and healthy! We're all headed to bed. If you have any questions or concerns, please don't hesitate to contact us!
On Wednesday night, we will upgrade FogBugz On Demand to version 8.9.79 as well as perform some database index maintenance for Kiln On Demand. Kiln On Demand will go first, with the FogBugz On Demand upgrade coming immediately afterward. The total maintenance window will be 90 minutes.
The Kiln On Demand index maintenance will start promptly at 0200 (2200 EDT). Customers should expect a service interruption of no more than 15 minutes while the index maintenance is underway. While I fully expect us to start on time, customers should be prepared for this service interruption to occur at any time during the maintenance window.
After Kiln On Demand is confirmed to be back online and functioning normally (better than normally, hopefully!), we will begin the FogBugz On Demand upgrade. Customers should expect between 2 and 5 minutes of service interruption while their accounts are upgraded, depending on the size of the account. The extended time is due to a database schema upgrade. This interruption may occur at any time during the maintenance window.
If you have any questions or concerns about the upgrade or the maintenance, please don't hesitate to contact us.