Fog Creek System Status

Resolved: FogBugz On Demand Performance Problems (Sept. 28 - Sept 30)

We had, and subsequently resolved, an issue with slowly degrading performance of some aspects of FogBugz On Demand. In particular, the sending/receiving of email and the indexing of cases would have performed very poorly. Based on logs, the problem's inception was Friday, Sept 28th at approximately 1615 UTC (1215 EDT), and shouldn't have become become noticeable until Saturday evening/Sunday morning, about the same time as our maintenance. Because our maintenance involved completely disabling these features, it masked the symptoms of the existing problem, so wasn't caught. It was the following morning that it was clear a problem existed.

We worked with it Sunday when it became clear it wasn't actually related to the maintenance, and cleared the problem up at 0300 UTC, Monday Oct. 1 (Sept. 30, 2300, EDT). Sorry for the trouble! Hopefully we caught this before its existence became too problematic.

FogBugz and Kiln On Demand Maintenance Completed

We've completed our previously announced maintenance without incident. No customers had downtime exceeding our 30 minute window (and most were closer to 30 seconds).

Service degradation for On Demand

UPDATE: At 03:27 EDT this morning, one of our shared infrastructure servers ran out of disk space. This caused two of Kiln's storage servers to begin responding to incoming requests extremely slowly. This caused a large backlog of requests, some of which were dropped, causing 500 and 503 errors to be returned. At 05:05 EDT, our on-call sysadmin responded to the alerts from the Kiln load balancers and cleared the issue. Outstanding requests were then processed, and Kiln service returned to normal within a few minutes.

We have two basic follow up items from this event:
1) Fix log rotation which caused disk space to filled up.
2) Insulate the Kiln servers from a failures on the shared infrastructure server.

 

Original Post:

From approximately 3:00 AM - 5:00 AM EDT (7:00 - 09:00 UTC), the On Demand infrastructure experienced some network degradation causing some open connections to time out or fail abruptly. We're investigating further to assess the entire impact. Once we know more about the issue, we'll post an update here. We'll be making sure that this does not occur again, if it's within our control, and post a root case analysis. 

At this time, it appears that the network degradation has been resolved. If you still see issues, please contact us at customer-service@fogcreek.com. 

FogBugz and Kiln On Demand Maintenance Sept. 30, 0200 UTC (Sept. 29, 2200 EDT)

We will be, once again, performing maintenance on our On Demand infrastructure. The customer-impacting portion of the maintenance will be starting promptly at 0200 UTC (2200 EDT).

This will be similar to the maintenance attempted last weekend. We determined that the cause of the errors we encountered was not due to the maintenance itself, and have corrected the remaining issues. We're also using a less-invasive method to perform the customer-impacting maintenance. Individual customers will be going down for approximately 30 minutes each, which may occur at any time between 0200 and 0500 UTC (2200 and 0100 EDT), depending on where they fall in the list of customers receiving maintenance.

Emergency maintenance completed successfully

Our previously announced emergency maintenance was completed without any incident. There was no customer impact. How great is that?!

On Demand Emergency Maintenance Tonight (Sept. 27, 0130 UTC)

Tonight we will be perfoming emergency hardware maintenance some of our On Demand infrastructure. The maintenance will begin at approximately 0130 UTC (2130 EDT). The maintenance itself will be brief and we're hoping for zero impact to customers.

However, there's a chance there will be a notable slowdown for some of our customers which will last approximately 30 minutes. I apologize for the trouble we've had lately; hopefully tonight's work won't be too off-putting for anyone (except the poor engineer who drew the short straw).

FogBugz and Kiln On Demand returned to normal operation

We were able to complete repairs of the FogBugz and Kiln architecture tonight. We completed at approximately 2330 EDT (0330 UTC). There was slightly more downtime than we wanted, but we needed to be certain things were completely finished.

I sincerely apologize for the trouble we caused over the weekend. However, there's no indication any of our planned maintenance on Saturday was the culprit: We caught the wrong end Hyperthreading on two of our servers. Disabling it fixed the timeout issues.

Should you want a more in-depth description of what happened, please don't hesitate to let us know and I will personally provide it. For now, our engineers and myself are tired and world-weary.

FogBugz and Kiln On Demand Maintenance Aborted [UPDATES]

Update, 1520 EDT: We've rolled back everything from last night we can get our hands on without doing a reboot of the affected servers, but haven't seen any improvement. As such, tonight at 2200 EDT (0200 UTC) we'll be rebooting our impacted servers. Customers currently on those servers will go offline for the duration, which will be approximately 25 minutes (if you take into account SQL Server recovery time for the databases involved; most users will be up faster than that). Our primary concern is that this problem will get geometrically or even expontentially worse once the additional week-day load is added, so we're anxious to get it solved before then. Otherwise, we would give a longer warning before the reboot.

Update, 1200 EDT: After the rollback, there appears to be a condition causing intermittent SQL timeouts for FogBugz and Kiln On Demand. We're rolling back the portions of the maintenance that looked good at the time, but there's no clear cause-and-effect at present. The errors shouldn't be crippling, but are certainly undesirable. We're hard at work tracking down the cause to initiate repairs. Sorry for the trouble, everyone!

Partway through the maintenance announced earlier this week, we encountered a condition that was fixable, but would have pushed the announced downtime beyond the limit. As such, I decided to call an abort at 0239 UTC (2239 EDT) and all services were restored by 0245 UTC (2245 EDT).

Very little of the completed maintenance had to be rolled back at this point, so once we resolve this roadblock, we will be making another attempt which will, ideally, be much more smooth. A new maintenance announcement will be released at that point.

FogBugz and Kiln On Demand Infrastructure Maintenance - Sept. 23, 0200 UTC (Sept. 22, 2200 EDT)

We will be performing server maintenance on Saturday night that will impact FogBugz and Kiln On Demand customers. There will be downtime associated with this maintenance. We are doing everything we can to minimize the downtime, but the current estimate is that the downtime will last about 1 hour. This downtime may occur at any time during the maintenance window.

Maintenance will begin promptly at 0200 UTC (2200 EDT) and last until 0500 UTC (0100 EDT).

Not all customers will be going offline. However, in the coming weeks we will be repeating this maintenance and impacting different groups of customers. We will make additional announcements prior to those maintenances.

I understand this is a lot of downtime. If that night won't work, please get in touch with our support department and we will attempt to find a better schedule.

FogBugz On Demand: "Error in Kiln Plugin" - Resolved

A fix has been deployed to resolve the problem of benign Kiln errors showing in FogBugz only On Demand accounts.

Thanks for your patience while we tested and released this fix!