Fog Creek Network Status

FogBugz and Kiln On Demand Upgrade on Sunday, 2012-01-29, at 03:00 GMT

At 03:00 Sunday morning (GMT) we will upgrade all accounts to FogBugz 8.8.6 and Kiln to 2.8.10. 

FogBugz release notes:

• Updated Plugins
  • Report Generator
  • FB Scratchout
  • CaseToSalesforce
  • Creately
  • Clark Kent
  • DoLater
• BugzScout improvements
  • Collect 50 BugzScouts, then turn off automatically
  • Can turn collection back on to get another 50 BugzScouts
  • Last Occurrence field lets you search, filter, sort on last BugzScout occurrence, whether it was recorded or not
• Performance improvements to reduce database load
• Backend improvements to search indexing to minimize the number of corrupt indices
• Screenshot tool uses your default browser for logging in
• Group headers in list view now toggle selection of all cases in the group
• Bug fixes
  • Screenshot tool lets you log in again
  • A user's homepage can now be saved
  • Email notifications for incoming non-autosorted mails without a text part now display correctly
  • Many minor cosmetic fixes

Kiln release notes:

• The Kiln client is now available for Mac OS X. If you're on Mac OS X, you can now use Tortoise, complete with full auto-update capabilities, drag-and-drop install, and more.
• Reviews are now live. If you edit a review, everyone else in the review sees it immediately, automatically. No more hitting "Reload" every few seconds.
• You can now create a review at push time by using the new push --review command.
• Problems with diff timeouts have largely been fixed. This is a tough one; a lot of you like to make absolutely massive changesets, which pose interesting problems for us. We're not all the way there, but the situation's a lot better.
• Problems with non-ASCII file encodings causing source code content indexing to hang have been fixed. Note that non-ASCII file names may not be properly indexed, but it shouldn't hang your Kiln Storage Service, either. (We're working on a new elasticsearch-based code search solution that'll be available son and will solve this problem completely.)
• Problems with IE8 and IE9 have been fixed. As a reminder, IE7 is no longer supported.
• Firefox 10 (its current beta, anyway) works again with Kiln.
• We've landed numerous performance improvements, especially for very large Kiln accounts and their activity feeds.
• Lots of small bug fixes and UI improvements, including an improved URL scheme for projects and repositories.

No database upgrade is required, so the process should be very quick.

FogBugz On Demand Email Processing Returns to Normal

Today's email processing delay has been corrected.

We're in the process of rebuilding the server farm that runs asynchronous maintenance tasks like search indexing and email processing for FogBugz On Demand. It turns out that the email processing task for cases@youraccount.fogbugz.com addresses silently failed with our new configuration, but we didn't notice until we brought down the last of the old servers today.

We implemented a quick-fix about half an hour ago and most delayed emails should have been processed by now. A permanent fix that includes noisy failure will go out with our next FogBugz On Demand deployment.

Delay in Processing of Some Email Messages in FogBugz On Demand

Since 12:42 PM EST today, FogBugz On Demand accounts have not been processing incoming email messages directed to cases@{your FogBugz site}.fogbugz.com. All incoming mail is still present, but has not yet been processed by FogBugz. We are actively working on a solution to this problem and will post to status.fogcreek.com when it is resolved.

Outgoing mail and additional custom mailboxes, e.g. bugs@example.com, are not affected by this issue.

Brief FogBugz On Demand Performance Degradation

Between 16:32 and 16:45 EST, one of our database servers suffered massive performance issues that customers on that server may have seen as slowness, timeouts, or FogBugz error messages.

Our monitors warned us this morning of an expected drive failure on this server. To get the drive replaced, Dell asked that I run their Dell E-Support Tool (DSET) to gather the relevant log entries. We've run this in the past with no issues and were assured that it was designed to operate unobtrusively on production machines. Today, unfortunately, it obtruded. Within minutes of requesting a report, memory owned by SQL Server began paging out. This brought the machine to a near-standstill.

SQL Server typically eats up about 44GB of memory on this particular server. When I finally managed to kill the DSET process, it had fallen to just over 3GB. The root issue was resolved at this point, but slowness lingered until about 16:45 as SQL Server reclaimed enough memory to function properly.

I apologize for the outage. Naturally, we won't be running this reporting tool again outside a contained test environment.

Maintenance Complete

Tonight's maintenance was completed without incident.

Infrastructure and FogBugz Maintenance on Sunday, 2012-01-08, at 05:00 GMT

This Sunday, between 05:00 and 08:00 GMT, we will perform maintenance on our network infrastructure and migrate some FogBugz databases between servers. The infrastructure maintenance should be very brief, but may cause hiccups in any of our services. As usual, the FogBugz database moves are on a per-account basis and may occur at any time during the maintenance window.

Trello Outage Postmortem

Trello.com was down for about 25 minutes on Tuesday, January 3.

Our monitors alerted us to intermittent connection failures at about 10:55 EST and a bandaid fix was rolled out at 11:20. The cause of the actual outage was a dramatic increase in the number of TCP connections open against Trello from our usual 20-25k to our firewall's 100k ceiling. We doubled this limit to try to relieve pressure while we diagnosed the problem, but hit the ceiling again within seconds of making the change.

Eventually, we narrowed the issue down to a problem with Websockets and, at 11:20, forced all clients to fall back on AJAX polling, ending the outage. 

The root problem was tracked back to a recent change in a client Websocket library that caused the application to repeatedly open duplicate connections. This has been fixed and a new version deployed. At this point, Trello is running Websockets for compatible browsers and our state tables look healthy.

We apologize for the inconvience. Thanks for bearing with us as we make Trello better and better.

Trello Issue Resolved, All Services Up

Today's Trello issue has been resolved. We're still digging into the root cause but do not expect a recurrence. We'll post an update once we have more information.

Network event impacting Trello.com

We presently are experiencing a network event which is impacting Trello.com. Our system administrators are hard at work tracking it down. We will post an update and post-mortem as soon as they are available. Sorry for the bother and thank you for bearing with us!

Kiln Full-Text Search Offline for Next 20 Minutes

A Kiln server responsible for full-text search will be unavailable for about 20 minutes (starting at 4:10pm EST, 9:10pm UTC) as a part of the preventative maintenance described at http://status.fogcreek.com/2011/12/kiln-search-will-be-out-of-date-for-some-repositories.html.

Thank you for your understanding.