Fog Creek Network Status

FogBugz On Demand Upgrade to 8.8.21: Sunday May 13, 0200 UTC

At 02:00 Sunday morning (UTC) we will upgrade all accounts to FogBugz 8.8.21. This is a maintenance release that fixes an internal facing bug. 

No database upgrade is required, so the process should be very quick.

May 5-6 Network Maintenance Post-Mortem

You may recall some issues stemming from the network maintenance performed last week. We've been investigating since, and here's our post-mortem.

What was the impact?

The network was offline with all services down for approximately two hours. This was one hour longer than our anticipated downtime, but still fell within the maintenance window that we had generated.

So, what happened?

During the process of rearchitecting our switch fabric's spanning tree (moving from a more control-centric per-vlan spanning tree to a faster-failover rapid spanning tree, ironically to keep downtime to a minimum), we suddenly lost access to our equipment. I was the one present on site at the datacenter, so began investigating.

• A network loop had formed between several switches.
• The gateways controlling access to the switch management network were isolated from each other, generating a split-brain scenario. Neither were accessible due to a sudden traffic flood.
• The flood was the result of a multi-switch BPDU (bridge protocol data unit) flood, indicating a spanning-tree flap. This is most likely what was changing the loop domain.

How was it repaired?

Switch reboots and some "hands-on" rearchitecting of the switches enabled us to complete the migration to RSTP (we were interrupted in the middle) and stop the flood. After the packet TTL died off and the switches were able to rebuild their MAC tables accurately, connectivity was restored. The rest of the time spent was on restoring services which had suffered.

What's to prevent this from happening again?

We're taking several steps to prevent a recurrence:

1. We're evaluating our equipment. We haven't found standards-based reasons for the BPDU flood; that's exactly what shouldn't be happening. If it's a bug, that means firmware updates or new equipment. If it's an odd convolution of events, that means rearchitecting our equipment to remove one of the necessary members of the convolution.
2. We're changing our maintenance planning to be more comprehensive. We identified some holes in the maintenance plan that failed to account for such a catastrophic breakdown. We're redrawing the line on the continuum of "safe to perform" and "keeps the maintenance window small." As a general rule, plans that optimize for short maintenance windows are less safe, whereas plans that are safe tend to have longer windows and longer downtime. We're aiming more toward "safe" now than we were. In a practical sense, our customers won't see the difference... except when we avert a catastrophe, rather than suffer through it.

 Finally, I would like to once again apologize for any trouble this maintenance caused.

FogBugz upgrade and network maintenance completed

The work (referenced here) has been completed, but not without incident.

One of our web servers ran into a little trouble during the FogBugz upgrade. There was approximately 15 minutes of additional downtime for those users.

The network maintenance likewise ran into trouble. Services were offline for significantly longer than intended. Most services were back by 0530 UTC (0130 EDT) and complete functionality was returned by 0600 UTC (0200 EDT). We deeply apologize for the trouble caused by this maintenance. We're currently investigating the root cause and will provide a post mortem as soon as the root cause has been definitively determined.

FogBugz On Demand Upgrade and Network Maintenance: Sunday, May 6 0200 UTC

We will be performing an account upgrade to Fogbugz 8.8.20 on Sunday, May 6 at 0200 UTC (Saturday, May 5, 2200 EDT). The complete process will take approximately one hour and individual customers can expect downtime lasting between 5 and 15 minutes any time during that hour.

Additionally, upon completion of the upgrade, we will be beginning the postponed network maintenance from two weeks prior. This maintenance will involve downtime for all services (FogBugz and Kiln On Demand, Copilot, Trello, and the Fog Creek Website) lasting approximately one hour. This hour may begin at any time between 0200 and 0500, UTC (2200 and 0100, EDT). We will make every effort to keep this downtime to the bare minimum.

All services will be restored by 0600 UTC (0200 EDT).

Release notes for 8.8.20:

•  Add the Intro Case plugin which sends new users an explanatory case
•  Fix up how custom fields with dropdowns work
•  Remove broken MySQL database backup option
•  When converting HTML to plain text, strip comments
•  Fix IE8 label rendering bug
•  Allow 24 hour times to be specified for Do Later
•  Fix case creation in FogBugz Mobile
•  In LDAP let community users set their password
•  Don't display Version or Computer when they are disabled
•  Disable AutoSubscribe for Community users
•  Show an error if the user enters a bogus bug id in the working on dialog
•  When you reset auto-sort, also reset what is pending training
•  Show error message when entering an invalid estimate
•  Don't close the code snippets dialog when you click outside
•  Generate valid HTML for tables of contents in the wiki
•  Make the recent items query faster
•  Plugin Updates:
  •  "Wiki Article: Case Relationships" version 2.4
  •  "Balsamiq Mockups" version 2.1.14

Kiln On Demand Upgrades

Over the next few weeks, we'll be moving all Kiln On Demand accounts to a brand-new high-performance infrastructure (just like our Student and Startup accounts). This upgrade will require about an hour of downtime during which Kiln On Demand will be unavailable.

To reduce the impact of this downtime, these upgrades are scheduled for Sunday, May 6th, 13th, and 20th, 2012. Each account will be upgraded between 9am and 6pm EDT (1300-2200 UTC) on one of those days.

If these maintenance periods would be distrupive to you, please contact us at customer-service@fogcreek.com -- we're happy to reschedule your upgrade for a more convenient time.

Thanks in advance for your patience as we make Kiln better!

FogBugz and Kiln On Demand Maintenance completed successfully

We have just completed both the FogBugz and Kiln On Demand deployment and maintenances (mentioned here and here, respectively).

Both maintenances were completed successfully. We detected minor difficulties during the Kiln maintenance, but no customer impact should have existed and the trouble lasted less than 2 minutes.

Kiln On Demand Maintenance: Sunday, April 29 0200 UTC

We will be performing maintenance on the backend systems that support Kiln On Demand on Sunday, April 29th at 2:00AM UTC (Saturday, April 28th at 10:00PM EDT).

This maintenance is expected to last approximately two hours and there should be no interruption in service for customers.

FogBugz On Demand Upgrade to 8.8.17: Sunday April 29, 0200 UTC

We will be performing an upgrade of all FogBugz On Demand accounts on Sunday, April 29th at 2:00AM UTC (Saturday, April 28th at 10:00PM EDT).

There is no database upgrade required, but we will be proceeding cautiously, so the entire process could take as long as two (2) hours with intermittent downtime for accounts during those hours. We don't expect the total downtime for any account to exceed 30 minutes.

The following are the list of changes we will be making:

FogBugz 8.8.17 Release Notes

• Fix an exception that prevents case creation when a plugin hooks to the AreaJoin or FixForJoin interfaces
• Dropdown fields behave better with values like "--"
• Performance fix for Mono and case linking
• Fix for admins on accounts with many thousands of users
• Added an admin notification for users that have unviewable cases
• Fix a time bug related to intervals over daylight savings time
• Typo fixes
• Plugin updates:
  • FogBugz Timesheet (v1.0.26.0)
  • Relative Time (v1.1.0)

Datacenter Maintenance: Completed (with caveats)

Our datacenter maintenance has been completed and was a success. However, some elements could not be completed in the time alloted, so have been postponed. We'll make a separate post at a later date announcing the new window for those changes.

Issue resolved; All services up

The email and FTP outage noted earlier has been resolved. The sever that provides these services crashed with what appears to be a RAID firmware bug. It was out of commission for approximately 15 minutes. No data was lost, though incoming emails may be delayed for a few more minutes as remote servers retry sending.

We'll schedule a maintenance in the near future to implement a permanent fix for this issue.