At 02:00 Sunday morning (UTC) we will upgrade all accounts to FogBugz 8.8.21. This is a maintenance release that fixes an internal facing bug.
No database upgrade is required, so the process should be very quick.
At 02:00 Sunday morning (UTC) we will upgrade all accounts to FogBugz 8.8.21. This is a maintenance release that fixes an internal facing bug.
No database upgrade is required, so the process should be very quick.
Posted by Shawn Hargan at 10:45 AM | Permalink
You may recall some issues stemming from the network maintenance performed last week. We've been investigating since, and here's our post-mortem.
What was the impact?
The network was offline with all services down for approximately two hours. This was one hour longer than our anticipated downtime, but still fell within the maintenance window that we had generated.
So, what happened?
During the process of rearchitecting our switch fabric's spanning tree (moving from a more control-centric per-vlan spanning tree to a faster-failover rapid spanning tree, ironically to keep downtime to a minimum), we suddenly lost access to our equipment. I was the one present on site at the datacenter, so began investigating.
How was it repaired?
Switch reboots and some "hands-on" rearchitecting of the switches enabled us to complete the migration to RSTP (we were interrupted in the middle) and stop the flood. After the packet TTL died off and the switches were able to rebuild their MAC tables accurately, connectivity was restored. The rest of the time spent was on restoring services which had suffered.
What's to prevent this from happening again?
We're taking several steps to prevent a recurrence:
Finally, I would like to once again apologize for any trouble this maintenance caused.
Posted by Bradford Ley at 05:05 PM | Permalink
The work (referenced here) has been completed, but not without incident.
One of our web servers ran into a little trouble during the FogBugz upgrade. There was approximately 15 minutes of additional downtime for those users.
The network maintenance likewise ran into trouble. Services were offline for significantly longer than intended. Most services were back by 0530 UTC (0130 EDT) and complete functionality was returned by 0600 UTC (0200 EDT). We deeply apologize for the trouble caused by this maintenance. We're currently investigating the root cause and will provide a post mortem as soon as the root cause has been definitively determined.
Posted by Bradford Ley at 02:52 AM | Permalink
We will be performing an account upgrade to Fogbugz 8.8.20 on Sunday, May 6 at 0200 UTC (Saturday, May 5, 2200 EDT). The complete process will take approximately one hour and individual customers can expect downtime lasting between 5 and 15 minutes any time during that hour.
Additionally, upon completion of the upgrade, we will be beginning the postponed network maintenance from two weeks prior. This maintenance will involve downtime for all services (FogBugz and Kiln On Demand, Copilot, Trello, and the Fog Creek Website) lasting approximately one hour. This hour may begin at any time between 0200 and 0500, UTC (2200 and 0100, EDT). We will make every effort to keep this downtime to the bare minimum.
All services will be restored by 0600 UTC (0200 EDT).
Release notes for 8.8.20:
Posted by Bradford Ley at 12:34 PM | Permalink
Over the next few weeks, we'll be moving all Kiln On Demand accounts to a brand-new high-performance infrastructure (just like our Student and Startup accounts). This upgrade will require about an hour of downtime during which Kiln On Demand will be unavailable.
To reduce the impact of this downtime, these upgrades are scheduled for Sunday, May 6th, 13th, and 20th, 2012. Each account will be upgraded between 9am and 6pm EDT (1300-2200 UTC) on one of those days.
If these maintenance periods would be distrupive to you, please contact us at customer-service@fogcreek.com -- we're happy to reschedule your upgrade for a more convenient time.
Thanks in advance for your patience as we make Kiln better!
Posted by Kevin Gessner at 12:34 PM | Permalink
We have just completed both the FogBugz and Kiln On Demand deployment and maintenances (mentioned here and here, respectively).
Both maintenances were completed successfully. We detected minor difficulties during the Kiln maintenance, but no customer impact should have existed and the trouble lasted less than 2 minutes.
Posted by Bradford Ley at 11:20 PM | Permalink
We will be performing maintenance on the backend systems that support Kiln On Demand on Sunday, April 29th at 2:00AM UTC (Saturday, April 28th at 10:00PM EDT).
This maintenance is expected to last approximately two hours and there should be no interruption in service for customers.
Posted by Bradford Ley at 11:17 AM | Permalink
We will be performing an upgrade of all FogBugz On Demand accounts on Sunday, April 29th at 2:00AM UTC (Saturday, April 28th at 10:00PM EDT).
There is no database upgrade required, but we will be proceeding cautiously, so the entire process could take as long as two (2) hours with intermittent downtime for accounts during those hours. We don't expect the total downtime for any account to exceed 30 minutes.
The following are the list of changes we will be making:
FogBugz 8.8.17 Release Notes
Posted by Bradford Ley at 12:44 PM | Permalink
Our datacenter maintenance has been completed and was a success. However, some elements could not be completed in the time alloted, so have been postponed. We'll make a separate post at a later date announcing the new window for those changes.
Posted by Bradford Ley at 12:34 AM | Permalink
The email and FTP outage noted earlier has been resolved. The sever that provides these services crashed with what appears to be a RAID firmware bug. It was out of commission for approximately 15 minutes. No data was lost, though incoming emails may be delayed for a few more minutes as remote servers retry sending.
We'll schedule a maintenance in the near future to implement a permanent fix for this issue.
Posted by Shawn Hargan at 11:25 AM | Permalink