Fog Creek System Status

Kiln On Demand Upgrade Successful

Last night's Kiln On Demand Upgrade to 2.9.19 was successful and resulted in zero downtime for any customers.

If your account was not upgraded, don't fret: We will be continuing the upgrade of customers in the very near future. Further announcements to come.

FogBugz On Demand Upgrade and Infrastructure Maintenance: July 1, 0200 UTC (June 30, 2200 EDT)

This weekend we will be upgrading all FogBugz On Demand accounts to version 8.8.36. Additionally, we will be performing some routine server maintenance in our datacenter. The FogBugz upgrade will begin promptly at 0200 UTC on July 1 (2200 EDT on June 30). Some customers may experience 5-10 minutes of downtime during the upgrade process, but all efforts are being made to keep this a zero-downtime affair.

Once the upgrade is completed, infrastructure maintenance will begin and last approximately 20 minutes. Customers may notice slower response times for certain requests.

FogBugz 8.8.36 Release Notes:

• When replying, use the received subject for the reply's subject
• Update password hashes for old accounts
• German translation fix
• Fix error when plugins called UrlAsFormFields
• Fix errors when plugins stored attachments
• Properly link text of the form "subcase #" in event descriptions
• Performance tweaks for FBOD infrastructure

Plugin Updates:

• Original Opener v 1.0.5
  • Format original opener properly when exporting to CSV
• Intro Case v 1.3.0
  • Don't create intro cases that users can't see

Kiln On Demand Upgrade: June 28th, 0230 UTC ( June 27th, 2230 EDT)

We have addressed all remaining concerns (again!) and are prepared to upgrade a small subset of Kiln On Demand accounts to Kiln 2.9. There is no expected service interruption for this upgrade. Previous concerns about infrastructure items have been addressed to everyone's satisfaction.

The release notes remain the same as the last notice:

• Revamped code review system.
• Totally rebuilt, faster search infrastructure using ElasticSearch.
• Faster and more reliable server-side branching and pushing.
• Push and pull a single head on a multi-headed repository in Kiln.
• Support for Mercurial's largefiles extension, the successor of kbfiles.
• Major performance improvements, on the server and in your browser.
• Countless bug fixes and stability improvements, such as:
  • Fix spurious "your session has expired" errors.
  • Kiln is now powered by Mercurial 2.2.2, the latest stable release. Our thanks as always to the entire Mercurial community.
• This version also includes a backwards-incompatible, breaking change to the Reviews API.

We will begin at approximately 0230 UTC (2230 EDT).

June 25th/26th Maintenance Cancelled

We've aborted the maintenance announced earlier today.

We began to see some evidence in the infrastructure that an upgrade would be ill-advised without further investigation. It has no impact on current service, but we didn't want to pursue an upgrade until it's resolved. Thus, we've opted to cancel the maintenance for tonight.

As a side note, the astute among you may have noticed we have a lot of schedule changes in the last month or so. The reason is not due to new problems in the infrastructure and software. We are trying to approach deploys andinfrastructure changes with a greater measure of caution and transparency. However, this many schedule changes may cause some trouble or confusion for some customers. If we're causing you undue distress, please let us know by contacting us!

Kiln On Demand Upgrade: June 26th, 0200 UTC ( June 25th, 2200 EDT)

We have addressed all remaining concerns and are prepared to upgrade a small subset of Kiln On Demand accounts to Kiln 2.9. There is no expected service interruption for this upgrade. The release notes remain the same as the last notice:

• Revamped code review system.
• Totally rebuilt, faster search infrastructure using ElasticSearch.
• Faster and more reliable server-side branching and pushing.
• Push and pull a single head on a multi-headed repository in Kiln.
• Support for Mercurial's largefiles extension, the successor of kbfiles.
• Major performance improvements, on the server and in your browser.
• Countless bug fixes and stability improvements, such as:
  • Fix spurious "your session has expired" errors.
  • Kiln is now powered by Mercurial 2.2.2, the latest stable release. Our thanks as always to the entire Mercurial community.
• This version also includes a backwards-incompatible, breaking change to the Reviews API.

We will begin promptly at 0200 UTC (2200 EDT).

June 24th Maintenance Complete

We've completed our scheduled maintenance with no incident, but some portions were deferred due to stability concerns.

• We have deferred the Kiln On Demand upgrade due to some issues uncovered in staging. We will post another announcement when that upgrade will be completed.
• We have deferred some portions of our infrastruture maintenance to coordinate with a larger maintenance effort to keep downtime to a minimum. Again, another announcement will be made when we have a final schedule.

All customers and services are now online and functioning as expected.

Kiln on Demand Upgrade and Infrastructure Maintenance Sunday 2012-06-24 02:00 UTC

We will be performing an account upgrade to Kiln 2.9 on Sunday June 24th at 02:00 UTC (Saturday June 23rd 22:00 EDT) The complete process will take approximately one hour and individual customers can expect downtime lasting between 5 and 15 minutes any time during that hour.

Once the account upgrade is complete we will be performing preventative hardware maintenance on our infrastructure. The complete process will take approximately two hours and certain customers may experience downtime lasting between 15 and 30 minutes anytime during those two hours.

• Revamped code review system.
• Totally rebuilt, faster search infrastructure using elasticsearch.
• Faster and more reliable server-side branching and pushing.
• Push and pull a single head on a multi-headed repository in Kiln.
• Support for Mercurial's largefiles extension, the successor of kbfiles.
• Major performance improvements, on the server and in your browser.
• Countless bug fixes and stability improvements, such as:
  • Fix spurious "your session has expired" errors.
  • Kiln is now powered by Mercurial 2.2.2, the latest stable release. Our thanks as always to the entire Mercurial community.
• This version also includes a backwards-incompatible, breaking change to the Reviews API.

 

 

Issue Resolved; All Services Up

This network event was resolved at approximately 14:30 EDT. 

The problem lasted from about 14:00 EDT to 14:30 EDT. During that time, you may have seen intermittent timeouts when trying to reach Fog Creek's hosted services. The essential problem was a traffic flood. It took longer than usual to diagnose and correct because the flood was so large; it overwhelmed our monitors and gateways both (from the fragmentary data I collected, it appears to have been at least 40x our usual baseline).

Short-term, I've blocked the offending IP in our firewalls. Long-term, we'll use the collected traffic samples to devise a rate-limiting strategy that can block this kind of traffic on the fly.

I apologize for the interruption. At this point, all monitors are green and all services are up.

Network Impacting Event

We are seeing a tremendous spike in traffic that is causing intermittent failure in reaching our network. I am investigating the issue and will post additional information as it becomes available.

Maintenance Complete

Maintenance tonight has been completed successfully. All services are up and gauges are in the green. Enjoy the weekend.