Fog Creek Network Status

FogBugz On Demand One Hour Maintenance - 8/2/2009 at 04:00 - 05:00 GMT

In an on going effort to get the most out of our FogBugz 7 systems, we will spend an hour this Sunday performing several key configuration changes to our back end servers.  During this hour, you will see a maintenance page while trying to visit your FogBugz On Demand site.  

We will post another update upon completion.

Your patience is much appreciated!

All Systems Are Up; FogBugz On Demand Maintenance is Complete

FogBugz On Demand was down for maintenance tonight from approximately 12:00 AM until 12:30 AM.  After that, a number of our customers were moved to other database servers in order to decrease load and improve the user experience.

FogBugz On Demand Maintenance for this Sunday, 7/26/2009 at 04:00 GMT

At 04:00 GMT on 07/26/2009, we will perform maintenance on FogBugz On Demand, which will complete by 08:00 GMT.  Our goal is to give you all a few bug fixes and a lot more performance.

The plan is as follows:

1. add an additional 32GB of RAM to a number of our database servers
2. upgrade all accounts to the latest FogBugz 7 generation
3. move a percentage of our On Demand customers to new database servers

After the first step has happened (ending by 01:00 GMT), your account will be available with minimal interruption.  If your account is slated to be moved, you may see brief outage screen for a few minutes, and then service will be restored. 

Please contact us with any questions, and know that we appreciate your patience.

FogBugz 7 On Demand Performance Issues and Mitigation

With FogBugz 7 being officially live in the On Demand environment, we've noticed that our database servers are under higher load than before.  Because of this, we have gotten a few emails from customers stating that their FogBugz On Demand site seems slower.

This coming weekend, we will be migrating a number of accounts to help spread the load across several new database servers (more on that in a future post).  However, if you are experiencing performance problems now, please send us an email so that we can work with you to move your account ASAP.

Ongoing FogBugz On Demand Upgrades

We will be upgrading FogBugz On Demand accounts on a rolling basis over the next few days. If your account becomes temporarily inaccessible during the upgrade process, it will remain so for only about 15 minutes while the server addresses refresh.  If your downtime lasts more than 30 minutes, please call us.

All Services Up; Earlier FogBugz On Demand 7 Beta Users Outage

As noted a few moments ago, our FogBugz On Demand Beta customers experienced an outage.  It began around 4:10 PM (EST), and ended at 4:25 PM (EST). 

The exact cause is unknown and is being investigated, however we believe it to be related to some recent changes in the NIC teaming configuration.  All changes have been reverted to the known good config, service has been restored and verified, and we do not expect this issue to appear again.

UPDATE: FogBugz On Demand 7 Beta Users - FIXED

UPDATE: Issue now fixed.

Started at 4:10PM EST.  We're looking into it.