Fog Creek Network Status

re: On Demand Slowness

Regarding this previous incident, we have determined that the slowdown was caused by a customer sending nearly 60,000 BugzScout events to their FogBugz On Demand account.  A BugzScout submission is an expensive operation, and this large batch was causing other customer accounts to respond slowly.

We're going to begin attacking this at the network level (layer 3/4) by slowly calibrating our firewall rate-limiting settings, as well as seeing what we can do at the application (layer 7) level.  As I type this, we're exploring ways to implement some amount of layer-7 accounting that would enable us to spot a dangerous spike in expensive requests and to automatically put an end to them.

On Demand Slowness for Some Customers

A few people have called in noting that their On Demand accounts are performing slowly.  I have been able to pin all of them to one data center and am investiging.  I will post with an update once I have more data.

Service Interruption for Some On Demand Customers

Earlier today some of our On Demand customers experienced a series of errors after logging into their FogBugz On Demand account.  

While the root cause is still unknown (the web server wasn't showing us any reason why it was malfunctioning), we do recognize that we can catch this error before it becomes wide spread.  We will expand our monitoring system this week to dig further into On Demand and look for errors when performing standard operations (opening a case, looking at filters, etc).