Regarding this previous incident, we have determined that the slowdown was caused by a customer sending nearly 60,000 BugzScout events to their FogBugz On Demand account. A BugzScout submission is an expensive operation, and this large batch was causing other customer accounts to respond slowly.
We're going to begin attacking this at the network level (layer 3/4) by slowly calibrating our firewall rate-limiting settings, as well as seeing what we can do at the application (layer 7) level. As I type this, we're exploring ways to implement some amount of layer-7 accounting that would enable us to spot a dangerous spike in expensive requests and to automatically put an end to them.
