A few of our FogBugz On Demand users may have received an HTTP 500 response when they hit their accounts. The outage began at approx 5:15 PM EST and was mitigated by 5:20 PM EST. We bumped into an unexpected memory limitation on one of our application servers. I have a short-to-mid-term work around in place until we are able to replace all of those boxes with 64-bit systems.
The ultimate lesson to all of you who desire to go into the application hosting business: run 64-bit on *everything*.
We're currently experiencing issues related to search on FogBugz On Demand that may be generating errors for some customers. We're investigating...
[UPDATE] A server that should not have been in our load balancer, accidentally got pushed into the pool. So anyone that was getting that server as their web server was seeing errors. We've taken that server out of the pool now and are doing more investigation, but things are stable again.
Just a note to everyone: our NYC colo will be performing a power generator test @ 7:00 AM EST on Wednesday, January 20th, 2008. This is a routine test that ensures that the power system is functioning properly, and we are not expecting any trouble.