Starting very early this morning, a number of users began running into issues when trying to access either FogBugz On Demand's API or BugzScout. This problem was resolved upon discovery at 6 AM.
FogBugz On Demand has a pool of servers dedicated to only handling BugzScout and API requests, which helps keep the load down on the servers that are servining up the UI to our customers. Upon further research, it was discovered that one of our API / BugzScout servers had a configuration mismatch that was causing the reported redirect loop.
I see two failures here. The first is obviously the configuration mismatch, which we have corrected by manually adjusting the configuration values. This bug was introduced in the original deployment / configuration of that specific server, and our deployment scripts and procedures have been updated to include the proper steps so that it doesn't happen again.
The second failure was one of monitoring. Normally, our monitoring system has the capability to discover services throughout our network and sets up monitoring automatically. Howerver, there are a few services that require some manual configuration in order to be monitored correctly. I have updated our documentation to ensure that this doesn't get missed in the future, and we are also working on a visualization system that will help us audit and detect imbalances like this.
