Partial Copilot Outage

Wednesday, January 23, at approximately 4:00pm EST Copilot users began having trouble connecting to the Fog Creek servers.

1. Why? One Reflector stopped responding to user requests, which went unnoticed by Fog Creek staff for longer than it should have.
2. Why? There was no automated notification until we heard about it from customers.
3. Why? The Reflectors were down and therefore not reporting bugs, and so it was impossible to be notified.
4. Why? Because there is no external monitoring system for the Reflectors.
5. Why? All prior bugs have been reported from within the Reflector by BugzScout, and nothing was set up to monitor it externally.

So how are we going to fix this? Fortunately, the solution is pretty simple. We are going to create a monitor that will regularly log in to Copilot from both sides and ensure that data is flowing between the clients.  If this process fails for any reason, we will be notified of it and can take action before it affects users.