This outage has been resolved. What appears to be a faulty network conneciton to one of our gateways began flapping this morning at about 09:37 EDT. Our secondary gateway took over but, for reasons we're still investigating, allowed the primary to reclaim the shared IP addresses as soon as its network connection was back up. At 09:48 EDT, we fixed this issue by forcing the primary gateway into a passive state and routing all traffic through our secondary.
Customers would have experienced slowness for a short while longer while all of our web servers recycled their SQL connection pools to handle the new routes.
The permanent fix for this will come in two stages. Both of these are already planned and the first already scheduled:
1 - We're inserting a pair of redundant switches between our gateways and our ISP. This will prevent a disruptive gateway failover during something as simple as a network link flap which should make these types of issues largely transparent to our users.
2 - We're replacing our gateways with a pair that handles failover much more seamlessly.
We're sorry for the interruption in service. If your'e still experiencing issues, please contact Support.
