Fog Creek Network Status

On Demand Outage Analysis

We have discovered the cause of the previous outage.

Our LA router was ultimately overwhelmed due to an excessive amount of syslog data (we're talking thousands of messages per second) being sent over an IPSec tunnel. IPSec is quite CPU intensive, and this large amount of traffic pushed things over the edge.

To remedy this, rate limiting will be put in place for syslog data. If a device ever generates a large amount of logging info again, we'll be able to block it before it ever hits the encrypted tunnel.

On Demand Outage

We are currently experiencing an outage with our LA data center, meaning that some of our customers will be experiencing connectivity issues. I am working with a tech right now, and will update as soon as we have more info.

Fog Creek Phone Outage for Friday, 9/12/2008

We are in the midst of a move to a brand new office, which means that phone service may be interrupted to our technical and sales support teams. All service will be fully restored by Monday morning.

If you are having issues or have any questions during this outage, please contact us via email.