Fog Creek System Status

OnDemand email and search-index delays

Due to a hung maintenance service, OnDemand emails and search-indexes were not processed between approximately 1:20PM and 3:30PM (EDT). No data was lost and everything should be flowing normally now. We've also added a new check to our monitoring system so we'll see this sooner if it happens again.

Copilot Maintenance this Sunday, October 26 @ 04:00 GMT - 04:59 GMT

In order to keep things running as smooth as possible, we will be patching the backend servers that service Copilot.  During this time you can expect brief periods of service interuption.

Scheduled Maintenance For Select On Demand Customers

Scheduled maintenance for some On Demand customers begins at midnight. Those that will be impacted were sent emails earlier this week.

For those that are interested, we are upgrading several components in our New York data center that will make our On Demand service even more reliable. Some of these upgrades include:

* redundant routers
* redundant connections to our ISP
* redundant network connections for every On Demand server

This maintenance is scheduled for 04:01 - 09:00 GMT.

re: NY Data Center Outage

update: we have been informed that the root cause was a failed UPS in the facility. They have re-routed power to a different unit to do further diagnostics.

Our ISP has corrected the issue (still waiting to get a root cause analysis) and services are now back online.

NY Data Center Outage

Our ISP appears to be having issues right now. I have them on the phone and will update this ASAP.

To be clear, this means that some of our On Demand customers are unable to reach their accounts and all Copilot services are down.