Fog Creek Network Status

Brief Copilot Maintenance on Saturday, 11/22/2008 @ 22:00 GMT

Hi!

This weekend I'll have to take the Copilot database offline for approximately 20 minutes while I apply a few patches to the database server.  We are in the process of scaling out our backend to increase availability, and this maintenance will get us much closer.

We appreciate your patience in the matter, and thank you for your patronage.  

re: Copilot / On Demand Slowness

The routing issue has been resolved and all services are functioning properly now. 

Copilot / On Demand Slowness

Due to issues with an upstream router, some of our Copilot and On Demand customers are experiencing performance issues.

Our ISP is aware of the issue and is working to resolve ASAP.

re: Copilot Maintenance Tonight

The maintenance has been completed and you will not experience any more service interruptions this morning.  I estimated 1.5 hours of total work, and it was completed in just under 30 minutes.  

As always, we appreciate your patience.

Copilot Maintenance Tonight

We will be performing maintenance on the Copilot service tonight / tomorrow morning at 1 AM Eastern.  During this time, Copilot customers may experience intermittent losses of connectivity.  We anticipate this outage to end by 2:30 AM.

While this outage is regrettable, we are performing this to make way for a more reliable version of Copilot.

Intermittent Out-of-Memory Errors for FogBugz On Demand (cont'd)

We have rolled back the memory quotas we introduced yesterday after finding that they were causing premature session timeouts, and we're deeply sorry for the inconvenience. We're still working on a fix to prevent the out-of-memory errors some of you saw yesterday, and will post as soon as we have more info.

Copilot Reflector Outage

update: this issue has been resolved.  

We are currently experiencing a reflector outage due to some locking issues on our databse server.  We are working to resolve, and expect things to be back to normal momentarly.

Intermittent Out-of-Memory Errors for FogBugz On Demand

We had a few reports that some of our On Demand customers experienced Out-of-Memory errors earlier this morning.  Our monitoring system caught it, and we immediately removed the problamatic server from the pool.

The offending process has been identified, and a fix is in the works.  In the meantime, quotas have been put in place to prevent the process from using up this amount of memory agian.

OnDemand email and search-index delays (cont'd)

RIght now it looks like this was due to some difficulty in handling a particular time zone (unrelated to the DST changeover). Developers have rolled out a patch for this bug, and we'll be watching this over night for any recurrence.

OnDemand email and search-index delays (cont'd)

This weekend, the New York instance of our FogBugz Maintenance Service began periodically spinning out. OnDemand customers based in that Data Center may see delays in search-indexing and email processing but, as before, no data is lost. We're working on tracking down the root cause and will post again when we've got more information.