Fog Creek System Status

Kiln On Demand Upgrades

Over the next few weeks, we'll be moving all Kiln On Demand accounts to a brand-new high-performance infrastructure (just like our Student and Startup accounts). This upgrade will require about an hour of downtime during which Kiln On Demand will be unavailable.

To reduce the impact of this downtime, these upgrades are scheduled for Sunday, May 6th, 13th, and 20th, 2012. Each account will be upgraded between 9am and 6pm EDT (1300-2200 UTC) on one of those days.

If these maintenance periods would be distrupive to you, please contact us at customer-service@fogcreek.com -- we're happy to reschedule your upgrade for a more convenient time.

Thanks in advance for your patience as we make Kiln better!

FogBugz and Kiln On Demand Maintenance completed successfully

We have just completed both the FogBugz and Kiln On Demand deployment and maintenances (mentioned here and here, respectively).

Both maintenances were completed successfully. We detected minor difficulties during the Kiln maintenance, but no customer impact should have existed and the trouble lasted less than 2 minutes.

Kiln On Demand Maintenance: Sunday, April 29 0200 UTC

We will be performing maintenance on the backend systems that support Kiln On Demand on Sunday, April 29th at 2:00AM UTC (Saturday, April 28th at 10:00PM EDT).

This maintenance is expected to last approximately two hours and there should be no interruption in service for customers.

FogBugz On Demand Upgrade to 8.8.17: Sunday April 29, 0200 UTC

We will be performing an upgrade of all FogBugz On Demand accounts on Sunday, April 29th at 2:00AM UTC (Saturday, April 28th at 10:00PM EDT).

There is no database upgrade required, but we will be proceeding cautiously, so the entire process could take as long as two (2) hours with intermittent downtime for accounts during those hours. We don't expect the total downtime for any account to exceed 30 minutes.

The following are the list of changes we will be making:

FogBugz 8.8.17 Release Notes

• Fix an exception that prevents case creation when a plugin hooks to the AreaJoin or FixForJoin interfaces
• Dropdown fields behave better with values like "--"
• Performance fix for Mono and case linking
• Fix for admins on accounts with many thousands of users
• Added an admin notification for users that have unviewable cases
• Fix a time bug related to intervals over daylight savings time
• Typo fixes
• Plugin updates:
  • FogBugz Timesheet (v1.0.26.0)
  • Relative Time (v1.1.0)

Datacenter Maintenance: Completed (with caveats)

Our datacenter maintenance has been completed and was a success. However, some elements could not be completed in the time alloted, so have been postponed. We'll make a separate post at a later date announcing the new window for those changes.

Issue resolved; All services up

The email and FTP outage noted earlier has been resolved. The sever that provides these services crashed with what appears to be a RAID firmware bug. It was out of commission for approximately 15 minutes. No data was lost, though incoming emails may be delayed for a few more minutes as remote servers retry sending.

We'll schedule a maintenance in the near future to implement a permanent fix for this issue.

Email and FTP outage

We're currently experiencing an outage covering inbound and outbound email for FogBugz On Demand as well as emails coming to our support department.

Outgoing mail from FogBugz On Demand will be queued in the account until the mail server is once again available. Incoming mail ought be queued by the sending mail server.

Additionally, this outage covers our FTP services, so anyone attempting to download their On Demand database or support files will not be able to do so until this is resolved.

Our engineers are working on a resolution presently. We will post again once services are restored. Sorry for any trouble this causes.

Datacenter Maintenance: 3:00AM UTC, April 21, 2012

We will be performing maintenance on our datacenter switch fabric. Due to the nature of the maintenance, customers can expect downtime between 3:00AM and 4:00AM UTC. The downtime will be done in a rolling fashion, so each customer should expect no more than 30 minutes of maximum downtime. This 30 minutes will be for any services, including Trello, FogBugz and Kiln On Demand, Copilot, and the Fog Creek websites. Every effort will be made to keep downtime to a bare minimum.

The purpose of the maintenance is to prepare for another, similar maintenance in the coming weeks (we will make a separate announcement). Separating the two results in a lower total downtime.

FogBugz On Demand Email Processing Returns to Normal

Today's email processing delay has been corrected.

Delay in Processing of Some Email Messages in FogBugz On Demand

Since 5:45 PM EST today, FogBugz On Demand accounts have not been processing incoming email messages directed to cases@{your FogBugz site}.fogbugz.com. All incoming mail is still present, but has not yet been processed by FogBugz. We are actively working on a solution to this problem and will post to status.fogcreek.com when it is resolved.

Outgoing mail and additional custom mailboxes, e.g. bugs@example.com, are not affected by this issue.