The power source changeover was completed without incident as we were able to fully mitigate the effects of the unhealthy UPS.
No services were impacted and all services are currently online.
« October 2012 | Main | December 2012 »
The power source changeover was completed without incident as we were able to fully mitigate the effects of the unhealthy UPS.
No services were impacted and all services are currently online.
Posted by Sven Henderson at 06:09 PM | Permalink
We have received confirmation that the power source cutover will occur today at 4:45 PM EDT (UTC -500). In preparation we have started switching to our alternative power source today at 4:15 PM EDT (UTC -500).
We have an alternative means of power and do not expect any downtime. The FogBugz and Kiln On Demand service may run slower and without redundancy for approximately 20 minutes.
Stay tuned for updates on Twitter @FogCreekStatus and here fogcreekstatus.com.
Posted by Derrick Miller at 04:19 PM | Permalink
As an update to our post for the On Demand Power Issues Advisory, we have contingency plans in place to maintain service availability. If those plans fail, we may have a brief service interruption as we implement the alternative power source.
The UPS technician will be arriving to the data center momentarily which will tell us more about the plans we need to execute.
Stay tuned on Twitter @FogCreekStatus and here fogcreekstatus.com for updates.
Posted by Derrick Miller at 03:31 PM | Permalink
We've been informed by our data center about some power issues to our infrastructure. We expect that our On Demand services for FogBugz and Kiln will remain running without interruption.
The electrical company, ConEd, needs to switch our power source from temporary to a permanent power source. During this switchover the Uninterruptable Power Supplies (UPS) will provide power.
However, the UPS powering one of our cabinets is currently unable to provide power during this power source changeover.
We are preparing multiple contingency plans to ensure that there will not be a service interruption. A UPS expert is en route to the data center to address the issues with the problematic UPS. Alternative means of power are also being prepared as a contingency to possible failure of this particular UPS.
We will post an update here at 3:30 PM EST (UTC -500) if there is going to be a service interruption.
As soon as the power source change is complete, we will post an update to @FogCreekStatus and here fogcreekstatus.com.
Posted by Derrick Miller at 02:25 PM | Permalink
UPDATE: As of 3:05am EST all services should be back online. More information on the outage will be provided following an investigation.
At approximately 2:35am EST one of our backend Kiln Mercurial servers became unresponsive, we are working on bringing it back up now.
Posted by Sven Henderson at 03:02 AM | Permalink
We will be performing additional migrations of our On Demand customers this weekend.
The maintenance window will open at 0300 UTC, Dec. 2nd (2200 EST, Dec. 1st) and will last three hours until 0600 UTC (0100 EST, Dec. 2nd). A small subset of our customers will experience service interruptions at that time. Individual customers should expect an interruption lasting between 5 and 30 minutes (depending on the size of the database).
We will also being doing maintenance on our customer account database. This will cause a service interruption for all customers lasting approximately 30 minutes, as well as prevent new accounts from being created during that time.
Posted by Bradford Ley at 04:21 PM | Permalink
On November 1st we posted that the data center situation has stabilized. Our next update was to let you know that we’re back on the electrical grid.
We're back on the electrical grid and have been for nearly a week. The power company has resumed power to the data center building on November 20th.
The power situation is back to normal, i.e. pre-Hurricane Sandy.
Posted by Derrick Miller at 10:51 AM | Permalink
The maintenance we announced earlier this week was completed without serious incident. Some customers may have noticed a brief period of degraded performance, but the impact was minor.
Additionally, the routine software maintenance for Kiln was completed smoothly.
Posted by Bradford Ley at 12:59 AM | Permalink
With our datacenter stabilized, we're going to continue our previously planned maintenances.
We will be migrating a subset of our customers to new hardware this weekend. The maintenance window opens promptly at 0300 UTC (2200 EST) and will continue until 0600 UTC (0100 EST). Service interruptions for individual accounts will be between 5 and 30 minutes (depending on the size of the account) and may occur at any time during the window. During the service interruption, both FogBugz and Kiln will be inaccessible. If you need to know if you will be part of the migration, please don't hestiate to contact us.
We will also be performing some routine software maintenance on our Kiln servers at the same time. This will cause approximately 5 minutes of interruption for all Kiln customers, and may occur at any time during the maintenance window.
Posted by Bradford Ley at 11:51 AM | Permalink
We've posted a Hurricane Sandy Wrap Up on the Fog Creek Software blog.
As always we don’t want your money if you’re not amazingly happy. If you were materially affected by this downtime, please email us at customer-service@fogcreek.com and we will make it right.
Posted by Mendy Berkowitz at 03:26 PM | Permalink
