Fog Creek Network Status

Kiln On Demand - Non-Admin Users Can Access Kiln Again, with Side Effects

Ealier today, we experienced an issue with Kiln On Demand where non-admin users couldn't access Kiln via the website or from the mercurial client. We've applied a patch to all Kiln On Demand accounts that addresses this issue.

A side effect of this bug caused user permissions related to repositories and Kiln projects to be in an unusable state. This does not affect site admins. We couldn't tell if a user simply had write access to a project or repository or if they had per-project or per-repository admin access. Because we couldn't tell the difference, we defaulted back to write-only access in these cases.

What this means for you: If you previously were a per-project or per-repository admin, your permissions on these projects and repositories have now been set to write only. You'll need to have a site administrator restore you as an admin on those particular projects and repositories. Users that are site admins are not affected by this issue.

We'll be following up on the status blog with more details later today to let you know exactly what happened.

Kiln On Demand - Non-Admin Users Can't Access Kiln

This is a separate issue from the intermittent outages experienced this morning. Several of our customers are reporing that non-admin users in Kiln are seeing "Kiln has overheated" error messages on their Kiln On Demand websites as well as 500 errors from the Kiln client. As a workaround, you can temporarily make the users who can't access Kiln site admins and then have them try to log in again.

We're actively investigating this issue and will post an update once it has been resolved.

 

Outage resolved; All services up

This outage has been resolved. What appears to be a faulty network conneciton to one of our gateways began flapping this morning at about 09:37 EDT. Our secondary gateway took over but, for reasons we're still investigating, allowed the primary to reclaim the shared IP addresses as soon as its network connection was back up. At 09:48 EDT, we fixed this issue by forcing the primary gateway into a passive state and routing all traffic through our secondary.

Customers would have experienced slowness for a short while longer while all of our web servers recycled their SQL connection pools to handle the new routes.

The permanent fix for this will come in two stages. Both of these are already planned and the first already scheduled:

1 - We're inserting a pair of redundant switches between our gateways and our ISP. This will prevent a disruptive gateway failover during something as simple as a network link flap which should make these types of issues largely transparent to our users.

2 - We're replacing our gateways with a pair that handles failover much more seamlessly. 

We're sorry for the interruption in service. If your'e still experiencing issues, please contact Support.

Intermittent Outages With FogBugz and Kiln On Demand

We're currently experiencing intermittent outages with FogBugz and Kiln On Demand. Symptoms may include your account is loading but very slowly. We're actively investigating this issue and will post more details as soon as we have them.

FogBugz and Kiln On Demand Upgrade Tonight

At 02:00 GMT we will upgrade FogBugz to 8.5.114 and Kiln to 2.5.159. 

Along with the usual assortment of minor bug fixes and performance enhancements, changes include:

FogBugz

• User accounts now have profile pictures, exposed when hovering over a user's name
• Plugins can now determine the status of a hosted account for both FogBugz and Kiln (free trial, enabled)
• Bug fixed where VS Addin occasionally crashed Visual Studio
• More robust screenshot tool uploads
• Faster attachment uploads
• Allow aliases for mailboxes that are longer than 256 characters

Kiln

• Both Kiln and its client tools have been fully updated to Mercurial 1.9, speeding up push times and providing many new features
• The Kiln BFiles plugin has received many improvements and bug fixes
• Code reviews show users' profile pictures, making it easier to see who said what
• Changesets now show visual diffs when images were changed
• The file browser allows showing a preview of image data
• A new feature, Access Tokens, allows continuous integration systems to work with Kiln without using up a Kiln or FogBugz account
• A new permission level, "Read + Branch", allows developers the ability to read central repositories and create branch repositories, but not to push to central repositories
• RSS feeds can now show diffs inline
• RSS feeds can now show pushes rather than individual changesets
• Code searches can now be limited to a project or a given file name
• A bug with Kiln caching that could result in a sticky notifications icon for administrators, or in users failing to show up in permissions tables, has been fixed

This release includes a database upgrade. For most accounts, the upgrade process will be too quick to notice. If you receive an "Account upgrading" message, it should disappear in just a minute or two.

Maintenance Retrospective

Sunday's On Demand maintenance took longer than expected because Microsoft SQL Server's shut-down and startup processes quiesce and scan each database for corruption. The time necessary for this check increases along with the number of databases on the system. In one extreme case, the server took two and and half hours to shut down properly.

Applying the hotfix in question required a stop and start of SQL Server itself, then a reboot. This timing didn't work out with the original window, so were only able to install the hotfix on two of the seven servers that needed it. We're researching ways to reduce the time it takes to restart SQL Server on these heavily populated servers and have already found some promising leads. We'll be rescheduling the remaining five servers soon, most likely coinciding with an upcoming power maintenance.

The good news is that we resolved a number of other issues with this maintenance. Most were internal, but one exciting networking change is the completion of our double-ring switching architecture. This means our network can survive the loss of any single cabinet with minimal impact to the others. 

Thanks for your patience as we make our On Demand services better. Look for more in the near future!

Maintenance Complete

Tonight's maintenance has been completed and FogBug and Kiln On Demand are accessible again. We'll post more details on Monday, after we've had some sleep.

Maintenance Continues

We are still working on tonight's maintenance.  Two of our SQL Servers are taking longer than the others to come back online.  We are aiming to have all systems back online 08:30 GMT.

Please stay tuned here for more updates.

Maintenance has begun

Tonight's maintenance has begun. All FogBugz and Kiln On Demand requests are redirecting to this blog. We will post updates as they become available. 

FogBugz and Kiln On Demand maintenance Sunday, 2011-07-17, at 04:00 GMT

The root cause for today's SQL server outage turns out to be a bug in MSSQL's handling of tempdb. There's a hotfix available for it. This Sunday, we will be taking down all On Demand SQL servers to apply the hotfix. Since we'll have them offline anyway, we'll also use this opportunity to apply some overdue Windows updates.

The maintenance window is slated for Sunday, July 17th, between 04:00 and 07:00 GMT. All FogBugz and Kiln On Demand services will be unavailable during these updates. This is a long outage and we hope to have things back up in much less time than allocated, but we'd prefer to play it safe with our scheduling.