« June 2012 | Main | August 2012 »
As of Tuesday, July 31st at approximately 1:30 PM ET (17:30 UTC), the Kiln On Demand file and code indexing service has been disabled to prevent index corruption due to competing network requests for the Kiln On Demand service.
Kiln On Demand remains available but without fully functional search for recently added files. Previously indexed files will remain searchable.
An update will be provided when we have more information.
Posted by Derrick Miller at 04:19 PM | Permalink
At around 6AM EDT (1000 UTC), we noticed our connection counts piling up in Kiln On Demand. Ultimately, to protect the service as a whole, we were forced to restart the load-balancer and SSH terminators. Some customers may have noticed their in-progress connections were aborted. Things appeared calm, but before we could research a full explanation, it occurred again at about 1030 EDT (1430 UTC) and thirty minutes later we restarted the load-balancers, again.
The service is currently in good shape. The running theory involves the code indexing processes, but we're still investigating. Things are looking good (again), but we're vigilantly searching for the complete answer and solution.
Sorry for any trouble we've caused today! We'll post again with a post-mortem when available.
Posted by Bradford Ley at 11:26 AM | Permalink
Today we experienced a DNS issue that prevented access to:
The DNS provider for these sites and services, Zerigo, suffered a severe DOS attack. Our monitoring first detected an intermittent latency problem at 11:30 PM last night. At 2:00 AM today it became consistent. Troubleshooting determined that it was caused by slow lookups to our DNS provider. The nature of the outage was such that we could not log in to retrieve our DNS records and move them until 9:30 AM. At that time, we moved our critical domains to Route 53 hosting. The switch over was completed by 10:25 AM, but users may continue to see residual issues until the TTL of the old NS records expire. This should be no later than 10:00 AM tomorrow.
If you used the local hosts file work-around posted in one of the updates above, you will need to undo it to start using the new Route 53 DNS. If you have any questions, please email us at customer-service@fogcreek.com
Following this issue, our sysadmin team will be reviewing our DNS architecture to address this single point of failure. In addition, we will update our out-of-hours incident procedures to make sure that status updates are posted in a more timely fashion. Finally, this status blog, while hosted outside of our network, used a domain that was affected by the issue. Going forward, you can access it at fogcreekstatus.com.
If you were materially affected by this downtime, please email us at customer-service@fogcreek.com and we'll make it right.
all times in US EDT (UTC-04:00)
Posted by Adam Wishneusky at 03:57 PM | Permalink
As of Friday afternoon (7/20/2012), customers attempting to log into their accounts at copilot.com began receiving an error message, "Uh oh. An error occurred." All existing invitation codes and new trial subscriptions continued to work during this time.
Because existing invitations and new unpaid subscriptions worked throughout the weekend, our monitoring system did not catch the error until this morning. Investigation found that part of the infrastructure shared by Fog Creek's services was corrupted by the process of rolling out a new Fog Creek product. We have taken steps to prevent this particular failure in the future.
If you were materially affected by this downtime, please email us at customer-service@fogcreek.com and we'll make it right.
Posted by Adam Wishneusky at 03:28 PM | Permalink
An update to the DNS problem affecting our services:
We have moved DNS for our main sites over to another provider, but we must wait for the TTL to expire on the old NS record which pointed to Zerigo. This may take up to a day. In the meantime, access to Copilot, Trello, FogBugz and Kiln On Demand is slowly resuming, but if you're still having trouble please try the workaround we posted previously, otherwise email us at customer-service@fogcreek.com.
P.S. We're sorry for the outage, thank you for your patience during this time.
Posted by Derrick Miller at 02:54 PM | Permalink
An update to the DNS problem affecting our services:
We have moved DNS for our main sites over to another provider, but we must wait for the TTL to expire on the old NS record which pointed to zerigo. This may take up to a day. In the meantime, please try the workaround we posted previously.
Posted by Adam Wishneusky at 11:39 AM | Permalink
To work around the the DNS issue we posted, edit your local hosts file to enter our server IP addresses.
On Windows, you can find the file by going to Start -> Run and entering the following:
%systemroot%\system32\drivers\etc
On MacOS, edit the file in TextEdit by running the following command in Terminal (enter your password when prompted). Make sure to copy and paste the following:
sudo /Applications/TextEdit.app/Contents/MacOS/TextEdit /etc/hosts
At the bottom of the file, enter these IPs:
64.34.80.167 web1copilot.fogcreek.com
64.34.80.167 webnlbcopilot.fogcreek.com
64.34.80.173 copilot.com
64.34.80.173 www.copilot.com
After accessing copilot.com, download the new executables for yourself and for the person receiving help. Email the executable to your customer/client/user and you should be able to connect. To do this, go to https://www.copilot.com/Helper/Go.aspx, at the bottom, click the link to Download the program for the person you are helping. Also, on the right-hand side click Download Now to get your new executable.
Posted by Adam Wishneusky at 11:37 AM | Permalink
To work around the the DNS issue we posted, edit your local hosts file to enter our server IP addresses.
On Windows, you can find the file by going to Start -> Run and entering the following:
%systemroot%\system32\drivers\etc
On MacOS, edit the file in TextEdit by running the following command in Terminal (enter your password when prompted). Make sure to copy and paste the following:
sudo /Applications/TextEdit.app/Contents/MacOS/TextEdit /etc/hosts
At the bottom of the file, enter the two IPs:
64.34.80.163 yoursite.fogbugz.com
64.34.80.179 yoursite.kilnhg.com
Change "yoursite.fogbugz.com" and "yoursite.kilnhg.com" to your FogBugz and Kiln URLs.
Posted by Adam Wishneusky at 09:45 AM | Permalink
Our DNS provider is currently experiencing a DDOS attack. If this interrupts your WebPutty access, please use this workaround. We will update this blog when the problem is resolved.
To access WebPutty, use http://cssfiddle.appspot.com instead of http://webputty.net
Posted by Adam Wishneusky at 09:40 AM | Permalink
To work around the the DNS issue we posted, edit your local hosts file to enter our server IP addresses.
On Windows, you can find the file by going to Start -> Run and entering the following:
%systemroot%\system32\drivers\etc
On MacOS, edit the file in TextEdit by running the following command in Terminal (enter your password when prompted). Make sure to copy and paste the following:
sudo /Applications/TextEdit.app/Contents/MacOS/TextEdit /etc/hosts
At the bottom of the file, enter the trello IP:
64.34.80.210 trello.com
Posted by Adam Wishneusky at 09:40 AM | Permalink
