CG Outage 2013-01-12
We're once again seeing fairly consistent connection errors to
CG for the last ~3 hours (starting around 2013-01-12 20:30
PST).
Mostly Curl error #28 "SSL connection timeout".
Our connection timeout remains unchanged in our PHP client library at the default of 10 seconds. It would be unfortunate if we needed to relax the connection timeout to an even longer period than that.
Is this a known issue? The amount and frequency of downtime recently is concerning.
Discussions are closed to public comments.
If you need help with Cheddar please
start a new discussion.
Keyboard shortcuts
Generic
| ? | Show this help |
|---|---|
| ESC | Blurs the current field |
Comment Form
| r | Focus the comment reply box |
|---|---|
| ^ + ↩ | Submit the comment |
You can use Command ⌘ instead of Control ^ on Mac
Support Staff 1 Posted by Marc Guyer on Jan 13, 2013 @ 10:28 AM
Hi Dan -- We're aware of the issue and we're working on it.
Support Staff 2 Posted by Marc Guyer on Jan 13, 2013 @ 11:05 AM
There's an apparent hardware failure. We're setting up for failover while we verify with Rackspace.
3 Posted by Dan Kamins on Jan 13, 2013 @ 06:34 PM
Thank you for the update. Can you speak to the scope of the problem?
I.e. the web interface seems to be mostly working (but the graph is flat and showing strange large negative numbers of credits in the dashboard). Is the problem limited to API? Certain API calls? Is there any data loss?
Support Staff 4 Posted by Marc Guyer on Jan 13, 2013 @ 07:37 PM
Hi Dan -- No data loss that we're aware of. The failover was clean. Accuracy of the slave data was verified as of yesterday afternoon. We're looking into the trouble with the graphs.
Also, the connection failures you recently experienced (10 days ago?) were likely due to this problem as well. It seems that IO on this particular server had been degrading over time until the load just couldn't take it anymore.
We'll prepare and blog a post-mortem as soon as we can.
Dean closed this discussion on Jan 31, 2013 @ 09:56 PM.