Outage this morning

[Derek [CT]]

Had a hiccup this morning that affected a large number of dedicated servers and many shared servers as well. More info is coming - we'll send out a notice to affected customers as well, with more details. Things should be coming back online now - total time of the  issue was about 20 minutes.  

UPDATE: One of the UPSs lost power, which caused the outage. Several segments were affected across the board (shared and dedicated, mail, database, etc.) including our phone server, which is why calls are slow getting connected.

Power was restored quickly but servers are coming online slowly as servers tend to do. We're working as fast as we can to get all servers back online...more info as it becomes available.

UPDATE 2: As mentioned, power was restored but we're still seeing issues with some servers recovering from the immediate shut down. A few database servers are being worked on, and there are some dedicated and VPS boxes that are being addressed as well. AND I renamed this forum topic so as to not appear to belittle the severity of the problem.

UPDATE 3: All but about a 5 dedicated servers are back online. Those 5 are getting the full attention of our Network Admins, so they should be back up shortly. No restoration of data was required. We are still working with our datacenter provider on what exactly occurred. I've been here 8 years and this is the only power-related issue I know of, so it's not a common issue.

[ak732]

Thanks for the info.  Was quite a hiccup.  The CT site and forums were down as well, at least from NJ and FL.  What was the problem?

[ak732]

My client's sites seem to be back up.  But my site is still down.

[justin.b]

I've still got a server down in the dedE segment - have submitted a support ticket just so its logged.

[Toddski07]

Yep...this stinks...all mail services are not working for my clients and Crystaltech says they don't have an ETA on when it will be working.  A little bit of info would help even if you are still working on the problem and don't exactly know when it will be up again.  Plus...it took several tries to get through the phone service only to learn that there is no additional info.

[cdeelsny]

Sorry, a hiccup is not 20 minutes and to this degree.

[Astralis]

This is the second time this has happened in a couple of months (not counting the Comcast issue, of course).

My sites are still down.

And like the last time, your phones are also offline.

Now that I read the forum, I won't call again nor submit a ticket.

[dani_17]

For what I see it was a power related issue, since our dedicated servers were rebooted. I hope you guys can fix it as soon as possible. I didn't even call or send a ticket, as this is a general issue CT is well aware of, plus, they're most probably have their phone lines busy and ticket system flooded.

[Derek [CT]]

Looks like a UPS lost power, which in turn caused the outage. It's back online now. Servers are coming back online as well, but some servers are taking longer than other. NOC and Server Ops Teams are working as quickly as possible to get servers online.

Still looking at how and why we had an issue with the UPS.

[Corobori]

On my side the websites with problems appear to be back online

[jamesedmunds]

Thanks for the update, Derek. About half of my two dozen or so sites are still down so I'll keep checking.

Also, can't connect to one or two MySQL servers through data administrator, presumably this is the same issue.

-James

[kurogers]

I think hiccup is an understatement.  20 Minutes is a lifetime when your customers can not reach you by web, email or phone.  I've have used your hosting service for many years and have recommended customers to CrystalTech (as recent as today actually).   Whatever the cause, it's a hot mess (hot mess > hiccup).

[HotBottleGuy]

 

Uh, pardon me... This was not a hiccup. This was full-on projectile vomiting. Some shared sites are still not up.

I don't understand how "UPS lost power" could cause an outage. I thought CT had bulletproof power backup capabilities. Guess not. You should have your prez redo the video to say that CT has total redundancy--oops, well except for power...

I love CT, don't get me wrong but someone's head should roll for this one.

[HotBottleGuy]

At 9:15am all my sites were back up. Not good guys. Fix this please.

[ak732]

While I agree that "hiccup" is a bit of a euphemism in this case, I think Derek's choice of terms is far outweighed by the goodness that is CT having this forum and posting about the problem quickly.  Given the choice between CT employees understandably wanting to employ a euphemism and not having them post anything at all, I'll live with the euphemism.

While I can easily overlook the terminology, I'm pretty curious about why a single failed UPS took down so many servers.  That seems kind of like a design flaw to me.  If I'm going to bitch about anything, it would be that.