Skip to Content

data center

Multi hour outage Sunday, October 15th: data center power meltdown

So, we're back. As you will have noticed, we were down for multiple hours today. The data center where our servers are located, and managed by the fine folks at WorkHabit, had their power supplies essentially melt down. A cascade happened which somehow caused the triple redundant power systems to overload and shut down one after the other, happening so quickly that the emergency generators didn't even have time to kick in. In short: the entire data center went dark in a few minutes.

This event also happened to take out a chunk of phone service -- including 911 for several hours! -- in the area (that's just us trying to feel better that the "big guys" took a hit as well). In any case, a full scramble went on at the data center, the system was plugged back into the city's electric grid, and systems came back up as we had people onsite manually restarting each machine. We expect that there will be some cleanup to do around database corruption issues: if you notice anything strange, please don't hesitate to email support, we'll be doing database checks over the next couple of days.

Also, now that our friends at Jaiku were bought by Google, looks like lots more people are checking them out. We posted a couple of updates as they happened to the Bryght channel on Jaiku -- we'll map that to some Bryght URL so it's easier to get to, but in the meantime you early adopters out there might want to check it out.

Syndicate content