Jul 27, 2012

Twitter's Fail-Over Setup Was an Epic Fail

Image representing Twitter as depicted in Crun...
Image via CrunchBase
You are probably aware of Twitter's major issues this week. If not, the super popular micro-blogging site seems to have some issues staying up. A year or so ago, this was a common occurrence, but over the last few months Twitter has built up their infrastructure to minimize down time.

Well they have been having issues this week. Most notably, yesterday morning they went down. According to ABC News:

At around 11:20 a.m. ET, Twitter users started to see their streams come to a screeching halt and were not able to send 140-character messages. The service did not come back online until 1:25 p.m. ET, Twitter confirmed.

They shouldn't have gone down though because like most online services they had set up their infrastructure with fail-over for redundancy. If a system goes down, the service is supposed to fail-over to the working system. This sort of setup is very common for systems requiring high availability. Twitter's fail-over setup failed though.

From Twitter's blog:


The cause of today’s outage came from within our data centers. Data centers are designed to be redundant: when one system fails (as everything does at one time or another), a parallel system takes over. What was noteworthy about today’s outage was the coincidental failure of two parallel systems at nearly the same time.  
I wish I could say that today’s outage could be explained by the Olympics or even a cascading bug. Instead, it was due to this infrastructural double-whammy. We are investing aggressively in our systems to avoid this situation in the future. 

It just goes to show you that even the biggest names in the business can have failures, and shows the importance of redundancy and if you can afford one, a DR warm or hot site. Hopefully enough CFO's use Twitter and can see that now, and will approve funds to improve redundancy in their organizations.

What were you doing while Twitter was down? Let us know in the comments.



Twitter Delicious Facebook Digg Stumbleupon Favorites More

 
Design by Free WordPress Themes | Bloggerized by Lasantha - Premium Blogger Themes | stopping spam