(or: How the Greatest Tool of the 1980’s Crippled a Generation, and how Hadoop and HBase help)

postnote: This isn’t about a complete death of the RDBMS. Just the death of the idea that it’s a tool meant for all your structured data storage needs.

(Like this? Check out our data platform startup, Drawn to Scale — Fill out the contact form and we’ll drop you a line!

Imagine someone told you that there was a piece of software out there which could change the way you do everything – and I mean everything. This code is so perpetually practical; it’s a Swiss Army Knife! Surely, it can handle the simple data storage and analytics you want to do on a ‘Social Media’ scale.  For example, you can:

• Store pieces of content for your website so it’s easy to manage
• Store content from EVERY website on the Internet
• Track your company’s financial transactions
• Track every credit card transaction for your multibillion-dollar banking conglomerate
• Store an address so it’s easy to find by a unique key
• Store every photo on Facebook so it’s accessible by a unique key
• Generate charts and graphs on how your business did in the last month
• Generate charts and graphs on key analytics for every piece of Social Media on the Internet.
• Analyze records of how people have accessed your website and attempt to simulate their behavior
• Analyze every click ever made on MySpace and optimize intrasite workflow.

…Well, in theory you can do the above.

Does it make sense that people can install the exact same software package and be able to do this, at all scales? Just throw bigger computers at the problem? It’s counter-intuitive to me, and it sure doesn’t work in reality. With the coming need to analyze things not on an enterprise-scale, but a Web Scale, social media is driving the final stake into the large analytical RDBMS (Relational Database Management System).

The ACIDy, Transactional, RDBMS doesn’t scale, and it needs to be relegated to the proper dustbin before it does any more damage to engineers trying to write scalable software.

Continue reading