Comments on: Social Media Kills the Database

By: What is NoSQL – Part 1 ? | Devguru

What is NoSQL – Part 1 ? | Devguru — Fri, 05 Mar 2010 04:16:05 +0000

[...] Social Media Kills the RDBMS by Bradford Stephens [...]

By: What is NoSQL database? Enter NoSQL East, conference of non-relational data stores — PaulStamatiou.com

Mon, 05 Oct 2009 14:29:58 +0000

[...] Social Media Kills the RDBMS by Bradford Stephens Slides from a NoSQL meetup [...]

By: How To Make Life Suck Less (While Making Scalable Systems) | Road to Failure

Wed, 09 Sep 2009 18:51:44 +0000

[...] Social Media Kills the Database [...]

By: Andrew Peterson

Andrew Peterson — Fri, 21 Aug 2009 15:04:29 +0000

You are right on, key-value, Hadoop, Cassandra, etc. are a must. But they don’t replace RDBMS. They replace what RDBMS don’t do well. When Dr. Codd wrote the relational concepts in 1969, hierarchal database were the norm, and accounting systems were the primary systems using a database. ACID and structure were critical. The RDBMS is great for maintaining an ACID driven, absolute relationship. Key-value is great for finding text based relationships. Like any tool, use both wisely for what they do well, not as a Swiss army knife.

By: Bradford

Bradford — Mon, 10 Aug 2009 22:42:38 +0000

We make the indexes with a MR job, and then Katta (distributed Lucene) pulls them out of HDFS onto a separate cluster optimized for serving searches.

By: Fred

Fred — Mon, 10 Aug 2009 14:38:52 +0000

Thanks very much for your reply! One thing that wasn’t clear to me is how exactly Lucene fits into this. Do you have Lucene using HFS as its directory storage and running directly on those 19 nodes, or are you extracting from HFS to a set of conventional (local file based) index shards?

By: Bradford

Bradford — Fri, 07 Aug 2009 03:33:28 +0000

Thanks for your comment! You’re right in that some of the OLAP/OLTP points are moot.

The cost and performance benefits are really nice so far. For an example, we have a $140,000 SQL Server + License, that is so full of data that it takes 24 hours to load and process a few GB of documents.

Yesterday, we loaded all 1TB of our data set into HBase, indexed it with Lucene, and deployed it to our search cluster in about 18 hours — and that’s with some random failures that set us back because we’re kinda new to Linux and things weren’t very efficient. This was done on a cluster of 19 nodes that cost $1700 each.

To do the same work on our SQL Server box took us over a week :)

Dev time wasn’t too bad — I knew Hadoop pretty well, but the team had to ramp up on learning Linux, HBase, Zookeeper, Lucene and the rest of the stack. It took our team of 5 people about 3 weeks for an end-to-end system prototype, which was pretty fragile. We’re going to spend several months to roll out a full “enterprise” solution, with proper monitoring, workflow, perf, etc. Which is really no different than any other large-scale application.

If you have more questions, feel free to e-mail me!

By: Fred

Fred — Fri, 07 Aug 2009 02:50:21 +0000

Interesting article! This is a thought-provoking discussion.

It sounds like the problem you’re trying to solve is more OLAP than OLTP, so some of your objections to DB transaction cost (for example) could be moot depending on the implementation. And your storage requirements (you mentioned hundreds of millions of docs), while large, don’t seem beyond the scope of what modern OLAP (or OLTP for that matter) systems can handle. You’re certainly right of course that there will be a substantial cost.

The Hadoop etc. solution doesn’t sound exactly simple or cheap either, though. Surely it’s cheaper than all your MS SQL Server boxes, but you still need a bunch of nodes, plus a fair amount of custom development to ingest data, perform the computations, and manage everything. How has the cost equation turned out? For example, how many nodes do you need in the new cluster to store everything? (And how many SQL Servers did you need before?) What’s the relative scale of the development effort — like, team of 3 for 3 months or team of 6 for 6 months etc.?

Anyway, thanks for sharing your experiences. It’s interesting to see the different ways people are trying to solve these problems.

By: Bradford

Bradford — Sun, 02 Aug 2009 15:27:49 +0000

That’s very interesting — one of my friends who went to TechEd was the source of the MSSQL Boxes fact. Now I’ll have to go double-check it’s legit :)

By: Gary

Gary — Sun, 02 Aug 2009 09:14:51 +0000

Database requirements of myspace are pretty tiny compared to the search giants.

According to the article below MySpace run off 500ish database servers. Much smaller scale than tens of thousands of the big search companies.

http://highscalability.com/myspace-architecture