Just wait until correlation database technologies get their feet under the DW/BI table.

Cheers, Martyn

But what most MR/Hadoop afficianadoes overlook is that the only way to use Hadoop is through a programmer. A million years ago, the SQL guys were like that. But today, the business user has cool tools to access the data, slice, dice, OLAP, pivot, etc. They dont need no stinking programmers. The business user does amazing things, makes incredible discoveries when they interact with the data direct. Hadoop has a lon way to go to get there. Hive and Pig are a good start, but quite young and immature.

The other thing Hadoop folks ignore is performance. They say “well, the database takes 20 minutes and Hadoop takes 2 hours but given that Hadoop is free, its worth it.” What they leave out is the 1-8 hours of programmer labor needed to get the Hadoop MR job built, tested, and finally running. We can have endless debate over every labor cost and job needed to run (ETL, load times, programming, operations, etc.) At the end of the day, the data warehouse stuff is strong, works well, and often already running.

I suggest a better way to think of this. Hadoop does not destroy or eliminate the data warehouse. And there are many things the DW does Hadoop/Hive/PIG cannot do and might never do. In contrast, Hadoop/Pig/Hive have a place in this world and its not fully decided what that is. We dont need to piss on one another to win. Pissing contests always end up with two stinkers. Hadoop/MR is a tool. Use it wisely and it will yield amazing things. the Data Warehouse is a tool. it too produces fantastic things. And dont give me that whining about cost — I’ve seen many multi-million dollar warehouses sold when the alternative was free or $20K. If the ROI is 800%, the cost of the warehouse means nothing. We only care that we have happy execs and users (emphasize users not programmers) AND that the right tool for the job was used.

Last, hadoop/MR has NOT proven itself in the market. A few dozens sites are doing great things. If you read Geoffrey Moores Crossing teh chasm, Hadoop/MR is at the beginng. Its called lunatic fringe, earliest adopters. Look at the Gratner Hype Cycles and Hadoop/MR hasn’t even hit the charts while cloud computing is roaring along. These things follow a consistent pattern. Hadoop/MR is not mainstream stuff and is primarily limited to a few dozen DOT.Coms. I’ve been doing this a long time and Hadoop/MR has all the earmarks of becoming mainstream in the next 5-7 years. Today, its still risky stuff, but marvelous in our eyes. Hadoop and EDW will sit side by side in the same site for the next 10-15 years. So why the fight? Use the right tool for the job.

Whole NoSQL moment and everything related to it isn’t about shedding everything existing and go new way on, it’s about making people aware, to let people out of local maxima and help them see the world beyond which is to realize “There is a more than one way to do it” (Perl mantra), and there always is.

Sticking always to a traditional approaches or systems to do some next generation or a different sort of job isn’t surely a way to go, we got to change the solution space when problem space changes.

I think future is about hybrid technology, when it wouldn’t even be require to be called hybrid anyways… We already see combinations of technology working complementary with each other: when Bradford’s mentioned: [Hadoop + Hbase + Hive] thing is in work, when Facebook’s [Hadoop + Casandra + Hive] is in work, when Linkedin’s [Hadoop + Voldemort + RDBMS (Oracle, MySQL)] is in work. (Reference: http://developer.yahoo.net/blog/archives/2009/06/nosql_meetup.html)

Line between traditional RDBMS & noSQL systems is already blurring, as we see HadoopDB (http://db.cs.yale.edu/hadoopdb/hadoopdb.pdf) combining power of [Hadoop + RDBMS (postgresql) + Hive] to cater the job. MongoDB and Yahoo Sherpa both are working to provide a scalable data storage system with as many friendly querying capabilities as possible. (Reference: http://developer.yahoo.net/blog/archives/2009/06/nosql_meetup.html)

Very soon I believe big vendors like Oracle are also going to introduce such parallel DBMS, with hybrid combination of some of these noSQL system approaches in the backend, as other close sourced data warehousing vendors GreenPlum (www.greenplum.com/technology/mapreduce) and Aster Data (http://www.asterdata.com/product/mapreduce.php) did.
(Reference: http://www.dbms2.com/2008/08/26/why-mapreduce-matters-to-sql-data-warehousing)

Future is of parallel DBMS functioning as a one package (Reference: http://www.computerworld.com/s/article/print/9131526/Researchers_Databases_still_beat_Google_s_MapReduce), where we don’t need to worry about integration of components making it work. Still when that’ll be here, one solution of course wouldn’t cater all problems, all problems will evolve with our solutions too; we would adapt and should continue picking the hat that (most closely) fits the head.

How large is your data size, how many concurrent requests do you have, what software are you using, and how much did everything cost?

You can e-mail me if you want, I’m truly interested :)

Actually, I’m doing real-time data warehousing now – and have peaks of 60,000+ queries running per day. And I expect over 99% of my queries to run in under a tenth of a second. And then there are those massive queries that take ten minutes.

But 2 hours? My users aren’t happy if their queries are taking that long – especially since it often takes a half-dozen iterations to get them perfect.

Do you work for Oracle? Do you have a vested interest in the status-quo for Data Warehouses to continue? A traditional data warehouse consultant?

The fact is that despite its substantial flaws (as you have pointed out) Hadoop SOMEHOW manages to get chosen for data warehousing by companies that seem to have SOME idea of what is going on.

And as far as “getting my facts straight” goes I’ll simply say “Facebook, Last.fm, IBM, Sun, Amazon, Cloudera, Yahoo, Google, Hulu, ImageShack, Joost, Baidu, Rackspace” and others, all listed at http://wiki.apache.org/hadoop/PoweredBy

While Hadoop can certainly complement existing data warehouses, Facebook has been using Hive in place of a data warehouse for over a year now. See http://www.dbms2.com/2009/05/11/facebook-hadoop-and-hive/ for more. In fact, the scale of their data warehouse and the number and diversity of users submitting queries makes it quite clear that Hadoop is a mature (potentially “upstanding”?) technology that’s seeing serious use in enterprises and labs around the world.

Of course, unlike other data warehousing products, you don’t have to just read about the technology. Go check it out for yourself and help us produce “differentiated value”: http://hadoop.apache.org.