Enjoy!

I can’t embed the video for some reason, here’s the Link .

A big thanks to Noah Campbell at LinkedIn for inviting me — it was a blast!

(Want to learn more? fill out this simple form and we’ll drop you a line!)

Imagine, for the sake of illustration, that you’re part of a new startup company that aggregates web content and does a lot of value-adding.

The company busily integrates gigabytes of data from various sources, analyzes it, then makes it available via search. There are three engineers: two working on an app, and one on data (administered with the ‘help’ of MySQL). A standalone machine processes incoming data, then feeds it to the database.

But there’s a problem…

Time goes on, you ingest more data, add new sources, etc. Soon, your database response times are getting longer for no apparent reason. Downtime is more frequent. Two more engineers come aboard to tune up the DB and rewrite some code.

Uh oh. Users are complaining. Management is sweating, and suddenly they’re paying attention to the data infrastructure.

$100,000 later, a state-of-the-art server arrives, and after some migration panic things are back to normal.

The business gets more popular, users demand more features… In a few months, the database is crawling again. Even more engineers are brought in and the whole dirty cycle starts again.

But that’s just the way it goes, right?

The Drawn to Scale Platform is designed to give companies room to grow. We discovered that Internet-scale software can be lean, fast, and even easy to use — all vital factors in building better systems that last. We believe in providing simple interfaces and controls, not SQL black magic (though we may support SQL in the future), and our platform offers seamless scalability to handle any peaks or spikes. We even provide a custom toolset that makes management and migration of existing data easy.

Traditional database servers are mature. They’re well understood, and well supported. But they also have performance ceilings most people won’t hit early on. This makes choosing them conventional wisdom, and a safe bet. But hit that ceiling and quite literally no amount of money will save you. Twitter learned this the hard way in 2008, and had to start over again. Not all companies survive such a rewrite.

My cofounder and I started Drawn to Scale so companies can move fast, developers can build incredible apps, and so no-one has to stay up late kicking database servers into shape.

(want to learn more? fill out this simple form and we’ll drop you a line! or Watch our Video and vote for “Drawn to Scale”.)

The result of all this is that you can now focus on building your business and applications, instead of spending precious resources maintaining and building infrastructure.

You don’t have to fight with your tools any more.

…let’s learn more!

Why are we doing this?

Our motivation comes from knowing that Internet-scale businesses need 100% agility and 100% presence, but existing solutions don’t really offer that. As a result, Drawn to Scale’s platform is:

• ready to build with from day zero
• built to run on commodity, cheap hardware
• able to scale out and resize on demand on the cloud or in your DC
• manageable by humans(!)

Just put your data in, and query it. It’s that easy.

Indestructibly lightweight storage

Data wants to be useful. But to be useful it has to be stored safely and turned around quickly for immediate, random access. To do this we use Hadoop and HBase, a fast and reliable storage tech that captures data of any type or size. Our implementation of HBase is self-managing and also super forgiving when it comes to organization, so it puts old-style “schemas” right back where they belong (ie yesterday).

Here’s an illustration of the kind of command we use to make data persistent and available:

Table.put(Column,ColumnFamily, Data);

Everything else that happens, like search preparation and backup, is helpfully automatic. Just send and go.

Optimized for the queries that matter

Software evolves fast because good ideas travel. Standardized form controls, standardized web page layout… over time certain patterns just rise to the top and stay there because they work. Database handling is no different. Regardless of how new and amazing an app might be, it’s likely the fundamental building blocks are familiar and constant.

Consider for a moment what you typically do with your data. It’s likely, if you’re maintaining any sort of web app or dashboard, that the vast majority of operations will be in the following list:

1. SELECT
2. GROUP BY
3. COUNT
4. SUM
5. Filtering/WHERE
6. A few others

The Drawn to Scale platform is fast and easy precisely because we made it to reflect this reality. We’ve created an indexing engine and query language specifically to do this one handful of things extremely right — things that common practice has proven to matter. Exactly what you need, and no more.

Search that doesn’t discriminate

Those operations we mentioned? They work just as well on raw text as they do on pre-structured data. Which means you can easily mix structured and unstructured data together and search or browse by attributes, the same way you might discover products at Amazon.com or Newegg. To achieve this we use search products like Lucene and bobo-browse, adapting them as part of a scalable architecture.

This combined directory/search ability — known as faceted search — also makes sense for web content aggregation, because it allows for related items to be pulled from non-uniform sources.

Availability in changing conditions

Data isn’t particularly useful if you can’t serve it on time. So as your data volume grows, or the number of hits on it increases, our platform can spin up additional server resources to balance out the load. The sky’s the limit when it comes to simultaneous requests (although we do recommend adding caches  to your apps wherever possible). And when traffic isn’t spiking, that extra capacity can come back to us and be saved for later.

What’s the value?

We believe in working smarter. By using a platform that addresses real requirements, you’re making more money and spending less of it. Your company gets to be moving faster and building what it needs to survive, and you’re freer to be a better developer, manager, or CEO.

We’re extremely excited to be rolling out this product over the coming months. To know more or become an advance customer, drop us a line on this form.

Want to see more info in a video? Check out this link and vote for “Drawn to Scale”.

Logging is perhaps the least sexy part of writing software, yet it’s surprisingly important. There’s lots of boring stuff that causes heated arguments (like whitespace, build systems, and version control). But no-one has impassioned, snarky blogs or tweets about what format your logs should be in. Much like cell phones or the Internet, we don’t realize how we need them until it’s too late.

We’re drowning in information from our servers. Especially when dealing with distributed/Big Data systems, which are growing incredibly fast. Even when building small server clusters, my co-founder Nick and I found it hard to learn what was going on. So we chatted, “Well, what’s the biggest step forward in web accessin the past 15 years? Search!” In a few brief moments, we set about sketching up some ideas for a generalized “log search” engine, and ended up naming it “LogSearch” (we’re literalists.)

Logging is something engineers have always needed, because software (no matter how experienced you are) is notoriously unpredictable. When you’ve got more than 10 computers, it’s difficult to keep track of what’s going on day-to-day, and even worse when Something Goes Wrong. When the inevitable crisis comes, you don’t want to spend a few hours poking through log files on 30 different machines. You want to be a hero and find the problem in 5 minutes, then get on with your life. Or maybe you want to take a longer view of things — run some interesting statistics on your logs, and study usage patterns.

Let’s ponder the implications of that for a second. What if you could log every action taken in your application, and analyze it to exactly how your application was used? Delicious.

All of us at Drawn to Scale have dealt with distributed systems for a while. We like to build stuff we would use. That’s why we built something that lets us:

• Be absurdly easy to use in the cloud or in a datacenter
• Easily ingest any amount of log data (hundreds of TBs)
• Find structured fields (like timestamps)
• Search these logs (we usually don’t know what we’re looking for when something goes wrong)
• Provide the actual text from the log itself
• Build trend charts (i.e., how many errors/day?)
• Send alerts when something looks “strange”
• Run Hadoop/MapReduce scripts to provide interesting analytics

If you had a log search engine, with various utilities built on top of it, you’d have an easy way to see what’s going on in the apps that drive your business. You could…

• Be aware of problems *before* they cause critical downtime
• Monitor trends so alerts can be sent for odd activities
• Examine long-term utilization patterns to optimize software
• Quickly diagnose when machines crash or become flaky
• Look smart and “in touch” with your systems!
• Save piles of money!
• Make piles of money!

Isn’t Monitoring Software Enough?

Engineers and ops people have a fantasy of coming into work, pulling up a dashboard on their web browser, and understanding everything that is going on. We want to be like Geordi in Star Trek, who can see everything on the Enterprise with a few finger presses. I wish there was such a thing! The closest we can get is ganglia, which monitors OS-level events (it can be hacked for apps). It’s pretty cool, but only so useful. Monitoring requires serious engineering effort – even if planned from the very beginning (it rarely is), there’s quite a bit of code that must be written and maintained. Monitoring services need to run on each box, and the central servers become a system of their own that needs to be reliable, redundant, and robust.

Logs, however, are a piece of cake. Here’s what usually happens.

1. Put a line of code similar to Log.Error(“Something horrible happened!”); in your application
2. The message and a Timestamp appear in a /logs directory on the disk
3. Logs are cleaned up every few days by an automated process

Since logs are so simple to implement, it’s easy for us lazy engineers to toss them into everything we do. The problem is that when they grow beyond a few hundred MB, it takes forever to search them for what you want, and it’s even harder if spread across multiple machines!

Why it’s Hard

So It’s easy to fill logs with lots of tasty info. But it’s hard to make sense of it all. It’s semi-structured text. This means in order to find what you want, you need to explore the data: you need to search it. The only tool most of us can use is grep — tedious and manual if you need to troubleshoot several dozen logs on each box. There’s also interesting information in your log that isn’t just raw text — dates, server addresses, error types, and more. So you have to build something to parse meaning out of it.

You end up having to design a distributed system just to parse handle logs. There’s commercial tools out there which claim to do this, but they fail at large scale.

Draw to Scale has built LogSearch from the beginning to handle Big Data.

How We Do It

The Drawn to Scale BigSearch platform is built on a ton of distributed and NoSQL tools, combined with quite a bit of Secret Sauce. Our goal is to provide an end-to-end data storage, search, and serving platform. To do this, however, we need to know what the system does, and prove its capabilities. That’s why LogSearch is a great tool to put on top of BigSearch.

We run in the cloud or in your datacenter. To run our system in the cloud (specifically, AWS), it’s rather simple: spin up a few instances with our machine image, and install a script on each of your boxes to find the logs and push them onto the LogSearch cluster. Since every component of our infrastructure is scalable, it can handle as much data as is thrown at it.

Everything is seamlessly processed with Hadoop, stored in HBase (a clone of Google’s BigTable), indexed, and made searchable in a few minutes. All you have to do is go to a webpage and search for what you want to find in the logs. In the near future, you can upload MapReduce, Pig, or Hive scripts to provide more complex analytics.

Are you excited? (I am, I wrote a whole friggin’ article). Want to learn more? Follow me on twitter at @lusciouspear, or drop me a line at info@drawntoscalehq.com. We’d love to show you a demo, or get your ideas on what would make this tool Really Awesome and Indispensable.

Distributed, scalable databases are desperately needed these days. From building massive data warehouses at a social media startup, to protein folding analysis at a biotech company, “Big Data” is becoming more important every day. While Hadoop has emerged as the de facto standard for handling big data problems, there are still quite a few distributed databases out there and each has their unique strengths.

Two databases have garnered the most attention: HBase and Cassandra. The split between these equally ambitious projects can be categorized into Features (things missing that could be added any at time), and Architecture (fundamental differences that can’t be coded away). HBase is a near-clone of Google’s BigTable, whereas Cassandra purports to being a “BigTable/Dynamo hybrid”.

In my opinion, while Cassandra’s “writes-never-fail” emphasis has its advantages, HBase is the more robust database for a majority of use-cases. Cassandra relies mostly on Key-Value pairs for storage, with a table-like structure added to make more robust data structures possible. And it’s a fact that far more people are using HBase than Cassandra at this moment, despite both being similarly recent.

Let’s explore the differences between the two in more detail…

CAP and You

This article at Streamy explains CAP theorem (Consistency, Availability, Partitioning) and how the BigTable-derived HBase and the Dynamo-derived Cassandra differ.

Before we go any further, let’s break it down as simply as possible:

• Consistency: “Is the data I’m looking at now the same if I look at it somewhere else?”
• Availability: “What happens if my database goes down?”
• Partitioning: “What if my data is on different networks?”

CAP posits that distributed systems have to compromise on each, and HBase values strong consistency and High Availability while Cassandra values Availability and Partitioning tolerance. Replication is one way of dealing with some of the design tradeoffs. HBase does not have replication yet, but that’s about to change — and Cassandra’s replication comes with some caveats and penalties.

Let’s go over some comparisons between these two datastores:

Feature Comparisons

Processing

HBase is part of the Hadoop ecosystems, so many useful distributed processing frameworks support it: Pig, Cascading, Hive, etc. This makes it easy to do complex data analytics without resorting to hand-coding. Efficiently running MapReduce on Cassandra, on the other hand, is difficult because all of its keys are in one big “space”, so the MapReduce framework doesn’t know how to split and divide the data natively. There needs to be some hackery in place to handle all of that.

In fact, here’s some code from a Cassandra/Hadoop Integration patch:

+ /*

+  FIXME This is basically a huge kludge because we needed access to

+ cassandra internals, and needed access to hadoop internals and so we

+ have to boot cassandra when we run hadoop. This is all pretty

+ fucking awful.

+

+  P.S. it does not boot the thrift interface.

+ */

This gives me The Fear.

Bottom line? Cassandra may be useful for storage, but not any data processing. HBase is much handier for that.

Installation & Ease of Use

Cassandra is only a Ruby gem install away. That’s pretty impressive. You still have to do quite a bit of manual configuration, however. HBase is a .tar (or packaged by Cloudera) that you need to install and setup on your own. HBase has thorough documentation, though, making the process a little more straightforward than it could’ve been.

HBase ships with a very nice Ruby shell that makes it easy to create and modify databases, set and retrieve data, and so on. We use it constantly to test our code. Cassandra does not have a shell at all — just a basic API. HBase also has a nice web-based UI that you can use to view cluster status, determine which nodes store various data, and do some other basic operations. Cassandra lacks this web UI as well as a shell, making it harder to operate. (ed: Apparently, there is now a shell and pretty basic UI — I just couldn’t find ‘em).

Overall Cassandra wins on installation, but lags on usability.

Architecture

The fundamental divergence of ideas and architecture behind Cassandra and HBase drives much of the controversy over which is better.

Off the bat, Cassandra claims that “writes never fail”, whereas in HBase, if a region server is down, writes will be blocked for affected data until the data is redistributed. This rarely happens in practice, of course, but will happen in a large enough cluster. In addition, HBase has a single point-of-failure (the Hadoop NameNode), but that will be less of an issue as Hadoop evolves. HBase does have row locking, however, which Cassandra does not.

Apps usually rely on data being accurate and unchanged from the time of access, so the idea of eventual consistency can be a problem. Cassandra, however, has an internal method of resolving up-to-dateness issues with vector clocks — a complex but workable solution where basically the latest timestamp wins. The HBase/BigTable puts the impetus of resolving any consistency conflicts on the application, as everything is stored versioned by timestamp.

Another architectural quibble is that Cassandra only supports one table per install. That means you can’t denormalize and duplicate your data to make it more usable in analytical scenarios. (edit: this was corrected in the latest release) Cassandra is really more of a Key Value store than a Data Warehouse. Furthermore, schema changes require a cluster restart(!). Here’s what the Cassandra JIRA says to do for a schema change:

1. Kill Cassandra
2. Start it again and wait for log replay to finish
3. Kill Cassandra AGAIN
4. Make your edits (now there is no data in the commitlog)
5. Manually remove the sstable files (-Data.db, -Index.db, and -Filter.db) for the CFs you removed, and rename files for CFs you renamed
6. Start Cassandra and your edits should take effect

With the lack of timestamp versioning, eventual consistency, no regions (making things like MapReduce difficult), and only one table per install, it’s difficult to claim that Cassandra implements the BigTable model.

Replication

Cassandra is optimized for small datacenters (hundreds of nodes) connected by very fast fiber. It’s part of Dynamo’s legacy from Amazon. HBase, being based on research originally published by Google, is happy to handle replication to thousands of planet-strewn nodes across the ’slow’, unpredictable Internet.

A major difference between the two projects is their approach to replication and multiple datacenters. Cassandra uses a P2P sharing model, whereas HBase (the upcoming version) employs more of a data+logs backup method, aka ‘log shipping’. Each has a certain elegance. Rather than explain this in words, here comes the drawings:

This first diagram is a model of the Cassandra replication scheme.

1. The value is written to the “Coordinator” node
2. A duplicate value is written to another node in the same cluster
3. A third and fourth value are written from the Coordinator to another cluster across the high-speed fiber
4. A fifth and sixth value are written from the Coordinator to a third cluster across the fiber
5. Any conflicts are resolved in the cluster by examining timestamps and determining the “best” value.

The major problem with this scheme is that there is no real-world auditability. The nodes are eventually consistent — if a datacenter (“DC”) fails, it’s impossible to tell when the required number of replicas will be up-to-date. This can be extremely painful in a live situation — when one of your DCs goes down, you often want to know *exactly* when to expect data consistency so that recovery operations can go ahead smoothly.

It’s important to note that Cassandra relies on high-speed fiber between datacenters. If your writes are taking 1 or 2 ms, that’s fine. But when a DC goes out and you have to revert to a secondary one in China instead of 20 miles away, the incredible latency will lead to write timeouts and highly inconsistent data.

Let’s take a look at the HBase replication model (note: this is coming in the .21 release):

What’s going on here:

1. The data is written to the HBase write-ahead-log in RAM, then it is then flushed to disk
2. The file on disk is automatically replicated due to the Hadoop Filesystem’s nature
3. The data enters a “Replication Log”, where it is piped to another Data Center.

With HBase/Hadoop’s deliberate sequence of events, consistency within a datacenter is high. There is usually only one piece of data around the same time period. If there are not, then HBase’s timestamps allow your code to figure out which version is the “correct” one, instead of it being chosen by the cluster. Due to the nature of the Replication Log, one can always tell the state of the data consistency at any time — a valuable tool to have when another data center goes down. In addition, using this structure makes it easy to recover from high-latency scenarios that can occur with inter-continental data transfer.

Knowing Which To Choose

The business context of Amazon and Google explains the emphasis on different functionality between Cassandra and HBase.

Cassandra expects High Speed Network Links between data centers. This is an artifact of Amazon’s Dynamo: Amazon datacenters were historically located very close to each other (dozens of miles apart) with very fast fiber optic cables between them. Google, however, had transcontinental datacenters which were connected only by the standard Internet, which means they needed a more reliable replication mechanism than the P2P eventual consistency.

If you need highly available writes with only eventual consistency, then Cassandra is a viable candidate for now. However, many apps are not happy with eventual consistency, and it is still lacking many features. Furthermore, even if writes do not fail, there is still cluster downtime associated with even minor schema changes. HBase is more focused on reads, but can handle very high read and write throughput. It’s much more Data Warehouse ready, in addition to serving millions of requests per second. The HBase integration with MapReduce makes it valuable, and versatile.

If you don’t know me you might think I’m an early riser. Alas… I’ve actually been up all night, unable to sleep, babysitting my Hadoop cluster and worrying about getting this new platform into production.

The past three months have been hellish — some amazing accomplishments, but plenty of painful mistakes. “You guys are working hard but it seems so effortless,” I heard from another team lead. If he only knew. Developing a distributed, concurrent platform for web-scale data  is far from effortless. A great deal of the time we spent ‘working hard’ was not new coding at all, but relentlessly beating on problems. I don’t always take my own advice, but I’m getting better. And so, I’ll try to keep the following in mind when planning what to do next:

Developing a platform for web-scale data analysis is hard work. Unlike well-worn areas like MVC on LAMP, big data (billions of records, intensive analysis) presents brand new challenges that need innovative engineering, testing strategies, and planning.

In recent months I’ve led my team at Visible Technologies through creating a new analytics platform based on Hadoop. This article lists a few lessons from our ordeal, which hopefully should help if you venture into similar territory. Here are my observations:

1. Big data is BIG
2. This is systems software, not an application (for now)
3. Learn the source, engage the community, contribute feedback
4. Scalable doesn’t imply cheap or easy. Just cheaper and easier.
5. It’s so much easier with smart, experienced people.

Let’s examine these in detail…

Big Data is BIG

“We’ve got two weeks to test our entire storage, indexing, and search workflow for up to one billion documents, with multi-lingual data”, I told my team. Our task seemed pretty easy, at the time. We had the workflow written up (in pieces at least), and had tested the system on a backup database of a few hundred million documents. Everything ran flawlessly — so far, so good.

Doubling the data size on a relatively untaxed computer cluster seemed like it should be painless. I’d run jobs to index hundreds of millions of documents before, no problem. When it came time to index a billion documents, though, it failed and kept failing. Iterative debugging processes become unsustainable when it takes 8 hours to run a test cycle. I found the answer, ten days later, in an obscure subroutine for sending a heartbeat back to a central server.

Scalable platforms may coordinate many machines to act as one, but that doesn’t mean you can engineer them the same way you normally would. Everything gets more difficult when multiple nodes are involved; more power, more problems.

The problem with big data is that it’s BIG. Things fail at scale, the same way they do on smaller projects. But with scale comes longer timeframes and more moving parts, and you need to plan testing strategies accordingly. The Hadoop/HBase world brings us back to the 70s concepts of batches and time-sharing. It’s scary to run jobs that take 8 hours to fail, and having the code of team members in various states of testing just adds to the nightmare.

If it takes a long time to fail, it’s going to take even longer to fix. Buy as much hardware as you can justify, and structure your test data to process a small amount before hitting bugs.

This is systems software, not an application (for now).

Plenty of MVC/LAMP or Rails programmers can write a good website, connect their markup through AJAX to a middle tier, and persist state to a DB. Most web-based software never gets more complex than this. It’s not always easy, especially with complex visualization and business logic thrown in — but it’s based on customizing well-delineated applications, and high-level shortcuts are plentiful.

Big data is not about applications yet. You’re not writing code to trigger APIs between standard modules, you’re working with the raw “stuff” of a distributed operating system, and digging around in the basement. Coding and debugging in this context requires a stronger handle on computer science theory than most web development demands.

As an example, here are a few problems we’ve stumbled upon:

1. Garbage collection pauses, and nodes lose heartbeat, when Java’s used memory heap exceeds 7GB.
2. Figuring out which garbage collection scheme to use (and which of the 10 associated parameters to tweak), requires a knowledge of the generational hypothesis of GC.
3. Microsecond race conditions can trip up processing on obscure data structures.
4. General headaches when dozens of threads need to share
5. Making everything an object can bog a cluster down. Instantiation is expensive and small objects are not cache-friendly.
6. It’s even important to understand the assumptions Java’s JIT Compiler makes about your code.

Note: Just because Hadoop requires you to work in Java, doesn’t mean it’s not systems code that you’re writing. Treat it with the same deference as a database written in C, which is something that’s likely to be very well architected from the beginning and highly optimized.

Learn the source, engage the community, contribute feedback

“Katta keeps failing when we try to deploy our 500M document index.”

“Why?”

“Not sure. Trying it again.”

An hour later, it failed again. So we made a configuration tweak. An hour later… more fail. For two days. Finally, we dove into the code. Two days might seem like a long wait to look deeper into a problem, but at scale it can take that long to eliminate all the random shots in the dark. We eventually traced the crash to a line of code that was timing out waiting for Zookeeper to give it some information. Once we were in the code, it took about 15 minutes to find the issue. There can be a mental barrier to examining the codebase of open source software (an ironic twist on not-invented-here syndrome), but mostly I’ve found them pretty easy to navigate.

This lesson of look fast, look early applies to pretty much any open source project, but doubly so for distributed ones because the time cost of incremental, scattershot debugging can be so high.

Along with studying OSS source code, you should engage the community around the platform you’re using. I’m extremely grateful to have the HBase team around on IRC and their mailing list. I usually get answers to even complex questions in a few minutes/hours. You can’t really engage the community without learning some of the source — you’ll be wasting everyone’s time. I’m trying to repay the HBase team for their kindness by submitting a few patches (although I wish I had the time to do much more, there’s a lot of hefty work to do).

Scalable doesn’t imply cheap or easy. Just cheaper and easier.

I was having a conversation with an associate about our new platform and the difficulties of developing in our current environment.

“Why do you need new machines? You have five. Isn’t that enough to prototype our new infrastructure? We only have two SQL servers,” he posited.

“The machines are processing so much data that they’re crashing because the CPU is unavailable waiting for I/O,”  I replied.

“Where’s the proof?”

It’s difficult to set expectations in emerging fields such as commoditized distributed computing, especially around hardware costs and platform capabilities. Somewhere along the way, “commodity hardware” became interpreted as “$500 machines”.

Would you run a high-volume, production MySQL server on a 2 core, 2 GB RAM Box from 2005? Of course not. Why would you want to run Hadoop or HBase on a machine of the same specs?  You can’t have a cluster that costs $5000 comprised of 10 boxes that cost $500. Nor do many of us want to pay $500,000 for Netezza hardware, or $5,000,000 to Oracle. A $50,000 cluster is cheap, and can handle a lot of data. Virtualization is not a panacea, either. If you need high-volume and high-avaliablity, you need to have control over the physical environment of the boxes. Running things in “the cloud” can have too much overhead, and shields you from important locality-awareness such as which rack you’re running on. It works for some, but you need to run the numbers to make sure you’re not losing too much performance or reliability.

Unlearning (and un-teaching) the habits of the Swiss Army RDBMS can be difficult. As mentioned in earlier articles, databases like Cassandra and HBase are not RDBMSs. You cannot just write SQL and expect everything to ‘just work’. Your storage and access paradigms need to be architected from the ground up to fit the data. Impedance mismatches between data structures and data storage are fatal at large scales, and lead to general engineering malaise — bad code, strange bugs, inconsistent performance, and much more. Not everything is a table, these days. It’s a tough sell to many of those who depended on an RDBMS to handle every aspect of data storage and manipulation.

As a friend of mine recently said: “To create something that is scalable, you have to add a certain level of complexity over and beyond the single node. Just like going to multi-threaded programming is so much more complex, so is going clustered.” But if you do it right, you’re not limited by data size any more.

It’s so much easier with smart and experienced people

I love my team dearly.  Still,  every day, I find myself wishing I knew someone who’d experienced the same gotchas I have; Java classpaths and Linux deployments, performance tuning, and general issues with highly concurrent software. There’s such a big difference between the problems of a typical 3-tier web app, and the problems that arise from distributed systems engineering.

Having said that, you don’t need to revamp your entire team just to move into this area. But having at least one person who’s been there before will save your whole operation weeks of Googling and downtime, especially if you can put them in a troubleshooter or coordinating role. And if you’re transitioning a former closed-source group to OSS distributed software, make sure everyone has their CS fundamentals down — that’s always a good start.

As you examine these sites, try to think about what kind of data they need to present to their users rapidly. There’s usually a selection of:

• Profiles (like users)
• Search results
• Sets of items by various criteria (all videos by one author, posts in a thread)
• Networks (“my friends”)
• Real-time updates “My status”
• Communications subsystems (messages to inboxes, e-mails, IMs)

Until recently, there’s been no other way to retrieve this data than with a classic RDBMS, which as we’ve seen earlier can be expensive and crippling to scale. Yet I’d estimate that a few use cases cover 80% of how data needs to be presented to these websites. The Swiss-Army RDBMS is costly overkill for what most sites *really* use it for. There needs to be a new type of database optimized for serving simple, web-scale data in real-time.


Let’s examine these operations, along with some examples.  Then, I’d like to propose a scalable engine I’m architecting that’s actually optimized to drive 80% of websites, built on HBase.



THE OPERATIONS

There’s a handful of operations that can both drive websites and provide low-latency analytics on information. These are:

• GROUP BY
• FILTER
• SORT
• COUNT
• SUM

There’s a whole class of operations called “additive” and “semi-additive” that are basically simple aggregations. These are used in a majority of DB analytics use cases.
By proper structuring of data storage and denormalization, along with narrowly focusing on a few operations, we can enable new class of scalable database and analytical engines.

I can’t recall a term which properly describes what this class of database does:

• Data Warehouse? That doesn’t work. Too big. I imagine getting in a forklift, driving to the data I need, loading it on a pallet, driving it to a staging area, unpacking the pallet, and eventually getting at the pieces I want. Forklifts are pretty awesome, granted. (Sometimes I fantasize smashing our current slow-as-agriculture SQL boxes with a forklift).
• Data Mart? Smaller, but still slow. I have to grab a grocery cart, walk to the aisle which has my data, pick it off a shelf, and then take it to a cashier so I can do what I want with it.
• Operational Data Store? Getting closer. Wikipedia says it “integrates data from many sources for reporting, usually atomic pieces of data captured in ‘real’ time”

I’m going to call it a Data Station: a place where you drive up with a car, get only exactly what you need as quickly as possible, and then leave.

THE HBASE DATA STATION

Hadoop and HBase are certainly scalable, but no part of their architecture is low-latency. You can serve webpages from HBase rows, but you can’t do any aggregations there.
There’s a few scalable databases (HBase, Cassandra) out there that handle a subset of this: you can retrieve data in all of them, and Lucene does faceted search, which implements several of the operations (notably grouping and counting). Lucene is fast and easy, but it’s pretty search-and-document oriented.

I think HBase is an excellent candidate for a storage engine. It’s scalable, fast (with the latest release), column oriented, and has a fairly intuitive codebase. That’s all HBase is – distributed storage and retrieval. It’s very impressive, but it delegates the duty of doing anything to your data out to the application. This makes it hard to crunch through lots of data with low latency, because you have to move the data somewhere.

What if we implemented a package that sat on top of where the data is stored on each server, performed your operations as close to the data as possible, and then sent the result to a master node for final aggregation? It’d almost resemble a MapReduce job. Let’s look at a diagram:



It’s pretty simple, and it won’t scale to infinite data per node — but it’s not meant to. I’m calling this “HBase Analytics” for lack of a better name. This solution is meant primarily to drive websites with a low latency, especially when you can’t pre-aggregate data. Implementation is somewhat straightforward. Without going into too much detail on the HBase architecture, you can ‘piggyback’ on top of how rows are scanned, and do your aggregation there. The point of this is to move data aggregation away from the app layer, and keep it as close to the data as possible.

In addition, these basic operations are commutative and associative. What that boils down to is that the order in which you read the data and make your aggregates is unimportant. Thus, you can make this process multithreaded.

Approximate consistency is a big performance boost as well. You can pass options to “trim” your data, so only the most significant items are sent. This is important for scenarios such as “Find the Top 10 Authors for these documents”. It might be a little lossy, but if I can cut my latency in half and get data that’s 90% correct, I’ll call it in a win in quite a few scenarios.Scaling to concurrent user demand is pretty simple: increase data replication, and provide more “master” servers to perform aggregation. This is in addition to the typical approach of intelligent caching, latency profiling, etc.

THE END

This solution isn’t maximally performant of course — though secondary indices could help with that too. But it is simple and scalable, keeping a “Data Station” mentality. Taking a look at 80% of big-data-driven websites, you can implement all their functionality with GROUP, COUNT, FILTER, SUM, SORT, and a few other additive and semi-additive operations. If that can be done in a way that’s truly scalable, then building “big data” websites can be done in a few weeks by a handful of people, instead of a few months with custom, hacked-together software. Stay tuned for news on the evolution of HBase Analytics.

http://www.slideshare.net/lusciouspear/building-a-business-on-hadoop-hbase-and-open-source-distributed-computing

PDF: Here

Video should be in up a day or so– having internet problems at home :) If the video quality is too poor, I’ll upload the audio track.

The talk focuses on two things: how to approach scalability from a problem solving standpoint, and how Visible Technologies used Hadoop, HBase, and Lucene to build a web-scale Business Intelligence platform.

When thinking about scalability, ask yourself a series of questions:

1. What makes me special? (What part of our problem are we trying to scale? How can we use it to our advantage?)
2. What can I sacrifice? (Compared to a traditional RDBMS, what are you willing to get rid of? Some sacrifices make scalability much easier).
3. How will my data be structured? (Not everything should be reduced to tables and rows. Do you have documents? Graphs? Videos?)

The slides and talk focus on how Visible answered those questions, and how they led us to our Hadoop stack architecture. In the future, I’ll be talking more about the process of thinking about scalability. Stay tuned!

If you wanted to generate intelligence or other OLAPish things from large amounts of data (TBs) three years ago, you faced a terrifying prospect: You bought a server (or several large servers) costing $x00,000, paid for software from Oracle, Sybase, or IBM, and then prayed to Data Jesus that it would meet your needs for a few years. You’d find all the data you could in all the disparate formats your company used and try and cram it into tables, have data analysts schedule SQL queries, and output your reports. This was by far the most common use case for “big data analysis”.

Then, Social Media happened, and small businesses needed to process TB, web-scale companies needed to process PB, and neither of them could afford the $millions to do it. With the advent of the Hadoop and HBase ecosystem, however soon anyone can scale their Data Warehousing in predictable, affordable ways.

The established and new vendors in this space may become irrelevant to a decent portion of businesses. They recognize there’s a problem, but their solutions so far will do nothing to address the fundamental issues. They won’t disappear, but the high-cost Enterprise Data Warehouse is no longer the only solution out there, and its significance will continue to dwindle.

Hadoop represents the first radical shift in a long time in an industry that was built around “pay big or go home”.

The Big Data Cheap Problem

With the rise of Social Media and the decreasing cost of storage, very small companies have a need for processing massive quantities of data. Furthermore, it’s easier than ever to write software to generate/output/process (which is it? I’m unclear) this data, thanks to languages like Ruby, frameworks like Spring, and scalability best practices. A few days’ work and a handful of engineers can net you a bare-bones Twitter clone, or a crawler to get the link graph of the entire Internet. This data growth can be far from linear. For example, the more users you have tweeting to each other, the more connections they form, and so on. You simply can’t analyze this much data in an RDBMS — but these small startups can’t spend millions on DWs, either. Since the analysis is a core of the business, internal hacked-together tools cobbled together from SQL boxes often emerged. What results is a temporary solution, not a platform. With Hadoop, HBase, and Hive, there’s now a free, scalable Data Warehousing platform that makes it possible to migrate a considerable portion of DW analytics. On top of that, it’s getting easier every day for Data Analysts to just hop in and start working in a supported environment, thanks to companies like Cloudera.

Vendor Approaches

Upstart enterprise DW vendors like Greenplum, Vertica, Aster, and Netezza are really struggling here. Not only are they fighting against a low-cost solution, but it’s more scalable and easier to tweak. A lot of established enterprises won’t care about this — they can pay money to purchase something that “just works”. Upstart Social Media companies can’t afford that, in cost or risk. They often have problems that don’t work for classic SQL (like PageRank’s graph traversals). Each has interesting approaches to indexing, scaling, and dealing with bottlenecks. All of these vendors require pricey hardware — from tens to hundreds of thousands of dollars. Several of these vendors claim to support “MapReduce”, but that’s a topic for a separate article.

A frequent thing I’ve seen in vendor presentations is claims of revolutionary speed increases for specific problems vs. Oracle. “This analysis took 12 hours to run on Oracle, but only 10 minutes on our product!”. I have no doubt that it’s true — but how much tweaking did it take to achieve that? Does performance on other queries degrade? For many types of large data warehouse analytics, the difference between 20 minutes from a Vendor solution and 1-2 hours from Hadoop may not be enough to justify the extra expenditure.

The frequent comparisons to Oracle is one of the lynchpins of understanding the state of the large DW vendors. Just making the price and performance improvements of a few dozen percent over Oracle would have a great selling point a few years ago. Now, they’re competing against something that is not just orders of magnitude cheaper but also more scalable. That’s tough.

The Killer Apps on the Cloud

Putting Hadoop on a Cloud is where one it really shines — one can spawn clusters to run analytics on-demand, instead of paying for large amounts of servers 24/7. There’s quite a few success stories here. The dating service eHarmony uses Hadoop + Amazon Elastic MR to create millions of “matches” between customer profiles. The cost for this turns out to be < $50 a day. A comparable vendor solution combined with hosting in a colo would be many times that amount.

Until HBase came along, you basically had no random access in Hadoop clusters. This was a harsh limit on ad-hoc and selective queries. HBase rectifies that, however. Instead of following a paradigm of “Index everything! Buy faster servers!”, you can denormalize, replicate, and sort your data on a primary key. If you have 7 columns in your data set, you may have to copy your data set 7 times if you want to slice and dice on every field — but disk space is cheap and scales extremely well.

Pairing well with HBase, Hive brings the flexibility of SQL to the scalability of MapReduce. Now, analysts can run queries just as they would any other DW. This abstraction on top of MapReduce is incredibly powerful — the millions who know SQL can hop right into using a free, scalable Data Warehouse for the first time in history.

The Future: Real-time Analytics in HBase?

There’s an interesting question I’ve been pondering lately — just what *can* you do in real-time on HBase? There’s no Hadoop MapReduce overhead, so you can achieve low latencies — but how much data can you crunch through in a handful of seconds? This could be useful for driving a web UI with some charts and dashboards on an arbitrary dataset. A lot of these types of queries are built from the fundamentals of GROUP, SORT, FILTER, COUNT, and SUM. I think a partial MapReduce algorithm could run on each node, pulling fields and grouping results (“This author appeared 12 times in this data set on this node”), and then combining those results on the master node that made the request. Is this process infinitely scalable with any query? Of course not — you’re limited by how quickly you can get data to the master node, and how complex the operations are. It’s still incredibly applicable towards a large variety of use cases, however.

And Finally…

Data Warehousing is more than just an analytical tool for large businesses — it’s an integral component of many startups, especially in the social media and BI fields. This need is driving one of the largest revolutions in the DW industry. Anyone who knows Java or SQL can derive complex intelligence from massive datasets for a fraction of the cost a few years ago. Enterprise DW vendors, used to competing with Oracle, now have to compete with something that is both free, powerful, and highly scalable.

postnote: This isn’t about a complete death of the RDBMS. Just the death of the idea that it’s a tool meant for all your structured data storage needs.

Imagine someone told you that there was a piece of software out there which could change the way you do everything – and I mean everything. This code is so perpetually practical; it’s a Swiss Army Knife! Surely, it can handle the simple data storage and analytics you want to do on a ‘Social Media’ scale.  For example, you can:

• Store pieces of content for your website so it’s easy to manage
• Store content from EVERY website on the Internet
• Track your company’s financial transactions
• Track every credit card transaction for your multibillion-dollar banking conglomerate
• Store an address so it’s easy to find by a unique key
• Store every photo on Facebook so it’s accessible by a unique key
• Generate charts and graphs on how your business did in the last month
• Generate charts and graphs on key analytics for every piece of Social Media on the Internet.
• Analyze records of how people have accessed your website and attempt to simulate their behavior
• Analyze every click ever made on MySpace and optimize intrasite workflow.

…Well, in theory you can do the above.

Does it make sense that people can install the exact same software package and be able to do this, at all scales? Just throw bigger computers at the problem? It’s counter-intuitive to me, and it sure doesn’t work in reality. With the coming need to analyze things not on an enterprise-scale, but a Web Scale, social media is driving the final stake into the large analytical RDBMS (Relational Database Management System).

The ACIDy, Transactional, RDBMS doesn’t scale, and it needs to be relegated to the proper dustbin before it does any more damage to engineers trying to write scalable software.

IN THE BEGINNING…

Traditional relational databases (like those used in finance) evolved in the 70’s and 80’s as a glorified spreadsheet-and-file-cabinet. The metaphor prevails today.  RDBMSs have made the modern business possible: being able to slice, dice, and analyze data in a neat, intuitive format is immensely powerful.

The problem is that the price of processing data in an RDBMS scales exponentially with the amount of data – getting twice the processing speed on twice the data rarely costs only twice as much. No matter what you do, something has to give. Eventually, you can try to shard out databases (like Oracle, Greenplum and Vertica), but their solutions still require expensive hardware, custom hash algorithms, compression, brittle analysis structures, and the licenses are enormously pricey.  With those sorts of limitations, it’s difficult to tackle “Web-Scale” problems.

In addition, developers have become crippled by being able to only think of data in terms of Rows and Columns. There’s a multitude of database paradigms: Graphs, Trees, Objects, and so on. Furthermore, databases limit developers to SQL, which is great for certain kinds of set mathematics and not much else. In order to overcome fundamental limitations of DBs, things like Conditionals, Iteration, String Manipulation, and more have been hacked into what was at first an elegant set mathematics language.

Algorithms (such as MapReduce) which make data analysis scalable are highly unintuitive in SQL. The limitations in the environment lead to a limitation in engineers’ ability to solve “big data” problems.

People treat Databases as The Only Way To Store And Process Lots Of Data, and this leads them down the Road To Failure…

THE CRIPPLING

RDBMSs are really just a storage abstraction on top of some B-trees, bitmap indices, disk sectors, etc. With SQL on top, they enable some pretty powerful (if limited) analytical expressions.

However, there are a few fundamental problems:

• The row/column approach limits you to one static storage paradigm
• Updating indices does not scale linearly… the more data you have, the longer it takes to put it in. You can’t delete data if your DB gets too large in a production environment, so performance continually decays until The Datapocalypse occurs (or you throw more money at the problem).
• There’s an immense amount of overhead with transactions.
• The internals are totally opaque for most DB solutions. Optimization, required for any performant solutions, becomes a “black art”.
• Backups are quite difficult to do – real-time mirrored, or periodic deltas.
• Normalization is the only way to keep data sizes from exploding, but joins are incredibly expensive. It’s also a bitch to shard out indices.
• The initial low latency for complex analysis lulls the average developer into a false sense of security – “Oh, when we need a larger DB, we’ll just get a larger machine”.
• Sooner or later, you hit physical limits on how many electrons you can pull from your SAS drives, dozens of processor cores, and RAM.

The Social Media space is littered with the corpses of those who have attempted to scale their RDBMSs (Twitter has teetered on the precipice several times). But even the Big Boys have dealt with the horror of DB scalability:

• Google (and Yahoo) once ran off of 40,000 MySQL Boxes (later replaced with minty MapReduce/GFS/HDFS goodness)
• MySpace has about xx,000 Microsoft SQL Servers (with license fees) to serve their content
• Facebook was spending $1,000,000/month+ for specialized database hardware just to serve their photos
• A company uses FPGAs to make disk access faster for databases, since they’ve reached their scalability limit on hardware.

OUR PROBLEM

My team at Visible Technologies focuses on ways to perform quasi-OLAP on discrete document fields in near real-time. This is part of a Social Media metrics and analysis framework. Our corpus may be hundreds of millions of documents, but we only have about 10-15 fields we care about.

The current system is built on top of a multitude of SQL Server instances on $x0,000 boxes, each of which contain so much data that data aging is quite difficult. Not to mention the ever-increasing latency on queries which drive user experiences.

HOW WE’RE SOLVING IT

Like most people who try to scale RDBMSs, we don’t actually need most of the features a modern database provides. Most of them prevent proper “shared-nothing scale-out”, and slow things down.

Many of our analytics and back-end ingestion processes are being supplanted by elegant MapReduce algorithms on top of Hadoop clusters.  We still have a need for low-latency calculations over moderate size metadata sets (MBs, not GBs) which drive BI Dashboards, however.

One of the ideas we’re prototyping is a distributed Key-Value store that we can perform set mathematics on. For our analytics, we just need to support a handful of operations. This could be built on top of HBase, Tokyo Cabinet, Project Voldemort, or a handful of other distributed data stores. There’s more things we’re prototyping (perhaps based off vendor solutions), but here’s our most interesting approach:

WHAT WE’RE SCRAPPING:

• Transactions. Our data is written in from a Hadoop cluster in large batches. If something fails, we’ll just grab the HDFS block and try again.
• Joins. Nothing is more evil than normalization when you need to shard data across multiple servers. If we need to search on 15 primary fields, we’re fine with copying our data set 15 times, with each field a primary key for its table.
• Backup and Complex Replication. All of our data is imported from HDFS. If high-availability is a must, we can simply use Zookeeper to keep track of what nodes die, and then bring up a new one and feed it the data needed in ~ 60 seconds. With scales of hundreds of millions of documents, no one will miss a few hundred thousand for that brief period of time.
• Consistency. If our users are analyzing millions of documents, they’re not going to care if there’s 15,000 unique Authors, or 15,001.

REQUIREMENTS:

• Fast GETs at the expense of PUTs.
• Support for SUM, COUNT, GROUP, and FILTER. That covers 95% of our use cases.
• Sharding that works and handles rebalancing and additions of new nodes elegantly
• We don’t want to allocate space for data we’re not storing. In addition, we need to rapidly be able to scan through ranges of single data values – like “AuthorName”. Perhaps a columnar or graph-oriented paradigm would best fit our needs.
• Any ‘reasonable’ query needs to return its results in 10 seconds or less.
• Setting user expectations properly and giving limits to their ’sandbox’, so we can store a few TB of data in fast-access storage instead of 100’s of TBs.
• Plays nice with our Hadoop / HBase cluster.

THE END

The RDBMS as we know it is simply not a scalable solution – its jack-of-all-trades nature leads to a fundamental weakness with scalability. Over the next few years, I think a number of solutions to our problem will emerge. A large portion of data analytics can be performed with SUM, COUNT, GROUP, and FILTER on denormalized data.

I’ll keep you updated on our progress over the next few months.