We wouldn’t care as much about NoSQL if it wasn’t called NoSQL. Even the word itself is triggering existential crises among some very smart, seasoned people.

But why?

…because the term NoSQL encapsulates just about every phase of database innovation in the last decade.

The inspiration for this post came from reading Chip and Dan Heath’s book called Made to Stick. Their theories, plus much buzzing from the anti-NoSQL camp about how “this is just a phase like XML DB” and “you obviously aren’t smart enough to understand the RDBMS”, has given me some fresh thoughts on why NoSQL is such a controversial thing.

According to the Heath brothers, a sticky idea is one that’s: simple, unexpected, concrete, credible, emotional, and a story. Now… the connection between psychology, computers, and social science can get pretty vague for someone untrained in two of those fields (me), but I hope this article will at least spark some interesting discussion.

So with all that in mind, let’s examine why NoSQL is such an awesome idea — and why it scares us.

(if you want a data platform that’s scalable, real-time, AND has a query language, check out Drawn to Scale)

Simple

“NoSQL” is the essence of simplicity. It’s one word. Yet it implies so much. First of all, it’s kind of modern-sounding, which is appropriate. But it’s also based on something that people know about (and in some cases, are quite happy to imagine being without).

When introducing new ideas, it’s best to build them upon previous ideas. It helps form a bridge and sets some expectations for what we should expect to see next. For example, let’s learn about Tapirs, because they’re badass:

“A tapir is a browsing mammal with a short, prehensile snout, belonging to a family of odd-toed ungulates. Tapirs inhabit jungle and forest regions of South America and Southern Asia. Most tapirs are about 7 ft long, stand about 3 ft high at the shoulder, and weigh about 500lbs. Their diet is omnivorous and they eat up to 85lbs of food a day.”

Now, if I were to ask you, “Are tapirs cuddly?”, you’d probably have to ponder that for a while. There’s nothing relevant to anchor your thoughts about it.

But what if I said, “A tapir is like a pig with a long snout that lives in the jungle.”?

Then you’d say, “Hells yeahs, Tapirs are cuddly”. Then you’d also know that they probably eat vegetation with that snout, and live in dense undergrowth for protection from predators.

Similarly, “NoSQL” immediately implies a database which does not use SQL as an interface. This is powerful. It raises many questions;

“How do I access data?”
“How does someone maintain it?”
“How can you store data without tables and rows?”
“Are there schema?”
“What do I click on?”
“Why the hell would someone wanna do that?”

There’s always a loss of fidelity with simple. Technically, any database could support SQL as a language (Drawn to Scale’s platform will). But you get a good start, and that’s worth quite a lot.

The word NoSQL fundamentally challenges everything we assume about databases. That’s worth embracing.

Unexpected

Unexpected events grab our attention, and keep it by raising questions. Humans have evolved to react and pay attention to changes in the environment. Imagine a class being taught on tapirs. The professor starts up a slide deck, shows a picture of the critter, and says “Tapirs are mammals.” zzzzzzzzz….

Now instead, imagine if the professor darkened the room and said “Tapirs are responsible for 78% of the world’s economic output.” Whoa! Serious business! You’d be interested. This is unexpected news. And it raises a mystery: why are tapirs so important? You’d need to know why.

Likewise, the word NoSQL is unexpected and creates a mystery. The concept steps on more than 25 years of data traditions.

“You can’t have no SQL… it makes no sense!”

Until now, using an RDBMS was as ordinary as wearing underwear. It’s just what you did, because why imagine the world any other way?

The logic goes: This is the way it is done. Data is stored in a database. It is structured as rows and columns. You use a language called SQL to access and manipulate that data. If you want it to go faster, hire consultants or buy a bigger server. If you need to do something else with data outside of the SQL language — tough luck. Go have fun with that.

The database industry is $150 billion. And most of that is spent on RDBMSs. Entire careers and corporations are built around tweaking queries and tables to get a few more % of space or speed.

NoSQL is terrifying. Everything these people have known for decades just became a little more irrelevant. RDBMSs are based on computer science fundamentals — but if you don’t know those fundamentals, it’s hard to understand why you’d need any other kind of database.

It’s always a bit surprising when the pillars of existence start to crack.

Concrete

It seems fairly intuitive that you’d need to make ideas concrete for them to be memorable and communicable, but engineers especially struggle with this. Staying abstract and technical allows us to communicate with each other much faster, at the expense of talking to anyone else. There’s a reason why whiteboards are so prevalent.

As we get more advanced in our studies of a topic, we tend to talk about it in a more abstract way. This makes it harder to communicate with the uninitiated. As an engineer founding a startup, I talk to a lot of CEOs and CTOs. I am painfully aware of this.

Yes, it’s hard to remember what it’s like to not know something… but there’s a point where it just comes off as wilful vaporousness. If you want to teach math, you don’t start up with lambda calculus and explain the recursive fundamentals of subtraction. You present something real, “If I have three Pikachus and put two in a blender, how many do I have left?”. That’s easier to remember.

Concrete ideas are more than memorable; they’re easier to communicate. Concrete ideas are like a common language. At my previous workplace , product management asked us to build something that “lets people explore all social media data. Like using Google, but you can drill down on things like Authors”. Using that as a concrete goal, my engineering team extrapolated “build a distributed, real-time database capable of faceted search that is fast no matter how much data you put in.”

It’s tricky to think about software fundamentals in a concrete way.

The suggested replacement term for NoSQL, “AltDB”, isn’t too useful. There’s nothing concrete about it. “Alternate Database? Alternate to what? Does that mean MySQL is an alternate to Postgres? Why do I need that?”

When I say “NoSQL”, something concrete is in there. “Well, shit. It’s not SQL. What do I do now?”

This inspires communication and thought. You may not know how a database does synchronous indexing, but you know it uses SQL, and that’s pretty important. Without that… there’s a lot of questions to be answered. And you won’t easily forget the term NoSQL.

Concrete things stick around much longer than abstract ideas.

Credible

NoSQL is especially powerful because many people are talking about it — and those who started the movement had some serious horsepower behind them. Not only that, but the movement seems to be pissing off the “right” kind of people.

External credibility is divided between authority and anti-authority figures. Both are important.

Authority figures have credibility. If the FDA tells you not to smoke because it’ll cause cancer, that’s believable. They’re a legitimate government institution.

Likewise, anti-authority figures add credibility to an idea. The classic example is the story of the Marlboro Man from old cigarette advertisements. Marlboro cigarettes projected a “tough, rugged, cool cowboy image”. In the 80s, the man was smoking through a tube in his throat and very much the opposite of a tobacco commercial. His misfortune helped strengthen the idea that maybe something was wrong here.

The NoSQL movement has advocates from academia and industry. There are quite a few academic papers about alternative data stores like Hadoop. In addition, Google led the NoSQL charge with their BigTable paper a few years ago. Google has pretty much the biggest “engineer cred”. If they say SQL doesn’t scale for their needs, that’s the truth. More and smaller companies are getting on the NoSQL bandwagon, like Twitter, LinkedIn, and Digg.

What’s just as interesting is the type of people NoSQL is aggravating. The folks who have the most invested in the status quo are not happy. They provide anti-authority credibility.

Database Administrators, 3-tier webapp designers, SQL tuners, analysts, data warehouse vendors, and more are all blogging furiously about the dire repercussions of non-traditional databases. This is often thinly veiled “concern” about “using untested technology”, or implications that engineers don’t REALLY understand relational databases, or they’d be using them.

Emotional

The strongest memories we have in life are the ones with the most intense emotion. NoSQL seems to elicit strong emotions from detractors and advocates alike.

NoSQL the word is not inherently stirring. But it strongly resonates with engineers who regularly feel crippled by the wrong tools for their job. Guys struggling with SQL problems seem to grasp the NoSQL term immediately. If you’ve been a similar situation, you know what it’s like. There’s a fundamental “wrongness”, like having to write with a different hand. It gnaws at you, even as you sleep. The world would be a fundamentally better place if you could just do it right.

On the other side, NoSQL elicits strong emotional responses from its detractors. It sort of eats at the core of their being. When you can no longer define yourself as a “database engineer” by knowing how to intimately tweak around the query planner, you begin to question tenets of your professional existence.

There’s no-one more unhappy than an engineer who can’t solve problems the right way. It’s a core part of our profession — build something that people want, the right way. Most engineers are builders, it’s why we studied this in college (or dug through the textbooks ourselves). When the RDBMS keeps you from building software and doing your duty as a professional, frustration and resentment begin to build against it.

It may sound corny, but the RDBMS paradigm has prevented people from realizing their potential and conquering intellectual challenges. NoSQL isn’t SQL — it’s not “that horrible thing”. The word invokes a sense of optimism, that these hard problems which kill RDBMSs *can* be solved through solid engineering.

Story

NoSQL is not a story of itself. It’s just a word. But it came about from the experiences of so many people. As we’ve seen, these experiences are a common thread between everyone who is interested in NoSQL.

The problems usually faced are:

• Large data volumes
• Non-row data structures (graphs, columns, images)
• Slow processing and query times
• Many simultaneous users
• Increasingly complex data
• Elastic data demand
• Affordable performance/scalability

When your business has one of these problems, you either have to change your business model, or adopt a NoSQL solution. That always leads to a story.

I’ve told my story in previous blog posts, but I’ll summarize it here. It’s not a special story — in fact, Drawn to Scale was inspired by hearing so many stories just like this.

My first week with my previous employer was eventful. It was the last week of 2007. The company provided business intelligence and brand management on social media — a worthy goal. Imagine being able to track topics, influencers, and everyone talking about your brand on the Internet. That’s incredibly powerful.

It was also utterly impossible with the company’s infrastructure. It would always be impossible. And they didn’t know it. So I had to tell them my first week there.

The workflow was basically:
1. Collect blog posts and tweets
2. Extract entities (Author Name, Comments) from unstructured text
3. Store this data in Microsoft SQL Server
4. Rework the data about 30 different ways to power some dashboards
5. Allow users to slice-and-dice, aggregate, and filter data.

All this was being done with a handful of inexpensive boxes and one pricey database server.

To collect about 10,000 sites a day, clean the data, load it, and then generate dashboards took about 24 hours. There was ~ 1 GB of data daily. That’s not “the entire social web”. That wouldn’t even cover Blogspot.

Not to mention, it had extreme reliability and performance issues. The server would go down once a week. It would slow down enough to be unusable to power the website. Customers would quit.

Many engineers insisted all we had to do was add more disks and processors. It helped for a while — cutting processing time in half, doubling uptime. For 10x the cost. But more data would flow in, and the system would collapse again. This had been happening for years (and would continue).

Eventually I just asked myself, “How did Google do this?”. That led me to the BigTable whitepaper, and then Hadoop and HBase. A quick prototype was powerful enough to earn me the right to hire a team, and I got some awesome people. We then built some distributed indexing and search. It also never crashed with high data volumes.

The platform we built there (which inspired Drawn to Scale’s platform) used a bunch of $1500 boxes to process in 5 minutes what used to take 10 hours. And queries could be updated in 250ms, instead of 24 hours. It was amazing, and never would have been possible with an RDBMS.

This story isn’t unique. You’ll see similar threads among many NoSQL users… even Google’s BigTable.

Due to the growth of the web and its massive data, and data-centric business, you’ll see more and more NoSQL solutions.

Conclusion

NoSQL isn’t controversial because of a few bloggers and tweets. There’s a ton of nice people in all ranges of opinion.

NoSQL is controversial because it fundamentally challenges so many aspects of business and software engineering.

Someone said to me recently: “NoSQL is popular in the same way NoDeath or NoTaxes would be popular. It’s the opposite of a well-known symbol for what some think is wrong with the world.”

I hear complaints about how you can’t frame things in terms of what they *aren’t*, like this post. But that is simply false. One of the best ways to highlight what something is, is by saying what it isn’t.

NoSQL certainly does that.

(P.S. if you want a data platform that’s scalable, real-time, AND has a query language, drop us a line at Drawn to Scale)