A Practical Introduction to RethinkDB

In this tutorial we'll cover how to work with RethinkDB, a document-based NoSQL database. This guide spans from RethinKDB installation to developing a simple application in Node.js that will show the database's real-time capabilities.

RethinkDB is an open-source database that, unlike traditional database systems, stores the information in JSON (JavaScript Object Notation) format. Because of this, it's considered a NoSQL (Not Only SQL) database, such as MongoDB, Cassandra, and CouchDB.

NoSQL databases share the following features:

• No required schema or table structure for storing the information.
• Distributed architecture so they are easy to scale horizontally (in clusters)
• Typically stray from the ACID model (Atomicity, Consistency, Isolation, Durability) of transactions. Follow the CAP theorem instead, which states that a distributed system cannot simultaneously provide the following guarantees:
  • Consistency (the same data can be read by all the clients of the system)
  • Availability (even in case of failure, clients receive a response)
  • Partition Tolerance (even in the case of network failure, the system continues to operate)

Many NoSQL databases sacrifice consistency in favor of availability and partition tolerance, offering the concept of eventual consistency, where the changes received in one part of the cluster are eventually replicated to the rest of it. Other NoSQL databases don't mind sacrificing availability or partition tolerance to prioritize consistency.

In the case of RethinkDB, it chooses to keep data consistent (over available) in the case of a network partition. This means that after a write operation, a read operation will always return the latest version of the data.

To make this work, in RethinkDB the data is assigned to a primary server (or individual partitions called shards), and all queries are routed to it. This is the default behavior, but if the data is replicated to other nodes within the cluster, we can instruct the query that it is acceptable to return out-of-date information. As a result, RethinkDB can direct the query to the closest available replica and not necessarily to the primary.

Besides all this, RethinkDB has the following unique features:

• It's easy to use and configure in a cluster with replication and sharding (the process of splitting a table across multiple machines).
• It has powerful query language, ReQL, that even supports joining tables (common in traditional database systems).
• It can also perform map-reduce and geospatial queries.
• It allows us to build real-time applications easily by publishing the table's changes as they occur. This is perhaps RethinkDB's most important and most popular feature.

Now that you know what is RethinkDB all about, let's install it.

Basically, you have three options to start using RethinkDB:

• Using Docker. There's an official RethinkDB repository on Docker Hub. If you have Docker version 1.11.2 or greater, just execute a command like the following:

Docker will automatically download the image and start an instance.

• Using an official installer package. RethinkDB has installers for Linux, OS X, and Windows. For Linux, there are official packages for Ubuntu, CentOS, and Debian, but you can find community packages for other distributions. For example, in Ubuntu you type the following commands:

The first line will export the variables from the file /etc/lsb-release. One of the is $DISTRIB_CODENAME that contains the version of Ubuntu you're running, and it's used to build the repository URL and save it with the tee command. The second line downloads the PGP key used to sign the RethinkDB packages and adds it to your system. The third command updates the package list, and the last command downloads and install the database.

If you're using OS X 10.9 (Mavericks) or above, maybe the easiest way is to use Homebrew, a popular package manager for OS X:

• Building it from source. Use this only if the previous options aren't available, or if you want to customize the database in some way, or if you want to run multiple versions of RethinkDB at the same time. To build from source, you need to install a bunch of dependencies, download and extract the source tarball (or clone it from Github) and build it depending on your platform. Find more information here.

Following any of these methods will successfully install RethinkDB. To start an instance of the database, just execute the following command:

This command will:

• Create a directory to store the data in your user directory (/home/user/rethinkdb_data for example)
• Start listening for client connections on port 28015
• Start listening for cluster connections on port 29015
• Start a web administration interface on port 8080

By default, this server will only listen for local connections, which means that clients on other machines won't be able to access the database (for security purposes). If you want global access, you can bind to all IPs with the option:

If you want to start a second instance to create a cluster on a single machine, execute this command:

This will create another instance on port 28016 (--port-offset 1), use /home/user/rethinkdb_data2 as its data directory, and connect to the first instance to create a cluster (--join localhost:29015).

If you're using two machines, start RethinkDB on one machine with the command:

And then, assuming that the IP of the first machine is 192.168.0.1, start the second instance on the other machine with the command:

For a complete list of options that can be used on the command line, type rethinkdb --help or read this page.

If you want to run RethinkDB as a service, you have to create a configuration file for initialization options and use init.d or systemd for Linux, or launchd in OS X (if used Homebrew for the installation, this is already done for you). You can refer to this page for more information.

Now that the database is running, we can go to https://localhost:8080 (or any IP your instance is bound to) to access the RethinkDB web interface:

If we click on the Tables option at the top of the page, we'll see the databases and tables we have on our server:

Since this is a new instance, we don't have any databases yet, so let's add one by clicking the +Add Database button and entering the name myDb and then adding a table by clicking the +Add Table button and entering the name myTable:

If we click on the Data Explorer section at the top of the page, the web interface will present an interactive shell where you can issue commands, for example, to query the table you just created:

We even get syntax auto-completion and contextual help.

If you're more of a command line person, there are two excellent projects that provide a Command Line Interface (CLI) to RethinkDB. They are both made in javascript with Node.js so you'll need to have node.js and npm (Node Package Manager) installed.

The first one is recli that can either take a ReQL expression as an argument or be used as a REPL (read–eval–print loop, or in other words, an interactive shell):

It uses configuration files and remembers commands that you run in the between REPL sessions.

The second one is ReQL CLI that just work as a REPL but has some helpful built-in commands:

Now that you know how to execute queries, let's know more about this ReQL language.

Just as in a traditional database, RethinkDB works with tables. But instead of working with records, it works with documents, which are objects written in JSON, such as in the following example:

As you can see, a document begins and ends with curly braces, it has keys and values separated by colons while key/value pairs are separated by commas.

The key is a string. Any Unicode character with the exception of the null character (\0) and the reserved characters . and $. The value can be of type:

• Number (any real number)
• String (any UTF-8 string)
• Binary object (like a file or an image)
• Date/Time (with millisecond precision)
• Boolean (true/false)
• Null (to explicitly denote the absence of a value)

Arrays and objects (which actually are embedded documents) are also accepted:

RethinkDB also has specific data types, such as streams (lazy loaded lists) and geometry data types (points, lines, and polygons).

ReQL is the language to make queries in RethinkDB using JSON documents. As you saw in the previous section, a simple ReQL query looks like this:

Let's break down this line of code. Think of ReQL as a fluent API (where functions are chained to provide readability). You start a query with the identifier r and with the . operator. Then you add operations at the end.

For instance, you choose a database: db("myDb")

If you don’t specify a database, the default one is test. table("myTable") selects the table.

This will return all the documents of that table.

But we don't have any document, we need to add one with insert():

The output will look like this:

Now if we query the table, the output will be:

RethinkDB generates a UUID (Universal Unique Identifier) as a primary key for the document if none is provided (with the key "id").

You can also insert more than one document in an insert command using an array (which is faster than inserting each one individually):

The output:

If you want to get a document, you can use get() passing its primary key:

The output:

If you want to get n number of documents, you can use limit(). For example, to get a single document (you might not get the same document every time):

Using filter() we can get all the documents that satisfy a condition. For example:

Returns the document with a name field with value Andrew. Documents that don't contain a name field are skipped.

RethinkDB is case-sensitive, so the previous query is different from the one below:

You have to be careful about data types. The following query won't return any document because the field age is defined as a number, not a string:

If you prefer the standard data table format, you can rewrite the first query as:

r.row("name") refers to the value of the field name of the current document. This syntax allows for more complex queries. For example, to get people over twenty-two years old:

To filter based on a field of an embedded object:

To filter based on two or more conditions:

To get people who are interested in reading:

Using pluck(), you can indicate the keys to be returned, for example:

This query will output:

To sort documents by the name field in ascending order:

To sort in descending order, add r.desc() in the orderBy()field:

There are two cases for making updates. When the document to be updated does not contain the field to be updated, the update is added to the document. For example, if you want to add the field gender to all the documents of myTable, execute this query:

The output will be:

If the field already exists, the value will be updated only:

The output:

To delete a document or set of documents, first you need to select them and then call delete():

Or:

The output of both commands will be:

To delete all the documents of a table:

This way, count(), or the number of documents that the table contains, will return zero:

We can delete the table itself:

The output:

And even delete the database:

The output:

Recreate the database with:

The output:

Now let's create a table with a more meaningful name:

The output:

Let's check that the database was created correctly by accessing the list of all databases on the server:

We can get the list of all the tables in our database:

You can also create (secondary) indexes on the fields of a table by using indexCreate():

This will create the index name for the name field.

For compound indexes, indexCreate() takes two parameters, the name of the index (name_age_index) and the field or array of fields that make up the index ([r.row("name"), r.row("age")]):

While get() is used with the primary key (index) of a document, getAll() is used to get documents with a (secondary) index:

getAll() takes the value of the fields and optionally, the name of the index to use.

The last feature we are going to cover is relationships between documents. There are two ways to model relationships between documents in RethinkDB:

• By using embedded arrays.
• By joining tables like in traditional relational database systems.

The interests field is an example of an embedded array:

However, we can model this relationship as separated tables. One way would be:

In this case, you need to create a secondary index on the foreing key:

Then, you can use the eqJoin() command as follows to join the tables:

The first parameter is the field of the left table (in this case people) that will be used to make the join, the second parameter is the table to join (in this case interests), and the third (and optional) parameter is the index in the table to join (again interests) to make the link.

The output will be:

The result set of eqJoin() is an array of objects. Each object returned set will be an object of the form { left: <left-document>, right: <right-document> }, where the field left contains the information from the left table in the query (in this case, people), and the field right contains the information from the right table in the query (in this case, interests). You can use zip() to merge those two fields together. See the last section of this page for more information.

Another way to model the relationship can be:

In this case, since we are simulating a subquery, the query is more complicated:

Many ReQL commands accept a function as a parameter. merge() joins two or more objects together to construct a new object with properties from all merged objects, in this case, tables people and interests.

The function used by merge() takes the document to be processed as an argument. It returns a new field that will be merged with the rest of the fields of the document of people (since this new field is called interest_ids, it will replace the original interest_ids field.)

Inside the function, the interests table is filtered, only if the id of an interest document is present in the interest_ids field of the people document, it will be returned.

The result of this query is:

RethinkDB has official client drivers (APIs) for the following languages:

• Javascript (Node.js)
• Ruby
• Python
• Java

However, there are community-supported drivers for a lot more languages.

For now, we are going to focus in the Node.js API, so you'll need to have installed the latest stable version of Node.js.

With NPM, the package manager used by Node.js to install dependencies, we can execute the following command to get RethinkDB's module:

This command will download RethinkDB and its dependencies. By default, NPM works in local mode, so it will install these files into the node_modules directory of the current directory.

RethinkDB API is very similar to ReQL. To use it, we import the module with:

And connect to the database with:

Optionally, r.connect also accepts the name of the database and authentication information.

Node.js is asynchronous by nature, so, to make a query, this function returns a promise, or an object that represents the eventual result of an operation. RethinkDB uses the [bluebird][bluebird](https://bluebirdjs.com/docs/getting-started.html) library for promises.

If the connection succeeds, the function passed to then() will be executed; otherwise, the function passed to error() will be executed.

If you don't want to work with promises, we can use a callback function as the second argument to r.connect() to have the same functionality:

Most functions of the API work by returning a promise or by taking a callback function as an argument.

Either way, once we have a valid connection, we can execute a ReQL query:

Notice that the only difference with executing queries in the Node.js API is that they require the run() function at the end to execute. This function takes the connection object as an argument and, optionally, a callback function that is executed when the query is complete. If no callback is provided, a promise is returned (like in the example).

RethinkDB's real-time functionality relies on something called changefeeds, which allow clients to receive changes on a table or document as they occur.

Almost any ReQL query can be converted into a changefeed with the changes() command.

As you can see, this query returns any changes made to people returning an object with two fields. old_val is the old version of the document, while new_val is the new version of the document. On an insert, old_val will be null; on a delete, new_val will be null. On an update, both old_val and new_val will be present.

Using Node.js:

changes() returns an infinite cursor that you can iterate to get the changes (with the same old_val/new_val object format).

You can also monitor a single document or a set of documents:

Depending on the frequency of the changes, you may want to control how often the changes are sent. You can do it with the squash option, for example:

The values of this parameter are true, false and a numeric value:

• With true, when multiple changes to the same document occur before a batch of notifications is sent, the changes are “squashed” into one change.
• With false, all changes will be sent to the client.
• A numeric value n is similar to true, but the server will wait n seconds to respond in order to squash as many changes together as possible to reduce network traffic.

If the database server becomes unavailable, the changefeed will be disconnected, and a runtime exception will be thrown.

Changefeeds allow us to build real-time applications, where users receive information as soon as it is available (with extremely low latency). A good example of this type of application is Twitter; tweets appear as soon as they get published, without users needing to refresh the page.

One solution to the problem of building real-time applications in the web is to continuously send requests to the server, waiting for a response. This is called polling. As you can imagine, polling is very inefficient.

With the arrival of HTML5, WebSockets has replaced polling to become one of the most popular ways to implement real-time functionality. Websockets is a technology thta opens a bidirectional communication channel between the client and the server so this can push data directly to the client when needed.

A popular library that implements WebSockets is Socket.IO. This library, combined with changefeeds, will allow us to build real-time web applications easily in Node.js.

We are going to build a simple voting app. The theme here is movies, but you actually can input any text you want. This app will not contain have authentication, vote throttling, or vote-base sorting.

It has been kept simple on purpose to demonstrate the integration of RethinkDB's API and how to develop a real-time application.

The stack is also simple: RethinkDB + Node.js + Expresss + EJS (as templating language) + Socket.IO + jQuery.

The full code and installation instructions are on this Github repository.

The final application looks like this:

This is the directory structure and files of the application:

• controllers/index.js contains the route configuration of the application. It receive the requests and call the model to send a response.
• models/movies.js communicates with RethinkDB to implement the database operations.
• node_modules contains the dependencies of the application.
• public/style.css contains the CSS classes to style the web page.
• public/voting.js contains the client-side javascript to respond to click events, form submision and receive Socket.IO events.
• views/index.ejs is the template of the web page.
• app.js is the main file of the Node.js application.
• config.js contains the information to connect to the database and the port where the server will listen for connections.

Let's start by creating a directory for the project, rethinkdb-example (or whatever you want to call it) and cd into it:

Execute npm init to initialize the directory as a Node.js project and enter some information about the project:

This will create a package.json file that contains the configuration of the project. Now, let's add the dependencies of the project to this file by executing this command:

In addition to downloading the latest versions of all these dependencies to the node_modules directory, the --save option of this command will add them to package.json:

In this file, you can also set up a shortcut to start the application so instead of executing node app.js, you execute the more generic command npm start:

For the config.js file, let's define (and export) the following variables:

If they are not defined as environment variables, they take the default values provided in this file (change them if you want).

In the main file of the application, app.js, import the modules we'll use:

Next, configure Express to indicate that we will receive JSON from the client and define the directory where the view templates can be found, the path for the static resources, and the view engine to use (EJS, which is a simple template language to generate HTML):

Next, we define the routes of the application, contained in the file controllers/index:

The routes are:

• / That will present the main page of the application, which presents the movies registered.
• /movie To register new movie via a POST requests
• /movie/like/:id To like a movie given its identifier
• /movie/like/:id To unlike a movie given its identifier

We can see those routes implemented in the file controllers/index:

Back in app.js, we then initialize the server to listen for connections in the port specified in the config.js file:

To avoid errors and minimize our number of commands, let's automate the process of creating the database and table used by the application.

In the file models.movie.js we'll have a setup method to perform these actions that will take a callback as an argument. The callback will be executed at the end when everything is set up -- more on that in a moment.

Let's also import RethinkDB's module and the configuration file:

Now let's connect with the RethinkDB database:

config.database (defined in config.js) just happens to have the same format required for the connect() method to connect to a database. To verify if the database exists, let's try to create it. If an error is thrown, it means that the database already exists:

After this, we can be sure that we are connected to the database. But we are working with promises (an operation that will be completed in the future), so we have to use the finally() method, which will be executed after the promise is fulfilled, even in the presence of an error:

We test if a table exists by performing a simple query. The name of the table is held in the MOVIES_TABLE variable. This time, using a callback instead of a promise, we'll create the table (in case of an error) or get the result of the query (in case the table already exists).

The structure of the documents stored in the movie table is very simple, we'll just store the title and the number of likes and unlikes, for example:

Now, remember that we are working with asynchronous operations, so we need a way to know when any of these operations is done. As such, we need to save the promise returned by these operations. Then, with the promise references in hand, we can hook up the code that will be executed when is safe to assume that the movies table exists.

This last piece of code will set up a changefeed that will monitor the table for changes:

In a change notification, the callback passed to the setup() function as an argument will be executed, passing to it the actual change.

This callback function is defined in app.js:

The job of this callback function is to emit a Socket.IO event to the client when RethinkDB's changefeed notifies us about a change. Remember from the previous section that the object returned by the changefeed has two fields, old_val and new_val. If both fields are set, we are receiving an update. If only new_val is set, we are receiving a new movie. Based on this, we send a different event.

These events are received by the client. public/voting.js is the file that contains the javascript code on the client side. Using jQuery, we'll set up Socket.IO to listen for events when the HTML document is ready:

As you can see, when an update event is received, we set the number of likes and unlikes. When a new movie is received, an HTML element is added to the movie container.

This way, when a user navigates to https://localhost:3000, the following code (from controllers/index.js) is executed:

model.getMovies() gets the documents of the movie table:

toArray() is a function that converts a cursor to an array, which is passed to a callback that renders the file index.ejs with it:

index.ejs is a simple HTML file that uses EJS as templating language and presents the movies retrieved (in no particular order) along with a form to post a new movie.

In EJS, the <% ... %> tag is used to control flow. In this case, to iterate over the movies collection:

Meanwhile, the <%= ... %> tag outputs the (escaped) value of a variable:

When you submit the form, this snippet of code in public/voting.js is executed:

It checks that we actually input a movie title (otherwise, an alert is sent), and then, an AJAX (Asynchronous JavaScript And XML) POST request is made to /movie. In the server side, the following code is executed:

Here a JSON object is constructed and sent to model.saveMovie() to be saved.

Finally, when we click the like or unlike buttons, the event is received by the javascript code on the client side (in the case of a like, but for an unlike is almost the same code):

Since the element (the movie) might not exist when at the time the click event was bound (because it was added later), we have to bind the event the container (.movie), which always exists, and then select the child of that element that will trigger the click event (span.btnLike). This way, the event will be received correctly.

Then, the line var movieId = $(this).parent('div').parent('div').parent('li')[0].id; will navigate through the DOM (Document Object Model) to find the element that contains the movie identifier. Since the "like" button is the source of the click event, the search starts relative to this element (that's why we have to go up three levels).

Once the request reaches the server, the following route is executed:

Since "like" and "unlike" are very similar, we extract the functionality of both to the private (because is not part of the exported section of the module) vote() function:

The action parameter contains the name of the field to update, so in the model.updateMovie() method:

To perform the update, we use a function instead of an object. Remember that in ReQL, some operations can take a function as an argument. As a result, we can dynamically specify the field to update with:

The movie document to be updated is passed as an argument to the function so we can get the current value and add one ("like" or "unlike") to set the new value.

This covers all the functionality of our basic application. To run it, start RethinkDB and then type npm start.

This concludes this introduction to RethinkDB. Once again, the code for the whole example application is on Github.