Auto-Incrementing Keys with NoSQL Solutions

Today I’d like to follow up on one of my previous posts on Couchbase Keys and talk a little bit about creating auto-incrementing keys in NoSQL solutions. I’ll focus on Couchbase as that is one of the NoSQL solutions I currently use but the same principles can also apply with other NoSQL solutions.

If you are coming from a SQL environment you are probably pretty familiar with auto-incrementing primary keys but in a NoSQL database there is no such thing as a primary key or auto-incrementing key so you have to make your own. There are also other key types you can create within a NoSQL database but what happens when you can’t come up with a good key solution? Let’s use a typical scenario that most websites all have; user accounts. You might think that you can use the username or email address or a combination of a username and email address or a phone number or the users first and last name or some other user account property but there are a few problems if you decide to go this route. What happens if the user wants to change their username or updates their email address or gets married and changes their last name or changes their phone number? You would then have to update your document key which could then lead to cascading changes and other issues. What happens if you miss a document that needed to be updated with the new key? You could end up with a real mess on your hands. This is where an auto-incrementing key comes in handy.

Another major benefit of using the auto-incrementing key is that it makes it really easy to enumerate through all of your documents. This also eliminates the need for you to use a view to retrieve your documents. You can read more about why using a view is not a great idea by checking out my previous post on Couchbase keys. I’ll show you an example of enumerating through all your documents later on in this post.

The implementation of an auto-incrementing key is going to use what’s called the Lookup Pattern. The first thing you want to do is create a master counter document. This document will hold the current number of documents that have been created and will be used to get the next id value to use in your key. To create your master counter document you will want to create a unique key; let’s go with something like user_account::count. This will make it easy to lookup this document and it is self documenting for what we are going to use it for. In Couchbase there is a client function that we can use called Increment to increase the value this document will hold. We only need to place an integer value in the document as the Increment method will handle incrementing this value when the increment client method is called. Let’s see a couple code examples below on how to create the master counter document. I’ve provided examples in both C# and Ruby.

C#

// initialize the counter document. this only needs to be done once.
var client = new CouchbaseClient();
client.Store(StoreMode.Add, "user_account::count", 0)

Ruby

# initialize the counter document. this only needs to be done once.
client = Couchbase.new
client.add("user_account::count", 0)

Once the master counter document is created you can increment the number in the document using the following line of code:

C#

current_counter_value = client.Increment("user_account::count")

Ruby

current_counter_value = client.incr("user_account::count")

As you can see it’s a very simple method to increment the value in the master counter document. Also, you don’t have to start the counter at 0, you can use whatever number you want. So if you want to start at 1000 just pass 1000 in instead of 0. The return value is the latest counter value so if the existing value before calling the Increment method was 5 then the return value would be 6. Once we have the latest counter value we can then use that value to create our user account document key. Continuing with the value of 6 we will then create a new user account document using a key of “user_account::6”. The next document key we create will be “user_account::7” and so on. Below is an entire example of creating the master counter document, incrementing the value and then creating a new user account document.

C#

// create the master counter document
var client = new CouchbaseClient();
client.Store(StoreMode.Add, "user_account::count", 0)

// increment the counter (This isn't actually necessary since we just initialized the document but 
// for illustrative purposes and completeness I wanted to show all the steps I've talked about so far.)
current_counter_value = client.Increment("user_account::count")

// create the document key of "user_account::1" since we just incremented our master counter document value
var docKey = "user_account::" + current_counter_value;
var user_acct = new UserAccount
{
	Key = docKey,
	FirstName = "Skippy",
	LastName = "JohnJones",
	Email = "skippyjj@email.com",
	PhoneNumber = "555-555-5123"
};

client.Store(StoreMode.Add, docKey, user_acct);
client.Disconnect();

Ruby

require 'rubygems'
require 'couchbase'

# create the master counter document
client = Couchbase.new
client.set("user_account::count", 0)

# increment the counter (This isn't actually necessary since we just initialized the document but 
# for illustrative purposes and completeness I wanted to show all the steps I've talked about so far.)
current_counter_value = client.incr("user_account::count")

# create the document key of "user_account::1" since we just incremented our master counter document value
docKey = "user_account::#{current_counter_value}"
user_acct = new UserAccount {
	:key => docKey,
	:first_name => "Skippy",
	:last_name => "JohnJones",
	:email => "skippyjj@email.com",
	:phone_number => "555-555-5123"
}

client.add(docKey, user_acct);

client.disconnect

That’s all there is to it to get started using an auto-incrementing key with Couchbase. You might be asking yourself how the heck do I lookup a document. Well I’m about to show you.

C#

// get the current counter value
current_counter_value = client.Get("user_account::count");
// form the key to get the latest user account document
var user_acct = client.Get("user_account::" + current_counter_value);

Ruby

# get the current counter value
current_counter_value = client.get("user_account::count")
# form the key to get the latest user account document
user_acct = client.get("user_account::#{current_counter_value}")

Once you have things up and running with this pattern, enumerating through all your documents is pretty easy. Here is an example on how to do that using the user_account info shown above.

C#

current_counter_value = client.Get("user_account::count");
for (int i = 0; i < current_counter_value; i++)
{
	var user_account = client.Get("user_account::" + i);
	// do something with each user_account document   
}

Ruby

# get the current counter value
current_counter_value = client.get("user_account::count")
[0..current_counter_value].each |i| do
	user_account = client.get("user_account::#{i}")
	# do something with the user account document
end

So that’s pretty much it. Overall, using the atomic counter and the lookup pattern in a NoSQL solution is a great way to create keys that can be used to retrieve the document in a very fast, efficient manner, while still creating a unique key that is easy to remember. Also keep in mind that atomic counter operations are always executed in order and are the fastest to perform get, set, increment and decrement operations. So now you have a simple way to generate unique auto incrementing keys just like you do in SQL databases.

Posted in C#, Couchbase, Ruby Tagged with: , , , , , ,

Leave a Reply