A quick look at the Redis source code

Having been writing predominantly Java and Scala for the last 7 years, my C skills are pretty rusty.  In fact they’re practically non existent.  Apart from the occasional hack I’ve not had the occasion to write much C since University.  There is a widely held opinion that reading other people’s code is an excellent way to learn, particularly if those people are experts or the codebase is held in high regard in terms of its quality.  I’ve decided to take a look at one such codebase.

redis-logo

Redis is an open source data structure server written in ANSI C.  ”Data structure server” is another way of saying really, really neat key-value store. Not only can you store simple values like strings against keys but also hashes (or maps, or dicts even), lists, sets and sorted sets.  We use Redis a lot at Top10, mostly for indexing hotels in (near) real time depending on their availability and price for the dates the user is searching on.  I’ve also discovered that it has a pretty easily understandable code base, even for a C newbie like myself.  The code is cleanly written, relatively small (around 45k lines of code), mostly single threaded, and with few dependencies.  The dependencies are all included with the source making building it as simple as cloning the repo and typing make.

I decided to dive straight in to the code by adding a new command to Redis.  Something simple that will give me an idea of how Redis handles a command and dispatches a response.  A command rand that accepts a single integer argument max and returns a random integer between 0 and max (exclusively).  Not an ideal use of a key value store but implementing it should be instructive.  I certainly won’t be submitting a pull request!

Disclaimer – as mentioned before I’m by no means an expert in C so take all the code and interpretation of code here on those terms.  Also I’m linking to the unstable branch of Redis on github so the links may be just that – unstable.  You’ll probably get more out of this post if you clone the Redis source yourself and follow along in your favourite editor, particularly if you compile and run the code changes found here.

The command table is found near the very top of src/redis.c.  It is an array of instances of the redisCommand struct.  redisCommand is defined in src/redis.h but there’s a very handy block comment explaining each of its fields above the declaration of redisCommandTable.  Here is the definition of the get command:

The first field "get" is the name of the command.  The second is a pointer to the function that implements the command (you can see the implementation in t_string.c).

The third field is the command’s arity (number of arguments it accepts).  Specifying this means the command lookup and execution code can prevalidate a request before passing control by calling the function pointer.  This reduces the error handling code necessary in each of the command functions.  The argument count appears to include the name of the command itself so the get command accepts two arguments; its name and the name of the key whose value should be fetched.

The fourth field, set to "r", is specifying that the command is read only and doesn’t modify any keys’ value or state.  There are a whole bunch of one letter flags that you can specify in this string that are explained in detail in the nearby block comment.  The field following this string  should always be set to zero, and will be computed later.  It’s simply a bitmask representation of the information implied by the string.

The sixth field is NULL because it is only necessary when you need complex logic to tell Redis which arguments of the command are actually keys.  A key implies a reference to a value stored in Redis as opposed to simple value parameters such as our max argument.  This allows Redis to extract the values of the keys (and check that they exist) before calling the command implementation.  If this field were used it would be a pointer to a function that would return an integer array of argument indexes (zunionInterGetKeys in db.c is an example of this).  In the case of the get command though (and most other command) this information can be conveyed with the next 3 integer fields.  There is only one argument to getCommand and it is a key.  Therefore the first argument that is a key is at index 1, the last argument that is a key is at index 1, and the step increment to find all the keys is; 1,1,1.

The last two fields of a redisCommand represent metrics about the command, are set by Redis and should always be initialised to 0.

Let’s add our rand command to the bottom of the table:

The command is called "rand", randCommand is the pointer to the implementation (not implemented yet) and it takes 2 arguments (the name and max).  As for the flags – it’s read only (r), returns random, non deterministic output (R) and can be called while Redis is still loading the database (l).  There are no key arguments.

The next step is to add the randCommand function prototype to src/redis.h.  A redis command implementation takes one argument, a redisClient struct that represents the command arguments but can also be used to send the response to the actual client:

This prototype ought to be placed in src/redis.h near the all the other command prototypes.  Grepping for this line:

will help you find where.

Let’s add an empty implementation to src/redis.c:

I added mine near to the infoCommand definition.  Now let’s run make

and run the server we’ve built (hint if you usually have an instance of redis running locally now would be a good time to stop it):

And let’s run a Redis client in another terminal and try out our command:

First let’s try out the error handling:

Good to see the arity checking working.  Let’s specify an argument this time:

… and Redis hangs.  I suppose that should have been expected given that we’re not responding with anything from the randCommand function.  Let’s ctrl-c the server and get back to the source.

We want to respond with a number so I dug around looking for an example of how to do that and found the zcardCommand in src/t_zset.c.  This command uses addReplyLongLong to reply to the client with a response that is a 64-bit integer (a long long).  Let’s try that:

Now when we make again and run the command:

OK, so it’s not very random but it’s a start.  Let’s parse our max argument from the command now and return a random value limited by max:

Whilst Redis uses primitive types and C strings throughout the codebase it also has its own internal object system for representing strings, longs and more complex types in a more generic fashion.  An example of Redis’s use of these objects is the representation of each command’s arguments.  Each command argument is stored as a Redis object in the argv array on the redisClient instance c.   To get a Redis object as a long I found an example in the getrangeCommand function in src/t_string.c that uses the getLongFromObjectOrReply function from src/object.c.

getLongFromObjectOrReply takes a redisClient instance, checks that the object in its second parameter is an object that represents a long, places the value of that long at the pointer specified by its third parameter and returns REDIS_OK.  If the argument is not a long (or overflows) it will return REDIS_ERR.  The beauty of this method is that if we receive REDIS_ERR we can just return from our randCommand function as any necessary error response will have already been sent to the client.  Let’s try out our command again:

Looks pretty good!  rand is an entirely pointless command but I learned quite a bit about Redis from implementing it and I hope you did too by following along.  Please use the comments to let me know if I’ve made any glaring errors in this post.  It would also be good to know if you’ve found this post useful or enjoyable.  I’ll certainly consider writing more posts like this about Redis or other open source codebases.

 Discuss on Hacker News

This entry was posted in C, Redis and tagged , , , , . Bookmark the permalink.

9 Responses to A quick look at the Redis source code

  1. Nice article! I’m sharing this, you’ll be featured in our next RedisWeekly !

  2. Pingback: A quick look at the Redis source code | Boardmad

  3. jeff says:

    Nice write up.

  4. Haridas N says:

    Easy to follow and very nicely explained. Thank you !

  5. Mike Leach says:

    Excellent article. Thanks for sharing!

  6. Liu Chao says:

    I am chinese and not good at english.But i read the article not hardly.I am expected your next aritcle about redis.Thank you so much.

  7. RajaRaviVarma says:

    We use Redis for so many things but never had an oppurtunity to peek into the source code.

    A school kid can understand this post. Excellent write up.

  8. alex smith says:

    another chinese here, this article is very neat !

  9. Buddy says:

    I enjoyed the article. I appreciated how you didn’t get hang up on any details and were able to produce something that I could read in just a few minutes. Also, like you I have some non-existent C practice so I learned a little about structs here. Thanks!

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">