Using Redis for storing item views

Redis is and advanced key-value database that allows you to save different types of data and is extremely fast on I/O operations. Redis stores everything in memory, but the data can be persisted by dumping the dataset to disk every once in a while or by adding each command to a log. Redis is very versatile compared to other key-value stores: It provides a set of powerful commands and supports diverse data structures such as strings, hashes, lists, sets, ordered sets, and even bitmaps or HyperLogLogs.

While SQL is best suitable for schema-defined persistent data storage, Redis offers numerous advantages when dealing with rapidly changing data, volatile storage, or when a quick cache is needed. Let's see how Redis can be used for building new functionality into our project.

Installing Redis

Download the latest Redis version from http://redis.io/download. Unzip the tar.gz file, enter the redis directory and compile Redis using the make command as follows:

cd redis-3.0.4
make

After installing it use the following shell command to initialize the Redis server:

src/redis-server

You should see an output that ends with the following lines:

# Server started, Redis version 3.0.4
* DB loaded from disk: 0.001 seconds
* The server is now ready to accept connections on port 6379

By default, Redis runs on port 6379, but you can also specify a custom port using the --port flag, for example redis-server --port 6655. When your server is ready, you can open the Redis client in another shell using the following command:

src/redis-cli

You should see the Redis client shell like the following:

127.0.0.1:6379>

The Redis client allows you to execute Redis commands directly from the shell. Let's try some commands. Enter the SET command in the Redis shell to store a value in a key:

127.0.0.1:6379> SET name "Peter"
OK

The previous command creates a name key with the string value "Peter" in the Redis database. The OK output indicates that the key has been saved successfully. Then, retrieve the value using the GET command as follows:

127.0.0.1:6379> GET name
"Peter"

You can also check if a key exists by using the EXISTS command. This command returns 1 if the given key exists, 0 otherwise:

127.0.0.1:6379> EXISTS name
(integer) 1

You can set the time for a key to expire using the EXPIRE command, which allows you to set time to live in seconds. Another option is using the EXPIREAT command that expects a Unix timestamp. Key expiration is useful to use Redis as a cache or to store volatile data:

127.0.0.1:6379> GET name
"Peter"
127.0.0.1:6379> EXPIRE name 2
(integer) 1

Wait for 2 seconds and try to get the same key again:

127.0.0.1:6379> GET name
(nil)

The (nil) response is a null response and means no key has been found. You can also delete any key using the DEL command as follows:

127.0.0.1:6379> SET total 1
OK
127.0.0.1:6379> DEL total
(integer) 1
127.0.0.1:6379> GET total
(nil)

These are just basic commands for key operations. Redis includes a large set of commands for other data types such as strings, hashes, sets, ordered sets, and so on. You can take a look at all Redis commands at http://redis.io/commands and all Redis data types at http://redis.io/topics/data-types.

Using Redis with Python

We need Python bindings for Redis. Install redis-py via pip using the command:

pip install redis==2.10.3

You can find the redis-py docs at http://redis-py.readthedocs.org/.

The redis-py offers two classes for interacting with Redis: StrictRedis and Redis. Both offer the same functionality. The StrictRedis class attempts to adhere to the official Redis command syntax. The Redis class extends StrictRedis overriding some methods to provide backwards compatibility. We are going to use the StrictRedis class since it follows the Redis command syntax. Open the Python shell and execute the following code:

>>> import redis
>>> r = redis.StrictRedis(host='localhost', port=6379, db=0)

This code creates a connection with the Redis database. In Redis, databases are identified by an integer index instead of a database name. By default, a client is connected to database 0. The number of available Redis databases is set to 16, but you can change this in the redis.conf file.

Now set a key using the Python shell:

>>> r.set('foo', 'bar')
True

The command returns True indicating that the key has been successfully created. Now you can retrieve the key using the get() command:

>>> r.get('foo')
'bar'

As you can see, the methods of StrictRedis follow the Redis command syntax.

Let's integrate Redis into our project. Edit the settings.py file of the bookmarks project and add the following settings to it:

REDIS_HOST = 'localhost'
REDIS_PORT = 6379
REDIS_DB = 0

These are the settings for the Redis server and the database that we will use for our project.

Storing item views in Redis

Let's store the total number of times an image has been viewed. If we did this using the Django ORM, it would involve an SQL UPDATE statement every time an image is displayed. Using Redis instead, we just need to increment a counter stored in memory, resulting in much better performance.

Edit the views.py file of the images application and add the following code to it:

import redis
from django.conf import settings

# connect to redis
r = redis.StrictRedis(host=settings.REDIS_HOST,
                      port=settings.REDIS_PORT,
                      db=settings.REDIS_DB)

Here we establish the Redis connection in order to use it in our views. Edit the image_detail view and make it look as follows:

def image_detail(request, id, slug):
    image = get_object_or_404(Image, id=id, slug=slug)
    # increment total image views by 1
    total_views = r.incr('image:{}:views'.format(image.id))
    return render(request,
                  'images/image/detail.html',
                  {'section': 'images',
                   'image': image,
                   'total_views': total_views})

In this view, we use the INCR command that increments the value of a key by 1 and sets the value to 0 before performing the operation if the key does not exist. The incr() method returns the value of the key after performing the operation and we store it in the total_views variable. We build the Redis key using a notation like object-type:id:field (for example image:33:id).

Note

The convention for naming Redis keys is to use a colon sign as separator for creating namespaced keys. By doing so, the key names are specially verbose and related keys share part of the same schema in their names.

Edit the image/detail.html template and add the following code to it after the existing <span class="count"> element:

<span class="count">
  <span class="total">{{ total_views }}</span>
  view{{ total_views|pluralize }}
</span>

Now open an image detail page in your browser and load it several times. You will see that each time the view is executed the total views displayed are incremented by 1. See the following example:

Storing item views in Redis

You have successfully integrated Redis into your project to store item counts.

Storing a ranking in Redis

Let's built some more functionality with Redis. We are going to create a ranking of the most viewed images in our platform. For building this ranking we will use Redis sorted sets. A sorted set is a non-repeating collection of strings in which every member is associated with a score. Items are sorted by their score.

Edit the views.py file of the images application and make the image_detail view look as follows:

def image_detail(request, id, slug):
    image = get_object_or_404(Image, id=id, slug=slug)
    # increment total image views by 1
    total_views = r.incr('image:{}:views'.format(image.id))
    # increment image ranking by 1
    r.zincrby('image_ranking', image.id, 1)
    return render(request,
                  'images/image/detail.html',
                  {'section': 'images',
                   'image': image,
                   'total_views': total_views})

We use the zincrby() command to store image views in a sorted set with the key image:ranking. We are storing the image id, and a score of 1 that will be added to the total score of this element in the sorted set. This will allow us to keep track of all image views globally and have a sorted set ordered by the total number of views.

Now create a new view to display the ranking of the most viewed images. Add the following code to the views.py file:

@login_required
def image_ranking(request):
    # get image ranking dictionary
    image_ranking = r.zrange('image_ranking', 0, -1,
                             desc=True)[:10]
    image_ranking_ids = [int(id) for id in image_ranking]
    # get most viewed images
    most_viewed = list(Image.objects.filter(
                           id__in=image_ranking_ids))
    most_viewed.sort(key=lambda x: image_ranking_ids.index(x.id))
    return render(request,
                  'images/image/ranking.html',
                  {'section': 'images',
                   'most_viewed': most_viewed})

This is the image_ranking view. We use the zrange() command to obtain the elements in the sorted set. This command expects a custom range by lowest and highest score. By using 0 as lowest and -1 as highest score we are telling Redis to return all elements in the sorted set. We also specify desc=True to retrieve the elements ordered by descending score. Finally, we slice the results using [:10] to get the first 10 elements with highest scores. We build a list of returned image IDs and we store it in the image_ranking_ids variable as a list of integers. We retrieve the Image objects for those IDs and force the query to be executed by using the list() function. It is important to force the queryset execution because next we use the sort() list method on it (at this point we need a list of objects instead of a queryset). We sort the Image objects by their index of appearance in the image ranking. Now we can use the most_viewed list in our template to display the 10 most viewed images.

Create a new image/ranking.html template file and add the following code to it:

{% extends "base.html" %}

{% block title %}Images ranking{% endblock %}

{% block content %}
  <h1>Images ranking</h1>
  <ol>
    {% for image in most_viewed %}
      <li>
        <a href="{{ image.get_absolute_url }}">
          {{ image.title }}
        </a>
      </li>
    {% endfor %}
  </ol>
{% endblock %}

The template is pretty straightforward, as we just iterate over the Image objects contained in the most_viewed list.

Finally create an URL pattern for the new view. Edit the urls.py file of the images application and add the following pattern to it:

url(r'^ranking/$', views.image_ranking, name='create'),

Open http://127.0.0.1:8000/images/ranking/ in your browser. You should be able to see an image ranking as follows:

Storing a ranking in Redis

Next steps with Redis

Redis is not a replacement for your SQL database but a fast in-memory storage that is more suitable for certain tasks. Add it to your stack and use it when you really feel it's needed. The following are some scenarios in which Redis suits pretty well:

  • Counting: As you have seen, it is very easy to manage counters with Redis. You can use incr() and incrby() for counting stuff.
  • Storing latest items: You can add items to the start/end of a list using lpush() and rpush(). Remove and return first/last element using lpop() / rpop().You can trim the list length using ltrim() to maintain its length.
  • Queues: In addition to push and pop commands, Redis offers blocking queue commands.
  • Caching: Using expire() and expireat() allows you to use Redis as a cache. You can also find third-party Redis cache backends for Django.
  • Pub/Sub: Redis provides commands for subscribing/unsubscribing and sending messages to channels.
  • Rankings and leaderboards: Redis sorted sets with scores make it very easy to create leaderboards.
  • Real-time tracking: Redis fast I/O makes it perfect for real-time scenarios.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset