Building a recommendation engine

A recommendation engine is a system that predicts the preference or rating that a user would give to an item. The system selects relevant items for the users based on their behavior and the knowledge it has about them. Nowadays, recommendation systems are used in many online services. They help users by selecting the stuff they might be interested in from the vast amount of available data irrelevant to them. Offering good recommendations enhances user engagement. E-commerce sites also benefit from offering relevant product recommendations by increasing their average sale.

We are going to create a simple yet powerful recommendation engine that suggests products that are usually bought together. We will suggest products based on historic sales, thus identifying products that are usually bought together. We are going to suggest complementary products in two different scenarios:

  • Product detail page: We will display a list of products that are usually bought with the given product. This will be displayed like: Users who bought this also bought X, Y, Z. We need a data structure that allows us to store the number of times that each product has been bought together with the product being displayed.
  • Cart detail page: Based on the products users add to the cart, we are going to suggest products that are usually bought together with these ones. In this case, the score we calculate to obtain related products has to be aggregated.

We are going to use Redis to store products that are purchased together. Remember that you already used Redis in Chapter 6, Tracking User Actions. If you haven't installed Redis yet, you can find installation instructions in that chapter.

Recommending products based on previous purchases

Now, we will recommend products to users based on what they have added to the cart. We are going to store a key in Redis for each product bought in our site. The product key will contain a Redis sorted set with scores. We will increment the score by 1 for each product bought together every time a new purchase is completed.

When an order is successfully paid for, we store a key for each product bought, including a sorted set of products that belong to the same order. The sorted set allows us to give scores for products that are bought together.

Edit the settings.py file of your project and add the following settings to it:

REDIS_HOST = 'localhost'
REDIS_PORT = 6379
REDIS_DB = 1

These are the settings required to establish a connection with the Redis server. Create a new file inside the shop application directory and name it recommender.py. Add the following code to it:

import redis
from django.conf import settings
from .models import Product

# connect to redis
r = redis.StrictRedis(host=settings.REDIS_HOST,
                      port=settings.REDIS_PORT,
                      db=settings.REDIS_DB)

class Recommender(object):

    def get_product_key(self, id):
        return 'product:{}:purchased_with'.format(id)

    def products_bought(self, products):
        product_ids = [p.id for p in products]
        for product_id in product_ids:
            for with_id in product_ids:
            # get the other products bought with each product
            if product_id != with_id:
                # increment score for product purchased together
                r.zincrby(self.get_product_key(product_id),
                          with_id,
                          amount=1)

This is the Recommender class that will allow us to store product purchases and retrieve product suggestions for a given product or products. The get_product_key() method receives an id of a Product object and builds the Redis key for the sorted set where related products are stored, which looks like product:[id]:purchased_with.

The products_bought() method receives a list of Product objects that have been bought together (that is, belong to the same order). In this method, we perform the following tasks:

  1. We get the product IDs for the given Product objects.
  2. We iterate over the product IDs. For each id, we iterate over the product IDs and skip the same product so that we get the products that are bought together with each product.
  3. We get the Redis product key for each product bought using the get_product_id() method. For a product with an ID of 33, this method returns the key product:33:purchased_with. This is the key for the sorted set that contains the product IDs of products that were bought together with this one.
  4. We increment the score of each product id contained in the sorted set by 1. The score represents the times another product has been bought together with the given product.

So we have a method to store and score the products that were bought together. Now we need a method to retrieve the products that are bought together for a list of given products. Add the following suggest_products_for() method to the Recommender class:

 def suggest_products_for(self, products, max_results=6):
    product_ids = [p.id for p in products]
    if len(products) == 1:
        # only 1 product
        suggestions = r.zrange(
                         self.get_product_key(product_ids[0]),
                         0, -1, desc=True)[:max_results]
    else:
        # generate a temporary key
        flat_ids = ''.join([str(id) for id in product_ids])
        tmp_key = 'tmp_{}'.format(flat_ids)
        # multiple products, combine scores of all products
        # store the resulting sorted set in a temporary key
        keys = [self.get_product_key(id) for id in product_ids]
        r.zunionstore(tmp_key, keys)
        # remove ids for the products the recommendation is for
        r.zrem(tmp_key, *product_ids)
        # get the product ids by their score, descendant sort
        suggestions = r.zrange(tmp_key, 0, -1, 
                               desc=True)[:max_results]
        # remove the temporary key
        r.delete(tmp_key)
    suggested_products_ids = [int(id) for id in suggestions]

    # get suggested products and sort by order of appearance
    suggested_products = list(Product.objects.filter(id__in=suggested_products_ids))
    suggested_products.sort(key=lambda x: suggested_products_ids.index(x.id))
    return suggested_products

The suggest_products_for() method receives the following parameters:

  • products: This is a list of Product objects to get recommendations for. It can contain one or more products.
  • max_results: This is an integer that represents the maximum number of recommendations to return.

In this method, we perform the following actions:

  1. We get the product IDs for the given Product objects.
  2. If only one product is given, we retrieve the ID of the products that were bought together with the given product, ordered by the total number of times that they were bought together. To do so, we use Redis' ZRANGE command. We limit the number of results to the number specified in the max_results attribute (6 by default).
  3. If more than one product is given, we generate a temporary Redis key built with the IDs of the products.
  4. We combine and sum all scores for the items contained in the sorted set of each of the given products. This is done using the Redis' ZUNIONSTORE command. The ZUNIONSTORE command performs a union of the sorted sets with the given keys, and stores the aggregated sum of scores of the elements in a new Redis key. You can read more about this command at http://redis.io/commands/ZUNIONSTORE. We save the aggregated scores in the temporary key.
  5. Since we are aggregating scores, we might obtain the same products we are getting recommendations for. We remove them from the generated sorted set using the ZREM command.
  6. We retrieve the IDs of the products from the temporary key, ordered by their score using the ZRANGE command. We limit the number of results to the number specified in the max_results attribute. Then we remove the temporary key.
  7. Finally, we get the Product objects with the given id and we order the products by the same order as the them.

For practical purposes, let's also add a method to clear the recommendations. Add the following method to the Recommender class:

    def clear_purchases(self):
        for id in Product.objects.values_list('id', flat=True):
            r.delete(self.get_product_key(id))

Let's try our recommendation engine. Make sure you include several Product objects in the database and initialize the Redis server using the following command from the shell:

src/redis-server

Open another shell, execute python manage.py shell, and write the following code to retrieve several products:

from shop.models import Product
black_tea = Product.objects.get(translations__name='Black tea')
red_tea = Product.objects.get(translations__name='Red tea')
green_tea = Product.objects.get(translations__name='Green tea')
tea_powder = Product.objects.get(translations__name='Tea powder')

Then, add some test purchases to the recommendation engine:

from shop.recommender import Recommender
r = Recommender()
r.products_bought([black_tea, red_tea])
r.products_bought([black_tea, green_tea])
r.products_bought([red_tea, black_tea, tea_powder])
r.products_bought([green_tea, tea_powder])
r.products_bought([black_tea, tea_powder])
r.products_bought([red_tea, green_tea])

We have stored the following scores:

black_tea: red_tea (2), tea_powder (2), green_tea (1) 
red_tea: black_tea (2), tea_powder (1), green_tea (1)
green_tea: black_tea (1), tea_powder (1), red_tea(1)
tea_powder: black_tea (2), red_tea (1), green_tea (1)

Let's take a look at the recommended products for a single product:

>>> r.suggest_products_for([black_tea])
[<Product: Tea powder>, <Product: Red tea>, <Product: Green tea>]
>>> r.suggest_products_for([red_tea])
[<Product: Black tea>, <Product: Tea powder>, <Product: Green tea>]
>>> r.suggest_products_for([green_tea])
[<Product: Black tea>, <Product: Tea powder>, <Product: Red tea>]
>>> r.suggest_products_for([tea_powder])
[<Product: Black tea>, <Product: Red tea>, <Product: Green tea>]

As you can see, the order for recommended products is based on their score. Let's get recommendations for multiple products with aggregated scores:

>>> r.suggest_products_for([black_tea, red_tea])
[<Product: Tea powder>, <Product: Green tea>]
>>> r.suggest_products_for([green_tea, red_tea])
[<Product: Black tea>, <Product: Tea powder>]
>>> r.suggest_products_for([tea_powder, black_tea])
[<Product: Red tea>, <Product: Green tea>]

You can see that the order of the suggested products matches the aggregated scores. For example, products suggested for black_tea and red_tea are tea_powder (2+1) and green_tea (1+1).

We have verified that our recommendation algorithm works as expected. Let's display recommendations for products on our site.

Edit the views.py file of the shop application and add the following import:

from .recommender import Recommender

Add the following code to the product_detail view just before the render() function:

r = Recommender()recommended_products = r.suggest_products_for([product], 4)

We get a maximum of four product suggestions. The product_detail view should now look as follows:

from .recommender import Recommender
def product_detail(request, id, slug):
    product = get_object_or_404(Product,
                                id=id,
                                slug=slug,
                                available=True)
    cart_product_form = CartAddProductForm()
    r = Recommender()
    recommended_products = r.suggest_products_for([product], 4)
    return render(request,
                  'shop/product/detail.html',
                  {'product': product,
                  'cart_product_form': cart_product_form,
                  'recommended_products': recommended_products})

Now edit the shop/product/detail.html template of the shop application and add the following code after {{ product.description|linebreaks }}:

{% if recommended_products %}
  <div class="recommendations">
    <h3>{% trans "People who bought this also bought" %}</h3>
    {% for p in recommended_products %}
      <div class="item">
        <a href="{{ p.get_absolute_url }}">
          <img src="{% if p.image %}{{ p.image.url }}{% else %}{% static "img/no_image.png" %}{% endif %}">
        </a>
        <p><a href="{{ p.get_absolute_url }}">{{ p.name }}</a></p>
      </div>
    {% endfor %}
  </div>
{% endif %}

Run the development server with the command python manage.py runserver and open http://127.0.0.1:8000/en/ in your browser. Click on any product to see its details page. You should see that recommended products are displayed below the product, as shown in the following image:

Recommending products based on previous purchases

We are also going to include product recommendations in the cart. The recommendation will be based on the products that the user has added to the cart. Edit the views.py inside the cart application directory and add the following import:

from shop.recommender import Recommender

Then, edit the cart_detail view to make it look as follows:

def cart_detail(request):
    cart = Cart(request)
    for item in cart:
        item['update_quantity_form'] = CartAddProductForm(
                          initial={'quantity': item['quantity'],
                                   'update': True})
    coupon_apply_form = CouponApplyForm()

    r = Recommender()
    cart_products = [item['product'] for item in cart]
    recommended_products = r.suggest_products_for(cart_products,
                                                  max_results=4)

    return render(request,
                  'cart/detail.html',
                  {'cart': cart,
                   'coupon_apply_form': coupon_apply_form,
                   'recommended_products': recommended_products})

Edit the cart/detail.html template of the cart application and add the following code just after the </table> HTML tag:

{% if recommended_products %}
  <div class="recommendations cart">
    <h3>{% trans "People who bought this also bought" %}</h3>
    {% for p in recommended_products %}
      <div class="item">
        <a href="{{ p.get_absolute_url }}">
          <img src="{% if p.image %}{{ p.image.url }}{% else %}{% static "img/no_image.png" %}{% endif %}">
        </a>
        <p><a href="{{ p.get_absolute_url }}">{{ p.name }}</a></p>
      </div>
    {% endfor %}
  </div>
{% endif %}

Open http://127.0.0.1:8000/en/ in your browser and add a couple of products to your cart. When you navigate to http://127.0.0.1:8000/en/cart/, you should see the aggregated product recommendations for the items in the cart as follows:

Recommending products based on previous purchases

Congratulations! You have built a complete recommendation engine using Django and Redis.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset