A Simple Tag

The simplest form of tag only exists in and of itself, typically to inject additional content into the page, based on some arguments. The node for this case is extremely simple, just taking and storing those arguments and formatting them into a string during rendering. Here’s how it would look if the filter from the previous section were instead implemented as a tag. In the template, this would look like {% first var_name 3 %}.

from django.template import Library, Node, Variable
 
register = Library()
 
class FirstNode(Node):
    def __init__(self, var, count):
        self.var = var
        self.count = count
 
    def render(self, context):
        value = self.var.resolve(context)
        return value[:self.count]

The function to compile it, on the other hand, is a bit more complicated. Tag functions, unlike filter functions, always take two arguments: the template parser object, and a token representing the text contained within the tag. It’s up to the compilation function to extract the necessary bits of information from these two objects.

For a simple tag like this, it’s not necessary to worry about the parser object, but the token is still necessary in order to get the argument, if one was specified.

The most important thing to know about the token is the split_contents() method, which intelligently breaks apart a tag’s declaration into individual components, including the tag’s name and its arguments.

It correctly handles variable references, quoted strings and numbers, though it doesn’t do any variable resolution, and leaves the quotes around quoted strings.

In order to get the two necessary bits of information out of our template tag, token.split_contents() is used to extract them from the declared string. Then, these can be coerced to the correct types and used to instantiate the node described previously.

@register.tag
def first(parser, token):
    var, count = token.split_contents()[1:]
    return FirstNode(Variable(var), int(count))

A Shortcut for Simple Tags

Thankfully, there’s a shortcut that makes this process a whole lot easier. The Library object contains another decorator method, simple_tag(), which handles simpler cases like this. Behind the scenes, it handles the parsing and resolution of arguments, and even the creation of the node class, so all that’s left for the template tag is a single function that looks quite similar to a variable filter.

from django.template import Library

 
register = Library()
 
@register.simple_tag
def first(value, count):
    return value[:count]

This is still of limited use, but there are many such situations where a simpler tag is necessary, and the shortcut can become quite a time-saver. For more advanced needs, manually creating the node offers much more power and flexibility.

Converting a Token to a String

Before diving into the full compilation function, first notice that tokens must be converted back into strings for Jinja to process them. Jinja uses a fairly similar syntax for its templates, so Django’s Lexer accurately identifies variable, block and comment tags. Though Jinja also creates tokens for those tags, tokens from the two template engines aren’t compatible with each other, so they must be converted back to strings. Jinja can then process them as it would any template from any source.

To accomplish this conversion, the node compilation function will rely on a separate function, which takes a token and returns a string. It works on the fact that django.template also contains constants for the beginning and ending portions of these tags. With this information and the structure of tokens, a suitable string can be created from a given token.

from django import template
 
def string_from_token(token):
    """
    Converts a lexer token back into a string for use with Jinja.
    """
    if token.token_type == template.TOKEN_TEXT:
        return token.contents
 
    elif token.token_type == template.TOKEN_VAR:
        return '%s %s %s' % (
            template.VARIABLE_TAG_START,
            token.contents,
            template.VARIABLE_TAG_END,
        )
 
    elif token.token_type == template.TOKEN_BLOCK:
        return '%s %s %s' % (
            template.BLOCK_TAG_START,
            token.contents,
            template.BLOCK_TAG_END,
        )
 
    elif token.token_type == template.TOKEN_COMMENT:
        return u'' # Django doesn't store the content of comments

This won’t produce an exact replica of the original template string. Some whitespace gets removed during Django’s Lexer processing and comments lose their contents entirely. All functional aspects of the tags are retained, so the template will still work as advertised, but know that some minor formatting issues may arise as the result of this technique.

Compiling to a Node

With a function in place to reproduce strings for the tokens within the jinja block, the next step is to generate a Node that will be used to render the content along with the rest of the template. When gathering up the content between an opening tag and its closing tag, compilation functions often make use of the Parser.parse() method, passing in the name of the end tag, which will return a list of Node objects representing the inner content.

Since Jinja tags can’t be processed using Django’s node functions, Parser.parse() will cause problems due to incorrect syntax. Instead, the Jinja compilation function must access the tokens directly, which can then be converted back to strings. There are no provided functions to do this entirely, but combining Parser.next_token() with some extra logic will work quite well.

The compilation function can loop over the available tokens, calling Parser.next_token() each time. This loop will execute until either an endjinja block token is found or there are no more tokens in the template. Once a token is obtained from the parser, it can be converted to a string and added to an internal template string that can be used to populate a JinjaNode.

import jinja2
from django import template
from django.base import TemplateSyntaxError
 
register = template.Library()
 
def jinja(parser, token):
    """
    Define a block that gets rendered by Jinja, rather than Django's templates.
    """
    bits = token.contents.split()
    if len(bits) != 1:
        raise TemplateSyntaxError("'%s' tag doesn't take any arguments." % bits[0])
 
    # Manually collect tokens for the tag's content, so Django's template
    # parser doesn't try to make sense of it.
    contents = []
    while 1:
        try:
            token = parser.next_token()
        except IndexError:
            # Reached the end of the template without finding the end tag
            raise TemplateSyntaxError("'endjinja' tag is required.")
        if token.token_type == template.TOKEN_BLOCK and 
           token.contents == 'endjinja':
            break
        contents.append(string_from_token(token))
    contents = ''.join(contents)
 
    return JinjaNode(jinja2.Template(contents))
jinja = register.tag(jinja)

Preparing the Jinja Template

Once the compilation function retrieves the Jinja template contents from the Django template tokens, a JinjaNode is created to access that template. Jinja provides its own Template object that compiles content into tangible objects, so it makes sense to use it when a JinjaNode is created.

Then, when it comes time to render the JinjaNode, all it takes is to render the compiled Jinja template and return that output back to Django’s template. This task is trickier than it may seem on the surface, because Django’s Context objects, which contain variables that should be passed to Jinja, don’t behave entirely like Python dictionaries. They support the common dictionary-style syntax for accessing keys, but internally, their structure is quite different from what Jinja expects.

To pass a nested Context object to the Jinja template properly, it must first be flattened to a single, standard Python dictionary. This can be done fairly easily, simply by looping through the individual dictionaries stored in the context and assigning them to a new dictionary, maintaining the precedence that Django itself uses: the first appearance of a particular key takes priority over any other instances of that same key. Only if a key doesn’t exist in the new Jinja context dictionary should it be added, so that no existing values get overwritten in the process.

Once the dictionary is available, that data can be passed to Jinja’s own Template.render() method. The result from that method is the properly rendered content that can be returned from the JinjaNode.render(), placing that content in the page.

import jinja2
 
class JinjaNode(template.Node):
    def __init__(self, template):
        self.template = template
 
    def render(self, django_context):
        # Jinja can't use Django's Context objects, so we have to
        # flatten it out to a single dictionary before using it.
        jinja_context = {}
        for layer in django_context:
            for key, value in layer.items():
                if key not in jinja_context:
                    jinja_context[key] = value
        return self.template.render(jinja_context)

Setting Up the Models

The first order of business is to make a place for templates to be stored in the database. In standard Django fashion, this is done with a model, with fields for the various properties of the template. For this application, a theme consists of a few various pieces of information:

• A block of text to be used as the content of the template
• A URL to a CSS file
• A user who created it
• A title, so other users can easily reference it, should it be made available for everyone
• An indication of whether it’s the site-wide default, so that users who haven’t yet selected a theme still have one to use

Most of this information will only be used by the theme object itself, as only the main block of text will be passed in to the template. It’s easy to think of a theme as a template in its own right, where it’s simultaneously a set of data that gets stored in the database and a set of instructions that are used to render HTML. Python provides a way to make that notion explicit and offers a simple way to deal with themes.

By using multiple inheritance, it’s possible for a theme to be both a model and a template, behaving in whichever way is necessary for the task at hand. The class inherits from django.db.models.Model and django.template.Template, and __init__() is overridden to initialize both sides separately:

from django.db import models
from django import template
from django.contrib.auth.models import User
 
from themes.managers import ThemeManager
 
class Theme(models.Model, template.Template):
    EDITING, PENDING, APPROVED = range(3)
    STATUS_CHOICES = (
        (EDITING, u'Editing'),
        (PENDING, u'Pending Approval'),
        (APPROVED, u'Approved'),
    )
    author = models.ForeignKey(User, related_name='authored_themes')
    title = models.CharField(max_length=255)
    template_string = models.TextField()
    css = models.URLField(null=True, blank=True)
    status = models.SmallIntegerField(choices=STATUS_CHOICES, default=EDITING)
    is_default = models.BooleanField()
 
    objects = ThemeManager()
 
    def __init__(self, *args, **kwargs):
        # super() won't work here, because the two __init__()
        # method signatures accept different sets of arguments
        models.Model.__init__(self, *args, **kwargs)
        template.Template.__init__(self, self.template_string,
                                   origin=repr(self), name=unicode(self))
    def save(self):
        if self.is_default:
            # Since only one theme can be the site-wide default, any new model that
            # is defined as default must remove the default setting from any other
            # theme before committing to the database.
            self.objects.all().update(is_default=False)
        super(Theme, self).save()
 
    def __unicode__(self):
        return self.title

That’s enough to get the themes themselves stored in the database, but it still doesn’t cover how a user can select a theme to use while browsing the site. Ordinarily, that would be set up as a ForeignKey on the model that references Theme, but since the User model is outside our control, something else will need to be done.

One way to store user-centric information, such as preferences, is to add a custom user model. Django’s official documentation³ covers this in detail, but the basic idea is that you can provide your own model to use in place of Django’s own user model. Your custom model can contain additional fields for anything user-related, including selected themes. A site can only have one custom user model, though, and it makes little sense to hijack that feature solely for the purpose of supporting themes. Instead, we can use a ManyToManyField to connect it to the User model.

class Theme(models.Model):
    ...  # All the other fields shown above
    users = models.ManyToManyField(User, through='SelectedTheme')
    ...  # All the other methods shown above
 
class SelectedTheme(models.Model):
    user = models.OneToOneField(User)
    theme = models.ForeignKey(Theme)

By using a OneToOneField, we can ensure that each user only appears once in the intermediary table. That way, each user can only have one selected theme. There are also some utility functions that can help manage this behavior a bit. There are actually two different methods that would be useful here, both for getting themes based on the user. One is for retrieving a user’s selected theme, while the other is for retrieving the themes a user has created.

from django.db import models
from django.conf import settings
 
class ThemeManager(models.Manager):
    def by_author(self, user):
        """
        A convenience method for retrieving the themes a user has authored.
        Since the only time we'll be retrieving themes by author is when
        they're being edited, this also limits the query to those themes
        that haven't yet been submitted for review.
        """
        return self.filter(author=self, status=self.model.EDITING)
 
    def get_current_theme(self, user):
        return SelectedTheme.objects.get(user=user).theme

With this manager in place, it’s easy to retrieve themes for a specific user, both those that user can edit, and the one that user should use when browsing the site. Having these shortcuts in place helps make views simpler, allowing them to focus on the business they really have to do. The whole point of a site-wide theme is that it’s used for every view, so clearly something else needs to be done to accommodate that.

Supporting Site-Wide Themes

Individual views have enough to worry about, and shouldn’t be responsible for managing themes. Instead, there needs to be a way to retrieve a user’s selected theme—or the default—and have that automatically applied to whatever template a view uses. Ideally, all this should happen without any changes to the views, so there’s little extra work that needs to be done.

This is a job best suited for a context processor, a concept described earlier in this chapter. By using a context processor, every view that uses RequestContext will automatically have access to the proper theme. This makes the ordinarily good advice of always using RequestContext now an absolute requirement. As will be seen in the next section, templates will explicitly rely on the theme being available, and failing to use RequestContext will violate that assumption.

The context processor required for this process is fairly straightforward, but it has to provide a few specific features. It must determine whether the current user is logged in or not, identify the user’s selected theme, fall back to a default theme if no theme is selected or if the user isn’t logged in, and it must return the proper theme so that it may be added to the template’s context. This code would be placed in a module called context_processors.py, in keeping with the conventions used within Django itself.

from django.conf import settings
 
from themes.models import Theme
def theme(request):
    if hasattr(request, 'user') and request.user.is_authenticated():
        # A valid user is logged in, so use the manager method
        theme = Theme.objects.get_current_theme(request.user)
    else:
        # The user isn't logged in, so fall back to the default
        theme = Theme.objects.get(is_default=True)
    name = getattr(settings, 'THEME_CONTEXT_NAME', 'theme')
    return {name: theme}

Note the use of hasattr() here in the test to see whether a user is logged in. That may seem unnecessary, but by adding that simple condition to the test, it allows this context processor to be used with no middleware requirements. Otherwise, it would always require django.contrib.auth.middleware.AuthenticationMiddleware, which places the user attribute on the request. If that middleware isn’t in use, every user will simply receive the default theme.

Also, note that the name of the context variable is driven by another new setting, this time called THEME_CONTEXT_NAME. This defaults to 'theme', so that it’s not necessary to supply a name explicitly unless that causes a clash with some other feature. This is a bit of a recurring theme (pun intended), because with an application that has to interact with a good deal outside of itself, such as user models and template contexts, it’s important to make sure conflicts are kept to a minimum.

With this file in place, the only thing left is to add 'themes.context_processors.theme' to the TEMPLATE_CONTEXT_PROCESSORS setting to make sure it gets applied to all the templates. Once the theme is made available to the template, it’s still necessary to make sure the template can access it and make use of it.

Setting Up Templates to Use Themes

The end goal of themes is to reorder the components of a page, so it’s important to identify what a “component” is. In terms of Django templates, this would mean a block of markup, identified by the {% block %} template tag. Each component of a page could be defined in a separate block, separating each bit into its own space.

With Django’s template inheritance, it’s possible to define blocks in one template that will be filled in with content from another template. This way, a page-specific template can define what goes in each block, while a base template can specify where those blocks are rendered, and what other markup gets placed around them. This would be an excellent way to reorder significant portions of a page, as long as there’s a way to dynamically specify where the base template places all the blocks.

Django supports template inheritance through the {% extends %} tag, which takes a single argument to identify the base template to extend. Typically, this is a hard-coded name of the template to use as a base. It can also take a context variable, containing a string to use as this base template. If that context variable points to a template instance, Django will use that instead of bothering to look up a template anywhere else.

Taking advantage of this in a template is easy; just put {% extends theme %} at the top of the template. If you’ve specified a THEME_CONTEXT_NAME explicitly for your site, make sure to change theme to whatever you’ve entered for that setting. That still only covers part of it. It’s still necessary to get the templates to make use of the blocks defined in the theme.

There’s no universal way to do this, since each site will have its own template inheritance setup, and its own set of blocks that every page will need to fill in. Typically, these blocks would be used for things like page title, navigation, page content and footers, but different sites may have different needs.

In addition, a site may have more blocks that can’t be rearranged, but are instead defined inside of other blocks. These wouldn’t be taken into consideration for our purposes at the moment, since themes are only concerned with blocks that can be moved around. Consider an application with the following blocks that can be customized:

• logo—The site’s logo, as an image
• title—The title of the current page
• search—A search box, possibly with advanced options
• navigation—A collection of links or other interface used for getting around the site
• sidebar—A bit of content related to the current page
• content—The flesh of the current page, whether that be a product listing, press release, search results or contact form
• footer—A copyright disclaimer, along with a few links for job openings, investor relations and contact information

Every theme must define all of these blocks in order to make sure the whole site gets displayed, so it’s important to outline them explicitly. Every template on the site needs to define content to be placed into these blocks, so that there’s always something to put in the right places. Many of those blocks aren’t specified to any particular page, so template inheritance comes to the rescue here as well.

By placing another template layer between the theme and the individual page, some blocks can be populated automatically for all pages, while others are left for individual pages to fill in. The individual page template still has final authority, with the ability to override any block with new content, if necessary. That just leaves the issue of making sure that templates do in fact define all the blocks required by the site’s inheritance scheme.

Validating and Securing Themes

Any time a site accepts input from users, it must be scrutinized to make sure that it fulfills a certain set of requirements and stays within acceptable limits. Themes are no exception there, but user-editable templates also represent a very real security risk. Django takes steps to ensure that templates can’t execute any common functions that make changes to the database, but there are a number of other things a template can do.

By default, only Django’s own data-altering methods are secured from templates by using the alters_data attribute. Any application’s models may define other methods that make changes to the database, and if those aren’t marked with alters_data, they’re fair game for use in templates. Even read-only access, if not kept in check, can be a problem. A theme is used on every page, and many pages will have access to a wide array of objects through model relationships.

There are so many ways to access things that should be kept private that no blacklist approach can ever hope to be complete. Instead, a whitelist approach is necessary, where themes are only allowed to use a small subset of features provided by Django’s template system. The trick is determining the right way to approach a problem like this.

On the surface, it may seem like regular expressions are the way to go. After all, Django itself uses a regular expression to parse templates and break them up into nodes, so surely it would be trivial to write a more limited expression to secure templates. That may be true for now, but remember that Django is constantly improving, and the future may bring new syntax to templates.

However unlikely that may be, if it does happen, no amount of careful crafting of our regular expression can predict what new syntax might be included in the future. Anything that slips past this protection has the potential to harm the site or divulge confidential information. That’s a lot to pin on the hope that the template syntax will remain constant.

Instead, we’ll rely on Django’s own regular expression to compile the template into a list of nodes, just like normal. Then, once it’s been compiled to a nodelist, it’s easy to peek at those nodes to make sure they’re all doing the right thing. Using this, forms can easily verify that the template defines all the right blocks and nothing else. Theme templates must:

• Inherit from the template referenced by the THEME_EXTENDS setting.
• Provide one block with the name referenced by a THEME_CONTAINER_BLOCK setting.
• Populate that block with all the blocks referenced in the THEME_BLOCKS setting.
• Provide no content in any of the THEME_BLOCKS blocks.
• Provide no other blocks than those mentioned in the THEME_BLOCKS setting.
• Contain no other tags whatsoever, only text.

from django import forms
from django import template
from django.template.loader_tags import BlockNode, ExtendsNode
from django.conf import settings
from theme import models
 
class ThemeForm(forms.ModelForm):
    title = forms.CharField()
    body = forms.CharField(widget=forms.Textarea)
 
    def clean_body(self):
        try:
            tpl = template.Template(self.cleaned_data['body'])
        except template.TemplateSyntaxError as e:
            # The template is invalid, which is an input error.
            raise forms.ValidationError(unicode(e))
 
        if [type(n) for n in tpl.nodelist] != [ExtendsNode] or 
            tpl.nodelist[0].parent_name != settings.THEME_EXTENDS:
                # No 'extends' tag was found
                error_msg = u"Template must extend '%s'" % settings.THEME_EXTENDS
                raise forms.ValidationError(error_msg)
 
        if [type(n) for n in tpl.nodelist[0].nodelist] != [BlockNode] or 
            tpl.nodelist[0].nodelist[0].name != settings.THEME_CONTAINER_BLOCK:
                # Didn't find exactly one block tag with the required name
                error_msg = u"Theme needs exactly one '%s' block" % 
                            settings.THEME_CONTAINER_BLOCK
                raise forms.ValidationError(error_msg)
 
        required_blocks = list(settings.THEME_BLOCKS[:])
        for node in tpl.nodelist[0].nodelist[0].nodelist:
            if type(node) is BlockNode:
                if node.name not in required_blocks:
                    error_msg = u"'%s' is not valid for themes." % node.name)
                    raise forms.ValidationError(error_msg)
                required_blocks.remove(node.name)
                if node.nodelist:
                    error_msg = u"'%s' block must be empty." % node.name)
                    raise forms.ValidationError(error_msg)
            elif type(node) is template.TextNode:
                # Text nodes between blocks are acceptable.
                pass
            else:
                # All other tags, including variables, are invalid.
                error_msg = u"Only 'extends', 'block' and plain text are allowed."
                raise forms.ValidationError(error_msg)
 
        if required_blocks:
            # Some blocks were missing from the template.
            blocks = ', '.join(map(repr, required_blocks))
            error_msg =  u"The following blocks must be defined: %s" % blocks
            raise forms.ValidationError(error_msg)
 
    class Meta:
        model = models.Theme

An Example Theme

Even with an application in place, it may be difficult to understand how a theme would be written to work with the site. Consider a site using this themes application with the following settings:

THEME_EXTENDS = 'base.html'
THEME_CONTEXT_NAME = 'theme'
THEME_CONTAINER_BLOCK = 'theme'
THEME_BLOCKS = (
    'title',
    'sidebar',
    'links',
)

The base.html template at the root of the inheritance chain might look like this:

<html>
<head>
<title>{% block title %}{% endblock %}</title>
<link rel="stylesheet" type="text/css" href="/style.css"/>
</head>
<body>{% block theme %}{% endblock %}</body>
</html>

A theme can then be written to fill in the application’s requirements: extend from base.html, provide a theme block and fill it with empty title, sidebar and links blocks. Unlike the other templates, this code would be stored in the database, as an instance of the Theme model.

{% extends 'base.html' %}
 
{% block theme %}
<h1>{% block title %}{% endblock %}</h1>
<ul id="links">{% block links %}{% endblock %}</ul>
<div id="content">{% block content %}{% endblock %}</div>
{% endblock %}

Now, individual templates for the rest of the site can be written to extend from the theme variable and fill in the title, sidebar and links blocks. Consider the template for the root of a real estate site:

{% extends theme %}
 
{% block title %}Acme Real Estate{% endblock %}
 
{% block links %}
<li><a href="{% url home_page %}">Home</a></li>
<li><a href="{% url property_list %}">Properties</a></li>
<li><a href="{% url about_page %}">About</a></li>
{% endblock %}
 
{% block content %}
<p>Welcome to Acme Real Estate!</p>
{% endblock %}

With all of these templates in place, loading up the root of the site will yield a full HTML document like the following:

<html>
<head>
<title>Acme Real Estate</title>
<link rel="stylesheet" type="text/css" href="/style.css"/>
</head>
<body>
<h1>Acme Real Estate</h1>
<ul id="links">
<li><a href="/">Home</a></li>
<li><a href="/properties/">Properties</a></li>
<li><a href="/about/">About</a></li>
</ul>
<div id="content">
<p>Welcome to Acme Real Estate!</p>
</div>
</body>
</html>