django.db.backends

This references the backend package’s base module, from which the entirety of the database can be accessed. Accessing the database backend in this manner ensures a unified, consistent interface, regardless of which database package is being used behind the scenes.

Django does a lot of work to make this level of access unnecessary, but there’s only so far it can go without overcomplicating things. When the ORM fails to offer some necessary bit of functionality—for example, updating one column based on the value of another column in pure SQL—it’s always possible to go straight to the source and peek at what’s really going on and adjust the standard behavior or replace it altogether.

Because this is really just an alias for a backend-specific module, the full import paths listed throughout this chapter are only valid when trying to access the database in this manner. When implementing a new backend, the package path will be specific to that backend. For instance, if a backend for connecting with IBM’s DB2² were placed in a package named db2, this module would actually be located at db2/base.py.

class DatabaseWrapper(BaseDatabaseWrapper):
    operators = {
        "exact": "= %s",
    }

Creation of New Structures

One of the more convenient features Django’s database connection provides is the ability to automatically create tables, columns, and indexes based solely on model definitions declared in Python. Along with a powerful database querying API, this is a key feature in avoiding the use of SQL code throughout an application, keeping it clean and portable.

While the SQL syntax itself is reasonably well standardized with regards to creation of data structures, the names and options available for individual field types are quite varied across different implementations. This is where Django’s database backends come in, providing a mapping of Django’s basic field types to the appropriate column types for that particular database.

This mapping is stored in the backend package’s creation module, which must contain a DatabaseCreation class that subclasses django.db.backends.creation.BaseDatabaseCreation. This class contains an attribute named data_types, with contains a dictionary with keys that match up with the available return values from the various Field subclasses and string values that will be passed to the database as the column’s definition.

The value can also be a Python format string, which will be given a dictionary of field attributes so that customized field settings can be used to determine how the column is created. For example, this is how CharField passes along the max_length attribute. While many field types have common attributes, the ones that are of most use to the column type are likely specific to each individual field. Consult the field’s source code to determine what attributes are available for use in this mapping.

There are a number of basic field types available as internal column types:

• AutoField—An auto-incrementing numeric field, used for primary keys when one isn’t defined explicitly in the model.
• BooleanField—A field representing just two possible values: on and off. If the database doesn’t have a separate column that represents this case, it’s also possible to use a single-character CharField to store "1" and "0" to simulate this behavior.
• CharField—A field containing a limited about of free-form text. Typically, this uses a variable-length string type in the database, using the extra max_length attribute to define the maximum length of a stored value.
• CommaSeparatedIntegerField—A field containing a list of integers, typically representing IDs, which are stored in a single string separated by commas. Because the list is stored as a string, this also uses a variable-length string type on the database side. Although some databases might have a more intelligent and efficient means of storing this type of data, the field’s code still expects a string of numbers, so the backend should always return one.
• DateField—A standard date, without any time information associated with it. Most databases should have a date column type, so this should be easy to support. Just make sure the column type used returns a Python datetime.date upon retrieval.
• DateTimeField—A date, but with associated time information attached, excluding time zones. Again, most reasonable databases will support this easily, but make sure the Python library for it returns a datetime.datetime when retrieving from the database.
• DecimalField—A fixed-precision decimal number. This is another example of using field attributes to define the database column because the max_digits and decimal_places field attributes should control the database column equivalents.
• FileField—The name and location of a file stored elsewhere. Django doesn’t support storing files as binary data in the database, so its files are referenced by a relative path and name, which is stored in the associated column. Because that’s text, this again uses a standard variable-length text field, which also utilizes the max_length field attribute.
• FilePathField—The name and path of a file in a storage system. This field is similar to FileField in many respects, but this is intended to allow users to choose from existing files, while FileField exists to allow saving new files. Because the data actually being stored is essentially the same format, it works the same way, using a variable-length string specified using the max_length attribute.
• FloatField—A field containing a floating point number. It doesn’t matter if the database stores the number with fixed precision internally, as long as the Python library returns a float for values stored in the column.
• IntegerField—A field containing a signed 32-bit integer.
• BigIntegerField—A field containing a signed 64-bit integer.
• IPAddressField—An Internet Protocol (IP) address, using the current IPv4³ standard, represented in Python as a string.
• GenericIPAddressField—An IP address using either the original IPv4 standard or the newer IPv6⁴ standard.
• NullBooleanField—A Boolean field that also allows NULL values to be stored in the database.
• PositiveIntegerField—A field containing an unsigned 32-bit integer.
• PositiveSmallIntegerField—A field containing an unsigned 8-bit integer.
• SmallIntegerField—A field containing a signed 8-bit integer.
• TextField—An unlimited-length text field, or at least the largest text field the database makes available. The max_length attribute has no effect on the length of this field.
• TimeField—A field representing the time of day, without any associated date information. The database library should return a datetime.time object for values in this column.

Introspection of Existing Structures

In addition to being able to create new table structures based on model information, it’s also possible to use an existing table structure to generate new models. This isn’t a perfect process because some model information doesn’t get stored in the table’s own definition, but it’s a great starting point for new projects that have to work with existing databases, usually to run alongside a legacy application that’s being phased out.

The backend should provide a module called introspection.py for this purpose, which provides a DatabaseIntrospection class with a number of methods for retrieving various details about the table structures. Each method receives an active database cursor; all arguments and return values of each of these methods are documented in the following list, as well as another mapping for picking the right field types based on the underlying column types.

• get_table_list(cursor)—Returns a list of table names that are present in the database.
• get_table_description(cursor, table_name)—Given the name of a specific table, found using get_table_list(), this returns a list of tuples, each describing a column in the table. Each tuple follows PEP-249’s standard for the cursor’s description attribute: (name, type_code, display_size, internal_size, precision, scale, null_ok). The type_code here is an internal type used by the database to identify the column type, which will be used by the reverse mapping described at the end of this section.
• get_relations(cursor, table_name)—Given a table’s name, this returns a dictionary detailing the relationships the table has with other tables. Each key is the column’s index in the list of all columns, while the associated value is a 2-tuple. The first item is the index of the related field according to its table’s columns, and the second item is the name of the associated table. If the database doesn’t provide an easy way to access this information, this function can instead raise NotImplementedError, and relationships will just be excluded from the generated models.
• get_key_columns(cursor, table_name)—Given a table’s name, this returns a list of columns that relate to other tables and how those references work. Each item in the list is a tuple consisting of the column name, the table it references, and the column within that referenced table.
• get_indexes(cursor, table_name)—Given the name of a table, this returns a dictionary of all the fields that are indexed in any way. The dictionary’s keys are column names, while the values are additional dictionaries. Each value’s dictionary contains two keys: 'primary_key' and 'unique', each of which is either True or False. If both are False, the column is still indicated as indexed by virtue of being in the outer dictionary at all; it’s just an ordinary index, without primary key or unique constraints. Like get_relations(), this can also raise NotImplementedError if there’s no easy way to obtain this information.

In addition to the preceding methods, the introspection class also provides a dictionary called data_types_reverse, which maps the type_code values in the dictionary returned from get_table_description(). The keys are whatever values are returned with as type_code, regardless of whether that’s a string, integer, or something else entirely. The values are strings containing the names of the Django fields that will support the associated column type.

DatabaseClient

Living in the database backend’s client.py module, this class is responsible for calling the command-line interface (shell) for the current database specified by DATABASE_ENGINE. This is called using the manage.py dbshell command, allowing users to manage the underlying tables’ structure and data manually if necessary.

The class consists of just a single method, runshell(), which takes no arguments. This method is then responsible for reading the appropriate database settings for the given backend and configuring a call to the database’s shell program.

DatabaseError and IntegrityError

Pulled in from {{ backend }}.base, these classes allow exceptions to be handled easily, while still being able to swap out databases. IntegrityError should be a subclass of DatabaseError, so that applications can just check for DatabaseError if the exact type of error isn’t important.

Third-party libraries that conform to PEP-249 will already have these classes available, so they can often just be assigned to the base module’s namespace and work just fine. The only time they would need to be subclassed or defined directly is if the library being used doesn’t behave in a way that’s similar to other databases supported by Django. Remember, it’s all about consistency across the entire framework.

get_user(user_id)

Any time a user’s ID is known in advance, whether from a session variable, a database record, or somewhere else entirely, the authentication backend is responsible for converting that ID into a usable django.contrib.auth.models.User instance. What it means to be an ID could be different for different backends, so the exact type of this argument might also change depending on the backend being used. For django.contrib.auth.backends.ModelBackend, the default that ships with Django, this is the database ID where the user’s information is stored. For others, it might be a username, a domain name, or something else entirely.

authenticate(**credentials)

When the user’s ID isn’t known, it’s necessary to ask for some credentials, with which the appropriate User account can be identified and retrieved. In the default case, these credentials are a username and password, but others may use a URL or a single-use token, for example. In the real world, the backend won’t accept arguments using the ** syntax, but rather it will accept just those arguments that make sense for it. However, because different backends will take different sets of credentials, there’s no single method definition that will suit all cases.

Storing User Information

One aspect of authentication that might not seem obvious is that all users must, for all intents and purposes, still be represented in Django as User objects in the django.contrib.auth application. This isn’t strictly required by Django as a framework, but most applications—including the provided admin interface—expect users to exist in the database and will make relationships with that model.

For backends that call out to external services for authentication, this means duplicating every user in Django’s database to make sure applications work correctly. On the surface, this sounds like a maintenance nightmare; not only does every existing user need to be copied, but new users need to be added and changes to user information should also be reflected in Django. If all this had to be managed by hand for all users, it would certainly be a considerable problem.

Remember, though, that the only real requirement for an authentication backend is that it receives the user’s credentials and returns a User object. In between, it’s all just standard Python, and the whole of Django’s model API is up for grabs. Once a user has been authenticated behind the scenes, the backend can simply create a new User if one doesn’t already exist. If one does exist, it can even update the existing record with any new information that’s updated in the “real” user database. This way, everything can stay in sync without having to do anything special for Django. Just administer your users using whatever system you’re already using, and let your authentication backend handle the rest.

The Base File Class

Regardless of source, destination or purpose, all files in Django are represented as instances of django.core.files.File. This works very much like Python’s own file object, but with a few additions and modifications for use on the Web and with large files. Subclasses of File can alter what goes on behind the scenes, but the following API is standard for all file types. The following attributes are available on all File objects:

• File.closed—A Boolean indicating whether the file has been closed. When instantiated, all File objects are open, and its contents can be accessed immediately. The close() method sets this to True, and the file must be reopened using open() before its contents can be accessed again.
• File.DEFAULT_CHUNK_SIZE—Typically an attribute of the file’s class rather than an instance of it, this determines what size chunks should be used with the chunks() method.
• File.mode—The access mode the file was opened with; defaults to 'rb'.
• File.name—The name of the file, including any given path relative to where it was opened.
• File.size—The size of the file’s contents, in bytes.

The following methods are also available on File objects:

• File.chunks(chunk_size=None)—Iterates over the file’s contents, yielding it in one or more smaller chunks to avoid filling up the server’s available memory with large files. If no chunk_size is provided, the DEFAULT_CHUNK_SIZE, which defaults to 64 KB, will be used.
• File.close()—Closes the file so its contents become inaccessible.
• File.flush()—Writes any new pending contents to the actual filesystem.
• File.multiple_chunks(chunk_size=None)—Returns True if the file is big enough to require multiple calls to chunks() to retrieve the full contents, or False if it can all be read in one pass. The chunk_size argument works the same as in chunks(). Note that this will not actually read the file at this point; it determines the value based on the file’s size.
• File.open(mode=None)—Reopens the file if it had been previously closed. The mode argument is optional and will default to whatever mode the file had used when it was last open.
• File.read(num_bytes=None)—Retrieves a certain number of bytes from the file. If called without a size argument, this will read the remainder of the file.
• File.readlines()—Retrieves the content of the file as a list of lines, as indicated by the presence of newline characters ( and ) in the file. These newline characters are left at the end of each line in this list.
• File.seek(position)—Moves the internal position of the file to the specified location. All read and write operations are relative to this position, so this allows different parts of the file to be accessed by the same code.
• File.tell()—Returns the position of the internal pointer, as the number of bytes from the beginning of the file.
• File.write(content)—Writes the specified contents to the file. This is only available if the file was opened in write mode (a mode beginning with 'w').
• File.xreadlines()—A generator version of readlines() yielding one line, including newline characters, at a time. In keeping with Python’s own transition away from xreadlines(), this functionality is also provided by iterating over the File object itself.

Handling Uploads

When accepting files from users, things get a little bit trickier, because these files shouldn’t necessarily be saved alongside the rest of your files until your code has had a chance to review them. To facilitate this, Django treats uploaded files a bit differently, using upload handlers to decide what subclass of File should be used to represent them. Each upload handler has a chance to step in during the upload and alter how Django proceeds.

Upload handlers are specified with the FILE_UPLOAD_HANDLERS setting, which takes a sequence of import paths. As uploaded files are being processed, Django calls various methods on each of these handlers in turn, so they can inspect the data as it comes in. There’s no need to all these directly, as it’s automatically handled by Django’s request processing code, but the API for new upload handlers provides ample opportunity to customize how incoming files are managed.

• FileUploadHandler.__init__(request)—The handler is initialized every time a request comes in with files attached, and the incoming request is passed in so the handler can decide if it needs to handle the files for the request. For example, if it’s designed to write details of the upload to the console of the development server, it might check if the DEBUG setting is True and if request.META['REMOTE_ADDR'] is in the INTERNAL_IPS setting. If a handler should always process every request, this doesn’t need to be defined manually; the inherited default will suffice for most cases.
• FileUploadHandler.new_file(field_name, file_name, content_type, content_length, charset=None)—This is called for each file submitted in the request, with various details about the file, but none of its actual content. The field_name is the form field name that was used to upload the file, while the file_name is the name of the file itself as reported by the browser. The content_type, content_length and charset are all properties of the file’s contents, but they should be taken with a grain of salt because they can’t be verified without accessing the file’s contents.While not strictly required, the primary function of this method is to set aside a place for the file’s content to be stored when received_data_chunk() is called. There’s no requirement on what type of storage is used, or what attribute is used for it, so nearly anything’s fair game. Common examples are temporary files or StringIO objects. Also, this method provides a way to decide whether certain features should be enabled, such as automatically generated thumbnails of images, determined by the content_type.
• FileUploadHandler.receive_data_chunk(raw_data, start)—This is one of only two required methods and is called repeatedly throughout the processing of the file, each time receiving a portion of the file’s contents as raw_data, with start being the offset within the file where that content was found. The amount of data called each time is based on the handler’s chunk_size attribute, which defaults to 64KiB.Once this method has completed processing the data chunk, it can also control how other handlers deal with that data. This is determined by whether the method returns any data or not, with any data returned being passed along to the next handler in line. If it returns None, Django will simply repeat the process with the next chunk of data.
• FileUploadHandler.file_complete(file_size)—As a complement to new_file(), this method is called when Django finds the end of the file in the request. Because this is also the only time the file’s total size can be known with certainty, Django gives each handler a chance to determine what to do with that information.This is the only other required method on an upload handler and should return an UploadedFile object if the file was processed by this handler. The UploadedFile returned will be used by the associated form as the content for the field used to upload the file. If the handler didn’t do anything with the file, for whatever reason, this can return None. However, be careful with this because at least one upload handler must return an UploadedFile to be used with forms.
• FileUploadHandler.upload_complete()—While file_complete() is called when each file is finished loading, upload_complete() is called once per request, after all uploaded files have been processed completely. If the handler needs to set up any temporary resources while dealing with all the files, this method is the place to clean up after itself, freeing up resources for the rest of the application.

Notice that many of the features made possible by these methods rely on one method knowing what decisions a previous method has already made, but there’s no obvious way to persist this information. Since handlers are instantiated on every incoming request and process files one at a time, it’s possible to simply set custom attributes on the handler object itself, which future method calls can read back to determine how to proceed.

For example, if __init__() sets self.activated to False, receive_data_chunk() can read that attribute to determine whether it should process the chunks it receives or just pass them through to the next handler in line. It’s also possible for new_file() to set the same or similar attribute, so those types of decisions can be made on a per-file basis as well as per-request.

Because each handler works in isolation from the others, there isn’t any standard imposed on which attributes are used or what they’re used for. Instead, interaction among the various installed upload handlers is handled by raising a number of exceptions in various situations. Proper operation of an upload handler doesn’t require the use of any of these, but they can greatly customize how a number of them can work together. Like FileUploadHandler, these are all available at django.core.files.uploadhander.

• StopUpload—Tells Django to stop processing all files in the upload, preventing all handlers from handling any more data than they’ve already processed. It also accepts a single optional argument, connection_reset, a Boolean indicating whether Django should stop without reading in the remainder of the input stream. The default value of False for this argument means that Django will read the entire request before passing control back to a form, while True will stop without reading it all in, resulting in a “Connection Reset” message shown in the user’s browser.
• SkipFile—Tells the upload process to stop processing the current file, but continue on with the next one in the list. This is a much more appropriate behavior if there were a problem with a single file in the request, which wouldn’t affect any other files that might be uploaded at the same time.
• StopFutureHandlers—Only valid if thrown from the new_file() method, this indicates the current upload handler will handle current file directly, and no other handlers should receive any data after it. Any handlers that process data before the handler that raises this exception will continue to execute in their original order, as determined by their placement within the FILE_UPLOAD_HANDLERS setting.

Storing Files

All file storage operations are handled by instances of StorageBase, which lives at django.core.files.storage, with the default storage system specified by an import path in the DEFAULT_FILE_STORAGE setting. A storage system encompasses all the necessary functions for dealing with how and where files are stored and retrieved. By using this extra layer, it’s possible to swap out which storage system is used, without having to make any changes to existing code. This is especially important when moving from development to production because production servers often have specialized needs for storing and serving static files.

To facilitate this level of flexibility, Django provides an API for dealing with files that goes beyond the standard open() function and associated file object provided by Python. Earlier in this chapter, Django’s File object was described, explaining what features are available for dealing with individual files. However, when looking to store, retrieve or list files, storage systems have a different set of tools available.

• Storage.delete(name)—Deletes a file from the storage system.
• Storage.exists(name)—Returns a Boolean indicating whether the specified name references a file that already exists in the storage system.
• Storage.get_valid_name(name)—Returns a version of the given name that’s suitable for use with the current storage system. If it’s already valid, it will be returned unchanged. One of only two methods with default implementations, this will return filenames suitable for a local filesystem, regardless of operating system.
• Storage.get_available_name(name)—Given a valid name, this returns a version of it that’s actually available for new files to be written, without overwriting any existing files. Being the other method with a default behavior, this will add underscores to the end of the requested name until an available name is found.
• Storage.open(name, mode='rb', mixin=None)—Returns an open File object, through which the file’s contents can be accessed. The mode accepts all the same arguments as Python’s open() function, allowing for both read and write access. The optional mixin argument accepts a class to be used alongside the File subclass provided by the storage system, to enable additional features on the file returned.
• Storage.path(name)—Returns the absolute path to the file on the local filesystem, which can be used with Python’s built-in open() function to access the file directly. This is provided as a convenience for the common case where files are stored on the local filesystem. For other storage systems, this will raise a NotImplementedError if there is no valid filesystem path at which the file can be accessed. Unless you’re using a library that only accepts file paths instead of open file objects, you should always open files using Storage.open(), which works across all storage systems.
• Storage.save(name, content)—Saves the given content to the storage system, preferably under the given name. This name will be passed through get_valid_name() and get_available_name() before being saved, and the return value of this method will be the name that was actually used to store the content. The content argument provided to this method should be a File object, typically as a result of a file upload.
• Storage.size(name)—Returns the size, in bytes, of the file referenced by name.
• Storage.url(name)—Returns an absolute URL where the file’s contents can be accessed directly by a Web browser.
• listdir(path)—Returns the contents of the directory specified by the path argument. The return value is a tuple containing two lists: the first for directories located at the path and the second for files located at that same path.

By default, Django ships with FileSystemStorage, which, as the name implies, stores files on the local filesystem. Typically this means the server’s hard drive, but there are many ways to map other types of filesystems to local paths, so there are already a number of possibilities. There are even more storage options available, though, and there are plenty of ways to customize how even the existing options behave. By subclassing StorageBase, it’s possible to make available a number of other options.

There are a number of things a storage system must provide, starting with most of these methods. One of those methods, get_available_name(), doesn’t strictly need to be supplied by the new storage class because its default implementation is suitable for many situations; overriding it is a matter of preference, not requirement. On the other hand, the get_valid_name() method has a default behavior that’s suitable for most backends, but some might have different file naming requirements and would require a new method to override it.

Two other methods, open() and save(), have still further requirements. By definition, both of these require special handling for each different storage system, but they shouldn’t be overridden directly in most situations. They provide additional logic beyond what’s necessary to store and retrieve files, and that logic should be maintained. Instead, they defer the interaction with the actual storage mechanism to _open() and _save(), respectively, which have a simpler set of expectations.

• Storage._open(name, mode='rb')—The name and mode arguments are the same as open(), but it no longer has the mixin logic to deal with, so _open() can focus solely on returning a File object suitable for accessing the requested file.
• Storage._save(name, content)—The arguments here are the same as save(), but the name provided here will have already gone through get_valid_name() and get_available_name(), and the content is guaranteed to be a File instance. This allows the _save() method to focus solely on committing the file’s content to the storage system with the given name.

In addition to providing these methods, most custom storage systems will also need to provide a File subclass with read() and write() methods that are designed to access the underlying data in the most efficient manner. The chunks() method defers to read() internally, so there shouldn’t need to be anything done there to make large files more memory-friendly for applications to work with. Keep in mind that not all filesystems allow reading or writing just part of a file, so the File subclass might also need to take additional steps to minimize both memory usage and network traffic in these situations.

Specifying a Backend

Specifying a cache backend in Django works quite a bit differently than other backends discussed in this chapter. Even though there are multiple configuration options to consider, there’s just one setting to control them all. This setting, CACHE_BACKEND, uses the URI syntax⁶ to accept all the necessary information in a way that can be parsed reliably. It can be split up into three separate parts, each with its own requirements.

CACHE_BACKEND = '{{ scheme }}://{{ host }}/?{{ arguments }}'

• The scheme portion specifies which backend code should be used to serve out the cache. Django ships with four backends that cover most cases—db, file, locmem, and memcached ⁷—which are well documented online and cover the majority of cases. For custom backends, this portion of the setting can also accept a full import path to a module that implements the protocol described in the next section.
• The host specifies where the cache should actually be stored, and its format will vary depending on the backend used. For example, db expects a single database name, file expects a full directory path, memcached expects a list of server addresses, and locmem doesn’t require anything at all. The host can also include by a trailing slash, which can help readability because it makes the whole setting look more like a URI.
• Arguments are optional and can be provided to customize how caching takes place within the backend. They’re provided using the query-string format, with one argument required for all backends: timeout, the number of seconds before an item should be removed from the cache. Two more arguments are also available for most backends (including all those supplied by Django except for memcached): max_entries, the total number of items that should be stored in the cache before culling old items; and cull_frequency, which controls how many items to purge from the cache when it reaches max_entries.
• One important thing to realize about cull_frequency is that its value isn’t actually how often items should be removed. Instead, the value is used in a simple formula, 1 / cull_frequency, which determines how many items are affected. So, if you’d like to purge 25% of the items at a time, that’s equivalent to 1/4, so you’d pass cull_frequency=4 as an argument to the cache backend, while half (1/2) of the entries would require passing cull_frequency=2. Essentially, cull_frequency is the number of times the cache must be culled to guarantee all items are purged.

Using the Cache Manually

In addition to the standard site-wide and per-view caching options, it’s also quite simple to use the cache directly, storing specific values so they can be retrieved later without having to perform expensive operations for data that doesn’t change often. This low-level API is available in a generic form through the cache object, living at django.core.cache. Most of the usefulness of this object comes from three methods—get(), set(), and delete()—which work mostly how you’d expect.

>>> cache.set('result', 2 ** 16 – 64 * 4)
>>> print cache.get('result')
65280
>>> cache.delete('result')
>>> print cache.get('result')
None

There are a few details about these methods that bear a little more explanation, and also some additional methods that prove useful. Here is a full list of the available methods, along with their functional details.

• CacheClass.set(key, value, timeout=None)—This sets the specified value in the cache, using the provided key. By default, the timeout for values to expire from the cache is determined by the timeout passed into the CACHE_BACKEND setting, but that can be overridden by specifying a different timeout as an argument to this method.
• CacheClass.get(key, default=None)—This method returns the value contained in the cache for the specified key. Normally, cache.get() returns None if the key doesn’t exist in the cache, but sometimes None is a valid value to have in the cache. In these cases, just set default to some value that shouldn’t exist in the cache, and that will be returned instead of None.
• CacheClass.delete(key)—This deletes the value associated with the given key.
• CacheClass.get_many(keys)—Given a list of keys, it returns a corresponding list of their values. For some backends, like memcached, this can provide a speed increase over calling cache.get() for each individual key.
• CacheClass.has_key(key)—This method returns True if the specified key has a value already in the cache or False if the key wasn’t set or has already expired.
• CacheClass.add(key, value, timeout=None)—This method only attempts to add a new key to the cache, using the specified value and timeout. If the given key already exists in the cache, this method will not update the cache to the new value.

A common idiom when working with cache is to first check to see if a value is already present in the cache and, if not, calculate it and store it in the cache. Then, the value can be retrieved from the cache regardless of whether it was there to begin with, making the code nice and simple. To make this a bit more Pythonic, the cache object also functions a bit like a dictionary, supporting the in operator as an alias for the has_key() method.

def get_complex_data(complex_data):
    if 'complex-data-key' not in cache:
        # Perform complex operations to generate the data here.
        cache.set('complex-data-key', complex_data)
    return cache.get('complex-data-key')

load_template_source(template_name, template_dirs=None)

While the loader can be called anything, the name Django uses for all of its template loaders is load_template_source, so it’s generally best to stick to that convention for ease of understanding. This is also typically placed in its own module, but again, the import path has to be supplied explicitly, so just make sure its location is well-documented.

The first argument is obviously the name of the template to be loaded, which is usually just a standard filename. This doesn’t have to map to an actual file, but views will typically request templates using a filename, so it’s up to the template loader to convert this name to whatever reference is used for templates. That may be database records, URLs pointing to external storage systems, or anything else your site might use to store and load templates.

The second argument to load_template_source() is a list of directories to use when searching for the template. Within Django itself, this is typically not provided, so the default of None is used, indicating that the TEMPLATE_DIRS setting should be used instead. A loader that uses the filesystem should always follow this behavior to maintain consistency with the way other template loaders work. If the loader retrieves templates from somewhere else, this argument can simply be ignored.

What goes on inside the template loader will be quite different from one template loader to the next, varying based on how each loader locates templates. Once a template is found, the loader must return a tuple containing two values: the template’s contents as a string, and a string indicating where the template was found. That second value is used to generate the origin argument to the new Template object, so that it’s easy to find a template if anything goes wrong.

If the given name doesn’t match any templates the loader knows about, it should raise the TemplateDoesNotExist exception, described in Chapter 6. This will instruct Django to move on to the next template loader in the list or to display an error if there are no more loaders to use.

load_template_source.is_usable

If the Python environment doesn’t have the requirements for a template loader to operate, Django also provides a way for the loader to indicate that it shouldn’t be used. This is useful if a template loader relies on a third-party library that hasn’t been installed. Adding an is_usable attribute to the function, set to True or False, will tell Django whether the template loader can be used.

load_template(template_name, template_dirs=None)

In addition to simply loading the source code for a template, this method is responsible for returning a template capable of being rendered. By default, the source returned from load_template_source() is processed by Django’s own template language, but this gives you a chance to replace that with something else entirely. This should still use the load_template_source() method internally to fetch the code for the template, so that users can separate the decision of where to find templates from how those templates should be interpreted.

The return value only needs one method to work properly: render(context). This render() method accepts a template context and returns a string generated by the template source code. The context passed in here works a lot like a standard dictionary, but Django’s contexts are actually a stack of dictionaries, so if you intend to pass this context into another template rendered, you probably need to flatten it to a single dictionary first.

flat_dict = {}
for d in context.dicts:
    flat_dict.update(d)

After this, you’ll have a single dictionary with all the values in it, which is usually suitable for most template languages.

Scanning Incoming Files for Viruses

For sites that allow users to upload files to be distributed to other users, a large amount of trust is placed on the quality of those incoming files. As with any form of user input, there must be a certain level of distrust in this information because there’s always someone out there who wants to do harm to your site and its users.

When looking to let users share specific types of files, it’s often easy to validate using third-party libraries designed to understand those files. Sharing arbitrary files, on the other hand, opens up a world of other possibilities, many of which put your site and its users at risk. Protecting against viruses is an important part of the safety of such an application, and Django’s upload handlers make this an extremely simple task.

For this example, we’ll use an excellent open source virus scanning application, ClamAV,⁸ which is designed for use in servers, along with pyclamd,⁹ a Python library for interacting with ClamAV. Together, these provide an easy-to-use interface for scanning any incoming file before it’s even passed to the rest of the application. If a virus is found, the offending file can simply be removed from the input stream immediately, before it can do any harm to anyone.

import pyclamd
from django.core.files import uploadhandler
from django.conf import settings
 
# Set up pyclamd to access running instance of clamavd, according to settings
host = getattr(settings, 'CLAMAV_HOST', 'localhost')
port = getattr(settings, 'CLAMAV_PORT', 3310)
pyclamd.init_network_socket(host, port)
 
class VirusScan(uploadhandler.FileUploadHandler):
    def receive_data_chunk(self, raw_data, start):        try:
          if pyclamd.scan_stream(raw_data):
            # A virus was found, so the file should
            # be removed from the input stream.
            raise uploadhandler.SkipFile()
        except pyclamd.ScanError:
            # Clam AV couldn't be contacted, so the file wasn't scanned.
            # Since we can't guarantee the safety of any files,
            # no other files should be processed either.
            raise uploadhander.StopUpload()
 
        # If everything went fine, pass the data along
        return raw_data
 
    def file_complete(self, file_size):
        # This doesn't store the file anywhere, so it should
        # rely on other handlers to provide a File instance.
        return None

Your application may have more specific requirements, like explaining to users which virus was found and that they should consider cleaning their own system before attempting to share files with others. The key to this example is how easy it is to implement this type of behavior, which might seem very difficult on the surface.