When using a list, you can change its contents by assigning to either a particular item (offset) or an entire section (slice):

>>> L = ['spam', 'Spam', 'SPAM!']
>>> L[1] = 'eggs'                 # Index assignment
>>> L
['spam', 'eggs', 'SPAM!']
>>> L[0:2] = ['eat', 'more']      # Slice assignment: delete+insert
>>> L                             # Replaces items 0,1
['eat', 'more', 'SPAM!']

Both index and slice assignments are in-place changes—they modify the subject list directly, rather than generating a new list object for the result. Index assignment in Python works much as it does in C and most other languages: Python replaces the object reference at the designated offset with a new one.

Slice assignment, the last operation in the preceding example, replaces an entire section of a list in a single step. Because it can be a bit complex, it is perhaps best thought of as a combination of two steps:

This isn’t what really happens, but it tends to help clarify why the number of items inserted doesn’t have to match the number of items deleted. For instance, given a list L that has the value [1,2,3], the assignment L[1:2]=[4,5] sets L to the list [1,4,5,3]. Python first deletes the 2 (a one-item slice), then inserts the 4 and 5 where the deleted 2 used to be. This also explains why L[1:2]=[] is really a deletion operation—Python deletes the slice (the item at offset 1), and then inserts nothing.

In effect, slice assignment replaces an entire section, or “column,” all at once. Because the length of the sequence being assigned does not have to match the length of the slice being assigned to, slice assignment can be used to replace (by overwriting), expand (by inserting), or shrink (by deleting) the subject list. It’s a powerful operation, but frankly, one that you may not see very often in practice. There are usually more straightforward ways to replace, insert, and delete (concatenation and the insert, pop, and remove list methods, for example), which Python programmers tend to prefer in practice.

Like strings, Python list objects also support type-specific method calls, many of which change the subject list in-place:

>>> L.append('please')                # Append method call: add item at end
>>> L
['eat', 'more', 'SPAM!', 'please']
>>> L.sort()                          # Sort list items ('S' < 'e')
>>> L
['SPAM!', 'eat', 'more', 'please']

Methods were introduced in Chapter 7. In brief, they are functions (really, attributes that reference functions) that are associated with particular objects. Methods provide type-specific tools; the list methods presented here, for instance, are generally available only for lists.

Perhaps the most commonly used list method is append, which simply tacks a single item (object reference) onto the end of the list. Unlike concatenation, append expects you to pass in a single object, not a list. The effect of L.append(X) is similar to L+[X], but while the former changes L in-place, the latter makes a new list.^[23]

Another commonly seen method, sort, orders a list in-place; it uses Python standard comparison tests (here, string comparisons), and by default sorts in ascending order. You can modify sort behavior by passing in keyword arguments—a special “name=value” syntax in function calls that specifies passing by name and is often used for giving configuration options. In sorts, the key argument gives a one-argument function that returns the value to be used in sorting, and the reverse argument allows sorts to be made in descending instead of ascending order:

>>> L = ['abc', 'ABD', 'aBe']
>>> L.sort()                                # Sort with mixed case
>>> L
['ABD', 'aBe', 'abc']
>>> L = ['abc', 'ABD', 'aBe']
>>> L.sort(key=str.lower)                   # Normalize to lowercase
>>> L
['abc', 'ABD', 'aBe']
>>>
>>> L = ['abc', 'ABD', 'aBe']
>>> L.sort(key=str.lower, reverse=True)     # Change sort order
>>> L
['aBe', 'ABD', 'abc']

The sort key argument might also be useful when sorting lists of dictionaries, to pick out a sort key by indexing each dictionary. We’ll study dictionaries later in this chapter, and you’ll learn more about keyword function arguments in Part IV.

One warning here: beware that append and sort change the associated list object in-place, but don’t return the list as a result (technically, they both return a value called None). If you say something like L=L.append(X), you won’t get the modified value of L (in fact, you’ll lose the reference to the list altogether!). When you use attributes such as append and sort, objects are changed as a side effect, so there’s no reason to reassign.

Partly because of such constraints, sorting is also available in recent Pythons as a built-in function, which sorts any collection (not just lists) and returns a new list for the result (instead of in-place changes):

>>> L = ['abc', 'ABD', 'aBe']
>>> sorted(L, key=str.lower, reverse=True)          # Sorting built-in
['aBe', 'ABD', 'abc']

>>> L = ['abc', 'ABD', 'aBe']
>>> sorted([x.lower() for x in L], reverse=True)    # Pretransform items: differs!
['abe', 'abd', 'abc']

Notice the last example here—we can convert to lowercase prior to the sort with a list comprehension, but the result does not contain the original list’s values as it does with the key argument. The latter is applied temporarily during the sort, instead of changing the values to be sorted. As we move along, we’ll see contexts in which the sorted built-in can sometimes be more useful than the sort method.

Like strings, lists have other methods that perform other specialized operations. For instance, reverse reverses the list in-place, and the extend and pop methods insert multiple items at the end of and delete an item from the end of the list, respectively. There is also a reversed built-in function that works much like sorted, but it must be wrapped in a list call because it’s an iterator (more on iterators later):

>>> L = [1, 2]
>>> L.extend([3,4,5])                # Add many items at end
>>> L
[1, 2, 3, 4, 5]
>>> L.pop()                          # Delete and return last item
5
>>> L
[1, 2, 3, 4]
>>> L.reverse()                      # In-place reversal method
>>> L
[4, 3, 2, 1]
>>> list(reversed(L))                # Reversal built-in with a result
[1, 2, 3, 4]

In some types of programs, the list pop method used here is often used in conjunction with append to implement a quick last-in-first-out (LIFO) stack structure. The end of the list serves as the top of the stack:

>>> L = []
>>> L.append(1)                      # Push onto stack
>>> L.append(2)
>>> L
[1, 2]
>>> L.pop()                          # Pop off stack
2
>>> L
[1]

The pop method also accepts an optional offset of the item to be deleted and returned (the default is the last item). Other list methods remove an item by value (remove), insert an item at an offset (insert), search for an item’s offset (index), and more:

>>> L = ['spam', 'eggs', 'ham']
>>> L.index('eggs')                  # Index of an object
1
>>> L.insert(1, 'toast')             # Insert at position
>>> L
['spam', 'toast', 'eggs', 'ham']
>>> L.remove('eggs')                 # Delete by value
>>> L
['spam', 'toast', 'ham']
>>> L.pop(1)                         # Delete by position
'toast'
>>> L
['spam', 'ham']

See other documentation sources or experiment with these calls interactively on your own to learn more about list methods.

Because lists are mutable, you can use the del statement to delete an item or section in-place:

>>> L
['SPAM!', 'eat', 'more', 'please']
>>> del L[0]                         # Delete one item
>>> L
['eat', 'more', 'please']
>>> del L[1:]                        # Delete an entire section
>>> L                                # Same as L[1:] = []
['eat']

Because slice assignment is a deletion plus an insertion, you can also delete a section of a list by assigning an empty list to a slice (L[i:j]=[]); Python deletes the slice named on the left, and then inserts nothing. Assigning an empty list to an index, on the other hand, just stores a reference to the empty list in the specified slot, rather than deleting it:

>>> L = ['Already', 'got', 'one']
>>> L[1:] = []
>>> L
['Already']
>>> L[0] = []
>>> L
[[]]

Although all the operations just discussed are typical, there are additional list methods and operations not illustrated here (including methods for inserting and searching). For a comprehensive and up-to-date list of type tools, you should always consult Python’s manuals, Python’s dir and help functions (which we first met in Chapter 4), or one of the reference texts mentioned in the Preface.

I’d also like to remind you one more time that all the in-place change operations discussed here work only for mutable objects: they won’t work on strings (or tuples, discussed in Chapter 9), no matter how hard you try. Mutability is an inherent property of each object type.

The last point in the prior list is important enough to demonstrate with a few examples. When you use lists, it is illegal to assign to an offset that is off the end of the list:

>>> L = []
>>> L[99] = 'spam'
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
IndexError: list assignment index out of range

Although you can use repetition to preallocate as big a list as you’ll need (e.g., [0]*100), you can also do something that looks similar with dictionaries that does not require such space allocations. By using integer keys, dictionaries can emulate lists that seem to grow on offset assignment:

>>> D = {}
>>> D[99] = 'spam'
>>> D[99]
'spam'
>>> D
{99: 'spam'}

Here, it looks as if D is a 100-item list, but it’s really a dictionary with a single entry; the value of the key 99 is the string 'spam'. You can access this structure with offsets much like a list, but you don’t have to allocate space for all the positions you might ever need to assign values to in the future. When used like this, dictionaries are like more flexible equivalents of lists.

In a similar way, dictionary keys are also commonly leveraged to implement sparse data structures—for example, multidimensional arrays where only a few positions have values stored in them:

>>> Matrix = {}
>>> Matrix[(2, 3, 4)] = 88
>>> Matrix[(7, 8, 9)] = 99
>>>
>>> X = 2; Y = 3; Z = 4           # ; separates statements
>>> Matrix[(X, Y, Z)]
88
>>> Matrix
{(2, 3, 4): 88, (7, 8, 9): 99}

Here, we’ve used a dictionary to represent a three-dimensional array that is empty except for the two positions (2,3,4) and (7,8,9). The keys are tuples that record the coordinates of nonempty slots. Rather than allocating a large and mostly empty three-dimensional matrix to hold these values, we can use a simple two-item dictionary. In this scheme, accessing an empty slot triggers a nonexistent key exception, as these slots are not physically stored:

>>> Matrix[(2,3,6)]
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
KeyError: (2, 3, 6)

Errors for nonexistent key fetches are common in sparse matrixes, but you probably won’t want them to shut down your program. There are at least three ways to fill in a default value instead of getting such an error message—you can test for keys ahead of time in if statements, use a try statement to catch and recover from the exception explicitly, or simply use the dictionary get method shown earlier to provide a default for keys that do not exist:

>>> if (2,3,6) in Matrix:              # Check for key before fetch
...     print(Matrix[(2,3,6)])         # See Chapter 12 for if/else
... else:
...     print(0)
...
0
>>> try:
...     print(Matrix[(2,3,6)])         # Try to index
... except KeyError:                   # Catch and recover
...     print(0)                       # See Chapter 33 for try/except
...
0
>>> Matrix.get((2,3,4), 0)             # Exists; fetch and return
88
>>> Matrix.get((2,3,6), 0)             # Doesn't exist; use default arg
0

Of these, the get method is the most concise in terms of coding requirements; we’ll study the if and try statements in more detail later in this book.

As you can see, dictionaries can play many roles in Python. In general, they can replace search data structures (because indexing by key is a search operation) and can represent many types of structured information. For example, dictionaries are one of many ways to describe the properties of an item in your program’s domain; that is, they can serve the same role as “records” or “structs” in other languages.

The following, for example, fills out a dictionary by assigning to new keys over time:

>>> rec = {}
>>> rec['name'] = 'mel'
>>> rec['age']  = 45
>>> rec['job']  = 'trainer/writer'
>>>
>>> print(rec['name'])
mel

Especially when nested, Python’s built-in data types allow us to easily represent structured information. This example again uses a dictionary to capture object properties, but it codes it all at once (rather than assigning to each key separately) and nests a list and a dictionary to represent structured property values:

>>> mel = {'name': 'Mark',
...        'jobs': ['trainer', 'writer'],
...        'web':  'www.rmi.net/˜lutz',
...        'home': {'state': 'CO', 'zip':80513}}

To fetch components of nested objects, simply string together indexing operations:

>>> mel['name']
'Mark'
>>> mel['jobs']
['trainer', 'writer']
>>> mel['jobs'][1]
'writer'
>>> mel['home']['zip']
80513

Although we’ll learn in Part VI that classes (which group both data and logic) can be better in this record role, dictionaries are an easy-to-use tool for simpler requirements.

import dbm
file = dbm.open("filename")    # Link to file
file['key'] = 'data'           # Store data by key
data = file['key']             # Fetch data by key

import cgi
form = cgi.FieldStorage()      # Parse form data
if 'name' in form:
    showReply('Hello, ' + form['name'].value)

As mentioned at the end of the prior section, dictionaries in 3.0 can also be created with dictionary comprehensions. Like the set comprehensions we met in Chapter 5, dictionary comprehensions are available only in 3.0 (not in 2.6). Like the longstanding list comprehensions we met briefly in Chapter 4 and earlier in this chapter, they run an implied loop, collecting the key/value results of expressions on each iteration and using them to fill out a new dictionary. A loop variable allows the comprehension to use loop iteration values along the way.

For example, a standard way to initialize a dictionary dynamically in both 2.6 and 3.0 is to zip together its keys and values and pass the result to the dict call. As we’ll learn in more detail in Chapter 13, the zip function is a way to construct a dictionary from key and value lists in a single call. If you cannot predict the set of keys and values in your code, you can always build them up as lists and zip them together:

>>> list(zip(['a', 'b', 'c'], [1, 2, 3]))        # Zip together keys and values
[('a', 1), ('b', 2), ('c', 3)]

>>> D = dict(zip(['a', 'b', 'c'], [1, 2, 3]))    # Make a dict from zip result
>>> D
{'a': 1, 'c': 3, 'b': 2}

In Python 3.0, you can achieve the same effect with a dictionary comprehension expression. The following builds a new dictionary with a key/value pair for every such pair in the zip result (it reads almost the same in Python, but with a bit more formality):

C:misc> c:python30python                      # Use a dict comprehension

>>> D = {k: v for (k, v) in zip(['a', 'b', 'c'], [1, 2, 3])}
>>> D
{'a': 1, 'c': 3, 'b': 2}

Comprehensions actually require more code in this case, but they are also more general than this example implies—we can use them to map a single stream of values to dictionaries as well, and keys can be computed with expressions just like values:

>>> D = {x: x ** 2 for x in [1, 2, 3, 4]}        # Or: range(1, 5)
>>> D
{1: 1, 2: 4, 3: 9, 4: 16}

>>> D = {c: c * 4 for c in 'SPAM'}               # Loop over any iterable
>>> D
{'A': 'AAAA', 'P': 'PPPP', 'S': 'SSSS', 'M': 'MMMM'}

>>> D = {c.lower(): c + '!' for c in ['SPAM', 'EGGS', 'HAM']}
>>> D
{'eggs': 'EGGS!', 'ham': 'HAM!', 'spam': 'SPAM!'}

Dictionary comprehensions are also useful for initializing dictionaries from keys lists, in much the same way as the fromkeys method we met at the end of the preceding section:

>>> D = dict.fromkeys(['a', 'b', 'c'], 0)        # Initialize dict from keys
>>> D
{'a': 0, 'c': 0, 'b': 0}

>>> D = {k:0 for k in ['a', 'b', 'c']}           # Same, but with a comprehension
>>> D
{'a': 0, 'c': 0, 'b': 0}

>>> D = dict.fromkeys('spam')                    # Other iterators, default value
>>> D
{'a': None, 'p': None, 's': None, 'm': None}

>>> D = {k: None for k in 'spam'}
>>> D
{'a': None, 'p': None, 's': None, 'm': None}

Like related tools, dictionary comprehensions support additional syntax not shown here, including nested loops and if clauses. Unfortunately, to truly understand dictionary comprehensions, we need to also know more about iteration statements and concepts in Python, and we don’t yet have enough information to address that story well. We’ll learn much more about all flavors of comprehensions (list, set, and dictionary) in Chapters 14 and 20, so we’ll defer further details until later. We’ll also study the zip built-in we used in this section in more detail in Chapter 13, when we explore for loops.

In 3.0 the dictionary keys, values, and items methods all return view objects, whereas in 2.6 they return actual result lists. View objects are iterables, which simply means objects that generate result items one at a time, instead of producing the result list all at once in memory. Besides being iterable, dictionary views also retain the original order of dictionary components, reflect future changes to the dictionary, and may support set operations. On the other hand, they are not lists, and they do not support operations like indexing or the list sort method; nor do they display their items when printed.

We’ll discuss the notion of iterables more formally in Chapter 14, but for our purposes here it’s enough to know that we have to run the results of these three methods through the list built-in if we want to apply list operations or display their values:

>>> D = dict(a=1, b=2, c=3)
>>> D
{'a': 1, 'c': 3, 'b': 2}

>>> K = D.keys()                   # Makes a view object in 3.0, not a list
>>> K
<dict_keys object at 0x026D83C0>
>>> list(K)                        # Force a real list in 3.0 if needed
['a', 'c', 'b']

>>> V = D.values()                 # Ditto for values and items views
>>> V
<dict_values object at 0x026D8260>
>>> list(V)
[1, 3, 2]

>>> list(D.items())
[('a', 1), ('c', 3), ('b', 2)]

>>> K[0]                           # List operations fail unless converted
TypeError: 'dict_keys' object does not support indexing
>>> list(K)[0]
'a'

Apart from when displaying results at the interactive prompt, you will probably rarely even notice this change, because looping constructs in Python automatically force iterable objects to produce one result on each iteration:

>>> for k in D.keys(): print(k)    # Iterators used automatically in loops
...
a
c
b

In addition, 3.0 dictionaries still have iterators themselves, which return successive keys—as in 2.6, it’s still often not necessary to call keys directly:

>>> for key in D: print(key)       # Still no need to call keys() to iterate
...
a
c
b

Unlike 2.X’s list results, though, dictionary views in 3.0 are not carved in stone when created—they dynamically reflect future changes made to the dictionary after the view object has been created:

>>> D = {'a':1, 'b':2, 'c':3}
>>> D
{'a': 1, 'c': 3, 'b': 2}

>>> K = D.keys()
>>> V = D.values()
>>> list(K)                        # Views maintain same order as dictionary
['a', 'c', 'b']
>>> list(V)
[1, 3, 2]

>>> del D['b']                     # Change the dictionary in-place
>>> D
{'a': 1, 'c': 3}

>>> list(K)                        # Reflected in any current view objects
['a', 'c']
>>> list(V)                        # Not true in 2.X!
[1, 3]

Also unlike 2.X’s list results, 3.0’s view objects returned by the keys method are set-like and support common set operations such as intersection and union; values views are not, since they aren’t unique, but items results are if their (key, value) pairs are unique and hashable. Given that sets behave much like valueless dictionaries (and are even coded in curly braces like dictionaries in 3.0), this is a logical symmetry. Like dictionary keys, set items are unordered, unique, and immutable.

Here is what keys lists look like when used in set operations. In set operations, views may be mixed with other views, sets, and dictionaries (dictionaries are treated the same as their keys views in this context):

>>> K | {'x': 4}                   # Keys (and some items) views are set-like
{'a', 'x', 'c'}

>>> V & {'x': 4}
TypeError: unsupported operand type(s) for &: 'dict_values' and 'dict'
>>> V & {'x': 4}.values()
TypeError: unsupported operand type(s) for &: 'dict_values' and 'dict_values'

>>> D = {'a':1, 'b':2, 'c':3}
>>> D.keys() & D.keys()            # Intersect keys views
{'a', 'c', 'b'}
>>> D.keys() & {'b'}               # Intersect keys and set
{'b'}
>>> D.keys() & {'b': 1}            # Intersect keys and dict
{'b'}
>>> D.keys() | {'b', 'c', 'd'}     # Union keys and set
{'a', 'c', 'b', 'd'}

Dictionary items views are set-like too if they are hashable—that is, if they contain only immutable objects:

>>> D = {'a': 1}
>>> list(D.items())                # Items set-like if hashable
[('a', 1)]
>>> D.items() | D.keys()           # Union view and view
{('a', 1), 'a'}
>>> D.items() | D                  # dict treated same as its keys
{('a', 1), 'a'}

>>> D.items() | {('c', 3), ('d', 4)}           # Set of key/value pairs
{('a', 1), ('d', 4), ('c', 3)}
>>> dict(D.items() | {('c', 3), ('d', 4)})     # dict accepts iterable sets too
{'a': 1, 'c': 3, 'd': 4}

For more details on set operations in general, see Chapter 5. Now, let’s look at three other quick coding notes for 3.0 dictionaries.

First of all, because keys does not return a list, the traditional coding pattern for scanning a dictionary by sorted keys in 2.X won’t work in 3.0. You must either convert to a list manually or use the sorted call introduced in Chapter 4 and earlier in this chapter on either a keys view or the dictionary itself:

>>> D = {'a':1, 'b':2, 'c':3}
>>> D
{'a': 1, 'c': 3, 'b': 2}

>>> Ks = D.keys()                            # Sorting a view object doesn't work!
>>> Ks.sort()
AttributeError: 'dict_keys' object has no attribute 'sort'

>>> Ks = list(Ks)                            # Force it to be a list and then sort
>>> Ks.sort()
>>> for k in Ks: print(k, D[k])
...
a 1
b 2
c 3

>>> D
{'a': 1, 'c': 3, 'b': 2}
>>> Ks = D.keys()                            # Or you can use sorted() on the keys
>>> for k in sorted(Ks): print(k, D[k])      # sorted() accepts any iterable
...                                          # sorted() returns its result
a 1
b 2
c 3

>>> D
{'a': 1, 'c': 3, 'b': 2}                     # Better yet, sort the dict directly
>>> for k in sorted(D): print(k, D[k])       # dict iterators return keys
...
a 1
b 2
c 3

Secondly, while in Python 2.6 dictionaries may be compared for relative magnitude directly with <, >, and so on, in Python 3.0 this no longer works. However, it can be simulated by comparing sorted keys lists manually:

sorted(D1.items()) < sorted(D2.items())      # Like 2.6 D1 < D2

Dictionary equality tests still work in 3.0, though. Since we’ll revisit this in the next chapter in the context of comparisons at large, we’ll defer further details here.

Finally, the widely used dictionary has_key key presence test method is gone in 3.0. Instead, use the in membership expression, or a get with a default test (of these, in is generally preferred):

>>> D
{'a': 1, 'c': 3, 'b': 2}

>>> D.has_key('c')                                        # 2.X only: True/False
AttributeError: 'dict' object has no attribute 'has_key'

>>> 'c' in D
True
>>> 'x' in D
False
>>> if 'c' in D: print('present', D['c'])                 # Preferred in 3.0
...
present 3

>>> print(D.get('c'))
3
>>> print(D.get('x'))
None
>>> if D.get('c') != None: print('present', D['c'])       # Another option
...
present 3

If you work in 2.6 and care about 3.0 compatibility, note that the first two changes (comprehensions and views) can only be coded in 3.0, but the last three (sorted, manual comparisons, and in) can be coded in 2.6 today to ease 3.0 migration in the future.