Before we can start improving performance, we need a reliable method to measure it. Python has a really nice module (timeit) with the specific purpose of measuring execution times of bits of code. It executes a bit of code many times to make sure there is as little variation as possible and to make the measurement fairly clean. It's very useful if you want to compare a few code snippets. Following are example executions:
# python3 -m timeit 'x=[]; [x.insert(0, i) for i in range(10000)]' 10 loops, best of 3: 30.2 msec per loop # python3 -m timeit 'x=[]; [x.append(i) for i in range(10000)]' 1000 loops, best of 3: 1.01 msec per loop # python3 -m timeit 'x=[i for i in range(10000)]' 1000 loops, best of 3: 381 usec per loop # python3 -m timeit 'x=list(range(10000))' 10000 loops, best of 3: 212 usec per loop
These few examples demonstrate the performance difference between list.insert, list.append, a list comprehension, and the list function. But more importantly, it demonstrates how to use the timeit command. Naturally, the command can be used with regular scripts as well, but the timeit module only accepts statements as strings to execute which is a bit of an annoyance. Luckily, you can easily work around that by wrapping your code in a function and just timing that function:
import timeit
def test_list():
return list(range(10000))
def test_list_comprehension():
return [i for i in range(10000)]
def test_append():
x = []
for i in range(10000):
x.append(i)
return x
def test_insert():
x = []
for i in range(10000):
x.insert(0, i)
return x
def benchmark(function, number=100, repeat=10):
# Measure the execution times
times = timeit.repeat(function, number=number, globals=globals())
# The repeat function gives `repeat` results so we take the min()
# and divide it by the number of runs
time = min(times) / number
print('%d loops, best of %d: %9.6fs :: %s' % (
number, repeat, time, function))
if __name__ == '__main__':
benchmark('test_list()')
benchmark('test_list_comprehension()')
benchmark('test_append()')
benchmark('test_insert()')When executing this, you will get something along the following lines:
# python3 test_timeit.py 100 loops, best of 10: 0.000238s :: test_list() 100 loops, best of 10: 0.000407s :: test_list_comprehension() 100 loops, best of 10: 0.000838s :: test_append() 100 loops, best of 10: 0.031795s :: test_insert()
As you may have noticed, this script is still a bit basic. While the regular version keeps trying until it reaches 0.2 seconds or more, this script just has a fixed number of executions. Unfortunately, the timeit module wasn't entirely written with re-use in mind, so besides calling timeit.main() from your script there is not much you can do to re-use that logic.
Personally, I recommend using IPython instead, as it makes measurements much easier:
# ipython3 In [1]: import test_timeit In [2]: %timeit test_timeit.test_list() 1000 loops, best of 3: 255 µs per loop In [3]: %timeit test_timeit.test_list_comprehension() 1000 loops, best of 3: 430 µs per loop In [4]: %timeit test_timeit.test_append() 1000 loops, best of 3: 934 µs per loop In [5]: %timeit test_timeit.test_insert() 10 loops, best of 3: 31.6 ms per loop
In this case, IPython automatically takes care of the string wrapping and passing of globals(). Still, this is all very limited and useful only for comparing multiple methods of doing the same thing. When it comes to full Python applications, there are more methods available.
The easiest way you can implement the %timeit function yourself is to simply call timeit.main:
import timeit timeit.main(args=['[x for x in range(1000000)]'])
The internals of the timeit module are nothing special. A basic version can be implemented with just an eval and a time.perf_counter (the highest resolution timer available in Python) combination:
import time
import functools
TIMEIT_TEMPLATE = '''
import time
def run(number):
%(setup)s
start = time.perf_counter()
for i in range(number):
%(statement)s
return time.perf_counter() - start
'''
def timeit(statement='pass', setup='pass', repeat=1, number=1000000,
globals_=None):
# Get or create globals
globals_ = globals() if globals_ is None else globals_
# Create the test code so we can separate the namespace
src = TIMEIT_TEMPLATE % dict(
statement=statement,
setup=setup,
number=number,
)
# Compile the source
code = compile(src, '<source>', 'exec')
# Define locals for the benchmarked code
locals_ = {}
# Execute the code so we can get the benchmark fuction
exec(code, globals_, locals_)
# Get the run function
run = functools.partial(locals_['run'], number=number)
for i in range(repeat):
yield run()The actual timeit code is a bit more advanced in terms of checking the input but this example roughly shows how the timeit.repeat function can be implemented.
To register your own function in IPython, you need to use some IPython magic. Note that the magic is not a pun. The IPython module that takes care of commands such as these is actually called magic. To demonstrate:
from IPython.core import magic
@magic.register_line_magic(line):
import timeit
timeit.main(args[line])To learn more about custom magic in IPython, take a look at the IPython documentation at https://ipython.org/ipython-doc/3/config/custommagics.html.