Search in book...
Toggle Font Controls
Create new playlist

Name your new playlist

Playlist description (optional)
Sign In

Email address

Password

Forgot Password?

or

Continue with Facebook

Continue with Google
Sign Up

Full Name

Email address

Confirm Email Address

Password

or

Continue with Facebook

Continue with Google

6
Python in Bigger Projects

WHAT YOU WILL LEARN IN THIS CHAPTER:

Testing your Python code
Debugging your Python code
Handling errors in your Python code
Structuring and releasing your Python code
Tuning the performance of your Python code

WROX.COM DOWNLOADS FOR THIS CHAPTER

You can find the wrox.com downloads for this chapter at www.wrox.com/go/pythonprojects on the Download Code tab. The code is in the Chapter 6 download, called Chapter 6.zip, and individually named according to the names throughout the chapter.

So far you’ve looked at many ways to use Python. You’ve made local scripts to handle small tasks, you’ve handled medium-sized tasks locally, and you’ve even made a small web app using Flask. But what if you find yourself in the midst of a larger project? Python, as you have seen by now, is a very powerful language. It’s also very open, meaning you, the developer, have access to all aspects of the language. This openness, however, makes testing your Python code more important than ever. Every object in Python is a first-class object, so you can change and manipulate any object available to you. Because you can change and manipulate objects, you must make sure to test and verify the logic of our code.

Python is not a “typed” language in the same way that C and Java are explicitly typed. You can pass objects around in Python and the interpreter will try to manipulate them to the best of its ability. If it cannot perform an operation on an object or data that is available, however, it raises an exception, which causes your program to crash. So, how can you prevent this? How can you write code, share that code, and guarantee that others can use it and that the code will function as expected? Testing.

Testing with the Doctest Module

The simplest form of testing in Python is the doctest module. This module is made for testing the simpler parts of your code, to verify that it will function as expected, as written in your document strings (triple quotes '''...''' or"""...""", single or double quotes will both work). Doctest tests are written like this:

  '''
  this function should take in a number and return its squared value
  >>> sq(3)
  9
  '''

  def sq(n):
 	return n*n

The usual way of writing doctest tests is to use the interpreter, write the code, and then run it in the interpreter. Then you copy and paste the interpreter text into the doctest string, as follows:

  Python 3.3.3 (default, Feb 14 2014, 12:35:03)
  [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> def sq(n):
  ... return n*n
  ...
  >>> sq(3)
  9
  >>>

So you would simply copy the following lines, and put them in your doctest strings:

  >>> sq(3)
  9

Doctest is not suitable for testing of large, complicated methods or functions. But it is really good at “contract programming.” By using doctest strings and saying “this function, when passed a 3 as an argument, will return a 9” and then calling the function, you are setting up a contract: if you pass a certain piece of data, the function will behave as you expect it to. However, you cannot test every possible outcome, so doctest will hit its limitations fairly quickly with larger projects.

In the following example you create and then run a small Python script with some doctest strings to test your code.

TRY IT OUT Creating and Executing Simple Doctest Tests

This Try It Out demonstrates how you can test a simple file that has a few functions using doctest, which houses testing strings in documentation strings, using triple quotes (''' ... ''').

Create a directory for Chapter 6 in your project directory, and then using your editor of choice, create a Python filenamed simple_doctest.py. Include the following function and test:
```
  def simple_math(x, y):
 	'''
 	>>> simple_math(1, 2)
 	3

 	>>> simple_math('k', 'v')
 	'kv'
 	'''

 	return x + y
```
You must have a space after the interpreter prompt (>>>) for the tests to run. Your first line with the interpreter prompt (>>> simple_math(1,2)') would not run properly if it were formatted as >>>simple_math(1,2). The space is mandatory.
Open a new Terminal window and from your Chapter 6 directory and run the following command:
```
  python -m doctest -v simple_doctest.py
```
Here you are calling Python, but by passing it the -m flag, you are telling Python you want to execute the file using a module—in this case the doctest module. The -v flag means that you want “verbose” output. If you take off the -v flag and rerun the code, you will see that it simply finishes silently, meaning the code runs, but then you are given another Terminal prompt and nothing further from the Python interpreter. Finally, the last argument is, of course, the file you are testing. With the -v flag, you should see the following output:
```
  ~chapter6$ python -m doctest -v simple_doctest.py
  Trying:
 	simple_math(1, 2)
  Expecting:
 	3
  ok
  Trying:
 	simple_math('k', 'v')
  Expecting:
 	'kv'
  ok
  1 items had no tests:
 	simple_doctest
  1 items passed all tests:
 	2 tests in simple_doctest.simple_math
  2 tests in 2 items.
  2 passed and 0 failed.
  Test passed.
```
Note that you must have a space after the interpreter prompt (>>>) for the tests to run.
Next, write these tests as if you wanted fully documented contract programming. In your editor, open your simple_doctest.py file and add these lines:
```
  def simple_math(x, y):
  	'''
  	This function will return x + y
  	we can use it on numbers. Passing 1 and 2:

  	>>> simple_math(1, 2)
  	3

  	We should get 3 as a return value
  	It will also work on strings. Passing the strings 'k' and 'v':

  	>>> simple_math('k', 'v')
  	'kv'

  	We should get 'kv'
  	'''

  	return x + y
```
Note that you must have a newline between your expected result and any documentation string that you are putting into the doctest string. So, when you have your expected 3 after your simple_math(1,2) call, you must have that newline in place before you specify the behavior you want. Otherwise, the interpreter will try to evaluate that line as expected output, therefore rendering that test a failure.

There are times where you will need to evaluate a value that cannot be consistently predicted (like an address in memory). Add the following to your simple_doctest.py file (after your first test is fine):

  class SimpleClass():
 	pass

  def class_testing_method_ahoy(obj):
  	''' Should return a list containing the object
  	>>> SimpleClass(class_testing_method_ahoy())
  	[<doctest_class_testing_method_ahoy.SimpleClass object at /
  	0x10382a390]
  	'''
  	return [obj]

Now run the tests and observe the output. You should see that your tests fail because the code is evaluating a location in memory that we cannot reliably predict each time. Note the memory addresses in your output.

  chapter6 $ python -m doctest -v simple_doctest.py
  Trying:
 	class_testing_method_ahoy(SimpleClass())
  Expecting:
 	[<doctest_class_testing_method_ahoy.SimpleClass object at /
 	0x10382a390>]
  ******************************************************************
  File "./simple_doctest.py", line 27, in /
  simple_doctest.class_testing_method_ahoy
  Failed example:
 	class_testing_method_ahoy(SimpleClass())
  Expected:
 	[<simple_doctest.SimpleClass object at 0x10382a390>]
  Got:
 	[<simple_doctest.SimpleClass object at 0x10af0fe50>]
  Trying:
 	simple_math(1, 2)
  Expecting:
 	3
  ok
  Trying:
 	simple_math('k', 'v')
  Expecting:
 	'kv'
  ok
  2 items had no tests:
 	simple_doctest
 	simple_doctest.SimpleClass
  1 items passed all tests:
 	2 tests in simple_doctest.simple_math
  ******************************************************************
  1 items had failures:
 	1 of 1 in simple_doctest.class_testing_method_ahoy
  3 tests in 4 items.
  2 passed and 1 failed.
  ***Test Failed*** 1 failures.

Doctest requires the actual output to match the expected output exactly. When we specify a memory address to Doctest as the expected output, the actual memory address received from the test must precisely match the declared expected value. When we:

  [<simple_doctest.SimpleClass object at 0x10382a390>]

Doctest wants an object at the memory location 0x10382a90, but you’re going to be creating a new object in a new memory location. You don’t really care about the memory location, only that the object is created. Doctest provides a way to work around this:

  >>> class_testing_method_ahoy(SimpleClass()) /
 	# doctest: +ELLIPSIS
  [<simple_doctest.SimpleClass object at 0x...>]

The ELLIPSIS option lets doctest know that what follows can be any value. This will return a successful test:

  Trying:
 	class_testing_method_ahoy(SimpleClass()) # doctest: +ELLIPSIS
  Expecting:
 	[<simple_doctest.SimpleClass object at 0x...>]
  ok
  Trying:
 	simple_math(1, 2)
  Expecting:
 	3
  ok
  Trying:
 	simple_math('k', 'v')
  Expecting:
 	'kv'
  ok
  2 items had no tests:
 	simple_doctest
 	simple_doctest.SimpleClass
  2 items passed all tests:
 	1 tests in simple_doctest.class_testing_method_ahoy
 	2 tests in simple_doctest.simple_math
  3 tests in 4 items.
  3 passed and 0 failed.
  Test passed.

The ELLIPSIS constant is also useful if you are checking that a list is returned, such as when using the range() method. Say you want to make sure you get back the numbers 1–4,590 when you call range(4589). Rather than print the entire list of 4,590 numbers, you can use the ELLIPSIS constant and simply have your result be [0, 1, ... , 4588, 4589]. Doctest has many of these constants for different situations. Refer to the full doctest documentation for a list of all of them.

How It Works

The doctest module is built into the Python language. It takes in strings that are usually copied directly from the interpreter and then evaluates those strings when the file is called. It does this by using the module (calling Python on the command line with the -m flag, followed by the module 'doctest' and then the filename).

Although doctest is good for evaluating whether your documentation strings are true and the code behaves as expected, it is not meant for thorough, robust testing of more complicated codebases. There are many other facets to the doctest API. You should check out the documentation to familiarize yourself with the full functionality of the module.

Testing with the Unittest Module

What if you need significant testing and you want to verify that your codebase is operating as expected? This is a job for the unittest module. This module is more robust than the doctest module, and will test your code thoroughly. Unittest is like the baseline testing module on which most testing libraries are based. It is also an excellent introduction to test-driven development (TDD) in Python.

The term unit test is not unique to Python. If you’re familiar with other languages and programming, you have no doubt heard of unit testing. Unit testing is simply testing your code in units. So, if you have five functions in your code, you want to have a minimum of five units in your testing harness for each unit of functionality in your codebase. Unit tests also consist of a test file, which contains all of your tests, written in the same structure or format as any other Python file. The only difference is that each test begins with test, and each test harness is a class from the Unittest.Test object. For example, if you have a function named login, and you want to test that function, create a test named test_login, which would then call your login function and run your tests against the output of that function.

Don’t forget that when you are writing unittest classes, you need to import the code module you’ll be testing into your test code. If you were testing users.py, you would need to import users into your test.py file, so that you can test the functions in the users module with your unittests.

You create unittest tests by creating classes that are subclasses to the TestCase class, as follows:

  import unittest

  class PythonProjectsTest(unittest.TestCase):
 	eturn

You want to put statements within your class that will be evaluated when the test is run and return an assertion value of True or False:

  import unittest

  class PythonProjectsTest(unittest.TestCase):
 	def test_to_fail(self):
 		self.failIf(False)

  if __name__ == '__main__':
 	unittest.main()

In the preceding example, you use the assertion method failIf() to evaluate the value in the parentheses. If the value is true, you will receive a failure message when you run the test. In this case, you’re passing in False, which will, of course, evaluate to false. Therefore, this test will return a failure.

If you run this test you should see the following output:

  ======================================================================
  FAIL: test_to_fail (__main__.PythonProjectsTest)
  ----------------------------------------------------------------------
  Traceback (most recent call last):
  File "<stdin>", line 3, in test_to_fail
  AssertionError: True is not false

  ----------------------------------------------------------------------
  Ran 1 test in 0.000s

  FAILED (failures=1)

If you change self.failIf(True) to self.failIf(False), you should see your output change to:

  ----------------------------------------------------------------------
  Ran 1 test in 0.000s
  OK

Note that unittest doesn’t evaluate whether a test is actually passing; it is simply evaluates whether an exception is thrown. Therefore, if an exception is not thrown, the test is considered OK. This could mean that your precise calculation, while returning a not-so precise number, shows as passing, or OK, not because the result is correct—which it isn’t—but simply because the test is not raising an exception.

Following are the three possible outcomes of unittest if it doesn’t actually have passing tests:

OK: The test is OK; no exception raised.
Fail: An AssertionError was raised (the test has failed).
Error: An exception was raised that is not an AssertionError.

The best way to understand unit testing and the unittest module is to just do some testing.

TRY IT OUT Building and Running Unit Tests Using the unittest Module

In this Try It Out, you will write functions and test them using the unittest module, to understand the architecture of the unittest module.

In your Chapter 6 directory, create the file ch6_example.py. This file contains some fairly useless functions, but they are easy to test:

  #ch6_example.py

  def first(chars):
 	chars.sort()
 	return chars[0]

  def last(chars):
 	chars.sort()
 	return chars[-1]

Create the test file and call it unittest_example.py. Import unittest and then, from your ch6_example.py file, import your two functions. Importing these functions directly means you won’t have to call ch6_example.first() or ch6_example.last() when testing them, and you can simply call first() and last(). Remember, this is called aliasing our functions into our code through importing.
```
  #unittest_example.py

  import unittest
  from ch6_example import first, last
```

Create two lists, one with numbers and one with strings. You’ll be using these lists to test your two different sort functions. Then, set up the testing class, inheriting from the unittest.TestCase class:

  #unittest_example.py

  import unittest
  from ch6_example import first, last

  list_nums = [7,9,5]
  list_chars = ['m', 'd', 'Z', 'l']import unittest

  class TestPPMath(unittest.TestCase):

Next, test a few assertions to see how they behave. Start with the most common: assertEqual. This test should pass, because when you sort your list of numbers, the first element in the list is 5, so this should return true:
```
  import unittest
  from ch6_example import first, last

  list_nums = [7,9,5]
  list_chars = ['m', 'd', 'Z', 'l']

  class TestPPMath(unittest.TestCase):

  def test_first(self):
 	self.assertEqual(first(list_nums), 5)
```
Remember: All testing functions that you want to run must begin with test.

Similar to AssertEqual, which checks equality, there is also assertTrue, which checks that the first value is the second value, and therefore true:

  import unittest
  from ch6_example import first, last

  list_nums = [7,9,5]
  list_chars = ['m', 'd', 'Z', 'l']

  class TestPPMath(unittest.TestCase):

 	def test_first(self):
 		self.assertEqual(first(list_nums), 5)

 	def test_last(self):
 		self.assertTrue(last(list_chars), 'm')

Unittest is only looking for exceptions, like the assertionError exception. You can use the failUnless() function to tell it to fail that test unless it is returning true:

  import unittest
  from ch6_example import first, last

  list_nums = [7,9,5]
  list_chars = ['m', 'd', 'Z', 'l']

  class TestPPMath(unittest.TestCase):

  	def test_first(self):
  		self.assertEqual(first(list_nums), 5)

  	def test_last(self):
  		self.assertTrue(last(list_chars), 'm')

  	def testFirstAgain(self):
  		self.failUnless(first(list_chars), 'Z')

If you want the test to fail if it’s true, you use the failIf() function, which fails if the inputs evaluate to true. So, this test should fail when you run it:

  import unittest
  from ch6_example import first, last

  list_nums = [7,9,5]
  list_chars = ['m', 'd', 'Z', 'l']

  class TestPPMath(unittest.TestCase):

  	def test_first(self):
  		self.assertEqual(first(list_nums), 5)

  	def test_last(self):
  		self.assertTrue(last(list_chars), 'm')

  	def testFirstAgain(self):
  		self.failUnless(first(list_chars), 'Z')

  	def testLastAgain(self):
  		self.failIf(last(list_nums), 9)

Finally, insert your __main__ check and run the unittest.main() method to actually test your new testing class:

  import unittest
  from ch6_example import first, last

  list_nums = [7,9,5]
  list_chars = ['m', 'd', 'Z', 'l']

  class TestPPMath(unittest.TestCase):

  	def test_first(self):
  		self.assertEqual(first(list_nums), 5)

  	def test_last(self):
  		self.assertTrue(last(list_chars), 'm')

  	def testFirstAgain(self):
  		self.failUnless(first(list_chars), 'Z')

  	def testLastAgain(self):
  		self.failIf(last(list_nums), 9)

  if __name__ == '__main__':
 	unittest.main()

How It Works

When you write tests, you’re simply creating static data to pass into functions you’ve already defined. You want to pass a known value to the function and then express, in your tests, the value you expect to be returned. If that value isn’t returned, the test should fail. If the value is returned, the test passes and the code moves on to the next testing function.

Some readers may quickly realize that testing with static data isn’t foolproof. What if the data that is passed in isn’t a type that you’ve tested? This is why writing good tests is important. One function in your program may have multiple tests, or one test could verify multiple situations.

Test-Driven Development in Python

A term that is becoming more and more popular in the Python community is test-driven development (TDD). What exactly does that mean? Although TDD is a very important topic when it comes to Python development, it is also a very robust topic. Therefore, this section gives only a very brief introduction of TDD so that you can familiarize yourself with the term and its basic definition.

TDD simply means writing your tests first. Most developers groan when they hear the word “testing.” They think it means longer development time and more effort on their part, and less of the fun stuff like writing the actual code that will make their project run. However, testing can be just as fun as the other stuff. And although it does require the developer to write more lines of code, it leads to better quality code and more maintainability later on in the project. Your future self and co-collaborators will thank you for taking the time to write tests first and develop against those tests.

So, how exactly does TDD work? Write tests! It’s really that simple. There is, of course, an art form to writing good tests, and it’s important that you take the time to study up and become familiar with proper TDD practices. Here are the basics:

Write tests first.
All tests should fail at first.
Write code.
Test code against tests.
Rewrite code.
Retest code against tests.
Repeat until all tests are passing.

This is the gist of TDD. You can probably see why doctest may not be the best answer for all testing situations. Once you have to test and retest, and you begin testing more complex ideas, doctest will hit its limitations. As stated, there is an art to writing effective tests, however, and that is where the beauty of TDD comes in.

Debugging Your Python Code

Most developers will likely tell you that they hate debugging. It’s tedious, persnickety, and can become rather boring or infuriating fairly quickly. It doesn’t have to be this way. Taking a new look at debugging and testing can make even the most cynical developer a little less irritated.

When you run into a bug with your code, rather than think about how annoying it is (don’t worry, it’s the natural reaction), think about how this is actually an opportunity to learn. Something is broken somewhere, some stone has gone unturned. This is your chance to find that stone, turn it over, and see what there is to see! You’re well on your way to becoming a seasoned programmer with every bug you squash.

Python makes debugging a little less of a hassle with the Python debugger module, or the pdb. If you read Chapter 5 and explored the Chrome Developer Tools, you may notice some similarities. If you’re a web developer by trade who is trying Python on for size, you’ll probably find that you like the pdb and it reminds you a little of your favorite web debugging software.

The pdb is fairly powerful in that it enables you to insert breakpoints in your code that will stop your code running, and drop you into a pdb prompt or terminal. This is very handy because you can then begin examining the data you have in scope at that moment. If you find an exception is being raised when a certain function is called, you can put a pdb() call in that function and then you can start to examine the data in an interactive interpreter in your terminal. Let’s try it out.

The following example illustrates using the pdb module for debugging your Python code.

TRY IT OUT Using the Python Debugger, or pdb module (pdb_example.py)

This Try It Out demonstrates how you can utilize the power of the pdb module to debug or examine your Python code.

Open the pdb_example.py file. You should see the following:

  #pdb_example.py

  class ExampleClass(object):

  	def __init__(self, name, number):
  		self.name = name
  		self.number = number

  	def example_entry(self):
 		return "The example name is {0} with the number {1}".format(self.name,
  		self.number)

  if __name__ == '__main__':
 	example = ExampleClass("Carla", 456)
 	return example.example_entry()

Import the pdb module:

  #pdb_example.py

  import pdb

  class ExampleClass(object):

  	def __init__(self, name, number):
  		self.name = name
  		self.number = number

  	def example_entry(self):
 		return "The example name is {0} with the number {1}".format(self.name,
  		self.number)

  if __name__ == '__main__':
 	example = ExampleClass("Carla", 456)
 	return example.example_entry()

The pdb module has many powerful features. The first one you look at is the .set_trace() method, so add a set_trace() to your code:

  #pdb_example.py

  import pdb

  class ExampleClass(object):

  	def __init__(self, name, number):
  		self.name = name
  		self.number = number

  	def example_entry(self):
 		pdb.set_trace()
 	return "The example name is {0} with the number {1}".format(self.name,
  self.number)

  if __name__ == '__main__':
 	example = ExampleClass("Carla", 456)

 	return example.example_entry()

Save the file. Now run your pdb_example.py file. You should be dropped into a pdb interpreter, which is noted with the (Pdb) prompt:

  chapter6$ python pdb_example.py
  > /Users/lcassell/Documents/Python_Companion/chapter6/pdb_example.py(13)
  example_entry()
  -> return "The example name is {0} with the number {1}".format( self.name,
  self.number)
  (Pdb)

Type n and press Enter/Return:

  (Pdb) n
  --Return--
  > /Users/lcassell/Documents/Python_Companion/chapter6/pdb_example.py(13)
  example_entry()->'The example ...he number 456'
  -> return "The example name is {0} with the number {1}".format( self.name,
  self.number)
  (Pdb)

What you’ve done is stepped down to the next (n) line in the file. Look at the pdb_example.py file and you’ll see that the set_trace() is placed before your return string:

  #pdb_example.py

  import pdb

  class ExampleClass(object):

  	def __init__(self, name, number):
  		self.name = name
  		self.number = number

  	def example_entry(self):
 		pdb.set_trace()
 	return "The example name is %s with the number %d" % name, number


  if __name__ == '__main__':
 	example = ExampleClass("Carla", 456)
 		example.example_entry()

This means that the program will break at that line and open a pdb interpreter so that you can examine your code. When you type n and then press Enter/Return, you’re moving to the next line in the code, which is your return statement. That line will execute and you’ll see the printout of the string (with some ellipses to indicate text that was left out for readability (/):

  > /Users/lcassell/Documents/Python_Companion/chapter6/pdb_example.py(13)
  example_entry()->'The example ...he number 456'
  -> return "The example name is {0} with the number {1}".format( self.name,
  self.number)
  (Pdb)

While still in the debugger, simply press Enter/Return again. You should see that the next line in the code is executed. It’s as if you’ve type n and Enter/Return again. The debugger retains your last command and will simply execute it with the Enter/Return key:
```
  (Pdb)
  --Return--
  > /Users/lcassell/Documents/Python_Companion/chapter6/pdb_example.py(19)<module>()
  ->None
  -> example.example_entry()
  (Pdb)
```
If you keep pressing Enter/Return, you’ll see that you simply step through the rest of the program until it completes and you’re back to your command prompt and out of the pdb environment:
```
  (Pdb)
  --Return--
  > /Users/lcassell/Documents/Python_Companion/chapter6/pdb_example.py(19)<module>()
  ->None
  -> example.example_entry()
  (Pdb)
  chapter6$
```

Start up the debugger again and run through some more handy commands:

  chapter6$ python pdb_example.py
  > /Users/lcassell/Documents/Python_Companion/chapter6/pdb_example.py(13)
  example_entry()
  -> return "The example name is {0} with the number {1}".format( self.name,
  self.number)
  (Pdb)

This time, print the value of some variables. At the debugger prompt, type p self.name and press Enter/Return:

  > /Users/lcassell/Documents/Python_Companion/chapter6/pdb_example.py(13)
  example_entry()
  -> return "The example name is {0} with the number {1}".format( self.name,
  self.number)
  (Pdb) p self.name
  'Carla'
  (Pdb)

You can use the print functionality by simply typing p followed by the variable name.

At your prompt, type locals() and press Enter/Return. You should see all objects that are in the local scope at that moment:
```
  (Pdb) p self.name
  'Carla'
  (Pdb) locals()
  {'self': <__main__.ExampleClass object at 0x106c66450>}
  (Pdb)
```
In this case your current local scope contains just your class, which is what it should be.

Type globals() and see what you have available in your global scope:

  (Pdb) locals()
  {'self': <__main__.ExampleClass object at 0x106c66450>}
  (Pdb) globals()
  {'example': <__main__.ExampleClass object at 0x106c66450>, '__builtins__': <module
  'builtins' (built-in)>, '__name__': '__main__', '__file__': 'pdb_example.py',
  'ExampleClass': <class '__main__.ExampleClass'>, 'pdb': <module 'pdb' from
  '/usr/local/Cellar/python3/3.3.3/Frameworks/Python.framework/Versions/3.3/
  lib/python3.3/pdb.py'>, '__package__': None, '__loader__': <_frozen_importlib.
  SourceFileLoader object at 0x106b9a410>, '__cached__': None, '__doc__': None}
  (Pdb)

Note that you have many things available to you, including your ExampleClass object, your pdb module (that you imported), and your local Python source. There may be times where you are debugging that you need to inspect what is in your local scope, to see if you have that data available to you. The locals() and globals() functions will be very useful during these times.

Type c and press Enter/Return. You should be taken out of the pdb, your code should complete, and you should see your normal command prompt. With the pdb, c simply continues running the program.
To quit the debugger without running the rest of your program, type q at the (Pdb) prompt and press Enter/Return prompt.

How It Works

The pdb is a built-in module in Python’s standard library. You simply import the pdb into your file, then either call the stack_trace() method to enter the debugger environment or call other methods to perform certain functions within the file at run time, which will then take you into the debugger interface. The pdb is incredibly useful for debugging code at run time and for examining the data in your code at certain points in a “live” environment. The pdb module contains many commands; consult a reference for a more robust list.

Handling Exceptions in Python

Python is an interpreted language, which means that there is no compiler to compile your code and find any logic or syntax errors before you run it. So how does Python handle this? Python uses exceptions to handle errors. This type of handling can mean that making one small mistake in your code can cause your entire program to fail. Because of this you want to test thoroughly, but on top of that, you also want to set up some fail-safes in case you encounter exceptions with your code during run time.

For example, if you try the following code in your interpreter,

  >>> def sum(a, b):
  ...	return a + b
  ...
  >>> sum("no", 4)

you’ll get the following error:

  Traceback (most recent call last):
 	File "<stdin>", line 1, in <module>
 	File "<stdin>", line 2, in sum
  TypeError: Can't convert 'int' object to str implicitly

As you can see, when you try to pass a string to a mathematic function, which can only operate on integers and floats, it throws a TypeError. This tells you that the data you sent to the function is not of the correct type. Because Python is not a strongly typed language, nor is it compiled, the only errors that you will get are exceptions, which will crop up at run time. When an exception is thrown at run time, your entire program will quit if there is no exception handling in place. It is imperative that you check for these sorts of “gotchas.” Not checking for them can render your code unusable, and that’s not a very good codebase to have!

A number of exceptions are built into the Python language. Here is a list of those exceptions:

  BaseException
  	+-- SystemExit
  	+-- KeyboardInterrupt
  	+-- GeneratorExit
  	+-- Exception
  	+-- StopIteration
  	+-- StandardError
  	|	+-- BufferError
  	|	+-- ArithmeticError
  	| |		+-- FloatingPointError
  	| |		+-- OverflowError
  	| |		+-- ZeroDivisionError
  	|	+-- AssertionError
  	|	+-- AttributeError
  	|	+-- EnvironmentError
  	| |		+-- IOError
  	| |		+-- OSError
  	| |		+-- WindowsError (Windows)
  	| |		+-- VMSError (VMS)
  	|	+-- EOFError
  	|	+-- ImportError
  	|	+-- LookupError
  	| |		+-- IndexError
  	| |		+-- KeyError
  	|	+-- MemoryError
  	|	+-- NameError
  	| |		+-- UnboundLocalError
  	|	+-- ReferenceError
  	|	+-- RuntimeError
  	| |		+-- NotImplementedError
  	|	+-- SyntaxError
  	| |	 +-- IndentationError
  	| |		+-- TabError
  	|	+-- SystemError
  	|	+-- TypeError
  	|	+-- ValueError
  	|		+-- UnicodeError
  	|			+-- UnicodeDecodeError
  	|			+-- UnicodeEncodeError
  	|			+-- UnicodeTranslateError
  	+-- Warning
  		+-- DeprecationWarning
  		+-- PendingDeprecationWarning
  		+-- RuntimeWarning
  		+-- SyntaxWarning
  		+-- UserWarning
  		+-- FutureWarning
  		+-- ImportWarning
  		+-- UnicodeWarning
  		+-- BytesWarning

With so much that can go wrong, how do you gracefully handle exceptions in Python? With a try-except block. The try-except block will try a piece of code and if the code throws one of the preceding exceptions, it will catch that exception and print out an error message, as defined in the base exception class, or you can even print your own error messages for each exception:

  >>> try:
  ...	sum("yes", 9)
  ... except TypeError:
  ...	print("Both inputs must be integers")
  ...
  Both inputs must be integers

You can also have try-except blocks handle exceptions so that your program doesn’t fail and you can continue moving down the stack:

  >>> try:
  ...	some_function()
  ... except:
  ...	graceful_function()
  ... else:
  ... next_function()

Sometimes you will want to run a function no matter if your try-catch catches an exception or runs. In that case you want to use the finally statement.

  >>> try:
  ...	some_function()
  ... except:
  ...	graceful_function()
  ... finally:
 	cleanup_function()

But what if you want your code to throw its own exceptions? What if you want to check for some certain type of data, and if that is not present, you want to alert the user? You can make custom exception classes to use on top of built-ins.

In the following example, you create and use customs exceptions.

TRY IT OUT Creating and Using Custom Exceptions in Python (exceptClass.py)

This Try It Out demonstrates how you can create and then use custom exceptions in your Python code.

Open exceptClass.py to familiarize yourself with the class you’ll be using:

  # exceptClass.py

  class TestClass(object):

  def __init__(self, name, number):
 	name = self.name
 	number = self.number

  def return_values(self):

 	print ("The values are: ", self.name, self.number)

Write the exception that you’ll throw if self.number isn’t a number. The first step to writing an exception is that it must be a class that inherits from the Exception class. Add the following lines to your exceptClass.py file:
```
  # exceptClass.py

  class TestClass(object):

  	def __init__(self, name, number):
 		self.name = name
 		self.number = number

  	def return_values(self):

 	print ("The values are: ", self.name, self.number)

 	class notANumber(Exception):
 		def __init__(self, value):
 			self.value = value

 		def __str__(self):
 			return repr(self.value)
```
Here you’ve created your own customized exception that will be thrown if the number attribute is not, in fact, a number. You’ve also overridden the __init__ function for the Exception class, and rather than using args you’re going to use value to catch the value that raised the exception. You are also overriding the __str__() method to output the self.value property using the repr() method call, which will give you the correct representation of the value that raised the exception (this is what will be printed out with your exception error message).
Next, change your return_values() method into something that can check whether self .number is an int. If the type of self.number isn’t an int, you want to raise your exception. Implement a very simple if/else statement to check in your try/catch:
```
  # exceptClass.py

  class TestClass(object):

  	def __init__(self, name, number):

 		self.name = name
 		self.number = number

  	def return_values(self):
 		try:
 			if (type(self.name) is int):
 				return "The values are: ", type(self.name), type(self.number)
 			else:
 				raise notANumber(self.number)
 		except notANumber as e:
 			print("The value for number must be an int you passed: ", e.value)

  class notANumber(Exception):
  	def __init__(self, value):
 		self.value = value

  	def __str__(self):
 		return repr(self.value)
```
What you are doing here is a simple check on the type of self.name. If it is not an int, you are raising the exception you defined earlier. Should the self.number property actually be an int, you’re simply returning a string that tells you the types of each property of your instance. If the type is not an int, notANumber will be raised and you’ll pass in self.number to be evaluated and output in your error message.
Now, run your script in interactive mode. Start up a Python interpreter, but do it using the -i flag and calling your exceptClass.py file, like so:
```
  $ python -i exceptClass.py
  >>>
```
When you use the -i flag when starting a Python interpreter, you can pass in a Python file and this imports the file you’ve passed in without having to explicitly import in the interpreter. This means you have both classes you’ve defined in yourexampleClass.py file, and you don’t have to namespace them with exampleClass.<foo>; you can simply call things.
Next, create a new instance of your TestClass, and pass in two strings (rather than a string and an integer):
```
  (ch3Ex2)$ python -i exceptClass.py
  >>> exampl = TestClass('string1', 'string2')
```

Call return_values() on your newly created instance and note the output:

  (ch3Ex2)$ python -i exceptClass.py
  >>> exampl = TestClass('string1', 'string2')
  >>> exampl.return_values()
  The value for number must be an int you passed: string

The try-except worked and caught that you were passing in a string rather than an integer

Create another instance and pass in a string and an integer; then call return_values() on that instance and note the output:

  >>> exm = TestClass('string1', 42)
  >>> exm.return_values()
  ('The values are: ', <class 'str'>, <class 'int'>)

How It Works

When you create an exception class, you’re really creating a subclass from the base Exception class that is built into Python. With this, you have control over how your own customized exceptions will behave when they are raised. You created a very simple class, and saw that when the exception was raised, your class will give feedback to the user as to what type of data was passed into your class.

As you can see, this feature can be incredibly powerful when writing larger projects. Hopefully this has given you enough of a glimpse into the formulation of exceptions that you can write your own, should the need arise.

Working on Larger Python Projects

When developing with Python you may find that different projects have different versions of different packages. What do you do when your local environment is Python 2.7, but that project you want to work on (or inherited) is 2.6? Or 3.4? This is a problem that many Python developers have encountered, so of course they created a solution. Enter virtualenv.

Virtualenv is a virtual environment for your Python projects. It enables you to create numerous Python instances and develop against all the libraries you need for certain projects. Say you want to work on a project that uses Python 2.7, which you have installed locally, but the project needs a different version of a library than what you have installed locally. The Python versions match up but the library’s versions do not. This is a job for virtualenv!

In this example, you create and then activate a virtualenv to create sandboxes for your individual Python projects.

TRY IT OUT Creating and Activating Virtualenvs

This Try It Out demonstrates how to install, activate, deactivate, and remove virtualenvs from your system.

Install virtualenv by using the commands appropriate for your system:

  OSX:
  brew install virtualenv

  Linux:
  apt-get install python-virtualenv
  pacman -s install python-virtualenv

  Windows (powershell users):
  pip install virtualenv

Move into the directory where you’ll be working. Some power users create a temp_env directory on their systems and create virtualenvs in that. This is a great workflow if you have many virutalenvs to manage. For your purposes, however, you’ll just keep things simple. Once you are in your directory, create your virtualenv:
```
  $ cd chapter6
  $ virtualenv ch6Ex3
  $
```
If you do a directory listing of the contents in the directory where you created your virtualenv, you should see a directory for the name of your environment (in this case ch6Ex3). You’ll be using that directory to activate your environment. This is also the directory that will house all of your installs and your Python code for this environment. To activate the new virtualenv, simply add the following command:
```
  $ source ch6Ex3/bin/activate
  (ch6Ex3)$
```
When you are in an active virtualenv, your command prompt will show the name of the virtualenv within parentheses before your command prompt. In this case you have (ch6Ex3)$.
Now let’s do an experiment. If you did the exercises in Chapter 5, you should have installed requests via pip install requests. Start a Python interpreter and see if you can use requests:
```
 	(ch6Ex3)$ python
  Python 2.7.5 (default, Aug 25 2013, 00:04:04)
  [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import requests
```
What version of the Python shell do you have? Is it 3.4? Or 2.7? How can you change that?

Once you press Enter/Return after importing requests, you should see an ImportError exception, declaring there are no module requests. This is because although you imported requests to your system-wide Python, you are not using that Python environment now, and you must reinstall requests if you want to use it in this virutalenv.

exit() out of the interpreter and pip install requests, while still in your virtualenv:

  (ch6Ex3)$ python
  Python 2.7.5 (default, Aug 25 2013, 00:04:04)
  [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import requests
  Traceback (most recent call last):
 	File "<stdin>", line 1, in <module>
  ImportError: No module named requests
  >>>exit()
  (ch6Ex3)$ pip install requests
 	Downloading/unpacking requests
  Downloading requests-2.2.1-py2.py3-none-any.whl (625kB): 625kB downloaded
  Installing collected packages: requests
  Successfully installed requests
  Cleaning up...
  (ch6Ex3)$

After you install a package you’re still in your virtualenv, and you’ll remain in your virtualenv until you deactivate that environment.

Note that even after installing a package, you are still in your virtualenv and will remain there until you deactivate that environment. Deactivate your environment like so:
```
 	(ch6Ex3)$ pip install requests
  Downloading/unpacking requests
 	Downloading requests-2.2.1-py2.py3-none-any.whl (625kB): 625kB downloaded
  Installing collected packages: requests
  Successfully installed requests
  Cleaning up...
  (ch6Ex3)$ deactivate
  $
```
You’ve successfully installed virtualenv, created a new virtualenv to use, installed a package for that environment, and even deactivated the virtualenv. What if you want to remove that environment altogether? Say you’re done with that project and you want to remove all those files you installed.

To remove a virtualenv, systematically remove the directory it created:

  	 (ch6Ex3)$ pip install requests
  Downloading/unpacking requests
 	Downloading requests-2.2.1-py2.py3-none-any.whl (625kB): 625kB downloaded
  Installing collected packages: requests
  Successfully installed requests
  Cleaning up...
  (ch6Ex3)$ deactivate
  $rm -rf ch6Ex3/
  $

How It Works

Virtualenv provides a way for Python developers to create environments that may have various version requirements. This helps to keep environments separate from others and allows the system to have sandboxes for development of multiple Python projects.

Oftentimes you have projects where more than one person is working in the environment. What happens when you have a long list of requirements that your project needs and you have four people working on the project, on different machines? Do you want to have your teammates simply type pip install <module_name> over and over? No, you do not.

Virtualenv has a very nice feature that enables you to make a requirements.txt file and put the packages needed for your program into the file. Anyone using your package can simply type pip install requirements.txt and get all the dependencies that your package requires! It really is that easy!

TRY IT OUT Creating a requirements.txt file to Simplify Adding Modules

This Try It Out demonstrates how to create a fake requirements.txt file and populate it with some popular packages.

Create a new virtualenv:

  $virtualenv ch6Ex3
  $source ch6Ex3/bin/activate
  (ch6Ex3)$

Write the requirements.txt file. In your Chapter 6 directory, create requirements.txt and add the following lines:
```
  BeautifulSoup==3.2.0
  requests
  https://github.com/django/django/tarball/master
```
These lines are all different. Usually, requirements.txt will have uniformity, but for illustrative purposes, these lines show the three most common ways to get a package installed via pip.

The first line (BeautifulSoup==3.2.0) shows that you want to install BeautifulSoup (a web scraping tool), but you want version 3.2.0, hence the double equal signs.

The second line (Requests) installs the current version.

The final line (https://github.com/django/django/tarball/master) indicates that you want to download and install the package at the URL provided. In this case you’ll be downloading and installing the entire Django project that is available on the master branch of the Django repository (this is a pretty big file, so be prepared for a short download wait).
Save this file and then activate your virtualenv and install those requirements:
```
  $ source ch6Ex3/bin/activate
  (ch6Ex3)$ pip install -r requirements.txt
```
You should see messages about downloading and installing the three packages we’ve provided. Once the packages are successfully downloaded and installed, you should see a “Cleaning up…” message, followed by your virtualenv prompt:
```
  Successfully installed Django
  Cleaning up...
  (ch6Ex3)$
```
Start up Python and see if you really do have those packages installed:
```
  Python 2.7.5 (default, Aug 25 2013, 00:04:04)
  [GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import requests
  >>> import BeautifulSoup
  >>> import django
  >>>
```
If you can import without an error being raised, you’ve successfully installed all of the requirements for your phantom project. Feel free to rm -rf ch6Ex3 virtualenv now. This will remove the virtualenv and all the packages you’ve installed, including the very large Django project.

How It Works

Requirements.txt is a feature of virtualenv that allows developers to include all the necessary libraries needed for their module to work with their Python package. This allows quick setup of environments so that developers can begin work quickly.

Releasing Python Packages

The __init__.py ('dunder, init, dunder') file is fairly important when releasing code out into the wild. For Python projects, __init__.py needs to be at each level of the codebase’s directory structure. For example, say you have a rather large codebase that has multiple .py files. You start by putting a __init__.py in the first layer of the directory structure:

  my_package
 	|----__init__.py
 		|---- my_package.py
 			|---- my_subpackage
 				|---- __init__.py
 				|---- my_subpackage.py

This tells the Python interpreter that you want to treat the directory as a Python package. The cool part is that you can leave the __init__.py file empty, or you can put configuration variables in it. Commonly, folks will import modules/libraries, or other configurations in their __init__.py file—basic setup work to help the package function.

So what happens when you create an __init__.py file and import something? How does Python’s namespacing work now? Suppose you have the following import statement in my_package/__init__.py :

  from file import File

When you want to call that import in the my_package.py file you would simply say:

  from my_package import File

Another use of the __init__.py file is to import all the modules that you’d like to import into the namespace of your package. You do this by assigning the __all__ variable to your subpackage in your package level __init__.py (the first one):

  __all__ = ['my_subpackage']

Doing this makes it so that when your users declare from my_package import * it will import all of the modules from my_subpackage.

Now that you have your code written, and your __init__.py files in place, what if you want to release this code out into the wild? What if you want to be able to install this module on other machines by simply typing pip install <package_name>?

PIP AND PYPI

You’ve been using pip to install third-party libraries and modules throughout this book. But just what is pip, and how does it work?

Pip is the Python package installer. It installs packages that are in the PyPI (pronounced pie-P-I, not pie-pie). The PyPI is the Python Package Index, also known as “The Cheese Shop” (another Monty Python reference), to more seasoned Python developers. This is where you can upload your own Python packages so that they will be available via pip install <package_name>. Sometimes, people will simply upload their own packages to PyPI because it’s easier for them to install those packages on multiple machines. Oftentimes people upload their packages because they hope it will be helpful to others.

To find out more about the Package Index, or to search the index, you can go to http://pypi.python.org/pypi. This is the main page for PyPI and has all the information you need to get started.

If you want to upload your own packages to the PyPI you’ll need to register with PyPI and then follow the tutorial, which is linked on the homepage. It really is that simple. You register, you upload your package, and then it will be available to you, shortly, via pip install <package_name>.

Keep in mind that when you upload a package to PyPI, it is readily available for anyone to download and use. This is why it is so important to practice good, Pythonic programming at all times. You never know when someone will download and use your module, and you want them to be able to use your creation with as little headache as possible.

Summary

We’ve looked over some of the basics of testing and packaging for your Python projects. You should now have a clear idea of just how most Python packages and modules/libraries are architected and created. A good exercise for the reader is to go back through the beginning of the book and work through the exercises using the concepts you’ve learned in this chapter. Can you rewrite the code in Chapter 3 to be test-driven? Can you package your Flask app from Chapter 5 and send it to another computer to be run and developed? You should try these things out so that you have a clear idea of just how all parts and pieces of Python packages are working together.

EXERCISES

In the zip file for this chapter, open the file markets.py and write a doctest string to test the value being returned by the function in the file. Can you think of a reason why a simple doctest string in this code could be incredibly useful for maintaining the code in the future?
Write a unittest for a function that will take a string and return that string reversed. Make sure the test fails, because you haven’t written the function to test, yet.
Write a function for your unittest that takes a string and returns the reverse of that string. Now, run your unittest against that function and modify the function until it passes.

WHAT YOU LEARNED IN THIS CHAPTER

TOPIC	DESCRIPTION
Unit test	Usually a function that is written in a separate testing script, that imports the code to be tested, and that tests each function in the imported code.
Virtualenv	Third-party software that allows developers to create system sandboxes for Python development, using customized versions of Python and Python libraries/modules.
TDD (Test-Driven Development	A development style where one writes tests first, which will fail, then writes the actual functioning code to make the tests past, therefore driving the development cycle based on testing first.
Pdb	Python Debugger, an interactive debugging module for Python.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.

Table of Contents for Chapter 6: Python in Bigger Projects

Create new playlist

Sign In

Sign Up

Testing with the Doctest Module

Testing with the Unittest Module

Test-Driven Development in Python

Debugging Your Python Code

Handling Exceptions in Python

Working on Larger Python Projects

Releasing Python Packages

Summary

EXERCISES

WHAT YOU LEARNED IN THIS CHAPTER

Table of Contents for
Chapter 6: Python in Bigger Projects