How do you convert between the different Python data types? What data quality issues arise when converting data types?

You convert between types using the type functions. Thus, to convert a float or string to an integer, you use the int() type function. To convert an object to a string, you use the str() type function. And so on.

When making the conversion, it is possible that you might lose some data in the process. For example, converting a floating-point number to an integer loses the decimal part of the number (for example, int(2.3) results in 2). If it’s important to retain the detail, you must retain a copy of the original as well.

Which of the Python collection types can be used as the key in a dictionary? Which of the Python data types can be used as a value in a dictionary?

Dictionary keys must be immutable. That means that of the basic Python types, integers, booleans, floats, strings, and tuples can all be used as keys (although floats are not recommended due to their imprecision, especially if you will be computing the key value rather than just storing it). Other custom types, such as frozenset, can also be used as keys provided they are immutable.

Dictionary values can be of any type regardless of mutability.

Write an example program using an if/elif chain involving at least four different selection expressions.

You could use any number of choices here. This example uses colors. Your if/elif/else code should look something like the following:

  (red, orange, yellow, green, blue, violet) = range(6)
  color = int(input('Type a number between 1 and 6'))-1
  if color == red:
      print ('You picked red')
  elif color == orange:
      print ('You picked orange')
  elif color == yellow:
      print ('You picked yellow')
  elif color == green:
      print ('You picked green')
  else:
      print('I don't like your color choice')

Write a Python for loop to repeat a message seven times.

  for n in range(7):
      print('Here is a message')

How can an infinite loop be created using a Python while loop? What potential problem might this cause? How could that issue be resolved?

An infinite loop is written using the while True: idiom.

The problem is that this is an infinite loop so it never ends. Sometimes that’s what you want, but often you need to exit if certain conditions occur. In those cases you can use an if check with a break statement. Here is a short loop that echoes the user input until the user enters an empty string:

  while True:
      message = input('Enter a message: ')
      if not message: break
      print(message)

Write a function that calculates the area of a triangle given the base and height measurements.

  def area_of_triangle(base, height):
          return 0.5 * base * height

Write a class that implements a rotating counter from 0 to 9. That is, the counter starts at 0, increments to 9, resets to 0 again, and repeats that cycle indefinitely. It should have increment()and reset()methods, the latter of which returns the current count then sets the count back to 0.

  class RotatingCounter:
      def __init__(self, start = 0)
          self.counter = 0

      def increment(self):
          self.counter += 1
          if self.counter > 9:
              self.counter = 0
          return self.counter

      def reset(self, value=0):
          current_value = self.counter
          if 0 < value < 9:
              self.counter = value
          else:
              raise ValueError('Value must be between 0 and 9')
          return current_value

Explore the os module to see what else you can discover about your computer. Be sure to read the relevant parts of the Python documentation for the os and stat modules.

Start the Python interpreter and type the following:

  >>> import os
  >>> os.nice(0)      # get relative process priority
  0
  >>> os.nice(1)      # change relative priority
  1
  >>> os.times()      # process times: system, user etc...
  posix.times_result(user=0.02, system=0.01,
  children_user=0.0, children_system=0.0, elapsed=1398831612.5)
  >>> os.isatty(0)    # is the file descriptor arg a tty?(0 = stdin)
  True
  >>> os.isatty(4)    # 4 is just an arbitrary test value
  False
  >>> os.getloadavg() # UNIX only - number of processes in queue
  (0.56, 0.49, 0.44)
  >>> os.cpu_count()  # New in Python 3.4
  4

There are many other functions you could try. For example: os.getpriority(), os.get_exec_path(), os.strerror(), and so on.

Try adding a new function to the file_tree module called find_dirs() that searches for directories matching a given regular expression. Combine both to create a third function, find_all(), that searches both files and directories.

See Chapter2.zip solutions/findfiles.py. The findfiles.py module included in the solutions download provides solutions to all three of the functions in the exercise as well as a couple of alternatives that you might find useful. The specific code for the examples is reproduced here:

  def find_dirs(pattern, base='.'):
      """Finds directories under base based on pattern
      Walks the filesystem starting at base and
      returns a list of directory names matching pattern"""

      regex = re.compile(pattern)
      matches = []
      for root, dirs, files in os.walk(base):
       for d in dirs:
  		 if regex.match(d):
  		     matches.append( path.join(root,d) )
      return matches
  def find_all(pattern, base='.'):
      """Finds files and folders under base based on pattern

      Returns the combined results of find_files and find_dirs"""
      matches = find_dirs(pattern,base)
      matches += find_files(pattern,base)
      return matches

Create another function, apply_to_files(), that applies a function parameter to all files matching the input pattern. You could, for example, use this function to remove all files matching a pattern, such as *.tmp , like this:

  findfiles.apply_to_files('.*.tmp', os.remove, 'TreeRoot')

See Chapter2.zip solutions/findfiles.py as described previously for Exercise 2.

  def apply_to_files(pattern, function, base='.'):
      ''' Apply function to any files matching pattern
      function should take a full file path as an argument
      the return value, if any, will be ignored '''

      regex = re.compile(pattern)
      errors = []
      for root, dirs, files in os.walk(base):
           for f in files:
               if regex.match(f):
                      try: function( path.join(root,f) )
                      except: errors.append(path.join(root,f))
      return errors

Write a program that loops over the first 128 characters and displays a message indicating whether or not the value is a control character (characters with ordinal values between 0x00 and 0x1F, plus 0x7F). Use ctypes to access the standard C library and call the iscntrl()function to determine if a given character is a control character. Note this is not one of the built-in test methods of the string type in Python.

See Chapter2.zip solutions/Ex2-4.py. The code for the iscntrl() function is provided here:

  import ctypes as ct
  # libc = ct.CDLL('libc.so.6') # in Linux
  libc = ct.cdll.msvcrt # in Windows

  for c in range(128):
      print(c, ' is a ctrl char' if libc.iscntrl(c) else 'is not a ctrl char')

To appreciate the work that pickle does for you, try building a simple serialization function for numbers, calledser_num(). It should accept any valid integer or float number as an argument and convert it into a byte string. You should also write a function to perform the reverse operation to read a byte string produced by your ser_num()function and convert it back to a number of the appropriate type. (Hint: You may find the struct module useful for this exercise.)

Create a file containing the following code (located in Exercise3_1.py in the Solutions folder of the .zip file):

  import struct
  def ser_num(n):
      '''
      ser_num(int|float) -> byte string

      convert n to a byte string if it is a float or int.
      ints are stored using their string representation,
      encoded as UTF-8, since they are arbitrarily long.
      floats are stored as C doubles

      Raise Type error for any other type.'''

      if isinstance(n, int):
           # convert to bytes using str()
           data = bytes('i','utf-8') + bytes(str(n),'utf-8')
      elif isinstance(n, float):
           # convert to bytes with struct.pack
           data = bytes('f','utf-8') + struct.pack('d', n)
  else: raise TypeError('Expecting int or float')
  return data
  def get_num(b):
  '''
  get_num(bytes) -> int|float
  convert bytestring b to an int of float'''
  flag = str(b[:1],'utf-8')
  data = b[1:]
  # convert to binary
  if flag == 'i':
  		 s = str(data, 'utf-8')
  		 return int(s)
  elif flag == 'f':
  		 return struct.unpack("d", data)[0]
  else: raise ValueError('Unrecognised byte string format')
  if __name__ == '__main__':
       e = 0.000000000000000001
       i = 1234567
       f = 3.1415926
       bi = ser_num(i)
       bf = ser_num(f)
       i == get_num(bi)
       f-e <= get_num(bf) <= f+e
       try: be = ser_num('a string')
       except TypeError: print('Type error on string')
       try: d = get_num(b'1234')
       except ValueError: print('Value Error on invalid bytes')

Write a version of the employee database example using shelve instead of SQLite. Populate the shelf with the sample data and write a function that lists the name of all employees earning more than a specified amount.

Create a file containing the following code (located in Exercise3_2.py in the Solutions folder of the .zip file):

  import shelve
  #'ID', 'Name', 'HireDate', 'Grade', 'ManagerID'
  employees = [
  ['1','John Brown', '2006-02-23', 'Foreman', ''],
  ['2','Fred Smith', '2014-04-03', 'Laborer', '1'],
  ['3','Anne Jones', '2009-06-17', 'Laborer', '1'],
  ]
  #'Grade','Amount'
  salaries = [
  ['Foreman', 60000],
  ['Laborer', 30000]
  ]

  def createDB(data, shelfname):
	  try:
  		 shelf = shelve.open(shelfname,'c')
  		 for datum in data:
  		 shelf[datum[0]] = datum
  finally:
  		 shelf.close()
  def readDB(shelfname):
  try:
  		 shelf = shelve.open(shelfname,'r')
  		 return [shelf[key] for key in shelf]
  finally:
  		 shelf.close()
  def with_salary(n):
       grades = [salary[0] for salary in readDB('salaryshelf') if salary[1] >= n]
       for staff in readDB('employeeshelf'):
       		 if staff[3] in grades:
       		 yield staff
 
  def main():
       print('Creating data files...')
       createDB(employees, 'employeeshelf')
       createDB(salaries, 'salaryshelf')
       print('Staff paid more than 30000:')
       for staff in with_salary(30000):
       		 print(staff[1])
       print('Staff paid more than 50000:')
       for staff in with_salary(50000):
       		 print(staff[1])

  if __name__ == "__main__": main()

Extend the lendydata.py module to provide CRUD functions for the loan table. Add an extra function, get_active_loans(), to list those loans that are still active (Hint: That means the DateReturned field is NULL.)

Add the following code (located in Exercise3_3.py in the Solutions folder of the .zip file) to the lendydata.py module:

  ##### CRUD functions for loans ######

  def insert_loan(item,borrower):
       query = '''
       insert into loan
       (itemID, BorrowerID, DateBorrowed, DateReturned )
       values (?, ?, date(?), date(?))'''

       cursor.execute(query, (item,borrower,'now',''))

  def get_loans():
       query = '''
       select id, itemID, BorrowerID, DateBorrowed, DateReturned
       from loan'''
       return cursor.execute(query).fetchall()

  def get_active_loans():
       query = '''
       select id, itemID, BorrowerID, DateBorrowed
       from loan
       where DateReturned is NULL'''
       return cursor.execute(query).fetchall()

  def get_loan_details(id):
       query = '''
       select itemID, BorrowerID, DateBorrowed, DateReturned
       from loan
       where id = ?'''
       return cursor.execute(query, (id,)).fetchall()[0]

  def update_loan(id, itemID=None, BorrowerID=None,
       		 DateBorrowed=None, DateReturned=None):
       query = '''
       update loan
       set itemID=?,BorrowerID=?,DateBorrowed=date(?),DateReturned=date(?)
       where id = ?'''
       data = get_loan_details(id)
       if not itemID: itemID = data[0]
       if not BorrowerID: BorrowerID = data[1]
       if not DateBorrowed: DateBorrowed = data[2]
       if not DateReturned: DateReturned = data[3]
       cursor.execute(query, (itemID,BorrowerID,DateBorrowed,DateReturned, id))

  def delete_loan(id):
       query = '''
       delete from loan
       where id = ?''' 
       cursor.execute(query,(id,))

You can test it using the following lines in the

  if __name__ == '__main__'

stanza (or you can import it and use it interactively):

  initDB()
  print('Testing loans

')
  insert_loan(1,3)
  print("Loans: ", get_loans())
  print("Active Loans: ", get_active_loans())
  print('Details of 4:',get_loan_details(4))
  update_loan(6,DateReturned='2014-06-23')
  print('Details of 6:',get_loan_details(6))
  delete_loan(6)
  print('All:',get_loans())
  closeDB()

Explore the Python statistics module to see what it offers (only available in Python 3.4 or later).

Open a Python 3.4 or later interpreter and type:

  >>> import statistics as stats
  >>> stats.mean(range(6))
  2.5
  >>> stats.median(range(6))
  2.5
  >>> stats.median_low(range(6))
  2
  >>> stats.median_high(range(6))
  3
  >>> stats.median_grouped(range(6))
  2.5
  >>> stats.mode(range(6))
  Traceback (most recent call last):
       File "<pyshell#13>", line 1, in <module>
            stats.mode(range(6))
       File "C:Python34libstatistics.py", line 434, in mode
            'no unique mode; found %d equally common values' % len(table)
  statistics.StatisticsError: no unique mode; found 6 equally common values
  >>> stats.mode(list(range(6))+[3])
  3
  >>> stats.pstdev(list(range(6))+[3])
  1.5907898179514348
  >>> stats.stdev(list(range(6))+[3])
  1.7182493859684491
  >>> stats.pvariance(list(range(6))+[3])
  2.5306122448979593
  >>> stats.variance(list(range(6))+[3])
  2.9523809523809526

Convert the oxo-logic.py module to reflect OOP design by creating a Game class.

See the following code (available in the Chapter4.zip file, in the Solutions folder as ex4-1.py):

  import os, random
  import oxo_data
  class Game():
       def __init__(self):
                 self.board = list(" " * 9)

       def save(self, game):
            ' save game to disk '
            oxo_data.saveGame(self.board)

       def restore(self):
            ''' restore previously saved game.
                If game not restored successfully return new game'''
           try:
            self.board = oxo_data.restoreGame()
            if len(self.board) != 9:
                 self.board = list(" " * 9)
            return self.board
            except IOError:
                 self.board = list(" " * 9)
                 return self.board

       def _generateMove(self):
                 ''' generate a random cell from those available.
                    If all cells are used return -1'''
                    options = [i for i in range(len(self.board)) if self.board[i] == " "]
                 if options:
                 return random.choice(options)
                 else: return -1

       def _isWinningMove(self):
                 wins = ((0,1,2), (3,4,5), (6,7,8),
                               (0,3,6), (1,4,7), (2,5,8),
                               (0,4,8), (2,4,6))
            game = self.board
            for a,b,c in wins:
                 chars = game[a] + game[b] + game[c]
                 if chars == 'XXX' or chars == 'OOO':
                      return True
                 return False

       def userMove(self,cell):
            if self.board[cell] != ' ':
                 raise ValueError('Invalid cell')
            else:
                 self.board[cell] = 'X'
            if self._isWinningMove():
                 return 'X'
            else:
            return ""

       def computerMove(self):
            cell = self._generateMove()
                 if cell == -1:
            return 'D'
                 self.board[cell] = 'O'
            if self._isWinningMove():
                 return 'O'
            else:
                 return ""

       def test():
            result = ""
            game = Game()
            while not result:
                 print(game.board)
                 try:
                      result = game.userMove( game._generateMove())
                 except ValueError:
                      print("Oops, that shouldn't happen")
                 if not result:
                      result = game.computerMove()

                 if not result: continue
                 elif result == 'D':
                      print("Its a draw")
                 else:
                      print("Winner is:", result)
                 print(game.board)

  if __name__ == "__main__":
       test()

Explore the Tkinter.filedialog module to get the name of a text file from a user and then display that file on screen.

Create or copy a text file into a folder. Change into that folder and start the Python interpreter. Type the following at the Python interpreter:

  >>> import tkinter.filedialog as fd
  >>> target = fd.askopenfilename()
  >>> for line in open(target):
  ... print(line, end='')
  ...
  <Your chosen file contents should appear here>

Replace the label in the first GUI example program with a Tix ScrolledText widget so that it displays the history of all the entries from the Entry widget.

The solution can be found in the download zip file in the Solutions folder as ex4-3.py. The code is shown here:

  import tkinter.tix as tk
  # create the event handler to clear the text
  def evClear():
       txt = stHistory.subwidget('text')
       txt.insert('end',eHello.get()+'
')
       eHello.delete(0, 'end')

  # create the top level window/frame
  top = tk.Tk()
  F = tk.Frame(top)
  F.pack(fill="both")

  # Now the frame with text entry
  fEntry = tk.Frame(F, border=1)
  eHello = tk.Entry(fEntry)
  eHello.pack(side="left")
  stHistory = tk.ScrolledText(fEntry, width=150, height=55)
  stHistory.pack(side="bottom", fill="x")
  fEntry.pack(side="top")

  # Finally the frame with the buttons.
  # We'll sink this one for emphasis
  fButtons = tk.Frame(F, relief="sunken", border=1)
  bClear = tk.Button(fButtons, text="Clear Text", command=evClear)
  bClear.pack(side="left", padx=5, pady=2)
  bQuit = tk.Button(fButtons, text="Quit", command=F.quit)
  bQuit.pack(side="left", padx=5, pady=2)
  fButtons.pack(side="top", fill="x")

  # Now run the eventloop
  F.mainloop()

Rewrite the first GUI example to be compatible with gettext and generate a new English version with different text on the controls.

The solution, based on Ex4-3.py, is found in the zip file under Solutions as ex4-4.py and messages_en.po.

The code, with changes in bold, is as shown:

  import tkinter.tix as tk

  #### gettext mods #####
  import gettext
  import locale
  locale.setlocale(locale.LC_ALL,'')
  filename="res/messages_{}.mo".format(locale.getlocale()[0][0:2])
  trans=gettext.GNUTranslations(open(filename,'rb'))
  trans.install()
  #######################

  # create the event handler to clear the text
  def evClear():
       txt = stHistory.subwidget('text')
       txt.insert('end',eHello.get()+'
')
       eHello.delete(0, 'end')

  # create the top level window/frame
  top = tk.Tk()
  F = tk.Frame(top)
  F.pack(fill="both")

  # Now the frame with text entry
  fEntry = tk.Frame(F, border=1)
  eHello = tk.Entry(fEntry)
  eHello.pack(side="left")
  stHistory = tk.ScrolledText(fEntry, width=150, height=55)
  stHistory.pack(side="bottom", fill="x")
  fEntry.pack(side="top")

  # Finally the frame with the buttons.
  # We'll sink this one for emphasis
  fButtons = tk.Frame(F, relief="sunken", border=1)
  bClear = tk.Button(fButtons, text=_("Clear Text"), command=evClear)
  bClear.pack(side="left", padx=5, pady=2)
  bQuit = tk.Button(fButtons, text=_("Quit"), command=F.quit)
  bQuit.pack(side="left", padx=5, pady=2)
  fButtons.pack(side="top", fill="x")

  # Now run the eventloop
  F.mainloop()

The edited messages_en.po file looks like this:

  # SOME DESCRIPTIVE TITLE.
  # Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
  # This file is distributed under the same license as the PACKAGE package.
  # FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
  #
  #, fuzzy
  msgid ""
  msgstr ""
  "Project-Id-Version: PACKAGE VERSION
"
  "Report-Msgid-Bugs-To: 
"
  "POT-Creation-Date: 2014-05-16 19:40+0100
"
  "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE
"
  "Last-Translator: FULL NAME <EMAIL@ADDRESS>
"
  "Language-Team: LANGUAGE <[email protected]>
"
  "Language: 
"
  "MIME-Version: 1.0
"
  "Content-Type: text/plain; charset=UTF-8
"
  "Content-Transfer-Encoding: 8bit
"

  #: ex4-4.py:34
  msgid "Clear Text"
  msgstr "Move to History"
  #: ex4-4.py:36
  msgid "Quit"
  msgstr "Exit"

Consider our code from earlier in this chapter:

  >>>for result in results['results']:
  ... id = result['id']
  ... print(id)
  ... print (result['id'])
  ... details = requests.get(market + id).json()
  ... print (details)
  ... print (details['marketdetails'])
  ... print (details['marketdetails']['GoogleLink'])

Using what you know about Python, can you figure out a way to create a list comprehension that will do the same thing as the preceding code? Remember that list comprehensions are constructed like this:

  [expression for item in list if conditional]

Here is the solution:

  print([requests.get(market + result['id'].json()['marketdetails']['GoogleLink'])
  for result in results['results']])

Using what you know so far about how to use files in Python, can you save the output of your call to the USDA’s API to a file on your machine, to parse later? (Saving it as a .txt file is fine.)

You can use the list comprehension above to simply write to a file as such:

  file = open("markets.txt", "w")
  file.write([requests.get(market +
  result['id'].json()['marketdetails']['GoogleLink'])
  for result in results['results']
])
  file.close

Can you find the docs for Flask that would help us to break our app into smaller, modularized files with our endpoints/views in a separate file, rather than having one big Python file with everything in it? (Hint: It is one concept/feature that Flask offers.)

Flask docs can be found at (http://flask.pocoo.org/docs/blueprints/#blueprints).

Blueprints are how we can separate our app into separate files so that we don’t have one large python file with every piece of functionality in it.

What other HTTP methods can you find? Can you find ways to use them in a Flask app?

Depending on how the reader researches the question, they will find there are a few other HTTP methods; GET, PUT, DELETE, OPTIONS. The second part of this exercise will vary, but the point is to get the reader reading docs and learning how to find answers, and explore what is available to them.

By reading the Requests docs, can you find the method call needed to output the HTML of a website by passing the URL to a requests method?

It’s not pretty, but it’s easy!

  >>> import requests
  >>> r = requests.get("http://www.python.org")
  >>> r.text

In the zip file for this chapter, open the file markets.py and write a doctest string to test the value being returned by the function in the file. Can you think of a reason why a simple doctest string in this code could be incredibly useful for maintaining the code in the future?

Depending on the static data you’ve decided to use, answers will vary, however here is one example using the ZIP code in the code (you may have changed this):

  import requests
  results = requests.get("http://search.ams.usda.gov/farmersmarkets/v1/data.svc/
  zipSearch?zip=46201").json()
  def get_details(results):
       '''
       >>> print(get_details(results))
       http://maps.google.com/?q=39.7776%2C%20-86.0782%20(%22Irvington+Farmers+
       Market+%22Error! Hyperlink reference not valid.

       		 '''
       market = "http://search.ams.usda.gov/farmersmarkets/v1/data.svc/mktDetail?id="
       for result in results['results']:
       		 id = result['id']
       		 details = requests.get(market+id).json()
       		 return details['marketdetails']['GoogleLink']

Write a unittest for a function that will take a string and return that string reversed. Make sure the test fails, because you haven’t written the function to test… yet.

  import unittest
  from reverse import rev
  class TestRev(unittest.TestCase):
       def test_rev(self):
            self.assertEqual(rev('robot'), 'tobor')

                           '''

Write a function for your unittest that takes a string and returns the reverse of that string. Now, run your unittest against that function and modify the function until it passes.

  #reverse.py
  def rev(chars):
                 chars.sort(reverse=True)
                 return chars