So just how would we go about writing such an FTP interface wrapper (he asks, rhetorically)? Given the ftplib library module, wrapping downloads of a particular file in a particular directory is straightforward. Connected FTP objects support two download methods:

We will meet the retrlines method in a later example; the getfile utility module in Example 13-4 always transfers in binary mode with retrbinary. That is, files are downloaded exactly as they were on the server, byte for byte, with the server’s line-feed conventions in text files (you may need to convert line feeds after downloads if they look odd in your text editor—see your editor or system shell commands for pointers, or write a Python script that opens and writes the text as needed).

#!/usr/local/bin/python
"""
Fetch an arbitrary file by FTP.  Anonymous FTP unless you pass a
user=(name, pswd) tuple. Self-test FTPs a test file and site.
"""

from ftplib  import FTP          # socket-based FTP tools
from os.path import exists       # file existence test

def getfile(file, site, dir, user=(), *, verbose=True, refetch=False):
    """
    fetch a file by ftp from a site/directory
    anonymous or real login, binary transfer
    """
    if exists(file) and not refetch:
        if verbose: print(file, 'already fetched')
    else:
        if verbose: print('Downloading', file)
        local = open(file, 'wb')                # local file of same name
        try:
            remote = FTP(site)                  # connect to FTP site
            remote.login(*user)                 # anonymous=() or (name, pswd)
            remote.cwd(dir)
            remote.retrbinary('RETR ' + file, local.write, 1024)
            remote.quit()
        finally:
            local.close()                       # close file no matter what
        if verbose: print('Download done.')     # caller handles exceptions

if __name__ == '__main__':
    from getpass import getpass
    file = 'monkeys.jpg'
    dir  = '.'
    site = 'ftp.rmi.net'
    user = ('lutz', getpass('Pswd?'))
    getfile(file, site, dir, user)

This module is mostly just a repackaging of the FTP code we used to fetch the image file earlier, to make it simpler and reusable. Because it is a callable function, the exported getfile.getfile here tries to be as robust and generally useful as possible, but even a function this small implies some design decisions. Here are a few usage notes:

Also notice that, despite its name, this module is very different from the getfile.py script we studied at the end of the sockets material in the preceding chapter. The socket-based getfile implemented custom client and server-side logic to download a server file to a client machine over raw sockets.

The new getfile here is a client-side tool only. Instead of raw sockets, it uses the standard FTP protocol to request a file from a server; all socket-level details are hidden in the simpler ftplib module’s implementation of the FTP client protocol. Furthermore, the server here is a perpetually running program on the server machine, which listens for and responds to FTP requests on a socket, on the dedicated FTP port (number 21). The net functional effect is that this script requires an FTP server to be running on the machine where the desired file lives, but such a server is much more likely to be available.

While we’re at it, let’s write a script to upload a single file by FTP to a remote machine. The upload interfaces in the FTP module are symmetric with the download interfaces. Given a connected FTP object, its:

Unlike the download interfaces, both of these methods are passed a file object as a whole, not a file object method (or other function). We will meet the storlines method in a later example. The utility module in Example 13-5 uses storbinary such that the file whose name is passed in is always uploaded verbatim—in binary mode, without Unicode encodings or line-feed translations for the target machine’s conventions. If this script uploads a text file, it will arrive exactly as stored on the machine it came from, with client line-feed markers and existing Unicode encoding.

#!/usr/local/bin/python
"""
Store an arbitrary file by FTP in binary mode.  Uses anonymous
ftp unless you pass in a user=(name, pswd) tuple of arguments.
"""

import ftplib                    # socket-based FTP tools

def putfile(file, site, dir, user=(), *, verbose=True):
    """
    store a file by ftp to a site/directory
    anonymous or real login, binary transfer
    """
    if verbose: print('Uploading', file)
    local  = open(file, 'rb')               # local file of same name
    remote = ftplib.FTP(site)               # connect to FTP site
    remote.login(*user)                     # anonymous or real login
    remote.cwd(dir)
    remote.storbinary('STOR ' + file, local, 1024)
    remote.quit()
    local.close()
    if verbose: print('Upload done.')

if __name__ == '__main__':
    site = 'ftp.rmi.net'
    dir  = '.'
    import sys, getpass
    pswd = getpass.getpass(site + ' pswd?')                # filename on cmdline
    putfile(sys.argv[1], site, dir, user=('lutz', pswd))   # nonanonymous login

Notice that for portability, the local file is opened in rb binary input mode this time to suppress automatic line-feed character conversions. If this is binary information, we don’t want any bytes that happen to have the value of the carriage-return character to mysteriously go away during the transfer when run on a Windows client. We also want to suppress Unicode encodings for nontext files, and we want reads to produce the bytes strings expected by the storbinary upload operation (more on input file modes later).

This script uploads a file you name on the command line as a self-test, but you will normally pass in real remote filename, site name, and directory name strings. Also like the download utility, you may pass a (username, password) tuple to the user argument to trigger nonanonymous FTP mode (anonymous FTP is the default).

It’s time for a bit of fun. To test, let’s use these scripts to transfer a copy of the Monty Python theme song audio file I have at my website. First, let’s write a module that downloads and plays the sample file, as shown in Example 13-6.

#!/usr/local/bin/python
"""
Usage: sousa.py.  Fetch and play the Monty Python theme song.
This will not work on your system as is: it requires a machine with Internet access
and an FTP server account you can access, and uses audio filters on Unix and your
.au player on Windows.  Configure this and playfile.py as needed for your platform.
"""

from getpass import getpass
from PP4E.Internet.Ftp.getfile  import getfile
from PP4E.System.Media.playfile import playfile

file = 'sousa.au'                      # default file coordinates
site = 'ftp.rmi.net'                   # Monty Python theme song
dir  = '.'
user = ('lutz', getpass('Pswd?'))

getfile(file, site, dir, user)         # fetch audio file by FTP
playfile(file)                         # send it to audio player

# import os
# os.system('getone.py sousa.au')      # equivalent command line

There’s not much to this script, because it really just combines two tools we’ve already coded. We’re reusing Example 13-4’s getfile to download, and Chapter 6’s playfile module (Example 6-23) to play the audio sample after it is downloaded (turn back to that example for more details on the player part of the task). Also notice the last two lines in this file—we can achieve the same effect by passing in the audio filename as a command-line argument to our original script, but it’s less direct.

As is, this script assumes my FTP server account; configure as desired (alas, this file used to be at the ftp.python.org anonymous FTP site, but that site went dark for security reasons between editions of this book). Once configured, this script will run on any machine with Python, an Internet link, and a recognizable audio player; it works on my Windows laptop with a broadband Internet connection, and it plays the music clip in Windows Media Player (and if I could insert an audio file hyperlink here to show what it sounds like, I would…):

C:...PP4EInternetFtp> sousa.py
Pswd?
Downloading sousa.au
Download done.

C:...PP4EInternetFtp> sousa.py
Pswd?
sousa.au already fetched

The getfile and putfile modules themselves can be used to move the sample file around too. Both can either be imported by clients that wish to use their functions, or run as top-level programs to trigger self-tests and command-line usage. For variety, let’s run these scripts from a command line and the interactive prompt to see how they work. When run standalone, the filename is passed in the command line to putfile and both use password input and default site settings:

C:...PP4EInternetFtp> putfile.py sousa.py
ftp.rmi.net pswd?
Uploading sousa.py
Upload done.

When imported, parameters are passed explicitly to functions:

C:...PP4EInternetFtp> python
>>> from getfile import getfile
>>> getfile(file='sousa.au', site='ftp.rmi.net', dir='.', user=('lutz', 'XXX'))
sousa.au already fetched

C:...PP4EInternetFtp> del sousa.au

C:...PP4EInternetFtp> python
>>> from getfile import getfile
>>> getfile(file='sousa.au', site='ftp.rmi.net', dir='.', user=('lutz', 'XXX'))
Downloading sousa.au
Download done.

>>> from PP4E.System.Media.playfile import playfile
>>> playfile('sousa.au')

Although Python’s ftplib already automates the underlying socket and message formatting chores of FTP, tools of our own like these can make the process even simpler.

As originally coded, our download and upload scripts comprise top-level script code that relies on global variables. Such a structure is difficult to reuse—code runs immediately on imports, and it’s difficult to generalize for varying contexts. Worse, it’s difficult to maintain—when you program by cut-and-paste of existing code, you increase the cost of future changes every time you click the Paste button.

To demonstrate how we might do better, Example 13-12 shows one way to refactor (reorganize) the download script. By wrapping its parts in functions, they become reusable in other modules, including our upload program.

#!/bin/env python
"""
##############################################################################
use FTP to copy (download) all files from a remote site and directory
to a directory on the local machine; this version works the same, but has
been refactored to wrap up its code in functions that can be reused by the
uploader, and possibly other programs in the future - else code redundancy,
which may make the two diverge over time, and can double maintenance costs.
##############################################################################
"""

import os, sys, ftplib
from getpass   import getpass
from mimetypes import guess_type, add_type

defaultSite = 'home.rmi.net'
defaultRdir = '.'
defaultUser = 'lutz'

def configTransfer(site=defaultSite, rdir=defaultRdir, user=defaultUser):
    """
    get upload or download parameters
    uses a class due to the large number
    """
    class cf: pass
    cf.nonpassive = False                 # passive FTP on by default in 2.1+
    cf.remotesite = site                  # transfer to/from this site
    cf.remotedir  = rdir                  # and this dir ('.' means acct root)
    cf.remoteuser = user
    cf.localdir   = (len(sys.argv) > 1 and sys.argv[1]) or '.'
    cf.cleanall   = input('Clean target directory first? ')[:1] in ['y','Y']
    cf.remotepass = getpass(
                    'Password for %s on %s:' % (cf.remoteuser, cf.remotesite))
    return cf

def isTextKind(remotename, trace=True):
    """
    use mimetype to guess if filename means text or binary
    for 'f.html,   guess is ('text/html', None): text
    for 'f.jpeg'   guess is ('image/jpeg', None): binary
    for 'f.txt.gz' guess is ('text/plain', 'gzip'): binary
    for unknowns,  guess may be (None, None): binary
    mimetype can also guess name from type: see PyMailGUI
    """
    add_type('text/x-python-win', '.pyw')                       # not in tables
    mimetype, encoding = guess_type(remotename, strict=False)   # allow extras
    mimetype  = mimetype or '?/?'                               # type unknown?
    maintype  = mimetype.split('/')[0]                          # get first part
    if trace: print(maintype, encoding or '')
    return maintype == 'text' and encoding == None              # not compressed

def connectFtp(cf):
    print('connecting...')
    connection = ftplib.FTP(cf.remotesite)           # connect to FTP site
    connection.login(cf.remoteuser, cf.remotepass)   # log in as user/password
    connection.cwd(cf.remotedir)                     # cd to directory to xfer
    if cf.nonpassive:                                # force active mode FTP
        connection.set_pasv(False)                   # most servers do passive
    return connection

def cleanLocals(cf):
    """
    try to delete all locals files first to remove garbage
    """
    if cf.cleanall:
        for localname in os.listdir(cf.localdir):    # local dirlisting
            try:                                     # local file delete
                print('deleting local', localname)
                os.remove(os.path.join(cf.localdir, localname))
            except:
                print('cannot delete local', localname)

def downloadAll(cf, connection):
    """
    download all files from remote site/dir per cf config
    ftp nlst() gives files list, dir() gives full details
    """
    remotefiles = connection.nlst()                  # nlst is remote listing
    for remotename in remotefiles:
        if remotename in ('.', '..'): continue
        localpath = os.path.join(cf.localdir, remotename)
        print('downloading', remotename, 'to', localpath, 'as', end=' ')
        if isTextKind(remotename):
            # use text mode xfer
            localfile = open(localpath, 'w', encoding=connection.encoding)
            def callback(line): localfile.write(line + '
')
            connection.retrlines('RETR ' + remotename, callback)
        else:
            # use binary mode xfer
            localfile = open(localpath, 'wb')
            connection.retrbinary('RETR ' + remotename, localfile.write)
        localfile.close()
    connection.quit()
    print('Done:', len(remotefiles), 'files downloaded.')

if __name__ == '__main__':
    cf = configTransfer()
    conn = connectFtp(cf)
    cleanLocals(cf)          # don't delete if can't connect
    downloadAll(cf, conn)

Compare this version with the original. This script, and every other in this section, runs the same as the original flat download and upload programs. Although we haven’t changed its behavior, though, we’ve modified the script’s software structure radically—its code is now a set of tools that can be imported and reused in other programs.

The refactored upload program in Example 13-13, for instance, is now noticeably simpler, and the code it shares with the download script only needs to be changed in one place if it ever requires improvement.

#!/bin/env python
"""
##############################################################################
use FTP to upload all files from a local dir to a remote site/directory;
this version reuses downloader's functions, to avoid code redundancy;
##############################################################################
"""

import os
from downloadflat_modular import configTransfer, connectFtp, isTextKind

def cleanRemotes(cf, connection):
    """
    try to delete all remote files first to remove garbage
    """
    if cf.cleanall:
        for remotename in connection.nlst():            # remote dir listing
            try:                                        # remote file delete
                print('deleting remote', remotename)    # skips . and .. exc
                connection.delete(remotename)
            except:
                print('cannot delete remote', remotename)

def uploadAll(cf, connection):
    """
    upload all files to remote site/dir per cf config
    listdir() strips dir path, any failure ends script
    """
    localfiles = os.listdir(cf.localdir)            # listdir is local listing
    for localname in localfiles:
        localpath = os.path.join(cf.localdir, localname)
        print('uploading', localpath, 'to', localname, 'as', end=' ')
        if isTextKind(localname):
            # use text mode xfer
            localfile = open(localpath, 'rb')
            connection.storlines('STOR ' + localname, localfile)
        else:
            # use binary mode xfer
            localfile = open(localpath, 'rb')
            connection.storbinary('STOR ' + localname, localfile)
        localfile.close()
    connection.quit()
    print('Done:', len(localfiles), 'files uploaded.')

if __name__ == '__main__':
    cf = configTransfer(site='learning-python.com', rdir='books', user='lutz')
    conn = connectFtp(cf)
    cleanRemotes(cf, conn)
    uploadAll(cf, conn)

Not only is the upload script simpler now because it reuses common code, but it will also inherit any changes made in the download module. For instance, the isTextKind function was later augmented with code that adds the .pyw extension to mimetypes tables (this file type is not recognized by default); because it is a shared function, the change is automatically picked up in the upload program, too.

This script and the one it imports achieve the same goals as the originals, but changing them for easier code maintenance is a big deal in the real world of software development. The following, for example, downloads the site from one server and uploads to another:

C:...PP4EInternetFtpMirror> python downloadflat_modular.py test
Clean target directory first?
Password for lutz on home.rmi.net:
connecting...
downloading 2004-longmont-classes.html to test2004-longmont-classes.html as text
...lines omitted...
downloading relo-feb010-index.html to test
elo-feb010-index.html as text
Done: 297 files downloaded.

C:...PP4EInternetFtpMirror> python uploadflat_modular.py test
Clean target directory first?
Password for lutz on learning-python.com:
connecting...
uploading test2004-longmont-classes.html to 2004-longmont-classes.html as text
...lines omitted...
uploading testzopeoutline.htm to zopeoutline.htm as text
Done: 297 files uploaded.

The function-based approach of the last two examples addresses the redundancy issue, but they are perhaps clumsier than they need to be. For instance, their cf configuration options object provides a namespace that replaces global variables and breaks cross-file dependencies. Once we start making objects to model namespaces, though, Python’s OOP support tends to be a more natural structure for our code. As one last twist, Example 13-14 refactors the FTP code one more time in order to leverage Python’s class feature.

#!/bin/env python
"""
##############################################################################
use FTP to download or upload all files in a single directory from/to a
remote site and directory;  this version has been refactored to use classes
and OOP for namespace and a natural structure;  we could also structure this
as a download superclass, and an upload subclass which redefines the clean
and transfer methods, but then there is no easy way for another client to
invoke both an upload and download;  for the uploadall variant and possibly
others, also make single file upload/download code in orig loops methods;
##############################################################################
"""

import os, sys, ftplib
from getpass   import getpass
from mimetypes import guess_type, add_type

# defaults for all clients
dfltSite = 'home.rmi.net'
dfltRdir = '.'
dfltUser = 'lutz'

class FtpTools:

    # allow these 3 to be redefined
    def getlocaldir(self):
        return (len(sys.argv) > 1 and sys.argv[1]) or '.'

    def getcleanall(self):
        return input('Clean target dir first?')[:1] in ['y','Y']

    def getpassword(self):
        return getpass(
               'Password for %s on %s:' % (self.remoteuser, self.remotesite))

    def configTransfer(self, site=dfltSite, rdir=dfltRdir, user=dfltUser):
        """
        get upload or download parameters
        from module defaults, args, inputs, cmdline
        anonymous ftp: user='anonymous' pass=emailaddr
        """
        self.nonpassive = False             # passive FTP on by default in 2.1+
        self.remotesite = site              # transfer to/from this site
        self.remotedir  = rdir              # and this dir ('.' means acct root)
        self.remoteuser = user
        self.localdir   = self.getlocaldir()
        self.cleanall   = self.getcleanall()
        self.remotepass = self.getpassword()

    def isTextKind(self, remotename, trace=True):
        """
        use mimetypes to guess if filename means text or binary
        for 'f.html,   guess is ('text/html', None): text
        for 'f.jpeg'   guess is ('image/jpeg', None): binary
        for 'f.txt.gz' guess is ('text/plain', 'gzip'): binary
        for unknowns,  guess may be (None, None): binary
        mimetypes can also guess name from type: see PyMailGUI
        """
        add_type('text/x-python-win', '.pyw')                    # not in tables
        mimetype, encoding = guess_type(remotename, strict=False)# allow extras
        mimetype  = mimetype or '?/?'                            # type unknown?
        maintype  = mimetype.split('/')[0]                       # get 1st part
        if trace: print(maintype, encoding or '')
        return maintype == 'text' and encoding == None           # not compressed

    def connectFtp(self):
        print('connecting...')
        connection = ftplib.FTP(self.remotesite)           # connect to FTP site
        connection.login(self.remoteuser, self.remotepass) # log in as user/pswd
        connection.cwd(self.remotedir)                     # cd to dir to xfer
        if self.nonpassive:                                # force active mode
            connection.set_pasv(False)                     # most do passive
        self.connection = connection

    def cleanLocals(self):
        """
        try to delete all local files first to remove garbage
        """
        if self.cleanall:
            for localname in os.listdir(self.localdir):    # local dirlisting
                try:                                       # local file delete
                    print('deleting local', localname)
                    os.remove(os.path.join(self.localdir, localname))
                except:
                    print('cannot delete local', localname)

    def cleanRemotes(self):
        """
        try to delete all remote files first to remove garbage
        """
        if self.cleanall:
            for remotename in self.connection.nlst():       # remote dir listing
                try:                                        # remote file delete
                    print('deleting remote', remotename)
                    self.connection.delete(remotename)
                except:
                    print('cannot delete remote', remotename)

    def downloadOne(self, remotename, localpath):
        """
        download one file by FTP in text or binary mode
        local name need not be same as remote name
        """
        if self.isTextKind(remotename):
            localfile = open(localpath, 'w', encoding=self.connection.encoding)
            def callback(line): localfile.write(line + '
')
            self.connection.retrlines('RETR ' + remotename, callback)
        else:
            localfile = open(localpath, 'wb')
            self.connection.retrbinary('RETR ' + remotename, localfile.write)
        localfile.close()

    def uploadOne(self, localname, localpath, remotename):
        """
        upload one file by FTP in text or binary mode
        remote name need not be same as local name
        """
        if self.isTextKind(localname):
            localfile = open(localpath, 'rb')
            self.connection.storlines('STOR ' + remotename, localfile)
        else:
            localfile = open(localpath, 'rb')
            self.connection.storbinary('STOR ' + remotename, localfile)
        localfile.close()

    def downloadDir(self):
        """
        download all files from remote site/dir per config
        ftp nlst() gives files list, dir() gives full details
        """
        remotefiles = self.connection.nlst()         # nlst is remote listing
        for remotename in remotefiles:
            if remotename in ('.', '..'): continue
            localpath = os.path.join(self.localdir, remotename)
            print('downloading', remotename, 'to', localpath, 'as', end=' ')
            self.downloadOne(remotename, localpath)
        print('Done:', len(remotefiles), 'files downloaded.')

    def uploadDir(self):
        """
        upload all files to remote site/dir per config
        listdir() strips dir path, any failure ends script
        """
        localfiles = os.listdir(self.localdir)       # listdir is local listing
        for localname in localfiles:
            localpath = os.path.join(self.localdir, localname)
            print('uploading', localpath, 'to', localname, 'as', end=' ')
            self.uploadOne(localname, localpath, localname)
        print('Done:', len(localfiles), 'files uploaded.')

    def run(self, cleanTarget=lambda:None, transferAct=lambda:None):
        """
        run a complete FTP session
        default clean and transfer are no-ops
        don't delete if can't connect to server
        """
        self.connectFtp()
        cleanTarget()
        transferAct()
        self.connection.quit()

if __name__ == '__main__':
    ftp = FtpTools()
    xfermode = 'download'
    if len(sys.argv) > 1:
        xfermode = sys.argv.pop(1)   # get+del 2nd arg
    if xfermode == 'download':
        ftp.configTransfer()
        ftp.run(cleanTarget=ftp.cleanLocals,  transferAct=ftp.downloadDir)
    elif xfermode == 'upload':
        ftp.configTransfer(site='learning-python.com', rdir='books', user='lutz')
        ftp.run(cleanTarget=ftp.cleanRemotes, transferAct=ftp.uploadDir)
    else:
        print('Usage: ftptools.py ["download" | "upload"] [localdir]')

In fact, this last mutation combines uploads and downloads into a single file, because they are so closely related. As before, common code is factored into methods to avoid redundancy. New here, the instance object itself becomes a natural namespace for storing configuration options (they become self attributes). Study this example’s code for more details of the restructuring applied.

Again, this revision runs the same as our original site download and upload scripts; see its self-test code at the end for usage details, and pass in a command-line argument to specify “download” or “upload.” We haven’t changed what it does, we’ve refactored it for maintainability and reuse:

C:...PP4EInternetFtpMirror> ftptools.py download test
Clean target dir first?
Password for lutz on home.rmi.net:
connecting...
downloading 2004-longmont-classes.html to test2004-longmont-classes.html as text
...lines omitted...
downloading relo-feb010-index.html to test
elo-feb010-index.html as text
Done: 297 files downloaded.


C:...PP4EInternetFtpMirror> ftptools.py upload test
Clean target dir first?
Password for lutz on learning-python.com:
connecting...
uploading test2004-longmont-classes.html to 2004-longmont-classes.html as text
...lines omitted...
uploading testzopeoutline.htm to zopeoutline.htm as text
Done: 297 files uploaded.

Although this file can still be run as a command-line script like this, its class is really now a package of FTP tools that can be mixed into other programs and reused. By wrapping its code in a class, it can be easily customized by redefining its methods—its configuration calls, such as getlocaldir, for example, may be redefined in subclasses for custom scenarios.

Perhaps most importantly, using classes optimizes code reusability. Clients of this file can both upload and download directories by simply subclassing or embedding an instance of this class and calling its methods. To see one example of how, let’s move on to the next section.

At this point, we could run whatever email tool we normally use to access our mailbox to verify the results of these two send operations; the two new emails should show up in our mailbox regardless of which mail client is used to view them. Since we’ve already written a Python script for reading mail, though, let’s put it to use as a verification tool—running the popmail script from the last section reveals our two new messages at the end of the mail list (again parts of the output have been trimmed to conserve space and protect the innocent here):

C:...PP4EInternetEmail> popmail.py
Password for pop.secureserver.net?
Connecting...
b'+OK <[email protected]>'
There are 4 mail messages in 5326 bytes
(b'+OK ', [b'1 1860', b'2 1408', b'3 1049', b'4 1009'], 32)
--------------------------------------------------------------------------------
[Press Enter key]

...first two mails omitted...

Received: (qmail 25683 invoked from network); 6 May 2010 14:12:07 −0000
Received: from unknown (HELO p3pismtp01-018.prod.phx3.secureserver.net) ([10.6.1
          (envelope-sender <[email protected]>)
          by p3plsmtp06-04.prod.phx3.secureserver.net (qmail-1.03) with SMTP
          for <[email protected]>; 6 May 2010 14:12:07 −0000
...more deleted...
Received: from [66.194.109.3] by smtp.mailmt.com (ArGoSoft Mail Server .NET v.1.
        for <[email protected]>; Thu, 06 May 2010 10:12:12 −0400
From: [email protected]
To: [email protected]
Date: Thu, 06 May 2010 14:11:07 −0000
Subject: A B C D E F G
Message-ID: <jdlohzf0j8dp8z4x06052010101212@SMTP>
X-FromIP: 66.194.109.3
X-Nonspam: None

Fiddle de dum, Fiddle de dee,
Eric the half a bee.


--------------------------------------------------------------------------------
[Press Enter key]
Received: (qmail 4634 invoked from network); 6 May 2010 14:16:57 −0000
Received: from unknown (HELO p3pismtp01-025.prod.phx3.secureserver.net) ([10.6.1
          (envelope-sender <[email protected]>)
          by p3plsmtp06-05.prod.phx3.secureserver.net (qmail-1.03) with SMTP
          for <[email protected]>; 6 May 2010 14:16:57 −0000
...more deleted...
Received: from [66.194.109.3] by smtp.mailmt.com (ArGoSoft Mail Server .NET v.1.
        for <[email protected]>; Thu, 06 May 2010 10:17:03 −0400
From: [email protected]
To: [email protected]
Date: Thu, 06 May 2010 14:16:31 −0000
Subject: testing smtpmail
Message-ID: <8fad1n462667fik006052010101703@SMTP>
X-FromIP: 66.194.109.3
X-Nonspam: None

Lovely Spam! Wonderful Spam!


--------------------------------------------------------------------------------
Bye.

Notice how the fields we input to our script show up as headers and text in the email’s raw text delivered to the recipient. Technically, some ISPs test to make sure that at least the domain of the email sender’s address (the part after “@”) is a real, valid domain name, and disallow delivery if not. As mentioned earlier, some servers also require that SMTP senders have a direct connection to their network and may require an authentication call with username and password (described near the end of the preceding section). In the second edition of the book, I used an ISP that let me get away with more nonsense, but this may vary per server; the rules have tightened since then to limit spam.

The first mail listed at the end of the preceding section was the one we sent with a fictitious sender address; the second was the more legitimate message. Like sender addresses, header lines are a bit arbitrary under SMTP. Our smtpmail script automatically adds From and To header lines in the message’s text with the same addresses that are passed to the SMTP interface, but only as a polite convention. Sometimes, though, you can’t tell who a mail was sent to, either—to obscure the target audience or to support legitimate email lists, senders may manipulate the contents of both these headers in the message’s text.

For example, if we change smtpmail to not automatically generate a “To:” header line with the same address(es) sent to the SMTP interface call:

text = ('From: %s
Date: %s
Subject: %s
' % (From, Date, Subj))

we can then manually type a “To:” header that differs from the address we’re really sending to—the “To” address list passed into the smtplib send call gives the true recipients, but the “To:” header line in the text of the message is what most mail clients will display (see smtpmail-noTo.py in the examples package for the code needed to support such anonymous behavior, and be sure to type a blank line after “To:”):

C:...PP4EInternetEmail> smtpmail-noTo.py
From? [email protected]
To?   [email protected]
Subj? a b c d e f g
Type message text, end with line=(ctrl + D or Z)
To: [email protected]

Spam; Spam and eggs; Spam, spam, and spam.
^Z
Connecting...
No errors.
Bye.

In some ways, the From and To addresses in send method calls and message header lines are similar to addresses on envelopes and letters in envelopes, respectively. The former is used for routing, but the latter is what the reader sees. Here, From is fictitious in both places. Moreover, I gave the real To address for the account on the server, but then gave a fictitious name in the manually typed “To:” header line—the first address is where it really goes and the second appears in mail clients. If your mail tool picks out the “To:” line, such mails will look odd when viewed.

For instance, when the mail we just sent shows up in my mailbox at learning-python.com, it’s difficult to tell much about its origin or destination in the webmail interface my ISP provides, as captured in Figure 13-5.

Figure 13-5. Anonymous mail in a web-mail client (see also ahead: PyMailGUI)

Furthermore, this email’s raw text won’t help unless we look closely at the “Received:” headers added by the machines it has been routed through:

C:...PP4EInternetEmail> popmail.py
Password for pop.secureserver.net?
Connecting...
b'+OK <[email protected]>'
There are 5 mail messages in 6364 bytes
(b'+OK ', [b'1 1860', b'2 1408', b'3 1049', b'4 1009', b'5 1038'], 40)
--------------------------------------------------------------------------------
[Press Enter key]

...first three mails omitted...


Received: (qmail 30325 invoked from network); 6 May 2010 14:33:45 −0000
Received: from unknown (HELO p3pismtp01-004.prod.phx3.secureserver.net) ([10.6.1
          (envelope-sender <[email protected]>)
          by p3plsmtp06-03.prod.phx3.secureserver.net (qmail-1.03) with SMTP
          for <[email protected]>; 6 May 2010 14:33:45 −0000
...more deleted...
Received: from [66.194.109.3] by smtp.mailmt.com (ArGoSoft Mail Server .NET v.1.
        for <[email protected]>; Thu, 06 May 2010 10:33:16 −0400
From: [email protected]
Date: Thu, 06 May 2010 14:32:32 −0000
Subject: a b c d e f g
To: [email protected]
Message-ID: <66koqg66e0q1c8hl06052010103316@SMTP>
X-FromIP: 66.194.109.3
X-Nonspam: None

Spam; Spam and eggs; Spam, spam, and spam.


--------------------------------------------------------------------------------
Bye.

Once again, though, don’t do this unless you have good cause. This demonstration is intended only to help you understand how mail headers factor into email processing. To write an automatic spam filter that deletes incoming junk mail, for instance, you need to know some of the telltale signs to look for in a message’s text. Spamming techniques have grown much more sophisticated than simply forging sender and recipient names, of course (you’ll find much more on the subject on the Web at large and in the SpamBayes mail filter written in Python), but it’s one common trick.

On the other hand, such To address juggling may also be useful in the context of legitimate mailing lists—the name of the list appears in the “To:” header when the message is viewed, not the potentially many individual recipients named in the send-mail call. As the next section’s example demonstrates, a mail client can simply send a mail to all on the list but insert the general list name in the “To:” header.

But in other contexts, sending email with bogus “From:” and “To:” lines is equivalent to making anonymous phone calls. Most mailers won’t even let you change the From line, and they don’t distinguish between the To address and header line. When you program mail scripts of your own, though, SMTP is wide open in this regard. So be good out there, OK?

>>> import time
>>> time.asctime()
'Wed May 05 17:52:05 2010'

import time
gmt = time.gmtime(time.time())
fmt = '%a, %d %b %Y %H:%M:%S GMT'
str = time.strftime(fmt, gmt)
hdr = 'Date: ' + str
print(hdr)

Date: Wed, 05 May 2010 21:49:32 GMT

>>> import email.utils
>>> email.utils.formatdate()
'Wed, 05 May 2010 21:54:28 −0000'
>>> email.utils.formatdate(localtime=True)
'Wed, 05 May 2010 17:54:52 −0400'
>>> email.utils.formatdate(usegmt=True)
'Wed, 05 May 2010 21:55:22 GMT'

Making a mail with attachments is a little more work, but not much: we just make a root Message and attach nested Message objects created from the MIME type object that corresponds to the type of data we’re attaching. The MIMEText class, for instance, is a subclass of Message, which is tailored for text parts, and knows how to generate the right types of header information when printed. MIMEImage and MIMEAudio similarly customize Message for images and audio, and also know how to apply Base64 and other MIME encodings to binary data. The root message is where we store the main headers of the mail, and we attach parts here, instead of setting the entire payload—the payload is a list now, not a string. MIMEMultipart is a Message that provides the extra header protocol we need for the root:

>>> from email.mime.multipart import MIMEMultipart      # Message subclasses
>>> from email.mime.text import MIMEText                # with extra headers+logic
>>>
>>> top = MIMEMultipart()                               # root Message object
>>> top['from'] = 'Art <[email protected]>'            # subtype default=mixed
>>> top['to']   = '[email protected]'
>>>
>>> sub1 = MIMEText('nice red uniforms...
')           # part Message attachments
>>> sub2 = MIMEText(open('data.txt').read())
>>> sub2.add_header('Content-Disposition', 'attachment', filename='data.txt')
>>> top.attach(sub1)
>>> top.attach(sub2)

When we ask for the text, a correctly formatted full mail text is returned, separators and all, ready to be sent with smtplib—quite a trick, if you’ve ever tried this by hand:

>>> text = top.as_string()    # or do: str(top) or print(top)
>>> print(text)
Content-Type: multipart/mixed; boundary="===============1574823535=="
MIME-Version: 1.0
from: Art <[email protected]>
to: [email protected]

--===============1574823535==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

nice red uniforms...

--===============1574823535==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="data.txt"

line1
line2
line3

--===============1574823535==--

If we are sent this message and retrieve it via poplib, parsing its full text yields a Message object just like the one we built to send. The message walk generator allows us to step through each part, fetching their types and payloads:

>>> text    # same as in prior interaction
'Content-Type: multipart/mixed; boundary="===============1574823535=="
MIME-Ver...'

>>> from email.parser import Parser
>>> msg = Parser().parsestr(text)
>>> msg['from']
'Art <[email protected]>'

>>> for part in msg.walk():
...     print(part.get_content_type())
...     print(part.get_payload())
...     print()
...
 multipart/mixed
[<email.message.Message object at 0x015EC610>,
<email.message.Message object at0x015EC630>]

text/plain
nice red uniforms...


text/plain
line1
line2
line3

Multipart alternative messages (with text and HTML renditions of the same message) can be composed and parsed in similar fashion. Because email clients are able to parse and compose messages with a simple object-based API, they are freed to focus on user-interface instead of text processing.

The first Unicode issue in Python3.1’s email package is nearly a showstopper in some contexts: the bytes strings of the sort produced by poplib for mail fetches must be decoded to str prior to parsing with email. Unfortunately, because there may not be enough information to know how to decode the message bytes per Unicode, some clients of this package may need to be generalized to detect whole-message encodings prior to parsing; in worst cases other than email that may mandate mixed data types, the current package cannot be used at all. Here’s the issue live:

>>> text    # from prior example in his section
'Content-Type: multipart/mixed; boundary="===============1574823535=="
MIME-Ver...'

>>> btext = text.encode()
>>> btext
b'Content-Type: multipart/mixed; boundary="===============1574823535=="
MIME-Ve...'

>>> msg = Parser().parsestr(text)           # email parser expects Unicode str
>>> msg = Parser().parsestr(btext)          # but poplib fetches email as bytes!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:Python31libemailparser.py", line 82, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
TypeError: initial_value must be str or None, not bytes

>>> msg = Parser().parsestr(btext.decode())               # okay per default
>>> msg = Parser().parsestr(btext.decode('utf8'))         # ascii encoded (default)
>>> msg = Parser().parsestr(btext.decode('latin1'))       # ascii is same in all 3
>>> msg = Parser().parsestr(btext.decode('ascii'))

This is less than ideal, as a bytes-based email would be able to handle message encodings more directly. As mentioned, though, the email package is not really fully functional in Python 3.1, because of its legacy str focus, and the sharp distinction that Python 3.X makes between Unicode text and byte strings. In this case, its parser should accept bytes and not expect clients to know how to decode.

Because of that, this book’s email clients take simplistic approaches to decoding fetched message bytes to be parsed by email. Specifically, full-text decoding will try a user-configurable encoding name, then fall back on trying common types as a heuristic, and finally attempt to decode just message headers.

This will suffice for the examples shown but may need to be enhanced for broader applicability. In some cases, encoding may have to be determined by other schemes such as inspecting email headers (if present at all), guessing from bytes structure analysis, or dynamic user feedback. Adding such enhancements in a robust fashion is likely too complex to attempt in a book’s example code, and it is better performed in common standard library tools in any event.

Really, robust decoding of mail text may not be possible today at all, if it requires headers inspections—we can’t inspect a message’s encoding information headers unless we parse the message, but we can’t parse a message with 3.1’s email package unless we already know the encoding. That is, scripts may need to parse in order to decode, but they need to decode in order to parse! The byte strings of poplib and Unicode strings of email in 3.1 are fundamentally at odds. Even within its own libraries, Python 3.X’s changes have created a chicken-and-egg dependency problem that still exists nearly two years after 3.0’s release.

Short of writing our own email parser, or pursuing other similarly complex approaches, the best bet today for fetched messages seems to be decoding per user preferences and defaults, and that’s how we’ll proceed in this edition. The PyMailGUI client of Chapter 14, for instance, will allow Unicode encodings for full mail text to be set on a per-session basis.

The real issue, of course, is that email in general is inherently complicated by the presence of arbitrary text encodings. Besides full mail text, we also must consider Unicode encoding issues for the text components of a message once it’s parsed—both its text parts and its message headers. To see why, let’s move on.

Our next email Unicode issue seems to fly in the face of Python’s generic programming model: the data types of message payload objects may differ, depending on how they are fetched. Especially for programs that walk and process payloads of mail parts generically, this complicates code.

Specifically, the Message object’s get_payload method we used earlier accepts an optional decode argument to control automatic email-style MIME decoding (e.g., Base64, uuencode, quoted-printable). If this argument is passed in as 1 (or equivalently, True), the payload’s data is MIME-decoded when fetched, if required. Because this argument is so useful for complex messages with arbitrary parts, it will normally be passed as true in all cases. Binary parts are normally MIME-encoded, but even text parts might also be present in Base64 or another MIME form if their bytes fall outside email standards. Some types of Unicode text, for example, require MIME encoding.

The upshot is that get_payload normally returns str strings for str text parts, but returns bytes strings if its decode argument is true—even if the message part is known to be text by nature. If this argument is not used, the payload’s type depends upon how it was set: str or bytes. Because Python 3.X does not allow str and bytes to be mixed freely, clients that need to use the result in text processing or store it in files need to accommodate the difference. Let’s run some code to illustrate:

>>> from email.message import Message
>>> m = Message()
>>> m['From'] = 'Lancelot'
>>> m.set_payload('Line?...')

>>> m['From']
'Lancelot'
>>> m.get_payload()                 # str, if payload is str
'Line?...'
>>> m.get_payload(decode=1)         # bytes, if MIME decode (same as decode=True)
b'Line?...'

The combination of these different return types and Python 3.X’s strict str/bytes dichotomy can cause problems in code that processes the result unless they decode carefully:

>>> m.get_payload(decode=True) + 'spam'                        # can't mix in 3.X!
TypeError: can't concat bytes to str
>>> m.get_payload(decode=True).decode() + 'spam'               # convert if required
'Line?...spam'

To make sense of these examples, it may help to remember that there are two different concepts of “encoding” for email text:

The email package handles email-style MIME encodings automatically when we pass decode=1 to fetch parsed payloads, or generate text for messages that have nonprintable parts, but scripts still need to take Unicode encodings into consideration because of Python 3.X’s sharp string types differentiation. For example, the first decode in the following refers to MIME, and the second to Unicode:

m.get_payload(decode=True).decode()  # to bytes via MIME, then to str via Unicode

Even without the MIME decode argument, the payload type may also differ if it is stored in different forms:

>>> m = Message(); m.set_payload('spam'), m.get_payload()      # fetched as stored
'spam'
>>> m = Message(); m.set_payload(b'spam'), m.get_payload()
b'spam'

Moreover, the same hold true for the text-specific MIME subclass (though as we’ll see later in this section, we cannot pass a bytes to its constructor to force a binary payload):

>>> from email.mime.text import MIMEText
>>> m = MIMEText('Line...?')
>>> m['From'] = 'Lancelot'
>>> m['From']
'Lancelot'
>>> m.get_payload()
'Line...?'
>>> m.get_payload(decode=1)
b'Line...?'

Unfortunately, the fact that payloads might be either str or bytes today not only flies in the face of Python’s type-neutral mindset, it can complicate your code—scripts may need to convert in contexts that require one or the other type. For instance, GUI libraries might allow both, but file saves and web page content generation may be less flexible. In our example programs, we’ll process payloads as bytes whenever possible, but decode to str text in cases where required using the encoding information available in the header API described in the next section.

More profoundly, text in email can be even richer than implied so far—in principle, text payloads of a single message may be encoded in a variety of different Unicode schemes (e.g., three HTML webpage file attachments, all in different Unicode encodings, and possibly different than the full message text’s encoding). Although treating such text as binary byte strings can sometimes finesse encoding issues, saving such parts in text-mode files for opening must respect the original encoding types. Further, any text processing performed on such parts will be similarly type-specific.

Luckily, the email package both adds character-set headers when generating message text and retains character-set information for parts if it is present when parsing message text. For instance, adding non-ASCII text attachments simply requires passing in an encoding name—the appropriate message headers are added automatically on text generation, and the character set is available directly via the get_content_charset method:

>>> s = b'Axe4B'
>>> s.decode('latin1')
'AäB'

>>> from email.message import Message
>>> m = Message()
>>> m.set_payload(b'Axe4B', charset='latin1')       # or 'latin-1': see ahead
>>> t = m.as_string()
>>> print(t)
MIME-Version: 1.0
Content-Type: text/plain; charset="latin1"
Content-Transfer-Encoding: base64

QeRC

>>> m.get_content_charset()
'latin1'

Notice how email automatically applies Base64 MIME encoding to non-ASCII text parts on generation, to conform to email standards. The same is true for the more specific MIME text subclass of Message:

>>> from email.mime.text import MIMEText
>>> m = MIMEText(b'Axe4B', _charset='latin1')
>>> t = m.as_string()
>>> print(t)
Content-Type: text/plain; charset="latin1"
MIME-Version: 1.0
Content-Transfer-Encoding: base64

QeRC

>>> m.get_content_charset()
'latin1'

Now, if we parse this message’s text string with email, we get back a new Message whose text payload is the Base64 MIME-encoded text used to represent the non-ASCII Unicode string. Requesting MIME decoding for the payload with decode=1 returns the byte string we originally attached:

>>> from email.parser import Parser
>>> q = Parser().parsestr(t)
>>> q
<email.message.Message object at 0x019ECA50>
>>> q.get_content_type()
'text/plain'
>>> q._payload
'QeRC
'
>>> q.get_payload()
'QeRC
'
>>> q.get_payload(decode=1)
b'Axe4B'

However, running Unicode decoding on this byte string to convert to text fails if we attempt to use the platform default on Windows (UTF8). To be more accurate, and support a wide variety of text types, we need to use the character-set information saved by the parser and attached to the Message object. This is especially important if we need to save the data to a file—we either have to store as bytes in binary mode files, or specify the correct (or at least a compatible) Unicode encoding in order to use such strings for text-mode files. Decoding manually works the same way:

>>> q.get_payload(decode=1).decode()
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-2: unexpected

>>> q.get_content_charset()
'latin1'
>>> q.get_payload(decode=1).decode('latin1')                    # known type
'AäB'
>>> q.get_payload(decode=1).decode(q.get_content_charset())     # allow any type
'AäB'

In fact, all the header details are available on Message objects, if we know where to look. The character set can also be absent entirely, in which case it’s returned as None; clients need to define policies for such ambiguous text (they might try common types, guess, or treat the data as a raw byte string):

>>> q['content-type']                                     # mapping interface
'text/plain; charset="latin1"'
>>> q.items()
[('Content-Type', 'text/plain; charset="latin1"'), ('MIME-Version', '1.0'),
('Content-Transfer-Encoding', 'base64')]

>> q.get_params(header='Content-Type')                    # param interface
[('text/plain', ''), ('charset', 'latin1')]
>>> q.get_param('charset', header='Content-Type')
'latin1'

>>> charset = q.get_content_charset()                     # might be missing
>>> if charset:
...    print(q.get_payload(decode=1).decode(charset))
...
AäB

This handles encodings for message text parts in parsed emails. For composing new emails, we still must apply session-wide user settings or allow the user to specify an encoding for each part interactively. In some of this book’s email clients, payload conversions are performed as needed—using encoding information in message headers after parsing and provided by users during mail composition.

On a related note, the email package also provides support for encoding and decoding message headers themselves (e.g., From, Subject) per email standards when they are not simple text. Such headers are often called Internationalized (or i18n) headers, because they support inclusion of non-ASCII character set text in emails. This term is also sometimes used to refer to encoded text of message payloads; unlike message headers, though, message payload encoding is used for both international Unicode text and truly binary data such as images (as we’ll see in the next section).

Like mail payload parts, i18n headers are encoded specially for email, and may also be encoded per Unicode. For instance, here’s how to decode an encoded subject line from an arguably spammish email that just showed up in my inbox; its =?UTF-8?Q? preamble declares that the data following it is UTF-8 encoded Unicode text, which is also MIME-encoded per quoted-printable for transmission in email (in short, unlike the prior section’s part payloads, which declare their encodings in separate header lines, headers themselves may declare their Unicode and MIME encodings by embedding them in their own content this way):

>>> rawheader = '=?UTF-8?Q?Introducing=20Top=20Values=3A=20A=20Special=20Selecti
on=20of=20Great=20Money=20Savers?='

>>> from email.header import decode_header        # decode per email+MIME
>>> decode_header(rawheader)
[(b'Introducing Top Values: A Special Selection of Great Money Savers', 'utf-8')]

>>> bin, enc = decode_header(rawheader)[0]        # and decode per Unicode
>>> bin, enc
(b'Introducing Top Values: A Special Selection of Great Money Savers', 'utf-8')
>>> bin.decode(enc)
'Introducing Top Values: A Special Selection of Great Money Savers'

Subtly, the email package can return multiple parts if there are encoded substrings in the header, and each must be decoded individually and joined to produce decoded header text. Even more subtly, in 3.1, this package returns all bytes when any substring (or the entire header) is encoded but returns str for a fully unencoded header, and uncoded substrings returned as bytes are encoded per “raw-unicode-escape” in the package—an encoding scheme useful to convert str to bytes when no encoding type applies:

>>> from email.header import decode_header

>>> S1 = 'Man where did you get that assistant?'
>>> S2 = '=?utf-8?q?Man_where_did_you_get_that_assistant=3F?='
>>> S3 = 'Man where did you get that =?UTF-8?Q?assistant=3F?='

# str: don't decode()
>>> decode_header(S1)
[('Man where did you get that assistant?', None)]

# bytes: do decode()
>>> decode_header(S2)
[(b'Man where did you get that assistant?', 'utf-8')]

# bytes: do decode() using raw-unicode-escape applied in package
>>> decode_header(S3)
[(b'Man where did you get that', None), (b'assistant?', 'utf-8')]

# join decoded parts if more than one
>>> parts = decode_header(S3)
>>> ' '.join(abytes.decode('raw-unicode-escape' if enc == None else enc)
...          for (abytes, enc) in parts)
'Man where did you get that assistant?'

We’ll use logic similar to the last step here in the mailtools package ahead, but also retain str substrings intact without attempting to decode.

One wrinkle pertaining to the prior section: for message headers that contain email addresses (e.g., From), the name component of the name/address pair might be encoded this way as well. Because the email package’s header parser expects encoded substrings to be followed by whitespace or the end of string, we cannot ask it to decode a complete address-related header—quotes around name components will fail.

To support such Internationalized address headers, we must also parse out the first part of the email address and then decode. First of all, we need to extract the name and address parts of an email address using email package tools:

>>> from email.utils import parseaddr, formataddr
>>> p = parseaddr('"Smith, Bob" <[email protected]>')      # split into name/addr pair
>>> p                                                # unencoded addr
('Smith, Bob', '[email protected]')
>>> formataddr(p)
'"Smith, Bob" <[email protected]>'

>>> parseaddr('Bob Smith <[email protected]>')             # unquoted name part
('Bob Smith', '[email protected]')
>>> formataddr(parseaddr('Bob Smith <[email protected]>'))
'Bob Smith <[email protected]>'

>>> parseaddr('[email protected]')                          # simple, no name
('', '[email protected]')
>>> formataddr(parseaddr('[email protected]'))
'[email protected]'

Fields with multiple addresses (e.g., To) separate individual addresses by commas. Since email names might embed commas, too, blindly splitting on commas to run each though parsing won’t always work. Instead, another utility can be used to parse each address individually: getaddresses ignores commas in names when spitting apart separate addresses, and parseaddr does, too, because it simply returns the first pair in the getaddresses result (some line breaks were added to the following for legibility):

>>> from email.utils import getaddresses
>>> multi = '"Smith, Bob" <[email protected]>, Bob Smith <[email protected]>, [email protected],
"Bob" <[email protected]>'

>>> getaddresses([multi])
[('Smith, Bob', '[email protected]'), ('Bob Smith', '[email protected]'), ('', '[email protected]'),
('Bob', '[email protected]')]

>>> [formataddr(pair) for pair in getaddresses([multi])]
['"Smith, Bob" <[email protected]>', 'Bob Smith <[email protected]>', '[email protected]',
'Bob <[email protected]>']

>>> ', '.join([formataddr(pair) for pair in getaddresses([multi])])
'"Smith, Bob" <[email protected]>, Bob Smith <[email protected]>, [email protected],
Bob <[email protected]>'

>>> getaddresses(['[email protected]'])     # handles single address cases too
('', '[email protected]')]

Now, decoding email addresses is really just an extra step before and after the normal header decoding logic we saw earlier:

>>> rawfromheader = '"=?UTF-8?Q?Walmart?=" <[email protected]>'

>>> from email.utils import parseaddr, formataddr
>>> from email.header import decode_header

>>> name, addr = parseaddr(rawfromheader)             # split into name/addr parts
>>> name, addr
('=?UTF-8?Q?Walmart?=', '[email protected]')

>>> abytes, aenc = decode_header(name)[0]             # do email+MIME decoding
>>> abytes, aenc
(b'Walmart', 'utf-8')

>>> name = abytes.decode(aenc)                        # do Unicode decoding
>>> name
'Walmart'

>>> formataddr((name, addr))                          # put parts back together
'Walmart <[email protected]>'

Although From headers will typically have just one address, to be fully robust we need to apply this to every address in headers, such as To, Cc, and Bcc. Again, the multiaddress getaddresses utility avoids comma clashes between names and address separators; since it also handles the single address case, it suffices for From headers as well:

>>> rawfromheader = '"=?UTF-8?Q?Walmart?=" <[email protected]>'
>>> rawtoheader = rawfromheader + ', ' + rawfromheader
>>> rawtoheader
'"=?UTF-8?Q?Walmart?=" <[email protected]>, "=?UTF-8?Q?Walmart?=" <newslet
[email protected]>'

>>> pairs = getaddresses([rawtoheader])
>>> pairs
[('=?UTF-8?Q?Walmart?=', '[email protected]'), ('=?UTF-8?Q?Walmart?=', 'ne
[email protected]')]

>>> addrs = []
>>> for name, addr in pairs:
...     abytes, aenc = decode_header(name)[0]      # email+MIME
...     name = abytes.decode(aenc)                 # Unicode
...     addrs.append(formataddr((name, addr)))     # one or more addrs
...
>>> ', '.join(addrs)
'Walmart <[email protected]>, Walmart <[email protected]>'

These tools are generally forgiving for unencoded content and return them intact. To be robust, though, the last portion of code here should also allow for multiple parts returned by decode_header (for encoded substrings), None encoding values for parts (for unencoded substrings), and str substring values instead of bytes (for fully unencoded names).

Decoding this way applies both MIME and Unicode decoding steps to fetched mails. Creating properly encoded headers for inclusion in new mails composed and sent is similarly straightforward:

>>> from email.header import make_header
>>> hdr = make_header([(b'Axc4Bxe4C', 'latin-1')])
>>> print(hdr)
AÄBäC
>>> print(hdr.encode())
=?iso-8859-1?q?A=C4B=E4C?=
>>> decode_header(hdr.encode())
[(b'Axc4Bxe4C', 'iso-8859-1')]

This can be applied to entire headers such as Subject, as well as the name component of each email address in an address-related header line such as From and To (use getaddresses to split into individual addresses first if needed). The header object provides an alternative interface; both techniques handle additional details, such as line lengths, for which we’ll defer to Python manuals:

>>> from email.header import Header
>>> h = Header(b'Axe4Bxc4X', charset='latin-1')
>>> h.encode()
'=?iso-8859-1?q?A=E4B=C4X?='
>>>
>>> h = Header('spam', charset='ascii')       # same as Header('spam')
>>> h.encode()
'spam'

The mailtools package ahead and its PyMailGUI client of Chapter 14 will use these interfaces to automatically decode message headers in fetched mails per their content for display, and to encode headers sent that are not in ASCII format. That latter also applies to the name component of email addresses, and assumes that SMTP servers will allow these to pass. This may encroach on some SMTP server issues which we don’t have space to address in this book. See the Web for more on SMTP headers handling. For more on headers decoding, see also file _test-i18n-headers.py in the examples package; it decodes additional subject and address-related headers using mailtools methods, and displays them in a tkinter Text widget—a foretaste of how these will be displayed in PyMailGUI.

Our last two email Unicode issues are outright bugs which we must work around today, though they will almost certainly be fixed in a future Python release. The first breaks message text generation for all but trivial messages—the email package today no longer supports generation of full mail text for messages that contain any binary parts, such as images or audio files. Without coding workarounds, only simple emails that consist entirely of text parts can be composed and generated in Python 3.1’s email package; any MIME-encoded binary part causes mail text generation to fail.

This is a bit tricky to understand without poring over email’s source code (which, thankfully, we can in the land of open source), but to demonstrate the issue, first notice how simple text payloads are rendered as full message text when printed as we’ve already seen:

C:...PP4EInternetEmail> python
>>> from email.message import Message            # generic message object
>>> m = Message()
>>> m['From'] = '[email protected]'
>>> m.set_payload(open('text.txt').read())       # payload is str text
>>> print(m)                                     # print uses as_string()
From: [email protected]

spam
Spam
SPAM!

As we’ve also seen, for convenience, the email package also provides subclasses of the Message object, tailored to add message headers that provide the extra descriptive details used by email clients to know how to process the data:

>>> from email.mime.text import MIMEText         # Message subclass with headers
>>> text = open('text.txt').read()
>>> m = MIMEText(text)                           # payload is str text
>>> m['From'] = '[email protected]'
>>> print(m)
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
From: [email protected]

spam
Spam
SPAM!

This works for text, but watch what happens when we try to render a message part with truly binary data, such as an image that could not be decoded as Unicode text:

>>> from email.message import Message            # generic Message object
>>> m = Message()
>>> m['From'] = '[email protected]'
>>> bytes = open('monkeys.jpg', 'rb').read()     # read binary bytes (not Unicode)
>>> m.set_payload(bytes)                         # we set the payload to bytes
>>> print(m)
Traceback (most recent call last):
  ...lines omitted...
  File "C:Python31libemailgenerator.py", line 155, in _handle_text
    raise TypeError('string payload expected: %s' % type(payload))
TypeError: string payload expected: <class 'bytes'>

>>> m.get_payload()[:20]
b'xffxd8xffxe0x00x10JFIFx00x01x01x01x00xx00xx00x00'

The problem here is that the email package’s text generator assumes that the message’s payload data is a Base64 (or similar) encoded str text string by generation time, not bytes. Really, the error is probably our fault in this case, because we set the payload to raw bytes manually. We should use the MIMEImage MIME subclass tailored for images; if we do, the email package internally performs Base64 MIME email encoding on the data when the message object is created. Unfortunately, it still leaves it as bytes, not str, despite the fact the whole point of Base64 is to change binary data to text (though the exact Unicode flavor this text should take may be unclear). This leads to additional failures in Python 3.1:

>>> from email.mime.image import MIMEImage       # Message sublcass with hdrs+base64
>>> bytes = open('monkeys.jpg', 'rb').read()     # read binary bytes again
>>> m = MIMEImage(bytes)                         # MIME class does Base64 on data
>>> print(m)
Traceback (most recent call last):
  ...lines omitted...
  File "C:Python31libemailgenerator.py", line 155, in _handle_text
    raise TypeError('string payload expected: %s' % type(payload))
TypeError: string payload expected: <class 'bytes'>

>>> m.get_payload()[:40]                         # this is already Base64 text
b'/9j/4AAQSkZJRgABAQEAeAB4AAD/2wBDAAIBAQIB'

>>> m.get_payload()[:40].decode('ascii')         # but it's still bytes internally!
'/9j/4AAQSkZJRgABAQEAeAB4AAD/2wBDAAIBAQIB'

In other words, not only does the Python 3.1 email package not fully support the Python 3.X Unicode/bytes dichotomy, it was actually broken by it. Luckily, there’s a workaround for this case.

To address this specific issue, I opted to create a custom encoding function for binary MIME attachments, and pass it in to the email package’s MIME message object subclasses for all binary data types. This custom function is coded in the upcoming mailtools package of this chapter (Example 13-23). Because it is used by email to encode from bytes to text at initialization time, it is able to decode to ASCII text per Unicode as an extra step, after running the original call to perform Base64 encoding and arrange content-encoding headers. The fact that email does not do this extra Unicode decoding step itself is a genuine bug in that package (albeit, one introduced by changes elsewhere in Python standard libraries), but the workaround does its job:

# in mailtools.mailSender module ahead in this chapter...
def fix_encode_base64(msgobj):
     from email.encoders import encode_base64
     encode_base64(msgobj)                # what email does normally: leaves bytes
     bytes = msgobj.get_payload()         # bytes fails in email pkg on text gen
     text  = bytes.decode('ascii')        # decode to unicode str so text gen works
     ...line splitting logic omitted...
     msgobj.set_payload('
'.join(lines))

>>> from email.mime.image import MIMEImage
>>> from mailtools.mailSender import fix_encode_base64      # use custom workaround
>>> bytes = open('monkeys.jpg', 'rb').read()
>>> m = MIMEImage(bytes, _encoder=fix_encode_base64)        # convert to ascii str
>>> print(m.as_string()[:500])
Content-Type: image/jpeg
MIME-Version: 1.0
Content-Transfer-Encoding: base64

/9j/4AAQSkZJRgABAQEAeAB4AAD/2wBDAAIBAQIBAQICAgICAgICAwUDAwMDAwYEBAMFBwYHBwcG
BwcICQsJCAgKCAcHCg0KCgsMDAwMBwkODw0MDgsMDAz/2wBDAQICAgMDAwYDAwYMCAcIDAwMDAwM
DAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAz/wAARCAHoAvQDASIA
AhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQA
AAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3
ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc

>>> print(m)   # to print the entire message: very long

Another possible workaround involves defining a custom MIMEImage class that is like the original but does not attempt to perform Base64 ending on creation; that way, we could encode and translate to str before message object creation, but still make use of the original class’s header-generation logic. If you take this route, though, you’ll find that it requires repeating (really, cutting and pasting) far too much of the original logic to be reasonable—this repeated code would have to mirror any future email changes:

>>> from email.mime.nonmultipart import MIMENonMultipart
>>> class MyImage(MIMENonMultipart):
...     def __init__(self, imagedata, subtype):
...         MIMENonMultipart.__init__(self, 'image', subtype)
...         self.set_payload(_imagedata)

...repeat all the base64 logic here, with an extra ASCII Unicode decode...
>>> m = MyImage(text_from_bytes)

Interestingly, this regression in email actually reflects an unrelated change in Python’s base64 module made in 2007, which was completely benign until the Python 3.X bytes/str differentiation came online. Prior to that, the email encoder worked in Python 2.X, because bytes was really str. In 3.X, though, because base64 returns bytes, the normal mail encoder in email also leaves the payload as bytes, even though it’s been encoded to Base64 text form. This in turn breaks email text generation, because it assumes the payload is text in this case, and requires it to be str. As is common in large-scale software systems, the effects of some 3.X changes may have been difficult to anticipate or accommodate in full.

By contrast, parsing binary attachments (as opposed to generating text for them) works fine in 3.X, because the parsed message payload is saved in message objects as a Base64-encoded str string, not bytes, and is converted to bytes only when fetched. This bug seems likely to also go away in a future Python and email package (perhaps even as a simple patch in Python 3.2), but it’s more serious than the other Unicode decoding issues described here, because it prevents mail composition for all but trivial mails.

The flexibility afforded by the package and the Python language allows such a workaround to be developed external to the package, rather than hacking the package’s code directly. With open source and forgiving APIs, you rarely are truly stuck.

Our final email Unicode issue is as severe as the prior one: changes like that of the prior section introduced yet another regression for mail composition. In short, it’s impossible to make text message parts today without specializing for different Unicode encodings.

Some types of text are automatically MIME-encoded for transmission. Unfortunately, because of the str/bytes split, the MIME text message class in email now requires different string object types for different Unicode encodings. The net effect is that you now have to know how the email package will process your text data when making a text message object, or repeat most of its logic redundantly.

For example, to properly generate Unicode encoding headers and apply required MIME encodings, here’s how we must proceed today for common Unicode text types:

>>> m = MIMEText('abc', _charset='ascii')             # pass text for ascii
>>> print(m)
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

abc
>>> m = MIMEText('abc', _charset='latin-1')           # pass text for latin-1
>>> print(m)                                          # but not for 'latin1': ahead
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

abc
>>> m = MIMEText(b'abc', _charset='utf-8')            # pass bytes for utf8
>>> print(m)
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64

YWJj

This works, but if you look closely, you’ll notice that we must pass str to the first two, but bytes to the third. That requires that we special-case code for Unicode types based upon the package’s internal operation. Types other than those expected for a Unicode encoding don’t work at all, because of newly invalid str/bytes combinations that occur inside the email package in 3.1:

>>> m = MIMEText('abc', _charset='ascii')
>>> m = MIMEText(b'abc', _charset='ascii')            # bug: assumes 2.X str
Traceback (most recent call last):
  ...lines omitted...
  File "C:Python31libemailencoders.py", line 60, in encode_7or8bit
    orig.encode('ascii')
AttributeError: 'bytes' object has no attribute 'encode'

>>> m = MIMEText('abc', _charset='latin-1')
>>> m = MIMEText(b'abc', _charset='latin-1')          # bug: qp uses str
Traceback (most recent call last):
  ...lines omitted...
  File "C:Python31libemailquoprimime.py", line 176, in body_encode
    if line.endswith(CRLF):
TypeError: expected an object with the buffer interface

>>> m = MIMEText(b'abc', _charset='utf-8')
>>> m = MIMEText('abc', _charset='utf-8')             # bug: base64 uses bytes
Traceback (most recent call last):
  ...lines omitted...
  File "C:Python31libemailase64mime.py", line 94, in body_encode
    enc = b2a_base64(s[i:i + max_unencoded]).decode("ascii")
TypeError: must be bytes or buffer, not str

Moreover, the email package is pickier about encoding name synonyms than Python and most other tools are: “latin-1” is detected as a quoted-printable MIME type, but “latin1” is unknown and so defaults to Base64 MIME. In fact, this is why Base64 was used for the “latin1” Unicode type earlier in this section—an encoding choice that is irrelevant to any recipient that understands the “latin1” synonym, including Python itself. Unfortunately, that means that we also need to pass in a different string type if we use a synonym the package doesn’t understand today:

>>> m = MIMEText('abc', _charset='latin-1')            # str for 'latin-1'
>>> print(m)
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

abc
>>> m = MIMEText('abc', _charset='latin1')
Traceback (most recent call last):
  ...lines omitted...
  File "C:Python31libemailase64mime.py", line 94, in body_encode
    enc = b2a_base64(s[i:i + max_unencoded]).decode("ascii")
TypeError: must be bytes or buffer, not str

>>> m = MIMEText(b'abc', _charset='latin1')            # bytes for 'latin1'!
>>> print(m)
Content-Type: text/plain; charset="latin1"
MIME-Version: 1.0
Content-Transfer-Encoding: base64

YWJj

There are ways to add aliases and new encoding types in the email package, but they’re not supported out of the box. Programs that care about being robust would have to cross-check the user’s spelling, which may be valid for Python itself, against that expected by email. This also holds true if your data is not ASCII in general—you’ll have to first decode to text in order to use the expected “latin-1” name because its quoted-printable MIME encoding expects str, even though bytes are required if “latin1” triggers the default Base64 MIME:

>>> m = MIMEText(b'Axe4B', _charset='latin1')
>>> print(m)
Content-Type: text/plain; charset="latin1"
MIME-Version: 1.0
Content-Transfer-Encoding: base64

QeRC

>>> m = MIMEText(b'Axe4B', _charset='latin-1')
Traceback (most recent call last):
  ...lines omitted...
  File "C:Python31libemailquoprimime.py", line 176, in body_encode
    if line.endswith(CRLF):
TypeError: expected an object with the buffer interface

>>> m = MIMEText(b'Axe4B'.decode('latin1'), _charset='latin-1')
>>> print(m)
MIME-Version: 1.0
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

A=E4B

In fact, the text message object doesn’t check to see that the data you’re MIME-encoding is valid per Unicode in general—we can send invalid UTF text but the receiver may have trouble decoding it:

>>> m = MIMEText(b'Axe4B', _charset='utf-8')
>>> print(m)
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64

QeRC

>>> b'Axe4B'.decode('utf8')
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-2: unexpected...

>>> import base64
>>> base64.b64decode(b'QeRC')
b'Axe4B'
>>> base64.b64decode(b'QeRC').decode('utf')
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-2: unexpected...

So what to do if we need to attach message text to composed messages if the text’s datatype requirement is indirectly dictated by its Unicode encoding name? The generic Message superclass doesn’t help here directly if we specify an encoding, as it exhibits the same encoding-specific behavior:

>>> m = Message()
>>> m.set_payload('spam', charset='us-ascii')
>>> print(m)
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

spam
>>> m = Message()
>>> m.set_payload(b'spam', charset='us-ascii')
AttributeError: 'bytes' object has no attribute 'encode'

>>> m.set_payload('spam', charset='utf-8')
TypeError: must be bytes or buffer, not str

Although we could try to work around these issues by repeating much of the code that email runs, the redundancy would make us hopelessly tied to its current implementation and dependent upon its future changes. The following, for example, parrots the steps that email runs internally to create a text message object for ASCII encoding text; unlike the MIMEText class, this approach allows all data to be read from files as binary byte strings, even if it’s simple ASCII:

>>> m = Message()
>>> m.add_header('Content-Type', 'text/plain')
>>> m['MIME-Version'] = '1.0'
>>> m.set_param('charset', 'us-ascii')
>>> m.add_header('Content-Transfer-Encoding', '7bit')
>>> data = b'spam'
>>> m.set_payload(data.decode('ascii'))           # data read as bytes here
>>> print(m)
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

spam
>>> print(MIMEText('spam', _charset='ascii'))     # same, but type-specific
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

spam

To do the same for other kinds of text that require MIME encoding, just insert an extra encoding step; although we’re concerned with text parts here, a similar imitative approach could address the binary parts text generation bug we met earlier:

>>> m = Message()
>>> m.add_header('Content-Type', 'text/plain')
>>> m['MIME-Version'] = '1.0'
>>> m.set_param('charset', 'utf-8')
>>> m.add_header('Content-Transfer-Encoding', 'base64')
>>> data = b'spam'
>>> from binascii import b2a_base64               # add MIME encode if needed
>>> data = b2a_base64(data)                       # data read as bytes here too
>>> m.set_payload(data.decode('ascii'))
>>> print(m)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64

c3BhbQ==

>>> print(MIMEText(b'spam', _charset='utf-8'))    # same, but type-specific
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64

c3BhbQ==

This works, but besides the redundancy and dependency it creates, to use this approach broadly we’d also have to generalize to account for all the various kinds of Unicode encodings and MIME encodings possible, like the email package already does internally. We might also have to support encoding name synonyms to be flexible, adding further redundancy. In other words, this requires additional work, and in the end, we’d still have to specialize our code for different Unicode types.

Any way we go, some dependence on the current implementation seems unavoidable today. It seems the best we can do here, apart from hoping for an improved email package in a few years’ time, is to specialize text message construction calls by Unicode type, and assume both that encoding names match those expected by the package and that message data is valid for the Unicode type selected. Here is the sort of arguably magic code that the upcoming mailtools package (again in Example 13-23) will apply to choose text types:

>>> from email.charset import Charset, BASE64, QP
>>> for e in ('us-ascii', 'latin-1', 'utf8', 'latin1', 'ascii'):
...     cset = Charset(e)
...     benc = cset.body_encoding
...     if benc in (None, QP):
...         print(e, benc, 'text')              # read/fetch data as str
...     else:
...         print(e, benc, 'binary')            # read/fetch data as bytes
...
us-ascii None text
latin-1 1 text
utf8 2 binary
latin1 2 binary
ascii None text

We’ll proceed this way in this book, with the major caveat that this is almost certainly likely to require changes in the future because of its strong coupling with the current email implementation.

The email package in Python 3.1 provides powerful tools for parsing and composing mails, and can be used as the basis for full-featured mail clients like those in this book with just a few workarounds. As you can see, though, it is less than fully functional today. Because of that, further specializing code to its current API is perhaps a temporary solution. Short of writing our own email parser and composer (not a practical option in a finitely-sized book!), some compromises are in order here. Moreover, the inherent complexity of Unicode support in email places some limits on how much we can pursue this thread in this book.

In this edition, we will support Unicode encodings of text parts and headers in messages composed, and respect the Unicode encodings in text parts and mail headers of messages fetched. To make this work with the partially crippled email package in Python 3.1, though, we’ll apply the following Unicode policies in various email clients in this book:

These are not necessarily complete solutions. For example, some of this edition’s email clients allow for Unicode encodings for both text attachments and mail headers, but they do nothing about encoding the full text of messages sent beyond the policies inherited from smtplib and implement policies that might be inconvenient in some use cases. But as we’ll see, despite their limitations, our email clients will still be able to handle complex email tasks and a very large set of emails.

Again, since this story is in flux in Python today, watch this book’s website for updates that may improve or be required of code that uses email in the future. A future email may handle Unicode encodings more accurately. Like Python 3.X, though, backward compatibility may be sacrificed in the process and require updates to this book’s code. For more on this issue, see the Web as well as up-to-date Python release notes.

Although this quick tour captures the basic flavor of the interface, we need to step up to larger examples to see more of the email package’s power. The next section takes us on the first of those steps.

This is also where we open and add attachment files, generate message text, and save sent messages to a local file. Most attachment files are opened in binary mode, but as we’ve seen, some text attachments must be opened in text mode because the current email package requires them to be str strings when message objects are created. As we also saw earlier, the email package requires attachments to be str text when mail text is later generated, possibly as the result of MIME encoding.

To satisfy these constraints with the Python 3.1 email package, we must apply the two fixes described earlier— part file open calls select between text or binary mode (and thus read str or bytes) based upon the way email will process the data, and MIME encoding calls for binary data are augmented to decode the result to ASCII text. The latter of these also splits the Base64 text into lines here for binary parts (unlike email), because it is otherwise sent as one long line, which may work in some contexts, but causes problems in some text editors if the raw text is viewed.

Beyond these fixes, clients may optionally provide the names of the Unicode encoding scheme associated with the main text part and each text attachment part. In Chapter 14’s PyMailGUI, this is controlled in the mailconfig user settings module, with UTF-8 used as a fallback default whenever user settings fail to encode a text part. We could in principle also catch part file decoding errors and return an error indicator string (as we do for received mails in the mail fetcher ahead), but sending an invalid attachment is much more grievous than displaying one. Instead, the send request fails entirely on errors.

Finally, there is also new support for encoding non-ASCII headers (both full headers and names of email addresses) per a client-selectable encoding that defaults to UTF-8, and the sent message save file is opened in the same mailconfig Unicode encoding mode used to decode messages when they are fetched.

The latter policy for sent mail saves is used because the sent file may be opened to fetch full mail text in this encoding later by clients which apply this encoding scheme. This is intended to mirror the way that clients such as PyMailGUI save full message text in local files to be opened and parsed later. It might fail if the mail fetcher resorted to guessing a different and incompatible encoding, and it assumes that no message gives rise to incompatibly encoded data in the file across multiple sessions. We could instead keep one save file per encoding, but encodings for full message text probably will not vary; ASCII was the original standard for full mail text, so 7- or 8-bit text is likely.

"""
###############################################################################
send messages, add attachments (see __init__ for docs, test)
###############################################################################
"""

import mailconfig                                      # client's mailconfig
import smtplib, os, mimetypes                          # mime: name to type
import email.utils, email.encoders                     # date string, base64
from .mailTool import MailTool, SilentMailTool         # 4E: package-relative

from email.message          import Message             # general message, obj->text
from email.mime.multipart   import MIMEMultipart       # type-specific messages
from email.mime.audio       import MIMEAudio           # format/encode attachments
from email.mime.image       import MIMEImage
from email.mime.text        import MIMEText
from email.mime.base        import MIMEBase
from email.mime.application import MIMEApplication     # 4E: use new app class


def fix_encode_base64(msgobj):
    """
    4E: workaround for a genuine bug in Python 3.1 email package that prevents
    mail text generation for binary parts encoded with base64 or other email
    encodings;  the normal email.encoder run by the constructor leaves payload
    as bytes, even though it's encoded to base64 text form;  this breaks email
    text generation which assumes this is text and requires it to be str;  net
    effect is that only simple text part emails can be composed in Py 3.1 email
    package as is - any MIME-encoded binary part cause mail text generation to
    fail;  this bug seems likely to go away in a future Python and email package,
    in which case this should become a no-op;  see Chapter 13 for more details;
    """

    linelen = 76  # per MIME standards
    from email.encoders import encode_base64

    encode_base64(msgobj)                # what email does normally: leaves bytes
    text = msgobj.get_payload()          # bytes fails in email pkg on text gen
    if isinstance(text, bytes):          # payload is bytes in 3.1, str in 3.2 alpha
        text = text.decode('ascii')      # decode to unicode str so text gen works

    lines = []                           # split into lines, else 1 massive line
    text  = text.replace('
', '')       # no 
 present in 3.1, but futureproof me!
    while text:
        line, text = text[:linelen], text[linelen:]
        lines.append(line)
    msgobj.set_payload('
'.join(lines))


def fix_text_required(encodingname):
    """
    4E: workaround for str/bytes combination errors in email package;  MIMEText
    requires different types for different Unicode encodings in Python 3.1, due
    to the different ways it MIME-encodes some types of text;  see Chapter 13;
    the only other alternative is using generic Message and repeating much code;
    """
    from email.charset import Charset, BASE64, QP

    charset = Charset(encodingname)   # how email knows what to do for encoding
    bodyenc = charset.body_encoding   # utf8, others require bytes input data
    return bodyenc in (None, QP)      # ascii, latin1, others require str


class MailSender(MailTool):
    """
    send mail: format a message, interface with an SMTP server;
    works on any machine with Python+Inet, doesn't use cmdline mail;
    a nonauthenticating client: see MailSenderAuth if login required;
    4E: tracesize is num chars of msg text traced: 0=none, big=all;
    4E: supports Unicode encodings for main text and text parts;
    4E: supports header encoding, both full headers and email names;
    """
    def __init__(self, smtpserver=None, tracesize=256):
        self.smtpServerName = smtpserver or mailconfig.smtpservername
        self.tracesize = tracesize

    def sendMessage(self, From, To, Subj, extrahdrs, bodytext, attaches,
                                      saveMailSeparator=(('=' * 80) + 'PY
'),
                                      bodytextEncoding='us-ascii',
                                      attachesEncodings=None):
        """
        format and send mail: blocks caller, thread me in a GUI;
        bodytext is main text part, attaches is list of filenames,
        extrahdrs is list of (name, value) tuples to be added;
        raises uncaught exception if send fails for any reason;
        saves sent message text in a local file if successful;

        assumes that To, Cc, Bcc hdr values are lists of 1 or more already
        decoded addresses (possibly in full name+<addr> format); client
        must parse to split these on delimiters, or use multiline input;
        note that SMTP allows full name+<addr> format in recipients;
        4E: Bcc addrs now used for send/envelope, but header is dropped;
        4E: duplicate recipients removed, else will get >1 copies of mail;
        caveat: no support for multipart/alternative mails, just /mixed;
        """

        # 4E: assume main body text is already in desired encoding;
        # clients can decode to user pick, default, or utf8 fallback;
        # either way, email needs either str xor bytes specifically;

        if fix_text_required(bodytextEncoding):
            if not isinstance(bodytext, str):
                bodytext = bodytext.decode(bodytextEncoding)
        else:
            if not isinstance(bodytext, bytes):
                bodytext = bodytext.encode(bodytextEncoding)

        # make message root
        if not attaches:
            msg = Message()
            msg.set_payload(bodytext, charset=bodytextEncoding)
        else:
            msg = MIMEMultipart()
            self.addAttachments(msg, bodytext, attaches,
                                     bodytextEncoding, attachesEncodings)

        # 4E: non-ASCII hdrs encoded on sends; encode just name in address,
        # else smtp may drop the message completely; encodes all envelope
        # To names (but not addr) also, and assumes servers will allow;
        # msg.as_string retains any line breaks added by encoding headers;

        hdrenc = mailconfig.headersEncodeTo or 'utf-8'        # default=utf8
        Subj = self.encodeHeader(Subj, hdrenc)                # full header
        From = self.encodeAddrHeader(From, hdrenc)            # email names
        To   = [self.encodeAddrHeader(T, hdrenc) for T in To] # each recip
        Tos  = ', '.join(To)                                  # hdr+envelope

        # add headers to root
        msg['From']    = From
        msg['To']      = Tos                        # poss many: addr list
        msg['Subject'] = Subj                       # servers reject ';' sept
        msg['Date']    = email.utils.formatdate()   # curr datetime, rfc2822 utc
        recip = To
        for name, value in extrahdrs:               # Cc, Bcc, X-Mailer, etc.
            if value:
                if name.lower() not in ['cc', 'bcc']:
                    value = self.encodeHeader(value, hdrenc)
                    msg[name] = value
                else:
                    value = [self.encodeAddrHeader(V, hdrenc) for V in value]
                    recip += value                     # some servers reject ['']
                    if name.lower() != 'bcc':          # 4E: bcc gets mail, no hdr
                        msg[name] = ', '.join(value)   # add commas between cc

        recip = list(set(recip))                       # 4E: remove duplicates
        fullText = msg.as_string()                     # generate formatted msg

        # sendmail call raises except if all Tos failed,
        # or returns failed Tos dict for any that failed

        self.trace('Sending to...' + str(recip))
        self.trace(fullText[:self.tracesize])                  # SMTP calls connect
        server = smtplib.SMTP(self.smtpServerName, timeout=15) # this may fail too
        self.getPassword()                                     # if srvr requires
        self.authenticateServer(server)                        # login in subclass
        try:
            failed = server.sendmail(From, recip, fullText)    # except or dict
        except:
            server.close()                                     # 4E: quit may hang!
            raise                                              # reraise except
        else:
            server.quit()                                      # connect + send OK
        self.saveSentMessage(fullText, saveMailSeparator)      # 4E: do this first
        if failed:
            class SomeAddrsFailed(Exception): pass
            raise SomeAddrsFailed('Failed addrs:%s
' % failed)
        self.trace('Send exit')

    def addAttachments(self, mainmsg, bodytext, attaches,
                                      bodytextEncoding, attachesEncodings):
        """
        format a multipart message with attachments;
        use Unicode encodings for text parts if passed;
        """
        # add main text/plain part
        msg = MIMEText(bodytext, _charset=bodytextEncoding)
        mainmsg.attach(msg)

        # add attachment parts
        encodings = attachesEncodings or (['us-ascii'] * len(attaches))
        for (filename, fileencode) in zip(attaches, encodings):
            # filename may be absolute or relative
            if not os.path.isfile(filename):             # skip dirs, etc.
                continue

            # guess content type from file extension, ignore encoding
            contype, encoding = mimetypes.guess_type(filename)
            if contype is None or encoding is not None:  # no guess, compressed?
                contype = 'application/octet-stream'     # use generic default
            self.trace('Adding ' + contype)

            # build sub-Message of appropriate kind
            maintype, subtype = contype.split('/', 1)
            if maintype == 'text':                       # 4E: text needs encoding
                if fix_text_required(fileencode):        # requires str or bytes
                    data = open(filename, 'r', encoding=fileencode)
                else:
                    data = open(filename, 'rb')
                msg = MIMEText(data.read(), _subtype=subtype, _charset=fileencode)
                data.close()

            elif maintype == 'image':
                data = open(filename, 'rb')              # 4E: use fix for binaries
                msg  = MIMEImage(
                       data.read(), _subtype=subtype, _encoder=fix_encode_base64)
                data.close()

            elif maintype == 'audio':
                data = open(filename, 'rb')
                msg  = MIMEAudio(
                       data.read(), _subtype=subtype, _encoder=fix_encode_base64)
                data.close()

            elif maintype == 'application':              # new  in 4E
                data = open(filename, 'rb')
                msg  = MIMEApplication(
                       data.read(), _subtype=subtype, _encoder=fix_encode_base64)
                data.close()

            else:
                data = open(filename, 'rb')              # application/* could
                msg  = MIMEBase(maintype, subtype)       # use this code too
                msg.set_payload(data.read())
                data.close()                             # make generic type
                fix_encode_base64(msg)                   # was broken here too!
               #email.encoders.encode_base64(msg)        # encode using base64

            # set filename (ascii or utf8/mime encoded) and attach to container
            basename = self.encodeHeader(os.path.basename(filename)) # oct 2011
            msg.add_header('Content-Disposition',
                           'attachment', filename=basename)
            mainmsg.attach(msg)

        # text outside mime structure, seen by non-MIME mail readers
        mainmsg.preamble = 'A multi-part MIME format message.
'
        mainmsg.epilogue = ''  # make sure message ends with a newline

    def saveSentMessage(self, fullText, saveMailSeparator):
        """
        append sent message to local file if send worked for any;
        client: pass separator used for your application, splits;
        caveat: user may change the file at same time (unlikely);
        """
        try:
            sentfile = open(mailconfig.sentmailfile, 'a',
                                  encoding=mailconfig.fetchEncoding)    # 4E
            if fullText[-1] != '
': fullText += '
'
            sentfile.write(saveMailSeparator)
            sentfile.write(fullText)
            sentfile.close()
        except:
            self.trace('Could not save sent message')    # not a show-stopper

    def encodeHeader(self, headertext, unicodeencoding='utf-8'):
        """
        4E: encode composed non-ascii message headers content per both email
        and Unicode standards, according to an optional user setting or UTF-8;
        header.encode adds line breaks in header string automatically if needed;
        """
        try:
            headertext.encode('ascii')
        except:
            try:
                hdrobj = email.header.make_header([(headertext, unicodeencoding)])
                headertext = hdrobj.encode()
            except:
                pass         # auto splits into multiple cont lines if needed
        return headertext    # smtplib may fail if it won't encode to ascii

    def encodeAddrHeader(self, headertext, unicodeencoding='utf-8'):
        """
        4E: try to encode non-ASCII names in email addresess per email, MIME,
        and Unicode standards; if this fails drop name and use just addr part;
        if cannot even get addresses, try to decode as a whole, else smtplib
        may run into errors when it tries to encode the entire mail as ASCII;
        utf-8 default should work for most, as it formats code points broadly;

        inserts newlines if too long or hdr.encode split names to multiple lines,
        but this may not catch some lines longer than the cutoff (improve me);
        as used, Message.as_string formatter won't try to break lines further;
        see also decodeAddrHeader in mailParser module for the inverse of this;
        """
        try:
            pairs = email.utils.getaddresses([headertext])   # split addrs + parts
            encoded = []
            for name, addr in pairs:
                try:
                    name.encode('ascii')         # use as is if okay as ascii
                except UnicodeError:             # else try to encode name part
                    try:
                        uni  = name.encode(unicodeencoding)
                        hdr  = email.header.make_header([(uni, unicodeencoding)])
                        name = hdr.encode()
                    except:
                        name = None              # drop name, use address part only
                joined = email.utils.formataddr((name, addr))  # quote name if need
                encoded.append(joined)

            fullhdr = ', '.join(encoded)
            if len(fullhdr) > 72 or '
' in fullhdr:      # not one short line?
                fullhdr = ',
 '.join(encoded)            # try multiple lines
            return fullhdr
        except:
            return self.encodeHeader(headertext)

    def authenticateServer(self, server):
        pass  # no login required for this server/class

    def getPassword(self):
        pass  # no login required for this server/class


################################################################################
# specialized subclasses
################################################################################

class MailSenderAuth(MailSender):
    """
    use for servers that require login authorization;
    client: choose MailSender or MailSenderAuth super
    class based on mailconfig.smtpuser setting (None?)
    """
    smtpPassword = None    # 4E: on class, not self, shared by poss N instances
    
    def __init__(self, smtpserver=None, smtpuser=None, tracesize=256):
        MailSender.__init__(self, smtpserver, tracesize)
        self.smtpUser = smtpuser or mailconfig.smtpuser
        #self.smtpPassword = None # 4E: makes PyMailGUI ask for each send!

    def authenticateServer(self, server):
        server.login(self.smtpUser, self.smtpPassword)

    def getPassword(self):
        """
        get SMTP auth password if not yet known;
        may be called by superclass auto, or client manual:
        not needed until send, but don't run in GUI thread;
        get from client-side file or subclass method
        """
        if not self.smtpPassword:
            try:
                localfile = open(mailconfig.smtppasswdfile)
                MailSenderAuth.smtpPassword = localfile.readline()[:-1] # 4E
                self.trace('local file password' + repr(self.smtpPassword))
            except:
                MailSenderAuth.smtpPassword = self.askSmtpPassword()    # 4E

    def askSmtpPassword(self):
        assert False, 'Subclass must define method'

class MailSenderAuthConsole(MailSenderAuth):
    def askSmtpPassword(self):
        import getpass
        prompt = 'Password for %s on %s?' % (self.smtpUser, self.smtpServerName)
        return getpass.getpass(prompt)

class SilentMailSender(SilentMailTool, MailSender):
    pass   # replaces trace

This module deals strictly in email text; parsing email after it has been fetched is delegated to a different module in the package. Moreover, this module doesn’t cache already loaded information; clients must add their own mail-retention tools if desired. Clients must also provide password input methods or pass one in, if they cannot use the console input subclass here (e.g., GUIs and web-based programs).

The loading and deleting tasks use the standard library poplib module in ways we saw earlier in this chapter, but notice that there are interfaces for fetching just message header text with the TOP action in POP if the mail server supports it. This can save substantial time if clients need to fetch only basic details for an email index. In addition, the header and full-text fetchers are equipped to load just mails newer than a particular number (useful once an initial load is run), and to restrict fetches to a fixed-sized set of the mostly recently arrived emails (useful for large inboxes with slow Internet access or servers).

This module also supports the notion of progress indicators—for methods that perform multiple downloads or deletions, callers may pass in a function that will be called as each mail is processed. This function will receive the current and total step numbers. It’s left up to the caller to render this in a GUI, console, or other user interface.

Additionally, this module is where we apply the session-wide message bytes Unicode decoding policy required for parsing, as discussed earlier in this chapter. This decoding uses an encoding name user setting in the mailconfig module, followed by heuristics. Because this decoding is performed immediately when a mail is fetched, all clients of this package can assume message text is str Unicode strings—including any later parsing, display, or save operations. In addition to the mailconfig setting, we also apply a few guesses with common encoding types, though it’s not impossible that this may lead to problems if mails decoded by guessing cannot be written to mail save fails using the mailconfig setting.

As described, this session-wide approach to encodings is not ideal, but it can be adjusted per client session and reflects the current limitations of email in Python 3.1—its parser requires already decoded Unicode strings, but fetches return bytes. If this decoding fails, as a last resort we attempt to decode headers only, as either ASCII (or other common format) text or the platform default, and insert an error message in the email body—a heuristic that attempts to avoid killing clients with exceptions if possible (see file _test-decoding.py in the examples package for a test of this logic). In practice, an 8-bit Unicode encoding such as Latin-1 will probably suffice in most cases, because ASCII was the original requirement of email standards.

In principle, we could try to search for encoding information in message headers if it’s present, by parsing mails partially ourselves. We might then take a per-message instead of per-session approach to decoding full text, and associate an encoding type with each mail for later processing such as saves, though this raises further complications, as a save file can have just one (compatible) encoding, not one per message. Moreover, character sets in email headers may refer to individual components, not the entire email’s text. Since most mails will conform to 7- or 8-bit standards, and since a future email release will likely address this issue, extra complexity is probably not warranted for this case in this book.

Also keep in mind that the Unicode decoding performed here is for the entire mail text fetched from a server. Really, this is just one part of the email encoding story in the Unicode-aware world of today. In addition:

When you start studying this example, you’ll also notice that Example 13-24 devotes substantial code to detecting synchronization errors between an email list held by a client and the current state of the inbox at the POP email server. Normally, POP assigns relative message numbers to email in the inbox, and only adds newly arrived emails to the end of the inbox. As a result, relative message numbers from an earlier fetch may usually be used to delete and fetch in the future.

However, although rare, it is not impossible for the server’s inbox to change in ways that invalidate previously fetched message numbers. For instance, emails may be deleted in another client, and the server itself may move mails from the inbox to an undeliverable state on download errors (this may vary per ISP). In both cases, email may be removed from the middle of the inbox, throwing some prior relative message numbers out of sync with the server.

This situation can result in fetching the wrong message in an email client—users receive a different message than the one they thought they had selected. Worse, this can make deletions inaccurate—if a mail client uses a relative message number in a delete request, the wrong mail may be deleted if the inbox has changed since the index was fetched.

To assist clients, Example 13-24 includes tools, which match message headers on deletions to ensure accuracy and perform general inbox synchronization tests on demand. These tools are useful only to clients that retain the fetched email list as state information. We’ll use these in the PyMailGUI client in Chapter 14. There, deletions use the safe interface, and loads run the on-demand synchronization test; on detection of synchronization errors, the inbox index is automatically reloaded. For now, see Example 13-24 source code and comments for more details.

Note that the synchronization tests try a variety of matching techniques, but require the complete headers text and, in the worst case, must parse headers and match many header fields. In many cases, the single previously fetched message-id header field would be sufficient for matching against messages in the server’s inbox. However, because this field is optional and can be forged to have any value, it might not always be a reliable way to identify messages. In other words, a same-valued message-id may not suffice to guarantee a match, although it can be used to identify a mismatch; in Example 13-24, the message-id is used to rule out a match if either message has one, and they differ in value. This test is performed before falling back on slower parsing and multiple header matches.

"""
###############################################################################
retrieve, delete, match mail from a POP server (see __init__ for docs, test)
###############################################################################
"""

import poplib, mailconfig, sys               # client's mailconfig on sys.path
print('user:', mailconfig.popusername)       # script dir, pythonpath, changes

from .mailParser import MailParser                 # for headers matching (4E: .)
from .mailTool   import MailTool, SilentMailTool   # trace control supers (4E: .)

# index/server msgnum out of synch tests
class DeleteSynchError(Exception): pass            # msg out of synch in del
class TopNotSupported(Exception): pass             # can't run synch test
class MessageSynchError(Exception): pass           # index list out of sync

class MailFetcher(MailTool):
    """
    fetch mail: connect, fetch headers+mails, delete mails
    works on any machine with Python+Inet; subclass me to cache
    implemented with the POP protocol; IMAP requires new class;
    4E: handles decoding of full mail text on fetch for parser;
    """
    def __init__(self, popserver=None, popuser=None, poppswd=None, hastop=True):
        self.popServer   = popserver or mailconfig.popservername
        self.popUser     = popuser   or mailconfig.popusername
        self.srvrHasTop  = hastop
        self.popPassword = poppswd  # ask later if None

    def connect(self):
        self.trace('Connecting...')
        self.getPassword()                          # file, GUI, or console
        server = poplib.POP3(self.popServer, timeout=15)
        server.user(self.popUser)                   # connect,login POP server
        server.pass_(self.popPassword)              # pass is a reserved word
        self.trace(server.getwelcome())             # print returned greeting
        return server

    # use setting in client's mailconfig on import search path;
    # to tailor, this can be changed in class or per instance;
    fetchEncoding = mailconfig.fetchEncoding

    def decodeFullText(self, messageBytes):
        """
        4E, Py3.1: decode full fetched mail text bytes to str Unicode string;
        done at fetch, for later display or parsing (full mail text is always
        Unicode thereafter);  decode with per-class or per-instance setting, or
        common types;  could also try headers inspection, or intelligent guess
        from structure; in Python 3.2/3.3, this step may not be required: if so,
        change to return message line list intact; for more details see Chapter 13;

        an 8-bit encoding such as latin-1 will likely suffice for most emails, as
        ASCII is the original standard;  this method applies to entire/full message
        text, which is really just one part of the email encoding story: Message
        payloads and Message headers may also be encoded per email, MIME, and
        Unicode standards; see Chapter 13 and mailParser and mailSender for more;
        """
        text = None
        kinds =  [self.fetchEncoding]             # try user setting first
        kinds += ['ascii', 'latin1', 'utf8']      # then try common types
        kinds += [sys.getdefaultencoding()]       # and platform dflt (may differ)
        for kind in kinds:                        # may cause mail saves to fail
            try:
                text = [line.decode(kind) for line in messageBytes]
                break
            except (UnicodeError, LookupError):   # LookupError: bad name
                pass

        if text == None:
            # try returning headers + error msg, else except may kill client;
            # still try to decode headers per ascii, other, platform default;

            blankline = messageBytes.index(b'')
            hdrsonly  = messageBytes[:blankline]
            commons   = ['ascii', 'latin1', 'utf8']
            for common in commons:
                try:
                    text = [line.decode(common) for line in hdrsonly]
                    break
                except UnicodeError:
                    pass
            else:                                                  # none worked
                try:
                    text = [line.decode() for line in hdrsonly]    # platform dflt?
                except UnicodeError:
                    text = ['From: (sender of unknown Unicode format headers)']
            text += ['', '--Sorry: mailtools cannot decode this mail content!--']
        return text

    def downloadMessage(self, msgnum):
        """
        load full raw text of one mail msg, given its
        POP relative msgnum; caller must parse content
        """
        self.trace('load ' + str(msgnum))
        server = self.connect()
        try:
            resp, msglines, respsz = server.retr(msgnum)
        finally:
            server.quit()
        msglines = self.decodeFullText(msglines)   # raw bytes to Unicode str
        return '
'.join(msglines)                 # concat lines for parsing

    def downloadAllHeaders(self, progress=None, loadfrom=1):
        """
        get sizes, raw header text only, for all or new msgs
        begins loading headers from message number loadfrom
        use loadfrom to load newly arrived mails only
        use downloadMessage to get a full msg text later
        progress is a function called with (count, total);
        returns: [headers text], [mail sizes], loadedfull?

        4E: add mailconfig.fetchlimit to support large email
        inboxes: if not None, only fetches that many headers,
        and returns others as dummy/empty mail; else inboxes
        like one of mine (4K emails) are not practical to use;
        4E: pass loadfrom along to downloadAllMsgs (a buglet);
        """
        if not self.srvrHasTop:                    # not all servers support TOP
            # naively load full msg text
            return self.downloadAllMsgs(progress, loadfrom)
        else:
            self.trace('loading headers')
            fetchlimit = mailconfig.fetchlimit
            server = self.connect()                # mbox now locked until quit
            try:
                resp, msginfos, respsz = server.list()   # 'num size' lines list
                msgCount = len(msginfos)                 # alt to srvr.stat[0]
                msginfos = msginfos[loadfrom-1:]         # drop already loadeds
                allsizes = [int(x.split()[1]) for x in msginfos]
                allhdrs  = []
                for msgnum in range(loadfrom, msgCount+1):          # poss empty
                    if progress: progress(msgnum, msgCount)         # run callback
                    if fetchlimit and (msgnum <= msgCount - fetchlimit):
                        # skip, add dummy hdrs
                        hdrtext = 'Subject: --mail skipped--

'
                        allhdrs.append(hdrtext)
                    else:
                        # fetch, retr hdrs only
                        resp, hdrlines, respsz = server.top(msgnum, 0)
                        hdrlines = self.decodeFullText(hdrlines)
                        allhdrs.append('
'.join(hdrlines))
            finally:
                server.quit()                          # make sure unlock mbox
            assert len(allhdrs) == len(allsizes)
            self.trace('load headers exit')
            return allhdrs, allsizes, False

    def downloadAllMessages(self, progress=None, loadfrom=1):
        """
        load full message text for all msgs from loadfrom..N,
        despite any caching that may be being done in the caller;
        much slower than downloadAllHeaders, if just need hdrs;

        4E: support mailconfig.fetchlimit: see downloadAllHeaders;
        could use server.list() to get sizes of skipped emails here
        too, but clients probably don't care about these anyhow;
        """
        self.trace('loading full messages')
        fetchlimit = mailconfig.fetchlimit
        server = self.connect()
        try:
            (msgCount, msgBytes) = server.stat()          # inbox on server
            allmsgs  = []
            allsizes = []
            for i in range(loadfrom, msgCount+1):         # empty if low >= high
                if progress: progress(i, msgCount)
                if fetchlimit and (i <= msgCount - fetchlimit):
                    # skip, add dummy mail
                    mailtext = 'Subject: --mail skipped--

Mail skipped.
'
                    allmsgs.append(mailtext)
                    allsizes.append(len(mailtext))
                else:
                    # fetch, retr full mail
                    (resp, message, respsz) = server.retr(i)  # save text on list
                    message = self.decodeFullText(message)
                    allmsgs.append('
'.join(message))        # leave mail on server
                    allsizes.append(respsz)                   # diff from len(msg)
        finally:
            server.quit()                                     # unlock the mail box
        assert len(allmsgs) == (msgCount - loadfrom) + 1      # msg nums start at 1
       #assert sum(allsizes) == msgBytes                      # not if loadfrom > 1
        return allmsgs, allsizes, True                        # not if fetchlimit

    def deleteMessages(self, msgnums, progress=None):
        """
        delete multiple msgs off server; assumes email inbox
        unchanged since msgnums were last determined/loaded;
        use if msg headers not available as state information;
        fast, but poss dangerous: see deleteMessagesSafely
        """
        self.trace('deleting mails')
        server = self.connect()
        try:
            for (ix, msgnum) in enumerate(msgnums):   # don't reconnect for each
                if progress: progress(ix+1, len(msgnums))
                server.dele(msgnum)
        finally:                                      # changes msgnums: reload
            server.quit()

    def deleteMessagesSafely(self, msgnums, synchHeaders, progress=None):
        """
        delete multiple msgs off server, but use TOP fetches to
        check for a match on each msg's header part before deleting;
        assumes the email server supports the TOP interface of POP,
        else raises TopNotSupported - client may call deleteMessages;

        use if the mail server might change the inbox since the email
        index was last fetched, thereby changing POP relative message
        numbers;  this can happen if email is deleted in a different
        client;  some ISPs may also move a mail from inbox to the
        undeliverable box in response to a failed download;

        synchHeaders must be a list of already loaded mail hdrs text,
        corresponding to selected msgnums (requires state);  raises
        exception if any out of synch with the email server;  inbox is
        locked until quit, so it should not change between TOP check
        and actual delete: synch check must occur here, not in caller;
        may be enough to call checkSynchError+deleteMessages, but check
        each msg here in case deletes and inserts in middle of inbox;
        """
        if not self.srvrHasTop:
            raise TopNotSupported('Safe delete cancelled')

        self.trace('deleting mails safely')
        errmsg  = 'Message %s out of synch with server.
'
        errmsg += 'Delete terminated at this message.
'
        errmsg += 'Mail client may require restart or reload.'

        server = self.connect()                       # locks inbox till quit
        try:                                          # don't reconnect for each
            (msgCount, msgBytes) = server.stat()      # inbox size on server
            for (ix, msgnum) in enumerate(msgnums):
                if progress: progress(ix+1, len(msgnums))
                if msgnum > msgCount:                            # msgs deleted
                    raise DeleteSynchError(errmsg % msgnum)
                resp, hdrlines, respsz = server.top(msgnum, 0)   # hdrs only
                hdrlines = self.decodeFullText(hdrlines)
                msghdrs = '
'.join(hdrlines)
                if not self.headersMatch(msghdrs, synchHeaders[msgnum-1]):
                    raise DeleteSynchError(errmsg % msgnum)
                else:
                    server.dele(msgnum)               # safe to delete this msg
        finally:                                      # changes msgnums: reload
            server.quit()                             # unlock inbox on way out

    def checkSynchError(self, synchHeaders):
        """
        check to see if already loaded hdrs text in synchHeaders
        list matches what is on the server, using the TOP command in
        POP to fetch headers text; use if inbox can change due to
        deletes in other client, or automatic action by email server;
        raises except if out of synch, or error while talking to server;

        for speed, only checks last in last: this catches inbox deletes,
        but assumes server won't insert before last (true for incoming
        mails); check inbox size first: smaller if just deletes;  else
        top will differ if deletes and newly arrived messages added at
        end;  result valid only when run: inbox may change after return;
        """
        self.trace('synch check')
        errormsg  = 'Message index out of synch with mail server.
'
        errormsg += 'Mail client may require restart or reload.'
        server = self.connect()
        try:
            lastmsgnum = len(synchHeaders)                      # 1..N
            (msgCount, msgBytes) = server.stat()                # inbox size
            if lastmsgnum > msgCount:                           # fewer now?
                raise MessageSynchError(errormsg)               # none to cmp
            if self.srvrHasTop:
                resp, hdrlines, respsz = server.top(lastmsgnum, 0)  # hdrs only
                hdrlines = self.decodeFullText(hdrlines)
                lastmsghdrs = '
'.join(hdrlines)
                if not self.headersMatch(lastmsghdrs, synchHeaders[-1]):
                    raise MessageSynchError(errormsg)
        finally:
            server.quit()

    def headersMatch(self, hdrtext1, hdrtext2):
        """"
        may not be as simple as a string compare: some servers add
        a "Status:" header that changes over time; on one ISP, it
        begins as "Status: U" (unread), and changes to "Status: RO"
        (read, old) after fetched once - throws off synch tests if
        new when index fetched, but have been fetched once before
        delete or last-message check;  "Message-id:" line is unique
        per message in theory, but optional, and can be anything if
        forged; match more common: try first; parsing costly: try last
        """
        # try match by simple string compare
        if hdrtext1 == hdrtext2:
            self.trace('Same headers text')
            return True

        # try match without status lines
        split1 = hdrtext1.splitlines()       # s.split('
'), but no final ''
        split2 = hdrtext2.splitlines()
        strip1 = [line for line in split1 if not line.startswith('Status:')]
        strip2 = [line for line in split2 if not line.startswith('Status:')]
        if strip1 == strip2:
            self.trace('Same without Status')
            return True

        # try mismatch by message-id headers if either has one
        msgid1 = [line for line in split1 if line[:11].lower() == 'message-id:']
        msgid2 = [line for line in split2 if line[:11].lower() == 'message-id:']
        if (msgid1 or msgid2) and (msgid1 != msgid2):
            self.trace('Different Message-Id')
            return False

        # try full hdr parse and common headers if msgid missing or trash
        tryheaders  = ('From', 'To', 'Subject', 'Date')
        tryheaders += ('Cc', 'Return-Path', 'Received')
        msg1 = MailParser().parseHeaders(hdrtext1)
        msg2 = MailParser().parseHeaders(hdrtext2)
        for hdr in tryheaders:                          # poss multiple Received
            if msg1.get_all(hdr) != msg2.get_all(hdr):  # case insens, dflt None
                self.trace('Diff common headers')
                return False

        # all common hdrs match and don't have a diff message-id
        self.trace('Same common headers')
        return True

    def getPassword(self):
        """
        get POP password if not yet known
        not required until go to server
        from client-side file or subclass method
        """
        if not self.popPassword:
            try:
                localfile = open(mailconfig.poppasswdfile)
                self.popPassword = localfile.readline()[:-1]
                self.trace('local file password' + repr(self.popPassword))
            except:
                self.popPassword = self.askPopPassword()

    def askPopPassword(self):
        assert False, 'Subclass must define method'


################################################################################
# specialized subclasses
################################################################################

class MailFetcherConsole(MailFetcher):
    def askPopPassword(self):
        import getpass
        prompt = 'Password for %s on %s?' % (self.popUser, self.popServer)
        return getpass.getpass(prompt)

class SilentMailFetcher(SilentMailTool, MailFetcher):
    pass   # replaces trace

This module also provides support for decoding message headers per email standards (both full headers and names in address headers), and handles decoding per text part encodings. Headers are decoded according to their content, using tools in the email package; the headers themselves give their MIME and Unicode encodings, so no user intervention is required. For client convenience, we also perform Unicode decoding for main text parts to convert them from bytes to str here if needed.

The latter main-text decoding merits elaboration. As discussed earlier in this chapter, Message objects (main or attached) may return their payloads as bytes if we fetch with a decode=1 argument, or if they are bytes to begin with; in other cases, payloads may be returned as str. We generally need to decode bytes in order to treat payloads as text.

In mailtools itself, str text part payloads are automatically encoded to bytes by decode=1 and then saved to binary-mode files to finesse encoding issues, but main-text payloads are decoded to str if they are bytes. This main-text decoding is performed per the encoding name in the part’s message header (if present and correct), the platform default, or a guess. As we learned in Chapter 9, while GUIs may allow bytes for display, str text generally provides broader Unicode support; furthermore, str is sometimes needed for later processing such as line wrapping and webpage generation.

Since this package can’t predict the role of other part payloads besides the main text, clients are responsible for decoding and encoding as necessary. For instance, other text parts which are saved in binary mode here may require that message headers be consulted later to extract Unicode encoding names for better display. For example, Chapter 14’s PyMailGUI will proceed this way to open text parts on demand, passing message header encoding information on to PyEdit for decoding as text is loaded.

Some of the to-text conversions performed here are potentially partial solutions (some parts may lack the required headers and fail per the platform defaults) and may need to be improved; since this seems likely to be addressed in a future release of Python’s email package, we’ll settle for our assumptions here.

"""
###############################################################################
parsing and attachment extract, analyse, save (see __init__ for docs, test)
###############################################################################
"""

import os, mimetypes, sys                       # mime: map type to name
import email.parser                             # parse text to Message object
import email.header                             # 4E: headers decode/encode
import email.utils                              # 4E: addr header parse/decode
from email.message import Message               # Message may be traversed
from .mailTool import MailTool                  # 4E: package-relative

class MailParser(MailTool):
    """
    methods for parsing message text, attachments

    subtle thing: Message object payloads are either a simple
    string for non-multipart messages, or a list of Message
    objects if multipart (possibly nested); we don't need to
    distinguish between the two cases here, because the Message
    walk generator always returns self first, and so works fine
    on non-multipart messages too (a single object is walked);

    for simple messages, the message body is always considered
    here to be the sole part of the mail;  for multipart messages,
    the parts list includes the main message text, as well as all
    attachments;  this allows simple messages not of type text to
    be handled like attachments in a UI (e.g., saved, opened);
    Message payload may also be None for some oddball part types;

    4E note: in Py 3.1, text part payloads are returned as bytes
    for decode=1, and might be str otherwise; in mailtools, text
    is stored as bytes for file saves, but main-text bytes payloads
    are decoded to Unicode str per mail header info or platform
    default+guess; clients may need to convert other payloads:
    PyMailGUI uses headers to decode parts saved to binary files;

    4E supports fetched message header auto-decoding per its own
    content, both for general headers such as Subject, as well as
    for names in address header such as From and To; client must
    request this after parse, before display: parser doesn't decode;
    """

    def walkNamedParts(self, message):
        """
        generator to avoid repeating part naming logic;
        skips multipart headers, makes part filenames;
        message is already parsed email.message.Message object;
        doesn't skip oddball types: payload may be None, must
        handle in part saves; some others may warrant skips too;
        """
        for (ix, part) in enumerate(message.walk()):    # walk includes message
            fulltype = part.get_content_type()          # ix includes parts skipped
            maintype = part.get_content_maintype()
            if maintype == 'multipart':                 # multipart/*: container
                continue
            elif fulltype == 'message/rfc822':          # 4E: skip message/rfc822
                continue                                # skip all message/* too?
            else:
                filename, contype = self.partName(part, ix)
                yield (filename, contype, part)

    def partName(self, part, ix):
        """
        extract filename and content type from message part;
        filename: tries Content-Disposition, then Content-Type
        name param, or generates one based on mimetype guess;
        """
        filename = part.get_filename()                # filename in msg hdrs?
        contype  = part.get_content_type()            # lowercase maintype/subtype
        if not filename:
            filename = part.get_param('name')         # try content-type name
        if not filename:
            if contype == 'text/plain':               # hardcode plain text ext
                ext = '.txt'                          # else guesses .ksh!
            else:
                ext = mimetypes.guess_extension(contype)
                if not ext: ext = '.bin'              # use a generic default
            filename = 'part-%03d%s' % (ix, ext)
        return (self.decodeHeader(filename), contype) # oct 2011: decode i18n fnames

    def saveParts(self, savedir, message):
        """
        store all parts of a message as files in a local directory;
        returns [('maintype/subtype', 'filename')] list for use by
        callers, but does not open any parts or attachments here;
        get_payload decodes base64, quoted-printable, uuencoded data;
        mail parser may give us a None payload for oddball types we
        probably should skip over: convert to str here to be safe;
        """
        if not os.path.exists(savedir):
            os.mkdir(savedir)
        partfiles = []
        for (filename, contype, part) in self.walkNamedParts(message):
            fullname = os.path.join(savedir, filename)
            fileobj  = open(fullname, 'wb')             # use binary mode
            content  = part.get_payload(decode=1)       # decode base64,qp,uu
            if not isinstance(content, bytes):          # 4E: need bytes for rb
                content = b'(no content)'               # decode=1 returns bytes,
            fileobj.write(content)                      # but some payloads None
            fileobj.close()                             # 4E: not str(content)
            partfiles.append((contype, fullname))       # for caller to open
        return partfiles

    def saveOnePart(self, savedir, partname, message):
        """
        ditto, but find and save just one part by name
        """
        if not os.path.exists(savedir):
            os.mkdir(savedir)
        fullname = os.path.join(savedir, partname)
        (contype, content) = self.findOnePart(partname, message)
        if not isinstance(content, bytes):          # 4E: need bytes for rb
            content = b'(no content)'               # decode=1 returns bytes,
        open(fullname, 'wb').write(content)         # but some payloads None
        return (contype, fullname)                  # 4E: not str(content)

    def partsList(self, message):
        """"
        return a list of filenames for all parts of an
        already parsed message, using same filename logic
        as saveParts, but do not store the part files here
        """
        validParts = self.walkNamedParts(message)
        return [filename for (filename, contype, part) in validParts]

    def findOnePart(self, partname, message):
        """
        find and return part's content, given its name;
        intended to be used in conjunction with partsList;
        we could also mimetypes.guess_type(partname) here;
        we could also avoid this search by saving in dict;
        4E: content may be str or bytes--convert as needed;
        """
        for (filename, contype, part) in self.walkNamedParts(message):
            if filename == partname:
                content = part.get_payload(decode=1)          # does base64,qp,uu
                return (contype, content)                     # may be bytes text

    def decodedPayload(self, part, asStr=True):
        """
        4E: decode text part bytes to Unicode str for display, line wrap,
        etc.; part is a Message; (decode=1) undoes MIME email encodings
        (base64, uuencode, qp), bytes.decode() performs additional Unicode
        text string decodings; tries charset encoding name in message
        headers first (if present, and accurate), then tries platform
        defaults and a few guesses before giving up with error string;
        """
        payload = part.get_payload(decode=1)           # payload may be bytes
        if asStr and isinstance(payload, bytes):       # decode=1 returns bytes
            tries = []
            enchdr = part.get_content_charset()        # try msg headers first!
            if enchdr:
                tries += [enchdr]                      # try headers first
            tries += [sys.getdefaultencoding()]        # same as bytes.decode()
            tries += ['latin1', 'utf8']                # try 8-bit, incl ascii
            for trie in tries:                         # try utf8 (windows dflt)
                try:
                    payload = payload.decode(trie)     # give it a shot, eh?
                    break
                except (UnicodeError, LookupError):    # lookuperr: bad name
                    pass
            else:
                payload = '--Sorry: cannot decode Unicode text--'
        return payload

    def findMainText(self, message, asStr=True):
        """
        for text-oriented clients, return first text part's str;
        for the payload of a simple message, or all parts of
        a multipart message, looks for text/plain, then text/html,
        then text/*, before deducing that there is no text to
        display;  this is a heuristic, but covers most simple,
        multipart/alternative, and multipart/mixed messages;
        content-type defaults to text/plain if not in simple msg;

        handles message nesting at top level by walking instead
        of list scans;  if non-multipart but type is text/html,
        returns the HTML as the text with an HTML type: caller
        may open in web browser, extract plain text, etc;  if
        nonmultipart and not text, there is  no text to display:
        save/open message content in UI; caveat: does not try
        to concatenate multiple inline text/plain parts if any;
        4E: text payloads may be bytes--decodes to str here;
        4E: asStr=False to get raw bytes for HTML file saves;
        """

        # try to find a plain text
        for part in message.walk():                            # walk visits message
            type = part.get_content_type()                     # if nonmultipart
            if type == 'text/plain':                           # may be base64,qp,uu
                return type, self.decodedPayload(part, asStr)  # bytes to str too?

        # try to find an HTML part
        for part in message.walk():
            type = part.get_content_type()                     # caller renders html
            if type == 'text/html':
                return type, self.decodedPayload(part, asStr)

        # try any other text type, including XML
        for part in message.walk():
            if part.get_content_maintype() == 'text':
                return part.get_content_type(), self.decodedPayload(part, asStr)

        # punt: could use first part, but it's not marked as text
        failtext = '[No text to display]' if asStr else b'[No text to display]'
        return 'text/plain', failtext

    def decodeHeader(self, rawheader):
        """
        4E: decode existing i18n message header text per both email and Unicode
        standards, according to its content; return as is if unencoded or fails;
        client must call this to display: parsed Message object does not decode;
        i18n header example: '=?UTF-8?Q?Introducing=20Top=20Values=20..Savers?=';
        i18n header example: 'Man where did you get that =?UTF-8?Q?assistant=3F?=';

        decode_header handles any line breaks in header string automatically, may
        return multiple parts if any substrings of hdr are encoded, and returns all
        bytes in parts list if any encodings found (with unencoded parts encoded as
        raw-unicode-escape and enc=None) but returns a single part with enc=None
        that is str instead of bytes in Py3.1 if the entire header is unencoded
        (must handle mixed types here); see Chapter 13 for more details/examples;

        the following first attempt code was okay unless any encoded substrings, or
        enc was returned as None (raised except which returned rawheader unchanged):
        hdr, enc = email.header.decode_header(rawheader)[0]
        return hdr.decode(enc) # fails if enc=None: no encoding or encoded substrs
        """
        try:
            parts = email.header.decode_header(rawheader)
            decoded = []
            for (part, enc) in parts:                      # for all substrings
                if enc == None:                            # part unencoded?
                    if not isinstance(part, bytes):        # str: full hdr unencoded
                        decoded += [part]                  # else do unicode decode
                    else:
                        decoded += [part.decode('raw-unicode-escape')]
                else:
                    decoded += [part.decode(enc)]
            return ' '.join(decoded)
        except:
            return rawheader         # punt!

    def decodeAddrHeader(self, rawheader):
        """
        4E: decode existing i18n address header text per email and Unicode,
        according to its content; must parse out first part of email address
        to get i18n part: '"=?UTF-8?Q?Walmart?=" <[email protected]>';
        From will probably have just 1 addr, but To, Cc, Bcc may have many;

        decodeHeader handles nested encoded substrings within an entire hdr,
        but we can't simply call it for entire hdr here because it fails if
        encoded name substring ends in " quote instead of whitespace or endstr;
        see also encodeAddrHeader in mailSender module for the inverse of this;

        the following first attempt code failed to handle encoded substrings in
        name, and raised exc for unencoded bytes parts if any encoded substrings;
        namebytes, nameenc = email.header.decode_header(name)[0]  (do email+MIME)
        if nameenc: name = namebytes.decode(nameenc)              (do Unicode?)
        """
        try:
            pairs = email.utils.getaddresses([rawheader])  # split addrs and parts
            decoded = []                                   # handles name commas
            for (name, addr) in pairs:
                try:
                    name = self.decodeHeader(name)                # email+MIME+Uni
                except:
                    name = None   # but uses encooded name if exc in decodeHeader
                joined = email.utils.formataddr((name, addr))     # join parts
                decoded.append(joined)
            return ', '.join(decoded)                             # >= 1 addrs
        except:
            return self.decodeHeader(rawheader)    # try decoding entire string

    def splitAddresses(self, field):
        """
        4E: use comma separator for multiple addrs in the UI, and
        getaddresses to split correctly and allow for comma in the
        name parts of addresses; used by PyMailGUI to split To, Cc,
        Bcc as needed for user inputs and copied headers;  returns
        empty list if field is empty, or any exception occurs;
        """
        try:
            pairs = email.utils.getaddresses([field])                # [(name,addr)]
            return [email.utils.formataddr(pair) for pair in pairs]  # [name <addr>]
        except:
            return ''   # syntax error in user-entered field?, etc.

    # returned when parses fail
    errorMessage = Message()
    errorMessage.set_payload('[Unable to parse message - format error]')

    def parseHeaders(self, mailtext):
        """
        parse headers only, return root email.message.Message object
        stops after headers parsed, even if nothing else follows (top)
        email.message.Message object is a mapping for mail header fields
        payload of message object is None, not raw body text
        """
        try:
            return email.parser.Parser().parsestr(mailtext, headersonly=True)
        except:
            return self.errorMessage

    def parseMessage(self, fulltext):
        """
        parse entire message, return root email.message.Message object
        payload of message object is a string if not is_multipart()
        payload of message object is more Messages if multiple parts
        the call here same as calling email.message_from_string()
        """
        try:
            return email.parser.Parser().parsestr(fulltext)       # may fail!
        except:
            return self.errorMessage     # or let call handle? can check return

    def parseMessageRaw(self, fulltext):
        """
        parse headers only, return root email.message.Message object
        stops after headers parsed, for efficiency (not yet used here)
        payload of message object is raw text of mail after headers
        """
        try:
            return email.parser.HeaderParser().parsestr(fulltext)
        except:
            return self.errorMessage

Here’s a run of the self-test script; it generates a lot of output, most of which has been deleted here for presentation in this book—as usual, run this on your own for further details:

C:...PP4EInternetEmailmailtools> selftest.py
config: ..mailconfig.py
user: [email protected]
Adding text/x-python
Sending to...['[email protected]']
Content-Type: multipart/mixed; boundary="===============0085314748=="
MIME-Version: 1.0
From: [email protected]
To: [email protected]
Subject: testing mailtools package
Date: Sat, 08 May 2010 19:26:22 −0000
X-Mailer: mailtools

A multi-part MIME format message.

--===============0085314748==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit

Here is my source code

--===============0085314748==
Content-Type: text/x-python; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="selftest.py"

"""
###############################################################################
self-test when this file is run as a program
###############################################################################
"""
...more lines omitted...

print(maintext)
input('Press Enter to exit')   # pause if clicked on Windows

--===============0085314748==--

Send exit
loading headers
Connecting...
Password for [email protected] on pop.secureserver.net?
b'+OK <[email protected]>'
(1, 7)
(2, 7)
(3, 7)
(4, 7)
(5, 7)
(6, 7)
(7, 7)
load headers exit
Received: (qmail 7690 invoked from network); 5 May 2010 15:29:43 −0000
Received: from unknown (HELO p3pismtp01-026.prod.phx3.secureserver.net) ([10.6.1
...more lines omitted...

load mail?y
load 1
Connecting...
b'+OK <[email protected]>'
Received: (qmail 7690 invoked from network); 5 May 2010 15:29:43 −0000
Received: from unknown (HELO p3pismtp01-026.prod.phx3.secureserver.net) ([10.6.1
...more lines omitted...

load mail?
loading full messages
Connecting...
b'+OK <[email protected]>'
(3, 7)
(4, 7)
(5, 7)
(6, 7)
(7, 7)
Received: (qmail 25683 invoked from network); 6 May 2010 14:12:07 −0000
Received: from unknown (HELO p3pismtp01-018.prod.phx3.secureserver.net) ([10.6.1
...more lines omitted...

Parsed: A B C D E F G
Fiddle de dum, Fiddle de dee,
Eric the half a bee.

Press Enter to exit

This program is used interactively, the same as the original. In fact, the output is nearly identical, so we won’t go into further details. Here’s a quick look at this script in action; run this on your own machine to see it firsthand:

C:...PP4EInternetEmail> pymail2.py
user: [email protected]
loading headers
Connecting...
Password for [email protected] on pop.secureserver.net?
b'+OK <[email protected]>'
1 of 7
2 of 7
3 of 7
4 of 7
5 of 7
6 of 7
7 of 7
load headers exit
[Pymail email client]
1:      1860 bytes
        From    =>[email protected]
        To      =>[email protected]
        Date    =>Wed, 5 May 2010 11:29:36 −0400 (EDT)
        Subject =>I'm a Lumberjack, and I'm Okay
2:      1408 bytes
        From    =>[email protected]
        To      =>[email protected]
        Date    =>Wed, 05 May 2010 08:33:47 −0700
        Subject =>testing
3:      1049 bytes
        From    =>[email protected]
        To      =>[email protected]
        Date    =>Thu, 06 May 2010 14:11:07 −0000
        Subject =>A B C D E F G
4:      1038 bytes
        From    =>[email protected]
        To      =>[email protected]
        Date    =>Thu, 06 May 2010 14:32:32 −0000
        Subject =>a b c d e f g
5:      957 bytes
        From    =>[email protected]
        To      =>maillist
        Date    =>Thu, 06 May 2010 10:58:40 −0400
        Subject =>test interactive smtplib
[Press Enter key]
6:      1037 bytes
        From    =>[email protected]
        To      =>[email protected]
        Date    =>Fri, 07 May 2010 20:32:38 −0000
        Subject =>Among our weapons are these
7:      3248 bytes
        From    =>[email protected]
        To      =>[email protected]
        Date    =>Sat, 08 May 2010 19:26:22 −0000
        Subject =>testing mailtools package
[Pymail] Action? (i, l, d, s, m, q, ?) l 7
load 7
Connecting...
b'+OK <[email protected]>'
-------------------------------------------------------------------------------
Here is my source code

-------------------------------------------------------------------------------
[Pymail] Action? (i, l, d, s, m, q, ?) d 7
[Pymail] Action? (i, l, d, s, m, q, ?) m
From? [email protected]
To?   [email protected]
Subj? test pymail2 send
Type message text, end with line="."
Run away! Run away!
.
Sending to...['[email protected]']
From: [email protected]
To: [email protected]
Subject: test pymail2 send
Date: Sat, 08 May 2010 19:44:25 −0000

Run away! Run away!

Send exit
[Pymail] Action? (i, l, d, s, m, q, ?) q
To be deleted: [7]
Delete?y
Deleting messages from server...
deleting mails
Connecting...
b'+OK <[email protected]>'

The messages in our mailbox have quite a few origins now—ISP webmail clients, basic SMTP scripts, the Python interactive command line, mailtools self-test code, and two console-based email clients; in later chapters, we’ll add even more. All their mails look the same to our script; here’s a verification of the email we just sent (the second fetch finds it already in-cache):

C:...PP4EInternetEmail> pymail2.py
user: [email protected]
loading headers
Connecting...
...more lines omitted...

[Press Enter key]
6:      1037 bytes
        From    =>[email protected]
        To      =>[email protected]
        Date    =>Fri, 07 May 2010 20:32:38 −0000
        Subject =>Among our weapons are these
7:      984 bytes
        From    =>[email protected]
        To      =>[email protected]
        Date    =>Sat, 08 May 2010 19:44:25 −0000
        Subject =>test pymail2 send
[Pymail] Action? (i, l, d, s, m, q, ?) l 7
load 7
Connecting...
b'+OK <[email protected]>'
-------------------------------------------------------------------------------
Run away! Run away!

-------------------------------------------------------------------------------
[Pymail] Action? (i, l, d, s, m, q, ?) l 7
-------------------------------------------------------------------------------
Run away! Run away!

-------------------------------------------------------------------------------
[Pymail] Action? (i, l, d, s, m, q, ?) q

Study pymail2’s code for more insights. As you’ll see, this version eliminates some complexities, such as the manual formatting of composed mail message text. It also does a better job of displaying a mail’s text—instead of blindly listing the full mail text (attachments and all), it uses mailtools to fetch the first text part of the message. The messages we’re using are too simple to show the difference, but for a mail with attachments, this new version will be more focused about what it displays.

Moreover, because the interface to mail is encapsulated in the mailtools package’s modules, if it ever must change, it will only need to be changed in that module, regardless of how many mail clients use its tools. And because the code in mailtools is shared, if we know it works for one client, we can be sure it will work in another; there is no need to debug new code.

On the other hand, pymail2 doesn’t really leverage much of the power of either mailtools or the underlying email package it uses. For example, things like attachments, Internationalized headers, and inbox synchronization are not handled at all, and printing of some decoded main text may contain character sets incompatible with the console terminal interface. To see the full scope of the email package, we need to explore a larger email system, such as PyMailGUI or PyMailCGI. The first of these is the topic of the next chapter, and the second appears in Chapter 16. First, though, let’s quickly survey a handful of additional client-side protocol tools.

The next listing shows this script being used to trigger a remote program. As before, if you don’t give the local filename explicitly, the script strips the base filename out of the filename argument. That’s not always easy or appropriate for program invocations—the filename can contain both a remote directory path at the front and query parameters at the end for a remote program invocation.

Given a script invocation URL and no explicit output filename, the script extracts the base filename in the middle by using first the standard urllib.parse module to pull out the file path, and then os.path.split to strip off the directory path. However, the resulting filename is a remote script’s name, and it may or may not be an appropriate place to store the data locally. In the first run that follows, for example, the script’s output goes in a local file called languages.py, the script name in the middle of the URL; in the second, we instead name the output CxxSyntax.html explicitly to suppress filename extraction:

C:...PP4EInternetOther> python http-getfile-urllib2.py localhost
                                /cgi-bin/languages.py?language=Scheme
http://localhost/cgi-bin/languages.py?language=Scheme languages.py
b'<TITLE>Languages</TITLE>
'
b'<H1>Syntax</H1><HR>
'
b'<H3>Scheme</H3><P><PRE>
'
b' (display "Hello World") (newline)  
'
b'</PRE></P><BR>
'
b'<HR>
'

C:...PP4EInternetOther> python http-getfile-urllib2.py localhost
                                /cgi-bin/languages.py?language=C++ CxxSyntax.html
http://localhost/cgi-bin/languages.py?language=C++ CxxSyntax.html
b'<TITLE>Languages</TITLE>
'
b'<H1>Syntax</H1><HR>
'
b'<H3>C  </H3><P><PRE>
'
b"Sorry--I don't know that language
"
b'</PRE></P><BR>
'
b'<HR>
'

The remote script returns a not-found message when passed “C++” in the last command here. It turns out that “+” is a special character in URL strings (meaning a space), and to be robust, both of the urllib scripts we’ve just written should really run the filename string through something called urllib.parse.quote, a tool that escapes special characters for transmission. We will talk about this in depth in Chapter 15, so consider this a preview for now. But to make this invocation work, we need to use special sequences in the constructed URL. Here’s how to do it by hand:

C:...PP4EInternetOther> python http-getfile-urllib2.py  localhost
                               /cgi-bin/languages.py?language=C%2b%2b CxxSyntax.html
http://localhost/cgi-bin/languages.py?language=C%2b%2b CxxSyntax.html
b'<TITLE>Languages</TITLE>
'
b'<H1>Syntax</H1><HR>
'
b'<H3>C++</H3><P><PRE>
'
b' cout &lt;&lt; "Hello World" &lt;&lt; endl;     
'
b'</PRE></P><BR>
'
b'<HR>
'

The odd %2b strings in this command line are not entirely magical: the escaping required for URLs can be seen by running standard Python tools manually—this is what these scripts should do automatically to be able to handle all possible cases well; urllib.parse.unquote can undo these escapes if needed:

C:...PP4EInternetOther> python
>>> import urllib.parse
>>> urllib.parse.quote('C++')
'c%2B%2B'

Again, don’t work too hard at understanding these last few commands; we will revisit URLs and URL escapes in Chapter 15, while exploring server-side scripting in Python. I will also explain there why the C++ result came back with other oddities like <<—HTML escapes for <<, generated by the tool cgi.escape in the script on the server that produces the reply, and usually undone by HTML parsers including Python’s html.parser module we’ll meet in Chapter 19:

>>> import cgi
>>> cgi.escape('<<')
'&lt;&lt;'

Also in Chapter 15, we’ll meet urllib support for proxies, and its support for client-side cookies. We’ll discuss the related HTTPS concept in Chapter 16—HTTP transmissions over secure sockets, supported by urllib.request on the client side if SSL support is compiled into your Python. For now, it’s time to wrap up our look at the Web, and the Internet at large, from the client side of the fence.