Chapter 14. The PyMailGUI Client

“Use the Source, Luke”

The preceding chapter introduced Python’s client-side Internet protocols tool set—the standard library modules available for email, FTP, network news, HTTP, and more, from within a Python script. This chapter picks up where the last one left off and presents a complete client-side example—PyMailGUI, a Python program that sends, receives, composes, and parses Internet email messages.

Although the end result is a working program that you can actually use for your email, this chapter also has a few additional agendas worth noting before we get started:

Client-side scripting

PyMailGUI implements a full-featured desktop GUI that runs on your machine and communicates with your mail servers when necessary. As such, it is a network client program that further illustrates some of the preceding chapter’s topics, and it will help us contrast server-side solutions introduced in the next chapter.

Code reuse

Additionally, PyMailGUI ties together a number of the utility modules we’ve been writing in the book so far, and it demonstrates the power of code reuse in the process—it uses a thread module to allow mail transfers to overlap in time, a set of mail modules to process message content and route it across networks, a window protocol module to handle icons, a text editor component, and so on. Moreover, it inherits the power of tools in the Python standard library, such as the email package; message construction and parsing, for example, is nearly trivial here.

Programming in the large

And finally, this chapter serves to illustrate realistic and large-scale software development in action. Because PyMailGUI is a relatively large and complete program, it shows by example some of the code structuring techniques that prove useful once we leave the realm of the small and artificial. For instance, object-oriented programming and modular design work well here to divide the system in smaller, self-contained units.

Ultimately, though, PyMailGUI serves to illustrate just how far the combination of GUIs, networking, and Python can take us. Like all Python programs, this system is scriptable—once you’ve learned its general structure, you can easily change it to work as you like, by modifying its source code. And like all Python programs, this one is portable—you can run it on any system with Python and a network connection, without having to change its code. Such advantages become automatic when your software is coded in an open source, portable, and readable language like Python.

Source Code Modules and Size

This chapter is something of a self-study exercise. Because PyMailGUI is fairly large and mostly applies concepts we’ve already learned, we won’t go into much detail about its actual code. Instead, it is listed for you to read on your own. I encourage you to study the source and comments and to run this program live to get a feel for its operation; example save-mail files are included so you can even experiment offline.

As you study and run this program, you’ll also want to refer back to the modules we introduced earlier in the book and are reusing here, to gain a full understanding of the system. For reference, here are the major examples that will see new action in this chapter:

Example 13-21: PP4E.Internet.Email.mailtools (package)

Server sends and receives, parsing, construction (Client-side scripting chapter)

Example 10-20: PP4E.Gui.Tools.threadtools.py

Thread queue management for GUI callbacks (GUI tools chapter)

Example 10-16: PP4E.Gui.Tools.windows.py

Border configuration for top-level window (GUI tools chapter)

Example 11-4: PP4E.Gui.TextEditor.textEditor.py

Text widget used in mail view windows, and in some pop ups (GUI examples chapter)

Some of these modules in turn use additional examples we coded earlier but that are not imported by PyMailGUI itself (textEditor, for instance, uses guimaker to create its windows and toolbar). Naturally, we’ll also be coding new modules here. The following new modules are intended to be potentially useful in other programs:

popuputil.py

Various pop-up windows, written for general use

messagecache.py

A cache manager that keeps track of mail already loaded

wraplines.py

A utility for wrapping long lines of messages

mailconfig.py

User configuration parameters—server names, fonts, and so on (augmented here)

html2text.py

A rudimentary parser for extracting plain text from HTML-based emails

Finally, the following are the new major modules coded in this chapter which are specific to the PyMailGUI program. In total, PyMailGUI itself consists of the ten modules in this and the preceding lists, along with a handful of less prominent source files we’ll see in this chapter:

SharedNames.py

Program-wide globals used by multiple files

ViewWindows.py

The implementation of View, Write, Reply, and Forward message view windows

ListWindows.py

The implementation of mail-server and local-file message list windows

PyMailGuiHelp.py

User-oriented help text, opened by the main window’s bar button

PyMailGui.py

The main, top-level file of the program, run to launch the main window

Code size

As a realistically scaled system, PyMailGUI’s size is also instructive. All told, PyMailGUI is composed of 18 new files: the 10 new Python modules in the two preceding lists, plus an HTML help file, a small configuration file for PyEdit pop ups, a currently unused package initialization file, and 5 short Python files in a subdirectory used for alternate account configuration.

Together, it contains some 2,400 new lines of program source code in 16 Python files (including comments and whitespace), plus roughly 1,700 lines of help text in one Python and one HTML file (in two flavors). This 4,100 new line total doesn’t include the four other book examples listed in the previous section that are reused in PyMailGUI. The reused examples themselves constitute 2,600 additional lines of Python program code—roughly 1,000 lines each for PyEdit and mailtools alone. That brings the grand total to 6,700 lines: 4,100 new + 2,600 reused. Of this total, 5,000 lines is in program code files (2,400 of which are new here) and 1,700 lines is help text.[54]

I obtained these lines counts with PyEdit’s Info pop up, and opened the files with the code button in the PyDemos entry for this program (the Source button in PyMailGUI’s own text-based help window does similar work). For the break down by individual files, see the Excel spreadsheet file linecounts.xls in the media subdirectory of PyMailGUI; this file is also used to test attachment sends and receives, and so appears near the end of the emails in file SavedEmailversion30-4E if opened in the GUI (we’ll see how to open mail save files in a moment).

Watch for the changes section ahead for size comparisons to prior versions. Also see the SLOC counter script in Chapter 6 for an alternative way to count source lines that is less manual, but can’t include all related files in a single run and doesn’t discriminate between program code and help text.

Code Structure

As these statistics probably suggest, this is the largest example we’ll see in this book, but you shouldn’t be deterred by its size. Because it uses modular and OOP techniques, the code is simpler than you may think:

  • Python’s modules allow us to divide the system into files that have a cohesive purpose, with minimal coupling between them—code is easier to locate and understand if your modules have a logical, self-contained structure.

  • Python’s OOP support allows us to factor code for reuse and avoid redundancy—as you’ll see, code is customized, not repeated, and the classes we will code reflect the actual components of the GUI to make them easy to follow.

For instance, the implementation of mail list windows is easy to read and change, because it has been factored into a common shared superclass, which is customized by subclasses for mail-server and save-file lists; since these are mostly just variations on a theme, most of the code appears in just one place. Similarly, the code that implements the message view window is a superclass shared by write, reply, and forward composition windows; subclasses simply tailor it for writing rather than viewing.

Although we’ll deploy these techniques in the context of a mail processing program here, such techniques will apply to any nontrivial program you’ll write in Python.

To help get you started, the PyMailGuiHelp.py module listed in part near the end of this chapter includes a help text string that describes how this program is used, as well as its major features. You can also view this help live in both text and HTML form when the program is run. Experimenting with the system, while referring to its code, is probably the best and quickest way to uncover its secrets.

Why PyMailGUI?

Before we start digging into the code of this relatively large system, some context is in order. PyMailGUI is a Python program that implements a client-side email processing user interface with the standard tkinter GUI toolkit. It is presented both as an instance of Python Internet scripting and as a realistically scaled example that ties together other tools we’ve already seen, such as threads and tkinter GUIs.

Like the pymail console-based program we wrote in Chapter 13, PyMailGUI runs entirely on your local computer. Your email is fetched from and sent to remote mail servers over sockets, but the program and its user interface run locally. As a result, PyMailGUI is called an email client: like pymail, it employs Python’s client-side tools to talk to mail servers from the local machine. Unlike pymail, though, PyMailGUI is a full-featured user interface: email operations are performed with point-and-click operations and advanced mail processing such as attachments, save files, and Internationalization is supported.

Like many examples presented in this text, PyMailGUI is a practical, useful program. In fact, I run it on all kinds of machines to check my email while traveling around the world teaching Python classes. Although PyMailGUI won’t put Microsoft Outlook out of business anytime soon, it has two key pragmatic features alluded to earlier that have nothing to do with email itself—portability and scriptability, which are attractive features in their own right and merit a few additional words here:

It’s portable

PyMailGUI runs on any machine with sockets and a Python with tkinter installed. Because email is transferred with the Python libraries, any Internet connection that supports Post Office Protocol (POP) and Simple Mail Transfer Protocol (SMTP) access will do. Moreover, because the user interface is coded with tkinter, PyMailGUI should work, unchanged, on Windows, the X Window System (Unix, Linux), and the Macintosh (classic and OS X), as long as Python 3.X runs there too.

Microsoft Outlook may be a more feature-rich package, but it has to be run on Windows, and more specifically, on a single Windows machine. Because it generally deletes email from a server as it is downloaded by default and stores it on the client, you cannot run Outlook on multiple machines without spreading your email across all those machines. By contrast, PyMailGUI saves and deletes email only on request, and so it is a bit friendlier to people who check their email in an ad hoc fashion on arbitrary computers (like me).

It’s scriptable

PyMailGUI can become anything you want it to be because it is fully programmable. In fact, this is the real killer feature of PyMailGUI and of open source software like Python in general—because you have full access to PyMailGUI’s source code, you are in complete control of where it evolves from here. You have nowhere near as much control over commercial, closed products like Outlook; you generally get whatever a large company decided you need, along with whatever bugs that company might have introduced.

As a Python script, PyMailGUI is a much more flexible tool. For instance, we can change its layout, disable features, and add completely new functionality quickly by changing its Python source code. Don’t like the mail-list display? Change a few lines of code to customize it. Want to save and delete your mail automatically as it is loaded? Add some more code and buttons. Tired of seeing junk mail? Add a few lines of text processing code to the load function to filter spam. These are just a few examples. The point is that because PyMailGUI is written in a high-level, easy-to-maintain scripting language, such customizations are relatively simple, and might even be fun.

At the end of the day, because of such features, this is a realistic Python program that I actually use—both as a primary email tool and as a fallback option when my ISP’s webmail system goes down (which, as I mentioned in the prior chapter, has a way of happening at the worst possible times).[55] Python scripting is an enabling skill to have.

Running PyMailGUI

Of course, to script PyMailGUI on your own, you’ll need to be able to run it. PyMailGUI requires only a computer with some sort of Internet connectivity (a PC with a broadband or dial-up account will do) and an installed Python with the tkinter extension enabled. The Windows port of Python has this capability, so Windows PC users should be able to run this program immediately by clicking its icon.

Two notes on running the system: first, you’ll want to change the file mailconfig.py in the program’s source directory to reflect your account’s parameters, if you wish to send or receive mail from a live server; more on this as we interact with the system ahead.

Second, you can still experiment with the system without a live Internet connection—for a quick look at message view windows, use the main window’s Open buttons to open saved-mail files included in the program’s SavedMail subdirectory. The PyDemos launcher script at the top of the book’s examples directory, for example, forces PyMailGUI to open saved-mail files by passing filenames on the command line. Although you’ll probably want to connect to your email servers eventually, viewing saved mails offline is enough to sample the system’s flavor and does not require any configuration file changes.

Presentation Strategy

PyMailGUI is easily the largest program in this book, but it doesn’t introduce many library interfaces that we haven’t already seen in this book. For instance:

  • The PyMailGUI interface is built with Python’s tkinter, using the familiar listboxes, buttons, and text widgets we met earlier.

  • Python’s email package is applied to pull-out headers, text, and attachments of messages, and to compose the same.

  • Python’s POP and SMTP library modules are used to fetch, send, and delete mail over sockets.

  • Python threads, if installed in your Python interpreter, are put to work to avoid blocking during potentially overlapping, long-running mail operations.

We’re also going to reuse the PyEdit TextEditor object we wrote in Chapter 11 to view and compose messages and to pop up raw text, attachments, and source; the mailtools package’s tools we wrote in Chapter 13 to load, send, and delete mail with a server; and the mailconfig module strategy introduced in Chapter 13 to support end-user settings. PyMailGUI is largely an exercise in combining existing tools.

On the other hand, because this program is so long, we won’t exhaustively document all of its code. Instead, we’ll begin with a quick look at how PyMailGUI has evolved, and then move on to describing how it works today from an end user’s perspective—a brief demo of its windows in action. After that, we’ll list the system’s new source code modules without many additional comments, for further study.

Like most of the longer case studies in this book, this section assumes that you already know enough Python to make sense of the code on your own. If you’ve been reading this book linearly, you should also know enough about tkinter, threads, and mail interfaces to understand the library tools applied here. If you get stuck, you may wish to brush up on the presentation of these topics earlier in the book.

Major PyMailGUI Changes

Like the PyEdit text editor of Chapter 11, PyMailGUI serves as a good example of software evolution in action. Because its revisions help document this system’s functionality, and because this example is as much about software engineering as about Python itself, let’s take a quick look at its recent changes.

New in Version 2.1 and 2.0 (Third Edition)

The 2.1 version of PyMailGUI presented in the third edition of the book in early 2006 is still largely present and current in this fourth edition in 2010. Version 2.1 added a handful of enhancements to version 2.0, and version 2.0 was a complete rewrite of the 1.0 version of the second edition with a radically expanded feature set.

In fact, the second edition’s version 1.0 of this program written in early 2000 was only some 685 total program lines long (515 lines for the GUI main script and 170 lines in an email utilities module), not counting related examples reused, and just 60 lines in its help text module. Version 1.0 was really something of a prototype (if not toy), written mostly to serve as a short book example.

Although it did not yet support Internationalized mail content or other 3.0 extensions, in the third edition, PyMailGUI 2.1 became a much more realistic and feature-rich program that could be used for day-to-day email processing. It grew by nearly a factor of three to be 1,800 new program source lines (plus 1,700 program lines in related modules reused, and 500 additional lines of help text). By comparison, version 3.0 by itself grew only by some 30% to be 2,400 new program source lines as described earlier (plus 2,500 lines in related modules, and 1,700 lines of help text). Statistically minded readers: consult file linecounts-prior-version.xls in PyMailGUI’s media subdirectory for a line counts breakdown for version 2.1 by file.

In version 2.1, among PyMailGUI’s new weapons were (and still are) these:

  • MIME multipart mails with attachments may be both viewed and composed.

  • Mail transfers are no longer blocking, and may overlap in time.

  • Mail may be saved and processed offline from a local file.

  • Message parts may now be opened automatically within the GUI.

  • Multiple messages may be selected for processing in list windows.

  • Initial downloads fetch mail headers only; full mails are fetched on request.

  • View window headers and list window columns are configurable.

  • Deletions are performed immediately, not delayed until program exit.

  • Most server transfers report their progress in the GUI.

  • Long lines are intelligently wrapped in viewed and quoted text.

  • Fonts and colors in list and view windows may be configured by the user.

  • Authenticating SMTP mail-send servers that require login are supported.

  • Sent messages are saved in a local file, which may be opened in the GUI.

  • View windows intelligently pick a main text part to be displayed.

  • Already fetched mail headers and full mails are cached for speed.

  • Date strings and addresses in composed mails are formatted properly.

  • View windows now have quick-access buttons for attachments/parts (2.1).

  • Inbox out-of-sync errors are detected on deletes, and on index and mail loads (2.1).

  • Save-mail file loads and deletes are threaded, to avoid pauses for large files (2.1).

The last three items on this list were added in version 2.1; the rest were part of the 2.0 rewrite. Some of these changes were made simple by growth in standard library tools (e.g., support for attachments is straightforward with the new email package), but most represented changes in PyMailGUI itself. There were also a few genuine fixes: addresses were parsed more accurately, and date and time formats in sent mails became standards conforming, because these tasks used new tools in the email package.

New in Version 3.0 (Fourth Edition)

PyMailGUI version 3.0, presented in this fourth edition of this book, inherits all of 2.1’s upgrades described in the prior section and adds many of its own. Changes are perhaps less dramatic in version 3.0, though some address important usability issues, and they seem collectively sufficient to justify assigning this version a new major release number. Here’s a summary of what’s new this time around:

Python 3.X port

The code was updated to run under Python 3.X only; Python 2.X is no longer supported without code changes. Although some of the task of porting to Python 3.X requires only minor coding changes, other idiomatic implications are more far reaching. Python 3.X’s new Unicode focus, for example, motivated much of the Internationalization support in this version of PyMailGUI (discussed ahead).

Layout improvements

View window forms are laid out with gridding instead of packed column frames, for better appearance and platform neutrality of email headers (see Chapter 9 for more details on form layout). In addition, list window toolbars are now arranged with expanding separators for clarity; this effectively groups buttons by their roles and scope. List windows are also larger when initially opened to show more.

Text editor fix for Tk change

Both the embedded text editor and some text editor instances popped up on demand are now forcibly updated before new text is inserted, for accurate initial positioning at line 1. See PyEdit in Chapter 11 for more on this requirement; it stems from a recent change (bug?) in either Tk or tkinter.

Text editor upgrades inherited

Because the PyEdit program is reused in multiple roles here, this version of PyMailGUI also acquires all its latest fixes by proxy. Most prominently, these include a new Grep external files search dialog and support for displaying, opening, and saving Unicode text. See Chapter 11 for details.

Workaround for Python 3.1 bug on traceback prints

In the obscure-but-all-too-typical category: the common function in SharedNames.py that prints traceback details had to be changed to work correctly under Python 3.X. The traceback module’s print_tb function can no longer print a stack trace to sys.stdout if the calling program is spawned from another on Windows; it still can as before if the caller was run normally from a shell prompt. Since this function is called from the main thread on worker thread exceptions, if allowed to fail any printed error kills the GUI entirely when it is spawned from the gadget or demo launchers.

To work around this, the function now catches exceptions when print_tb is called and in response runs it again with a real file instead of sys.stdout. This appears to be a Python 3.X regression, as the same code worked correctly in both contexts in Python 2.5 and 2.6. Unlike some similar issues, it has nothing to do with printing Unicode, as stack traces are all ASCII text. Even more baffling, directly printing to stdout in the same function works fine. Hey, if it were easy, they wouldn’t call it “work.”

Bcc addresses added to envelope but header omitted

Minor change: addresses entered in the user-selectable Bcc header line of edit windows are included in the recipients list (the “envelope”), but the Bcc header line itself is no longer included in the message text sent. Otherwise, Bcc recipients might be seen by some email readers and clients (including PyMailGUI), which defeats most of this header’s purpose.

Avoiding parallel fetches of the same mail

PyMailGUI loads only mail headers initially, and fetches a mail’s full text later when needed for viewing and other operations, allowing multiple fetches to overlap in time (they are run in parallel threads). Though unlikely, it was not impossible for a user to trigger a new fetch for a mail that was currently being fetched, by selecting the mail again during its download (clicking its list entry twice quickly sufficed to kick this off). Although the message cache updates performed in the parallel fetch threads appeared to be thread safe, this behavior seemed odd and wasted time.

To do better, this version now keeps track of all fetches in progress in the main thread, to avoid this overlap potential entirely—a message fetch in progress disables all new fetch requests that it is a part of, until its fetch completes. Multiple overlapping fetches are still allowed, as long as their targets do not intersect. A set is used to detect nondisjoint fetch requests. Mails already fetched and cached are not subject to this check and can always be selected irrespective of any fetches in progress.

Multiple recipients separated in GUI by commas, not semicolons

In the prior edition, “;” was used as the recipient character, and addresses were naively split on “;” on a send. This attempted to avoid conflicts with “,” commonly used in email names. Replies dropped the name part if it contained a “;” when extracting a To address, but it was not impossible that clashes could still arise if a “;” appeared both as the separator and in manually typed address’s name.

To improve, this edition uses “,” as the recipient separator, and fully parses email address lists with the email package’s getaddresses and parseaddr tools, instead of splitting naively. Because these tools fully parse the list’s content, “,” characters embedded in email address name parts are not mistakenly takes as address separators, and so do not clash. Servers and clients generally expect “,” separators, too, so this works naturally.

With this fix, commas can appear both as address separators as well as embedded in address name components. For replies, this is handled automatically: the To field is prefilled with the From in the original message. For sends, the split happens automatically in email tools for To, Cc, and Bcc headers fields (the latter two are ignored if they contain just the initial “?” when sent).

HTML help display

Help can now be displayed in text form in a GUI window, in HTML form in a locally running web browser, or both. User settings in the mailconfig module select which form or forms to display. The HTML version is new; it uses a simple translation of the help text with added links to sections and external sites and Python’s webbrowser module, discussed earlier in this book, to open a browser. The text help display is now redundant, but it is retained because the HTML display currently lacks its ability to open source file viewers.

Thread callback queue speedup

The global thread queue dispatches GUI update callbacks much faster now—up to 100 times per second, instead of the prior 10. This is due both to checking more frequently (20 timer events per second versus 10) and to dispatching more callbacks per timer event (5 versus the prior 1). Depending on the interleaving of queue puts and gets, this speeds up initial loads for larger mailboxes by as much as an order of magnitude (factor of 10), at some potential minor cost in CPU utilization. On my Windows 7 laptop, though, PyMailGUI still shows 0% CPU utilization in Task Manager when idle.

I bumped up the queue’s speed to support an email account having 4,800 inbox messages (actually, even more by the time I got around to taking screenshots for this chapter). Without the speedup, initial header loads for this account took 8 minutes to work through the 4,800 progress callbacks (4800 ÷ 10 ÷ 60), even though most reflected messages skipped immediately by the new mail fetch limits (see the next item). With the speedup, the initial load takes just 48 seconds—perhaps not ideal still, but this initial headers load is normally performed only once per session, and this policy strikes a balance between CPU resources and responsiveness. This email account is an arguably pathological case, of course, but most initial loads benefit from the faster speed.

See Chapter 10’s threadtools for most of this change’s code, as well as additional background details. We could alternatively loop through all queued events on each timer event, but this may block the GUI indefinitely if updates are queued quickly.

Mail fetch limits

Since 2.1, PyMailGUI loads only mail headers initially, not full mail text, and only loads newly arrived headers thereafter. Depending on your Internet and server speeds, though, this may still be impractical for very large inboxes (as mentioned, one of mine currently has some 4,800 emails). To support such cases, a new mailconfig setting can be used to limit the number of headers (or full mails if TOP is unsupported) fetched on loads.

Given this setting N, PyMailGUI fetches at most N of the most recently arrived mails. Older mails outside this set are not fetched from the server, but are displayed as empty/dummy emails which are mostly inoperative (though they can generally still be fetched on demand).

This feature is inherited from mailtools code in Chapter 13; see the mailconfig module ahead for the user setting associated with it. Note that even with this fix, because the threadtools queue system used here dispatches GUI events such as progress updates only up to 100 times per second, a 4,800 mail inbox still takes 48 seconds to complete an initially header load. The queue should either run faster still, or I should delete an email once in a while!

HTML main text extraction (prototype)

PyMailGUI is still somewhat plain-text biased, despite the emergence of HTML emails in recent years. When the main (or only) text part of a mail is HTML, it is displayed in a popped-up web browser. In the prior version, though, its HTML text was still displayed in a PyEdit text editor component and was still quoted for the main text of replies and forwards.

Because most people are not HTML parsers, this edition’s version attempts to do better by extracting plain text from the part’s HTML with a simple HTML parsing step. The extracted plain text is then displayed in the mail view window and used as original text in replies and forwards.

This HTML parser is at best a prototype and is largely included to provide a first step that you can tailor for your tastes and needs, but any result it produces is better than showing raw HTML. If this fails to render the plain text well, users can still fall back on viewing in the web browser and cutting and pasting from there into replies and forwards. See also the note about open source alternatives by this parser’s source code later in this chapter; this is an already explored problem domain.

Reply copies all original recipients by default

In this version, replies are really reply-to-all by default—they automatically prefill the Cc header in the replies composition window with all the original recipients of the message. To do so, replies extract all addresses among both the original To and Cc headers, and remove duplicates as well as the new sender’s address by using set operations. The net effect is to copy all other recipients on the reply. This is in addition to replying to the sender by initializing To with the original sender’s address.

This feature is intended to reflect common usage: email circulated among groups. Since it might not always be desirable, though, it can be disabled in mailconfig so that replies initialize just To headers to reply to the original sender only. If enabled, users may need to delete the Cc prefill if not wanted; if disabled, users may need to insert Cc addresses manually instead. Both cases seem equally likely. Moreover, it’s not impossible that the original recipients include mail list names, aliases, or spurious addresses that will be either incorrect or irrelevant when the reply is sent. Like the Bcc prefill described in the next item, the reply’s Cc initialization can be edited prior to sends if needed, and disabled entirely if preferred. Also see the suggested enhancements for this feature at the end of this chapter—allowing this to be enabled or disabled in the GUI per message might be a better approach.

Other upgrades: Bcc prefills, “Re” and “Fwd” case, list size, duplicate recipients

In addition, there have been smaller enhancements throughout. Among them: Bcc headers in edit windows are now prefilled with the sender’s address as a convenience (a common role for this header); Reply and Forward now ignore case when determining if adding a “Re:” or “Fwd:” to the subject would be redundant; mail list window width and height may now be configured in mailconfig; duplicates are removed from the recipient address list in mailtools on sends to avoid sending anyone multiple copies of the same mail (e.g., if an address appears in both To and Cc); and other minor improvements which I won’t cover here. Look for “3.0” and “4E” in program comments here and in the underlying mailtools package of Chapter 13 to see other specific code changes.

Unicode (Internationalization) support

I’ve saved the most significant PyMailGUI 3.0 upgrade for last: it now supports Unicode encoding of fetched, saved, and sent mails, to the extent allowed by the Python 3.1 email package. Both text parts of messages and message headers are decoded when displayed and encoded when sent. Since this is too large a change to explain in this format, the next section elaborates.

Version 3.0 Unicode support policies

The last item on the preceding list is probably the most substantial. Per Chapter 13, a user-configurable setting in the mailconfig module is used on a session-wide basis to decode full message bytes into Unicode strings when fetched, and to encode and decode mail messages stored in text-mode save files.

More visibly, when composing, the main text and attached text parts of composed mails may be given explicit Unicode encodings in mailconfig or via user input; when viewing, message header information of parsed emails is used to determine the Unicode types of both the main mail text as well as text parts opened on demand. In addition, Internationalized mail headers (e.g., Subject, To, and From) are decoded per email, MIME, and Unicode standards when displayed according to their own content, and are automatically encoded if non-ASCII when sent.

Other Unicode policies (and fixes) of Chapter 13’s mailtools package are inherited here, too; see the prior chapter for more details. In summation, here is how all these policies play out in terms of user interfaces:

Fetched emails

When fetching mails, a session-wide user setting is used to decode full message bytes to Unicode strings, as required by Python’s current email parser; if this fails, a handful of guesses are applied. Most mail text will likely be 7 or 8 bit in nature, since original email standards required ASCII.

Composed text parts

When sending new mails, user settings are used to determine Unicode type for the main text part and any text attachment parts. If these are not set in mailconfig, the user will instead be asked for encoding names in the GUI for each text part. These are ultimately used to add character set headers, and to invoke MIME encoding. In all cases, the program falls back on UTF-8 if the user’s encoding setting or input does not work for the text being sent—for instance, if the user has chosen ASCII for the main text of a reply to or forward of a non-ASCII message or for non-ASCII attachments.

Composed headers

When sending new mails, if header lines or the name component of an email address in address-related lines do not encode properly as ASCII text, we first encode the header per email Internationalization standard. This is done per UTF-8 by default, but a mailconfig setting can request a different encoding. In email address pairs, names which cannot be encoded are dropped, and only the email address is used. It is assumed that servers will respect the encoded names in email addresses.

Displayed text parts

When viewing fetched mail, Unicode encoding names in message headers are used to decode whenever possible. The main-text part is decoded into str Unicode text per header information prior to inserting it into a PyEdit component. The content of all other text parts, as well as all binary parts, is saved in bytes form in binary-mode files, from where the part may be opened later in the GUI on demand. When such on-demand text parts are opened, they are displayed in PyEdit pop-up windows by passing to PyEdit the name of the part’s binary-mode file, as well as the part’s encoding name obtained from part message headers.

If the encoding name in a text part’s header is absent or fails to decode, encoding guesses are tried for main-text parts, and PyEdit’s separate Unicode policies are applied to text parts opened on demand (see Chapter 11—it may prompt for an encoding if not known). In addition to these rules, HTML text parts are saved in binary mode and opened in a web browser, relying on the browser’s own character set support; this may in turn use tags in the HTML itself, guesses, or user encoding selections.

Displayed headers

When viewing email, message headers are automatically decoded per email standards. This includes both full headers such as Subject, as well as the name components of all email address fields in address-related headers such as From, To, and Cc, and allows these components to be completely encoded or contain encoded substrings. Because their content gives their MIME and Unicode encodings, no user interaction is required to decode headers.

In other words, PyMailGUI now supports Internationalized message display and composition for both payloads and headers. For broadest utility, this support is distributed across multiple packages and examples. For example, Unicode decoding of full message text on fetches actually occurs deep in the imported mailtool package classes. Because of this, full (unparsed) message text is always Unicode str here. Similarly, headers are decoded for display here using tools implemented in mailtools, but headers encoding is both initiated and performed within mailtools itself on sends.

Full text decoding illustrates the types of choices required. It is done according to the fetchEncoding variable in the mailconfig module. This user setting is used across an entire PyMailGUI session to decode fetched message bytes to the required str text prior to parsing, and to save and load full message text to save files. Users may set this variable to a Unicode encoding name string which works for their mails’ encodings; “latin-1”, “utf-8”, and “ascii” are reasonable guesses for most emails, as email standards originally called for ASCII (though “latin-1” was required to decode some old mail save files generated by the prior version). If decoding with this encoding name fails, other common encodings are attempted, and as a last resort the message is still displayed if its headers can be decoded, but its body is changed to an error message; to view such unlikely mails, try running PyMailGUI again with a different encoding.

In the negatives column, nothing is done about the Unicode format for the full text of sent mails, apart from that inherited from Python’s libraries (as we learned in Chapter 13, smtplib attempts to encode per ASCII when messages are sent, which is one reason that header encoding is required). And while mail content character sets are fully supported, the GUI itself still uses English for its labels and buttons.

As explained in Chapter 13, this program’s Unicode polices are a broad but partial solution, because the email package in Python 3.1, upon which PyMailGUI utterly relies for correct operation, is in a state of flux for some use cases. An updated version which handles the Python 3.X str/bytes distinctions more accurately and completely is likely to appear in the future; watch this book’s updates page (see the Preface) for future changes and improvements to this program’s Unicode policies. Hopefully, the current email package underlying PyMailGUI 3.0 will be available for some time to come.

Although there is still room for improvement (see the list at the end of this chapter), the PyMailGUI program is able to provide a full-featured email interface, represents the most substantial example in this book, and serves to demonstrate a realistic application of the Python language and software engineering at large. As its users often attest, Python may be fun to work with, but it’s also useful for writing practical and nontrivial software. This example, more than any other in this book, testifies the same. The next section shows how.

A PyMailGUI Demo

PyMailGUI is a multiwindow interface. It consists of the following:

  • A main mail-server list window opened initially, for online mail processing

  • One or more mail save-file list windows for offline mail processing

  • One or more mail-view windows for viewing and editing messages

  • PyEdit windows for displaying raw mail text, extracted text parts, and the system’s source code

  • Nonblocking busy state pop-up dialogs

  • Assorted pop-up dialogs for opened message parts, help, and more

Operationally, PyMailGUI runs as a set of parallel threads, which may overlap in time: one for each active server transfer, and one for each active offline save file load or deletion. PyMailGUI supports mail save files, automatic saves of sent messages, configurable fonts and colors, viewing and adding attachments, main message text extraction, plain text conversion for HTML, and much more.

To make this case study easier to understand, let’s begin by seeing what PyMailGUI actually does—its user interaction and email processing functionality—before jumping into the Python code that implements that behavior. As you read this part, feel free to jump ahead to the code listings that appear after the screenshots, but be sure to read this section, too; this, along with the prior discussion of version changes, is where some subtleties of PyMailGUI’s design are explained. After this section, you are invited to study the system’s Python source code listings on your own for a better and more complete explanation than can be crafted in English.

Getting Started

OK, it’s time to take the system out for a test drive. I’m going to run the following demo on my Windows 7 laptop. It may look slightly different on different platforms (including other versions of Windows) thanks to the GUI toolkit’s native-look-and-feel support, but the basic functionality will be similar.

PyMailGUI is a Python/tkinter program, run by executing its top-level script file, PyMailGui.py. Like other Python programs, PyMailGUI can be started from the system command line, by clicking on its filename icon in a file explorer interface, or by pressing its button in the PyDemos or PyGadgets launcher bar. However it is started, the first window PyMailGUI presents is captured in Figure 14-1, shown after running a Load to fetch mail headers from my ISP’s email server. Notice the “PY” window icon: this is the handiwork of window protocol tools we wrote earlier in this book. Also notice the non-ASCII subject lines here; I’ll talk about Internationalization features later.

PyMailGUI main server list window
Figure 14-1. PyMailGUI main server list window

This is the PyMailGUI main window—every operation starts here. It consists of:

  • A help button (the bar at the top)

  • A clickable email list area for fetched emails (the middle section)

  • A button bar at the bottom for processing messages selected in the list area

In normal operation, users load their email, select an email from the list area by clicking on it, and press a button at the bottom to process it. No mail messages are shown initially; we need to first load them with the Load button—a simple password input dialog is displayed, a busy dialog appears that counts down message headers being downloaded to give a status indication, and the index is filled with messages ready to be selected.

PyMailGUI’s list windows, such as the one in Figure 14-1, display mail header details in fixed-width columns, up to a maximum size. Mails with attachments are prefixed with a “*” in mail index list windows, and fonts and colors in PyMailGUI windows like this one can be customized by the user in the mailconfig configuration file. You can’t tell in this black-and-white book, but most of the mail index lists we’ll see are configured to be Indian red, view windows are light blue, pop-up PyEdit windows are beige instead of PyEdit’s normal light cyan, and help is steel blue. You can change most of these as you like, and PyEdit pop-up window appearance can be altered in the GUI itself (see Example 8-11 for help with color definition strings, and watch for alternative configuration examples ahead).

List windows allow multiple messages to be selected at once—the action selected at the bottom of the window is applied to all selected mails. For instance, to view many mails, select them all and press View; each will be fetched (if needed) and displayed in its own view window. Use the All check button in the bottom right corner to select or deselect every mail in the list, and Ctrl-Click and Shift-Click combinations to select more than one (the standard Windows multiple selection operations apply—try it).

Before we go any further, though, let’s press the help bar at the top of the list window in Figure 14-1 to see what sort of help is available; Figure 14-2 shows the text-based help window pop up that appears—one of two help flavors available.

PyMailGUI text help pop up
Figure 14-2. PyMailGUI text help pop up

The main part of this window is simply a block of text in a scrolled-text widget, along with two buttons at the bottom. The entire help text is coded as a single triple-quoted string in the Python program. As we’ll see in a moment, a fancier option which opens an HTML rendition of this text in a spawned web browser is also available, but simple text is sufficient for many people’s tastes.[56] The Cancel button makes this nonmodal (i.e., nonblocking) window go away. More interestingly, the Source button pops up PyEdit text editor viewer windows for all the source files of PyMailGUI’s implementation; Figure 14-3 captures one of these (there are many; this is intended as a demonstration, not as a development environment). Not every program shows you its source code, but PyMailGUI follows Python’s open source motif.

PyMailGUI text help source code viewer window
Figure 14-3. PyMailGUI text help source code viewer window

New in this edition, help is also displayed in HTML form in a web browser, in addition to or instead of the scrolled text display just shown. Choosing help in text, HTML, or both is controlled by a setting in the mailconfig module. The HTML flavor uses the Python webbrowser module to pop up the HTML file in a browser on the local machine, and currently lacks the source-file opening button of the text display version (one reason you may wish to display the text viewer, too). HTML help is captured in Figure 14-4.

PyMailGUI HTML help display (new in 3.0)
Figure 14-4. PyMailGUI HTML help display (new in 3.0)

When a message is selected for viewing in the mail list window by a mouse click and View press, PyMailGUI downloads its full text (if it has not yet been downloaded in this session), and a formatted email viewer window appears, as captured in Figure 14-5 for an existing message in my account’s inbox.

PyMailGUI view window
Figure 14-5. PyMailGUI view window

View windows are built in response to actions in list windows and take the following form:

  • The top portion consists of action buttons (Part to list all message parts, Split to save and open parts using a selected directory, and Cancel to close this nonmodal window), along with a section for displaying email header lines (From, To, and so on).

  • In the middle, a row of quick-access buttons for opening message parts, including attachments, appears. When clicked, PyMailGUI opens known and generally safe parts according to their type. Media types may open in a web browser or image viewer, text parts in PyEdit, HTML in a web browser, Windows document types per the Windows Registry, and so on.

  • The bulk of this window (its entire lower portion) is just another reuse of the TextEditor class object of the PyEdit program we wrote in Chapter 11—PyMailGUI simply attaches an instance of TextEditor to every view and compose window in order to get a full-featured text editor component for free. In fact, much on the window shown in Figure 14-5 is implemented by TextEditor, not by PyMailGUI.

Reusing PyEdit’s class this way means that all of its tools are at our disposal for email text—cut and paste, find and goto, saving a copy of the text to a file, and so on. For instance, the PyEdit Save button at the bottom left of Figure 14-5 can be used to save just the main text of the mail (as we’ll see later, clicking the leftmost part button in the middle of the screen affords similar utility, and you can also save the entire message from a list window). To make this reuse even more concrete, if we pick the Tools menu of the text portion of this window and select its Info entry, we get the standard PyEdit TextEditor object’s text statistics box shown in Figure 14-6—the same pop up we’d get in the standalone PyEdit text editor and in the PyView image view programs we wrote in Chapter 11.

PyMailGUI attached PyEdit info box
Figure 14-6. PyMailGUI attached PyEdit info box

In fact, this is the third reuse of TextEditor in this book: PyEdit, PyView, and now PyMailGUI all present the same text-editing interface to users, simply because they all use the same TextEditor object and code. PyMailGUI uses it in multiple roles—it both attaches instances of this class for mail viewing and composition, and pops up instances in independent windows for some text mail parts, raw message text display, and Python source-code viewing (we saw the latter in Figure 14-3). For mail view components, PyMailGUI customizes PyEdit text fonts and colors per its own configuration module; for pop ups, user preferences in a local textConfig module are applied.

To display email, PyMailGUI inserts its text into an attached TextEditor object; to compose email, PyMailGUI presents a TextEditor and later fetches all its text to ship over the Net. Besides the obvious simplification here, this code reuse makes it easy to pick up improvements and fixes—any changes in the TextEditor object are automatically inherited by PyMailGUI, PyView, and PyEdit.

In the third edition’s version, for instance, PyMailGUI supports edit undo and redo, just because PyEdit had gained that feature. And in this fourth edition, all PyEdit importers also inherit its new Grep file search, as well as its new support for viewing and editing text of arbitrary Unicode encodings—especially useful for text parts in emails of arbitrary origin like those displayed here (see Chapter 11 for more about PyEdit’s evolution).

Loading Mail

Next, let’s go back to the PyMailGUI main server list window, and click the Load button to retrieve incoming email over the POP protocol. PyMailGUI’s load function gets account parameters from the mailconfig module listed later in this chapter, so be sure to change this file to reflect your email account parameters (i.e., server names and usernames) if you wish to use PyMailGUI to read your own email. Unless you can guess the book’s email account password, the presets in this file won’t work for you.

The account password parameter merits a few extra words. In PyMailGUI, it may come from one of two places:

Local file

If you put the name of a local file containing the password in the mailconfig module, PyMailGUI loads the password from that file as needed.

Pop up dialog

If you don’t put a password filename in mailconfig (or if PyMailGUI can’t load it from the file for whatever reason), PyMailGUI will instead ask you for your password anytime it is needed.

Figure 14-7 shows the password input prompt you get if you haven’t stored your password in a local file. Note that the password you type is not shown—a show='*' option for the Entry field used in this pop up tells tkinter to echo typed characters as stars (this option is similar in spirit to both the getpass console input module we met earlier in the prior chapter and an HTML type=password option we’ll meet in a later chapter). Once entered, the password lives only in memory on your machine; PyMailGUI itself doesn’t store it anywhere in a permanent way.

PyMailGUI password input dialog
Figure 14-7. PyMailGUI password input dialog

Also notice that the local file password option requires you to store your password unencrypted in a file on the local client computer. This is convenient (you don’t need to retype a password every time you check email), but it is not generally a good idea on a machine you share with others, of course; leave this setting blank in mailconfig if you prefer to always enter your password in a pop up.

Once PyMailGUI fetches your mail parameters and somehow obtains your password, it will next attempt to pull down just the header text of all your incoming email from your inbox on your POP email server. On subsequent loads, only newly arrived mails are loaded, if any. To support obscenely large inboxes (like one of mine), the program is also now clever enough to skip fetching headers for all but the last batch of messages, whose size you can configure in mailconfig—they show up early in the mail list with subject line “--mail skipped--”; see the 3.0 changes overview earlier for more details.

To save time, PyMailGUI fetches message header text only to populate the list window. The full text of messages is fetched later only when a message is selected for viewing or processing, and then only if the full text has not yet been fetched during this session. PyMailGUI reuses the load-mail tools in the mailtools module of Chapter 13 to fetch message header text, which in turn uses Python’s standard poplib module to retrieve your email.

Threading Model

Now that we’re downloading mails, I need to explain the juggling act that PyMailGUI performs to avoid becoming blocked and support operations that overlap in time. Ultimately, mail fetches run over sockets on relatively slow networks. While the download is in progress, the rest of the GUI remains active—you may compose and send other mails at the same time, for instance. To show its progress, the nonblocking dialog of Figure 14-8 is displayed when the mail index is being fetched.

Nonblocking progress indicator: Load
Figure 14-8. Nonblocking progress indicator: Load

In general, all server transfers display such dialogs. Figure 14-9 shows the busy dialog displayed while a full text download of five selected and uncached (not yet fetched) mails is in progress, in response to a View action. After this download finishes, all five pop up in individual view windows.

Nonblocking progress indicator: View
Figure 14-9. Nonblocking progress indicator: View

Such server transfers, and other long-running operations, are run in threads to avoid blocking the GUI. They do not disable other actions from running in parallel, as long as those actions would not conflict with a currently running thread. Multiple mail sends and disjoint fetches can overlap in time, for instance, and can run in parallel with the GUI itself—the GUI responds to moves, redraws, and resizes during the transfers. Other transfers such as mail deletes must run all by themselves and disable other transfers until they are finished; deletes update the inbox and internal caches too radically to support other parallel operations.

On systems without threads, PyMailGUI instead goes into a blocked state during such long-running operations (it essentially stubs out the thread-spawn operation to perform a simple function call). Because the GUI is essentially dead without threads, covering and uncovering the GUI during a mail load on such platforms will erase or otherwise distort its contents. Threads are enabled by default on most platforms that run Python (including Windows), so you probably won’t see such oddness on your machine.

Threading model implementation

On nearly every platform, though, long-running tasks like mail fetches and sends are spawned off as parallel threads, so that the GUI remains active during the transfer—it continues updating itself and responding to new user requests, while transfers occur in the background. While that’s true of threading in most GUIs, here are two notes regarding PyMailGUI’s specific implementation and threading model:

GUI updates: exit callback queue

As we learned earlier in this book, only the main thread that creates windows should generally update them. See Chapter 9 for more on this; tkinter doesn’t support parallel GUI changes. As a result, PyMailGUI takes care to not do anything related to the user interface within threads that load, send, or delete email. Instead, the main GUI thread continues responding to user interface events and updates, and uses a timer-based event to watch a queue for exit callbacks to be added by worker threads, using the thread tools we implemented earlier in Chapter 10 (Example 10-20). Upon receipt, the main GUI thread pulls the callback off the queue and invokes it to modify the GUI in the main thread.

Such queued exit callbacks can display a fetched email message, update the mail index list, change a progress indicator, report an error, or close an email composition window—all are scheduled by worker threads on the queue but performed in the main GUI thread. This scheme makes the callback update actions automatically thread safe: since they are run by one thread only, such GUI updates cannot overlap in time.

To make this easy, PyMailGUI stores bound method objects on the thread queue, which combine both the function to be called and the GUI object itself. Since threads all run in the same process and memory space, the GUI object queued gives access to all GUI state needed for exit updates, including displayed widget objects. PyMailGUI also runs bound methods as thread actions to allow threads to update state in general, too, subject to the next paragraph’s rules.

Other state updates: operation overlap locks

Although the queued GUI update callback scheme just described effectively restricts GUI updates to the single main thread, it’s not enough to guarantee thread safety in general. Because some spawned threads update shared object state used by other threads (e.g., mail caches), PyMailGUI also uses thread locks to prevent operations from overlapping in time if they could lead to state collisions. This includes both operations that update shared objects in memory (e.g., loading mail headers and content into caches), as well as operations that may update POP message numbers of loaded email (e.g., deletions).

Where thread overlap might be an issue, the GUI tests the state of thread locks, and pops up a message when an operation is not currently allowed. See the source code and this program’s help text for specific cases where this rule is applied. Operations such as individual sends and views that are largely independent can overlap broadly, but deletions and mail header fetches cannot.

In addition, some potentially long-running save-mail operations are threaded to avoid blocking the GUI, and this edition uses a set object to prevent fetch threads for requests that include a message whose fetch is in progress in order to avoid redundant work (see the 3.0 changes review earlier).

For more on why such things matter in general, be sure to see the discussion of threads in GUIs in Chapters 5, 9, and 10. PyMailGUI is really just a concrete realization of concepts we’ve explored earlier.

Load Server Interface

Let’s return to loading our email: because the load operation is really a socket operation, PyMailGUI automatically connects to your email server using whatever connectivity exists on the machine on which it is run. For instance, if you connect to the Net over a modem and you’re not already connected, Windows automatically pops up the standard connection dialog. On the broadband connections that most of us use today, the interface to your email server is normally automatic.

After PyMailGUI finishes loading your email, it populates the main window’s scrolled listbox with all of the messages on your email server and automatically scrolls to the most recently received message. Figure 14-10 shows what the main window looks like after selecting a message with a click and resizing—the text area in the middle grows and shrinks with the window, revealing more header columns as it grows.

PyMailGUI main window resized
Figure 14-10. PyMailGUI main window resized

Technically, the Load button fetches all your mail’s header text the first time it is pressed, but it fetches only newly arrived email headers on later presses. PyMailGUI keeps track of the last email loaded, and requests only higher email numbers on later loads. Already loaded mail is kept in memory, in a Python list, to avoid the cost of downloading it again. PyMailGUI does not delete email from your server when it is loaded; if you really want to not see an email on a later load, you must explicitly delete it.

Entries in the main list show just enough to give the user an idea of what the message contains—each entry gives the concatenation of portions of the message’s Subject, From, Date, To, and other header lines, separated by | characters and prefixed with the message’s POP number (e.g., there are 13 emails in this list). Columns are aligned by determining the maximum size needed for any entry, up to a fixed maximum, and the set of headers displayed can be configured in the mailconfig module. Use the horizontal scroll or expand the window to see additional header details such as message size and mailer.

As we’ve seen, a lot of magic happens when downloading email—the client (the machine on which PyMailGUI runs) must connect to the server (your email account machine) over a socket and transfer bytes over arbitrary Internet links. If things go wrong, PyMailGUI pops up standard error dialog boxes to let you know what happened. For example, if you type an incorrect username or password for your account (in the mailconfig module or in the password pop up), you’ll receive the message in Figure 14-11. The details displayed here are just the Python exception type and exception data. Additional details, including a stack trace, show up in standard output (the console window) on errors.

PyMailGUI invalid password error box
Figure 14-11. PyMailGUI invalid password error box

Offline Processing with Save and Open

We’ve seen how to fetch and view emails from a server, but PyMailGUI can also be used in completely offline mode. To save mails in a local file for offline processing, select the desired messages in any mail list window and press the Save action button; as usual, any number of messages may be selected for saving together as a set. A standard file-selection dialog appears, like that in Figure 14-12, and the mails are saved to the end of the chosen text file.

Save mail selection dialog
Figure 14-12. Save mail selection dialog

To view saved emails later, select the Open action at the bottom of any list window and pick your save file in the selection dialog. A new mail index list window appears for the save file and it is filled with your saved messages eventually—there may be a slight delay for large save files, because of the work involved. PyMailGUI runs file loads and deletions in threads to avoid blocking the rest of the GUI; these threads can overlap with operations on other open save-mail files, server transfer threads, and the GUI at large.

While a mail save file is being loaded in a parallel thread, its window title is set to “Loading…” as a status indication; the rest of the GUI remains active during the load (you can fetch and delete server messages, view mails in other files, write new messages, and so on). The window title changes to the loaded file’s name after the load is finished. Once filled, a message index appears in the save file’s window, like the one captured in Figure 14-13 (this window also has three mails selected for processing).

List window for mail save file, multiple selections
Figure 14-13. List window for mail save file, multiple selections

In general, there can be one server mail list window and any number of save-mail file list windows open at any time. Save-mail file list windows like that in Figure 14-13 can be opened at any time, even before fetching any mail from the server. They are identical to the server’s inbox list window, but there is no help bar, the Load action button is omitted since this is not a server view, and all other action buttons are mapped to the save file, not to the server.

For example, View opens the selected message in a normal mail view window identical to that in Figure 14-5, but the mail originates from the local file. Similarly, Delete removes the message from the save file, instead of from the server’s inbox. Deletions from save-mail files are also run in a thread, to avoid blocking the rest of the GUI—the window title changes to “Deleting…” during the delete as a status indicator. Status indicators for loads and deletions in the server inbox window use pop ups instead, because the wait is longer and there is progress to display (see Figure 14-8).

Technically, saves always append raw message text to the chosen file; the file is opened in 'a' mode to append text, which creates the file if it’s new and writes at its end. The Save and Open operations are also smart enough to remember the last directory you selected; their file dialogs begin navigation there the next time you press Save or Open.

You can also save mails from a saved file’s window—use Save and Delete to move mails from file to file. In addition, saving to a file whose window is open for viewing automatically updates that file’s list window in the GUI. This is also true for the automatically written sent-mail save file, described in the next section.

Sending Email and Attachments

Once we’ve loaded email from the server or opened a local save file, we can process our messages with the action buttons at the bottom of list windows. We can also send new emails at any time, even before a load or open. Pressing the Write button in any list window (server or file) generates a mail composition window; one has been captured in Figure 14-14.

PyMailGUI write-mail compose window
Figure 14-14. PyMailGUI write-mail compose window

This window is much like the message view window we saw in Figure 14-5, except there are no quick-access part buttons in the middle (this window is a new mail). It has fields for entering header line detail, action buttons for sending the email and managing attachment files added to it when sent, and an attached TextEditor object component for writing and editing the main text of the new email.

The PyEdit text editor component at the bottom has no File menu in this role, but it does have a Save button—useful for saving a draft of your mail’s text in a file. You can cut and paste this temporary copy into a composition window later if needed to begin composing again from scratch. PyEdit’s separate Unicode policies apply to mail text drafts saved this way (it may ask for an encoding—see Chapter 11).

For write operations, PyMailGUI automatically fills the From line and inserts a signature text line (the last two lines shown), from your mailconfig module settings. You can change these to any text you like in the GUI, but the defaults are filled in automatically from your mailconfig. When the mail is sent, an email.utils call handles date and time formatting in the mailtools module in Chapter 13.

There is also a new set of action buttons in the upper left here: Cancel closes the window (if verified), and Send delivers the mail—when you press the Send button, the text you typed into the body of this window is mailed to all the addresses you typed into the To, Cc, and Bcc lines, after removing duplicates, and using Python’s smtplib module. PyMailGUI adds the header fields you type as mail header lines in the sent message (exception: Bcc recipients receive the mail, but no header line is generated).

To send to more than one address, separate them with a comma character in header fields, and feel free to use full “name” <address> pairs for recipients. In this mail, I fill in the To header with my own email address in order to send the message to myself for illustration purposes. New in this version, PyMailGUI also prefills the Bcc header with the sender’s own address if this header is enabled in mailconfig; this prefill sends a copy to the sender (in addition to that written to the sent-mail file), but it can be deleted if unwanted.

Also in compose windows, the Attach button issues a file selection dialog for attaching a file to your message, as in Figure 14-15. The Parts button pops up a dialog displaying files already attached, like that in Figure 14-16. When your message is sent, the text in the edit portion of the window is sent as the main message text, and any attached part files are sent as attachments properly encoded according to their type.

Attachment file dialog for Attach
Figure 14-15. Attachment file dialog for Attach
Attached parts list dialog for Parts
Figure 14-16. Attached parts list dialog for Parts

As we’ve seen, smtplib ultimately sends bytes to a server over a socket. Since this can be a long-running operation, PyMailGUI delegates this operation to a spawned thread, too. While the send thread runs, a nonblocking wait window appears and the entire GUI stays alive; redraw and move events are handled in the main program thread while the send thread talks to the SMTP server, and the user may perform other tasks in parallel, including other views and sends.

You’ll get an error pop up if Python cannot send a message to any of the target recipients for any reason, and the mail composition window will pop up so that you can try again or save its text for later use. If you don’t get an error pop up, everything worked correctly, and your mail will show up in the recipients’ mailboxes on their email servers. Since I sent the earlier message to myself, it shows up in mine the next time I press the main window’s Load button, as we see in Figure 14-17.

PyMailGUI main window after loading sent mail
Figure 14-17. PyMailGUI main window after loading sent mail

If you look back to the last main window shot, you’ll notice that there is only one new email now—PyMailGUI is smart enough to download only the one new message’s header text and tack it onto the end of the loaded email list. Mail send operations automatically save sent mails in a save file that you name in your configuration module; use Open to view sent messages in offline mode and Delete to clean up the sent mail file if it grows too large (you can also save from the sent-mail file to another file to copy mails into other save files per category).

Viewing Email and Attachments

Now let’s view the mail message that was sent and received. PyMailGUI lets us view email in formatted or raw mode. First, highlight (single-click) the mail you want to see in the main window, and press the View button. After the full message text is downloaded (unless it is already cached), a formatted mail viewer window like that shown in Figure 14-18 appears. If multiple messages are selected, the View button will download all that are not already cached (i.e., that have not already been fetched) and will pop up a view window for each selected. Like all long-running operations, full message downloads are run in parallel threads to avoid blocking.

PyMailGUI view incoming mail window
Figure 14-18. PyMailGUI view incoming mail window

Python’s email module is used to parse out header lines from the raw text of the email message; their text is placed in the fields in the top right of the window. The message’s main text is fetched from its body and stuffed into a new TextEditor object for display at the window bottom. PyMailGUI uses heuristics to extract the main text of the message to display, if there is one; it does not blindly show the entire raw text of the mail. HTML-only mail is handled specially, but I’ll defer details on this until later in this demo.

Any other parts of the message attached are displayed and opened with quick-access buttons in the middle. They are also listed by the Parts pop up dialog, and they can be saved and opened all at once with Split. Figure 14-19 shows this window’s Parts list pop up, and Figure 14-20 displays this window’s Split dialog in action.

Parts dialog listing all message parts
Figure 14-19. Parts dialog listing all message parts
Split dialog selection
Figure 14-20. Split dialog selection

When the Split dialog in Figure 14-20 is submitted, all message parts are saved to the directory you select, and known parts are automatically opened. Individual parts are also automatically opened by the row of quick-access buttons labeled with the part’s filename in the middle of the view window, after being saved to a temporary directory; this is usually more convenient, especially when there are many attachments.

For instance, Figure 14-21 shows the two image parts attached to the mail we sent open on my Windows laptop, in a standard image viewer on that platform; other platforms may open this in a web browser instead. Click the image filenames’ quick-access buttons just below the message headers in Figure 14-18 to view them immediately, or run Split to open all parts at once.

PyMailGUI opening image parts in a viewer or browser
Figure 14-21. PyMailGUI opening image parts in a viewer or browser

By this point, the photo attachments displayed in Figure 14-21 have really gotten around: they have been MIME encoded, attached, and sent, and then fetched, parsed, and MIME decoded. Along the way, they have moved through multiple machines—from the client, to the SMTP server, to the POP server, and back to the client, crossing arbitrary distances along the way.

In terms of user interaction, we attached the images to the email in Figure 14-14 using the dialog in Figure 14-15 before we sent the email. To access them later, we selected the email for viewing in Figure 14-17 and clicked on their quick-access button in Figure 14-18. PyMailGUI encoded the photos in Base64 form, inserted them in the email’s text, and later extracted and decoded it to get the original photos. With Python email tools, and our own code that rides above them, this all just works as expected.

Notice how in Figures 14-18 and 14-19 the main message text counts as a mail part, too—when selected, it opens in a PyEdit window, like that captured in Figure 14-22, from which it can be processed and saved (you can also save the main mail text with the Save button in the View window itself). The main part is included, because not all mails have a text part. For messages that have only HTML for their main text part, PyMailGUI displays plain text extracted from its HTML text in its own window, and opens a web browser to view the mail with its HTML formatting. Again, I’ll say more on HTML-only mails later.

Main text part opened in PyEdit
Figure 14-22. Main text part opened in PyEdit

Besides images and plain text, PyMailGUI also opens HTML and XML attachments in a web browser and uses the Windows Registry to open well-known Windows document types. For example, .doc and .docx, .xls and .xlsx, and .pdf files usually open, respectively, in Word, Excel, and Adobe Reader. Figure 14-23 captures the response to the lp4e-pref.html quick-access part button in Figure 14-18 on my Windows laptop. If you inspect this screenshot closely, or run live for a better look, you’ll notice that the HTML attachment is displayed in both a web browser and a PyEdit window; the latter can be disabled in mailconfig, but is on by default to give an indication of the HTML’s encoding.

The quick-access buttons in the middle of the Figure 14-18 view window are a more direct way to open parts than Split—you don’t need to select a save directory, and you can open just the part you want to view. The Split button, though, allows all parts to be opened in a single step, allows you to choose where to save parts, and supports an arbitrary number of parts. Files that cannot be opened automatically because of their type can be inspected in the local save directory, after both Split and quick-access button selections (pop up dialogs name the directory to use for this).

After a fixed maximum number of parts, the quick-access row ends with a button labeled “...”, which simply runs Split to save and open additional parts when selected. Figure 14-24 captures one such message in the GUI; this message is available in SavedMail file version30-4E if you want to view it offline—a relatively complex mail, with 11 total parts of mixed types.

Attached HTML part opened in a web browser
Figure 14-23. Attached HTML part opened in a web browser
View window for a mail with many parts
Figure 14-24. View window for a mail with many parts

Like much of PyMailGUI’s behavior, the maximum number of part buttons to display in view windows can be configured in the mailconfig.py user settings module. That setting specified eight buttons in Figure 14-24. Figure 14-25 shows what the same mail looks like when the part buttons setting has been changed to a maximum of five. The setting can be higher than eight, but at some point the buttons may become unreadable (use Split instead).

View window with part buttons setting decreased
Figure 14-25. View window with part buttons setting decreased

As a sample of other attachments’ behavior, Figures 14-26 and 14-27 show what happens when the sousa.au and chapter25.pdf buttons in Figures 14-24 and 14-18 are pressed on my Windows laptop. The results vary per machine; the audio file opens in Windows Media Player, MP3 files open in iTunes instead, and some platforms may open such files directly in a web browser.

An audio part opened by PyMailGUI
Figure 14-26. An audio part opened by PyMailGUI
A PDF part opened in PyMailGUI
Figure 14-27. A PDF part opened in PyMailGUI

Besides the nicely formatted view window, PyMailGUI also lets us see the raw text of a mail message. Double-click on a message’s entry in the main window’s list to bring up a simple unformatted display of the mail’s raw text (its full text is downloaded in a thread if it hasn’t yet been fetched and cached). Part of the raw version of the mail I sent to myself in Figure 14-18 is shown in Figure 14-28; in this edition, raw text is displayed in a PyEdit pop-up window (its prior scrolled-text display is still present as an option, but PyEdit adds tools such as searching, saves, and so on).

PyMailGUI raw mail text view window (PyEdit)
Figure 14-28. PyMailGUI raw mail text view window (PyEdit)

This raw text display can be useful to see special mail headers not shown in the formatted view. For instance, the optional X-Mailer header in the raw text display identifies the program that transmitted a message; PyMailGUI adds it automatically, along with standard headers like From and To. Other headers are added as the mail is transmitted: the Received headers name machines that the message was routed through on its way to our email server, and Content-Type is added and parsed by Python’s email package in response to calls from PyMailGUI.

And really, the raw text form is all there is to an email message—it’s what is transferred from machine to machine when mail is sent. The nicely formatted display of the GUI’s view windows simply parses out and decodes components from the mail’s raw text with standard Python tools, and places them in the associated fields of the display. Notice the Base64 encoding text of the image file at the end of Figure 14-28, for example; it’s created when sent, transferred over the Internet, and decoded when fetched to recreate the image’s original bytes. Quite a feat, but largely automatic with the code and libraries invoked.

Email Replies and Forwards and Recipient Options

In addition to reading and writing email, PyMailGUI also lets users forward and reply to incoming email sent from others. These are both just composition operations, but they quote the original text and prefill header lines as appropriate. To reply to an email, select its entry in the main window’s list and click the Reply button. If I reply to the mail I just sent to myself (arguably narcissistic, but demonstrative), the mail composition window shown in Figure 14-29 appears.

PyMailGUI reply compose window
Figure 14-29. PyMailGUI reply compose window

This window is identical in format to the one we saw for the Write operation, except that PyMailGUI fills in some parts automatically. In fact, the only thing I’ve added in this window is the first line in the text editor part; the rest is filled in by PyMailGUI:

  • The From line is set to your email address in your mailconfig module.

  • The To line is initialized to the original message’s From address (we’re replying to the original sender, after all).

  • The Subject line is set to the original message’s subject line, prepended with a “Re:”, the standard follow-up subject line form (unless it already has one, in uppercase or lowercase).

  • The optional Bcc line, if enabled in the mailconfig module, is prefilled with the sender’s address, too, since it’s often used this way to retain a copy (new in this version).

  • The body of the reply is initialized with the signature line in mailconfig, along with the original message’s text. The original message text is quoted with > characters and is prepended with a few header lines extracted from the original message to give some context.

  • Not shown in this example and new in this version, too, the Cc header in replies is also prefilled with all the original recipients of the message, by extracting addresses among the original To and Cc headers, removing duplicates, and removing your address from the result. In other words, Reply really is Reply-to-All by default—it replies to the sender and copies all other recipients as a group. Since the latter isn’t always desirable, it can be disabled in mailconfig so that replies only initialize To with the original sender. You can also simply delete the Cc prefill if not wanted, but you may have to add addresses to Cc manually if this feature is disabled. We’ll see reply Cc prefills at work later.

Luckily, all of this is much easier than it may sound. Python’s standard email module extracts all of the original message’s header lines, and a single string replace method call does the work of adding the > quotes to the original message body. I simply type what I wish to say in reply (the initial paragraph in the mail’s text area) and press the Send button to route the reply message to the mailbox on my mail server again. Physically sending the reply works the same as sending a brand-new message—the mail is routed to your SMTP server in a spawned send-mail thread, and the send-mail wait pop up appears while the thread runs.

Forwarding a message is similar to replying: select the message in the main window, press the Fwd button, and fill in the fields and text area of the popped-up composition window. Figure 14-30 shows the window created to forward the mail we originally wrote and received after a bit of editing.

PyMailGUI forward compose window
Figure 14-30. PyMailGUI forward compose window

Much like replies, forwards fill From with the sender’s address in mailconfig; the original text is automatically quoted in the message body again; Bcc is preset initially the same as From; and the subject line is preset to the original message’s subject prepended with the string “Fwd:”. All these lines can be changed manually before sending if you wish to tailor. I always have to fill in the To line manually, though, because a forward is not a direct reply—it doesn’t necessarily go back to the original sender. Further, the Cc prefill of original recipients done by Reply isn’t performed for forwards, because they are not a continuation of group discussions.

Notice that I’m forwarding this message to three different addresses (two in the To, and one manually entered in the Bcc). I’m also using full “name <address>” formats for email addresses. Multiple recipient addresses are separated with a comma (,) in the To, Cc, and Bcc header fields, and PyMailGUI is happy to use the full address form anywhere you type an address, including your own in mailconfig. As demonstrated by the first To recipient in Figure 14-30, commas in address names don’t clash with those that separate recipients, because address lines are parsed fully in this version. When we’re ready, the Send button in this window fires the forwarded message off to all addresses listed in these headers, after removing any duplicates to avoid sending the same recipient the same mail more than once.

I’ve now written a new message, replied to it, and forwarded it. The reply and forward were sent to my email address, too; if we press the main window’s Load button again, the reply and forward messages should show up in the main window’s list. In Figure 14-31, they appear as messages 15 and 16 (the order they appear in may depend on timing issues at your server, and I’ve stretched this horizontally in the GUI to try to reveal the To header of the last of these).

PyMailGUI mail list after sends and load
Figure 14-31. PyMailGUI mail list after sends and load

Keep in mind that PyMailGUI runs on the local computer, but the messages you see in the main window’s list actually live in a mailbox on your email server machine. Every time we press Load, PyMailGUI downloads but does not delete newly arrived emails’ headers from the server to your computer. The three messages we just wrote (14 through 16) will also appear in any other email program you use on your account (e.g., in Outlook or in a webmail interface). PyMailGUI does not automatically delete messages as they are downloaded, but simply stores them in your computer’s memory for processing. If we now select message 16 and press View, we see the forward message we sent, as in Figure 14-32.

This message went from my machine to a remote email server and was downloaded from there into a Python list from which it is displayed. In fact, it went to three different email accounts I have (the other two appear later in this demo—see Figure 14-45). The third recipient doesn’t appear in Figure 14-32 here because it was a Bcc blind-copy—it receives the message, but no header line is added to the mail itself.

PyMailGUI view forwarded mail
Figure 14-32. PyMailGUI view forwarded mail

Figure 14-33 shows what the forward message’s raw text looks like; again, double-click on a main window’s entry to display this form. The formatted display in Figure 14-32 simply extracts bits and pieces out of the text shown in the raw display form.

PyMailGUI view forwarded mail, raw
Figure 14-33. PyMailGUI view forwarded mail, raw

One last pointer on replies and forwards: as mentioned, replies in this version reply to all original recipients, assuming that more than one means that this is a continuation of a group discussion. To illustrate, Figure 14-34 shows an original message on top, a forward of it on the lower left, and a reply to it on the lower right. The Cc header in the reply has been automatically prefilled with all the original recipients, less any duplicates and the new sender’s address; the Bcc (enabled here) has also been prefilled with the sender in both. These are just initial settings which can be edited and removed prior to sends. Moreover, the Cc prefill for replies can be disabled entirely in the configuration file. Without it, though, you may have to manually cut-and-paste to insert addresses in group mail scenarios. Open this version’s mail save file to view this mail’s behavior live, and see the suggested enhancements later for more ideas.

Reply-to-all Cc prefills
Figure 14-34. Reply-to-all Cc prefills

Deleting Email

So far, we’ve covered every action button on list windows except for Delete and the All checkbox. The All checkbox simply toggles from selecting all messages at once or deselecting all (View, Delete, Reply, Fwd, and Save action buttons apply to all currently selected messages). PyMailGUI also lets us delete messages from the server permanently, so that we won’t see them the next time we access our inbox.

Delete operations are kicked off the same way as Views and Saves; just press the Delete button instead. In typical operation, I eventually delete email I’m not interested in, and save and delete emails that are important. We met Save earlier in this demo.

Like View, Save, and other operations, Delete can be applied to one or more messages. Deletes happen immediately, and like all server transfers, they are run in a nonblocking thread but are performed only if you verify the operation in a pop up, such as the one shown in Figure 14-35. During the delete, a progress dialog like those in Figures 14-8 and 14-9 provide status.

PyMailGUI delete verification on quit
Figure 14-35. PyMailGUI delete verification on quit

By design, no mail is ever removed automatically: you will see the same messages the next time PyMailGUI runs. It deletes mail from your server only when you ask it to, and then only if verified in the last pop up shown (this is your last chance to prevent permanent mail removal). After the deletions are performed, the mail index is updated, and the GUI session continues.

Deletions disable mail loads and other deletes while running and cannot be run in parallel with loads or other deletes already in progress because they may change POP message numbers and thus modify the mail index list (they may also modify the email cache). Messages may still be composed during a deletion, however, and offline save files may be processed.

POP Message Numbers and Synchronization

By now, we’ve seen all the basic functionality of PyMailGUI—enough to get you started sending and receiving simple but typical text-based messages. In the rest of this demo, we’re going to turn our attention to some of the deeper concepts in this system, including inbox synchronization, HTML mails, Internationalization, and multiple account configuration. Since the first of these is related to the preceding section’s tour of mail deletions, let’s begin here.

Though they might seem simple from an end-user perspective, it turns out that deletions are complicated by POP’s message-numbering scheme. We learned about the potential for synchronization errors between the server’s inbox and the fetched email list in Chapter 13, when studying the mailtools package PyMailGUI uses (near Example 13-24). In brief, POP assigns each message a relative sequential number, starting from one, and these numbers are passed to the server to fetch and delete messages. The server’s inbox is normally locked while a connection is held so that a series of deletions can be run as an atomic operation; no other inbox changes occur until the connection is closed.

However, message number changes also have some implications for the GUI itself. It’s never an issue if new mail arrives while we’re displaying the result of a prior download—the new mail is assigned higher numbers, beyond what is displayed on the client. But if we delete a message in the middle of a mailbox after the index has been loaded from the mail server, the numbers of all messages after the one deleted change (they are decremented by one). As a result, some message numbers might no longer be valid if deletions are made while viewing previously loaded email.

To work around this, PyMailGUI adjusts all the displayed numbers after a Delete by simply removing the entries for deleted mails from its index list and mail cache. However, this adjustment is not enough to keep the GUI in sync with the server’s inbox if the inbox is modified at a position other than after the end, by deletions in another email client (even in another PyMailGUI session), or by deletions performed by the mail server itself (e.g., messages determined to be undeliverable and automatically removed from the inbox). Such modifications outside PyMailGUI’s scope are uncommon, but not impossible.

To handle these cases, PyMailGUI uses the safe deletion and synchronization tests in mailtools. That module uses mail header matching to detect mail list and server inbox synchronization errors. For instance, if another email client has deleted a message prior to the one to be deleted by PyMailGUI, mailtools catches the problem and cancels the deletion, and an error pop up like the one in Figure 14-36 is displayed.

Safe deletion test detection of inbox difference
Figure 14-36. Safe deletion test detection of inbox difference

Similarly, both index list loads and individual message fetches run a synchronization test in mailtools, as well. Figure 14-37 captures the error generated on a fetch if a message has been deleted in another client since we last loaded the server index window. The same error is issued when this occurs during a load operation, but the first line reads “Load failed.”

Synchronization error after delete in another client
Figure 14-37. Synchronization error after delete in another client

In both synchronization error cases, the mail list is automatically reloaded with the new inbox content by PyMailGUI immediately after the error pop up is dismissed. This scheme ensures that PyMailGUI won’t delete or display the wrong message, in the rare case that the server’s inbox is changed without its knowledge. See mailtools in Chapter 13 for more on synchronization tests; these errors are detected and raised in mailtools, but triggered by calls made in the mail cache manager here.

Handling HTML Content in Email

Up to this point, we’ve seen PyMailGUI’s basic operation in the context of plain-text emails. We’ve also seen it handling HTML part attachments, but not the main text of HTML messages. Today, of course, HTML is common for mail content too. Because the PyEdit mail display deployed by PyMailGUI uses a tkinter Text widget oriented toward plain text, HTML content is handled specially:

  • For text/HTML alternative mails, PyMailGUI displays the plain text part in its view window and includes a button for opening the HTML rendition in a web browser on demand.

  • For HTML-only mails, the main text area shows plain text extracted from the HTML by a simple parser (not the raw HTML), and the HTML is also displayed in a web browser automatically.

In all cases, the web browser’s display of International character set content in the HTML depends upon encoding information in tags in the HTML, guesses, or user feedback. Well-formed HTML parts already have “<meta>” tags in their “<head>” sections which give the HTML’s encoding, but they may be absent or incorrect. We’ll learn more about Internationalization support in the next section.

Figure 14-38 gives the scene when a text/HTML alternative mail is viewed, and Figure 14-39 shows what happens when an HTML-only email is viewed. The web browser in Figure 14-38 was opened by clicking the HTML part’s button; this is no different than the HTML attachment example we saw earlier.

For HTML-only messages, though, behavior is new here: the view window on the left in Figure 14-39 reflects the results of extracting plain text from the HTML shown in the popped-up web browser behind it. The HTML parser used for this is something of a first-cut prototype, but any result it can give is an improvement on displaying raw HTML in the view window for HTML-only mails. For simpler HTML mails of the sort sent by individuals instead of those sent by mass-mailing companies (like those shown here), the results are generally good in tests run to date, though time will tell how this prototype parser fares in today’s unforgiving HTML jungle of nonstandard and nonconforming code—improve as desired.

Viewing text/HTML alternative mails
Figure 14-38. Viewing text/HTML alternative mails
Viewing HTML-only mails
Figure 14-39. Viewing HTML-only mails

One caveat here: PyMailGUI can today display HTML in a web browser and extract plain text from it, but it cannot display HTML directly in its own window and has no support for editing it specially. These are enhancements that will have to await further attention by other programmers who may find them useful.

Mail Content Internationalization Support

Our next advanced feature is something of an inevitable consequence of the Internet’s success. As described earlier when summarizing version 3.0 changes, PyMailGUI fully supports International character sets in mail content—both text part payloads and email headers are decoded for display and encoded when sent, according to email, MIME, and Unicode standards. This may be the most significant change in this version of the program. Regrettably, capturing this in screenshots is a bit of a challenge and you can get a better feel for how this pans out by running an Open on the following included mail save file, viewing its messages in formatted and raw modes, starting replies and forwards for them, and so on:

C:...PP4EInternetEmailPyMailGuiSavedMaili18n-4E

To sample the flavor of this support here, Figure 14-40 shows the scene when this file is opened, shown for variety here with one of the alternate account configurations described the next section. This figure’s index list window and mail view windows capture Russian and Chinese language messages sent to my email account (these were unsolicited email of no particular significance, but suffice as reasonable test cases). Notice how both message headers and text payload parts are decoded for display in both the mail list window and the mail view windows.

Internationalization support, headers and body decoded for display
Figure 14-40. Internationalization support, headers and body decoded for display

Figure 14-41 shows portions of the raw text of the two fetched messages, obtained by double-clicking their list entries (you can open these mails from the save file listed earlier if you have trouble seeing their details as shown in this book). Notice how the body text is encoded per both MIME and Unicode conventions—the headers at the top and text at the bottom of these windows show the actual Base64 and quoted-printable strings that must be decoded to achieve the nicely displayed output in Figure 14-40.

For the text parts, the information in the part’s header describes its content’s encoding schemes. For instance, charset="gb2312" in the content type header identifies a Chinese Unicode character set, and the transfer encoding header gives the part’s MIME encoding type (e.g. base64).

The headers are encoded per i18n standards here as well—their content self-describes their MIME and Unicode encodings. For example, the header prefix =?koi8-r?B means Russian text, Base64 encoded. PyMailGUI is clever enough to decode both full headers and the name fields of addresses for display, whether they are completely encoded (as shown here) or contain just encoded substrings (as shown by other saved mails in the version30-4E file in this example’s SavedMail directory).

Raw text of fetched Internationalized mails, headers and body encoded
Figure 14-41. Raw text of fetched Internationalized mails, headers and body encoded

As additional context, Figure 14-42 shows how these messages’ main parts appear when opened via their part buttons. Their content is saved as raw post-MIME bytes in binary mode, but the PyEdit pop ups decode according to passed-in encoding names obtained from the raw message headers. As we learned in Chapters 9 and 11, the underlying tkinter toolkit generally renders decoded str better than raw bytes.

Main text parts of Internationalized mails, decoded in PyEdit pop-ups
Figure 14-42. Main text parts of Internationalized mails, decoded in PyEdit pop-ups

So far, we’ve displayed Internationalized emails, but PyMailGUI allows us to send them as well, and handles any encoding tasks implied by the text content. Figure 14-43 shows the result of running replies and forwards to the Russian language email, with the To address changed to protect the innocent. Headers in the view window were decoded for display, encoded when sent, and decoded back again for display; text parts in the mail body were similarly decoded, encoded, and re-decoded along the way and headers are also decoded within the “>” quoted original text inserted at the end of the message.

Result of reply and forward with International character sets, re-decoded
Figure 14-43. Result of reply and forward with International character sets, re-decoded

And finally, Figure 14-44 shows a portion of the raw text of the Russian language reply message that appears in the lower right of the formatted view of Figure 14-43. Again, double-click to see these details live. Notice how both headers and body text have been encoded per email and MIME standards.

As configured, the body text is always MIME encoded to UTF-8 when sent if it fails to encode as ASCII, the default setting in the mailconfig module. Other defaults can be used if desired and will be encoded appropriately for sends; in fact, text that won’t work in the full text of email is MIME encoded the same way as binary parts such as images.

This is also true for non-Internationalized character sets—the text part of a message written in English with any non-ASCII quotes, for example, will be UTF-8 and Base64 encoded in the same way as the message in Figure 14-44, and assume that the recipient’s email reader will decode (any reasonable modern email client will). This allows non-ASCII text to be embedded in the full email text sent.

Message headers are similarly encoded per UTF-8 if they are non-ASCII when sent, so they will work in the full email text. In fact, if you study this closely you’ll find that the Subject here was originally encoded per a Russian Unicode scheme but is UTF-8 now—its new representation yields the same characters (code points) when decoded for display.

Raw text of sent Russian reply, headers and body re-encoded
Figure 14-44. Raw text of sent Russian reply, headers and body re-encoded

In short, although the GUI itself is still in English (its labels and the like), the content of emails displayed and sent support arbitrary character sets. Decoding for display is done per content where possible, using message headers for text payloads and content for headers. Encoding for sends is performed according to user settings and policies, using user settings or inputs, or a UTF-8 default. Required MIME and email header encodings are implemented in a largely automatic fashion by the underlying email package.

Not shown here are the pop-up dialogs that may be issued to prompt for text part encoding preferences on sends if so configured in mailconfig, and PyEdit’s similar prompts under certain user configurations. Some of these user configurations are meant for illustration and generality; the presets seem to work well for most scenarios I’ve run into, but your International mileage may vary. For more details, experiment with the file’s messages on your own and see the system’s source code.

Alternative Configurations and Accounts

So far, we’ve mostly seen PyMailGUI being run on an email account I created for this book’s examples, but it’s easy to tailor its mailconfig module for different accounts, as well as different visual effects. For example, Figure 14-45 captures the scene with PyMailGUI being run on three different email accounts I use for books and training. All three instances are run in independent processes here. Each main list window is displaying a different email account’s messages, and each customizes appearance or behavior in some fashion. The message view window at the top, opened from the server list window in the lower left also applies custom color and displayed headers schemes.

Alternative accounts and configurations
Figure 14-45. Alternative accounts and configurations

You can always change mailconfigs in-place for a specific account if you use just one, but we’ll later see how the altconfigs subdirectory applies one possible solution to allow configuring for multiple accounts such as these, completely external to the original source code. The altconfigs option renders the windows in Figure 14-45, and suffices as my launching interface; see its code ahead.

Multiple Windows and Status Messages

Finally, PyMailGUI is really meant to be a multiple-window interface—a detail that most of the earlier screenshots haven’t really done justice to. For example, Figure 14-46 shows PyMailGUI with the main server list window, two save-file list windows, two message view windows, and help. All these windows are nonmodal; that is, they are all active and independent, and do not block other windows from being selected, even though they are all running a single PyMailGUI process.

PyMailGUI multiple windows and text editors
Figure 14-46. PyMailGUI multiple windows and text editors

In general, you can have any number of mail view or compose windows up at once, and cut and paste between them. This matters, because PyMailGUI must take care to make sure that each window has a distinct text-editor object. If the text-editor object were a global, or used globals internally, you’d likely see the same text in each window (and the Send operations might wind up sending text from another window). To avoid this, PyMailGUI creates and attaches a new TextEditor instance to each view and compose window it creates, and associates the new editor with the Send button’s callback handler to make sure we get the right text. This is just the usual OOP state retention, but it acquires a tangible benefit here.

Though not GUI-related, PyMailGUI also prints a variety of status messages as it runs, but you see them only if you launch the program from the system command-line console window (e.g., a DOS box on Windows or an xterm on Linux) or by double-clicking on its filename icon (its main script is a .py, not a .pyw). On Windows, you won’t see these messages when PyMailGUI is started from another program, such as the PyDemos or PyGadgets launcher bar GUIs. These status messages print server information; show mail loading status; and trace the load, store, and delete threads that are spawned along the way. If you want PyMailGUI to be more verbose, launch it from a command line and watch:

C:...PP4EInternetEmailPyMailGui> PyMailGui.py
user: [email protected]
loading headers
Connecting...
b'+OK <[email protected]>'
load headers exit
synch check
Connecting...
b'+OK <[email protected]>'
Same headers text
loading headers
Connecting...
b'+OK <[email protected]>'
load headers exit
synch check
Connecting...
b'+OK <[email protected]>'
Same headers text
load 16
Connecting...
b'+OK <[email protected]>'
Sending to...['[email protected]', '[email protected]']
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
From: [email protected]
To: [email protected]
Subject: Already got one...
Date: Fri, 04 Jun 2010 06:30:26 −0000
X-Mailer: PyMailGUI 3.0 (Python)

> -----Origin
Send exit

You can also double-click on the PyMailGui.py filename in your file explorer GUI and monitor the popped-up DOS console box on Windows. Console messages are mostly intended for debugging, but they can be used to help understand the system’s operation as well.

For more details on using PyMailGUI, see its help display (press the help bar at the top of its main server list windows), or read the help string in the module PyMailGuiHelp.py, described in the next section.

PyMailGUI Implementation

Last but not least, we get to the code. PyMailGUI consists of the new modules listed near the start of this chapter, along with a handful of peripheral files described there. The source code for these modules is listed in this section. Before we get started, here are two quick reminders to help you study:

Code reuse

Besides the code here, PyMailGUI also gets a lot of mileage out of reusing modules we wrote earlier and won’t repeat here: mailtools for mail loads, composition, parsing, and delete operations; threadtools for managing server and local file access threads; the GUI section’s TextEditor for displaying and editing mail message text; and so on. See the example numbers list earlier in this chapter.

In addition, standard Python modules and packages such as poplib, smtplib, and email hide most of the details of pushing bytes around the Net and extracting and building message components. As usual, the tkinter standard library module also implements GUI components in a portable fashion.

Code structure

As mentioned earlier, PyMailGUI applies code factoring and OOP to leverage code reuse. For instance, list view windows are implemented as a common superclass that codes most actions, along with one subclass for the server inbox list window and one for local save-file list windows. The subclasses customize the common superclass for their specific mail media.

This design reflects the operation of the GUI itself—server list windows load mail over POP, and save-file list windows load from local files. The basic operation of list window layout and actions, though, is similar for both and is shared in the common superclass to avoid redundancy and simplify the code. Message view windows are similarly factored: a common view window superclass is reused and customized for write, reply, and forward view windows.

To make the code easier to follow, it is divided into two main modules that reflect the structure of the GUI—one for the implementation of list window actions and one for view window actions. If you are looking for the implementation of a button that appears in a mail view or edit window, for instance, see the view window module and search for a method whose name begins with the word on—the convention used for callback handler methods. A specific button’s text can also be located in name/callback tables used to build the windows. Actions initiated on list windows are coded in the list window module instead.

In addition, the message cache is split off into an object and module of its own, and potentially reusable tools are coded in importable modules (e.g., line wrapping and utility pop ups). PyMailGUI also includes a main module that defines startup window classes, a simple HTML to plain-text parser, a module that contains the help text as a string, the mailconfig user settings module (a version specific to PyMailGUI is used here), and a small handful of related files.

The following subsections present each of PyMailGUI’s source code files for you to study. As you read, refer back to the demo earlier in this chapter and run the program live to map its behavior back to its code.

One accounting note up-front: the only one of PyMailGUI’s 18 new source files not listed in this section is its __init__.py package initialization file. This file is mostly empty except for a comment string and is unused in the system today. It exists only for future expansion, in case PyMailGUI is ever used as a package in the future—some of its modules may be useful in other programs. As is, though, same-directory internal imports here are not package-relative, so they assume this system is either run as a top-level program (to import from “.”) or is listed on sys.path directly (to use absolute imports). In Python 3.X, a package’s directory is not included on sys.path automatically, so future use as a package would require changes to internal imports (e.g., moving the main script up one level and using from . import module throughout). See resources such as the book Learning Python for more on packages and package imports.

PyMailGUI: The Main Module

Example 14-1 defines the file run to start PyMailGUI. It implements top-level list windows in the system—combinations of PyMailGUI’s application logic and the window protocol superclasses we wrote earlier in the text. The latter of these define window titles, icons, and close behavior.

The main internal, nonuser documentation is also in this module, as well as command-line logic—the program accepts the names of one or more save-mail files on the command line, and automatically opens them when the GUI starts up. This is used by the PyDemos launcher of Chapter 10, for example.

Example 14-1. PP4EInternetEmailPyMailGuiPyMailGui.py
"""
##################################################################################
PyMailGui 3.0 - A Python/tkinter email client.
A client-side tkinter-based GUI interface for sending and receiving email.

See the help string in PyMailGuiHelp.py for usage details, and a list of
enhancements in this version.

Version 2.0 was a major, complete rewrite.  The changes from 2.0 (July '05)
to 2.1 (Jan '06) were quick-access part buttons on View windows, threaded
loads and deletes of local save-mail files, and checks for and recovery from
message numbers out-of-synch with mail server inbox on deletes, index loads,
and message loads.

Version 3.0 (4E) is a port to Python 3.X; uses grids instead of packed column
frames for better form layout of headers in view windows; runs update() after
inserting into a new text editor for accurate line positioning (see PyEdit
loadFirst changes in Chapter 11); provides an HTML-based version of its help
text; extracts plain-text from HTML main/only parts for display and quoting;
supports separators in toolbars; addresses both message content and header
Unicode/I18N encodings for fetched, sent, and saved mails (see Ch13 and Ch14);
and much more (see Ch14 for the full rundown on 3.0 upgrades); fetched message
decoding happens deep in the mailtools package, on mail cache load operations
here; mailtools also fixes a few email package bugs (see Ch13);

This file implements the top-level windows and interface.  PyMailGui uses a
number of modules that know nothing about this GUI, but perform related tasks,
some of which are developed in other sections of the book.  The mailconfig
module is expanded for this program.

==Modules defined elsewhere and reused here:==

mailtools (package)
    client-side scripting chapter
    server sends and receives, parsing, construction     (Example 13-21+)
threadtools.py
    GUI tools chapter
    thread queue manangement for GUI callbacks           (Example 10-20)
windows.py
    GUI tools chapter
    border configuration for top-level windows           (Example 10-16)
textEditor.py
    GUI programs chapter
    text widget used in mail view windows, some pop ups  (Example 11-4)

==Generally useful modules defined here:==

popuputil.py
    help and busy windows, for general use
messagecache.py
    a cache that keeps track of mail already loaded
wraplines.py
    utility for wrapping long lines of messages
html2text.py
    rudimentary HTML parser for extracting plain text
mailconfig.py
    user configuration parameters: server names, fonts, etc.

==Program-specific modules defined here:==

SharedNames.py
    objects shared between window classes and main file
ViewWindows.py
    implementation of view, write, reply, forward windows
ListWindows.py
    implementation of mail-server and local-file list windows
PyMailGuiHelp.py (see also PyMailGuiHelp.html)
    user-visible help text, opened by main window bar
PyMailGui.py
    main, top-level file (run this), with main window types
##################################################################################
"""

import mailconfig, sys
from SharedNames import appname, windows
from ListWindows import PyMailServer, PyMailFile


###############################################################################
# top-level window classes
#
# View, Write, Reply, Forward, Help, BusyBox all inherit from PopupWindow
# directly: only usage;  askpassword calls PopupWindow and attaches;  the
# order matters here!--PyMail classes redef some method defaults in the
# Window classes, like destroy and okayToExit: must be leftmost;  to use
# PyMailFileWindow standalone, imitate logic in PyMailCommon.onOpenMailFile;
###############################################################################

# uses icon file in cwd or default in tools dir
srvrname = mailconfig.popservername or 'Server'

class PyMailServerWindow(PyMailServer, windows.MainWindow):
    "a Tk, with extra protocol and mixed-in methods"
    def __init__(self):
        windows.MainWindow.__init__(self, appname, srvrname)
        PyMailServer.__init__(self)

class PyMailServerPopup(PyMailServer, windows.PopupWindow):
    "a Toplevel, with extra protocol and mixed-in methods"
    def __init__(self):
        windows.PopupWindow.__init__(self, appname, srvrnane)
        PyMailServer.__init__(self)

class PyMailServerComponent(PyMailServer, windows.ComponentWindow):
    "a Frame, with extra protocol and mixed-in methods"
    def __init__(self):
        windows.ComponentWindow.__init__(self)
        PyMailServer.__init__(self)

class PyMailFileWindow(PyMailFile, windows.PopupWindow):
    "a Toplevel, with extra protocol and mixed-in methods"
    def __init__(self, filename):
        windows.PopupWindow.__init__(self, appname, filename)
        PyMailFile.__init__(self, filename)


###############################################################################
# when run as a top-level program: create main mail-server list window
###############################################################################

if __name__ == '__main__':
    rootwin = PyMailServerWindow()              # open server window
    if len(sys.argv) > 1:                       # 3.0: fix to add len()
        for savename in sys.argv[1:]:
            rootwin.onOpenMailFile(savename)    # open save file windows (demo)
        rootwin.lift()                          # save files loaded in threads
    rootwin.mainloop()

SharedNames: Program-Wide Globals

The module in Example 14-2 implements a shared, system-wide namespace that collects resources used in most modules in the system and defines global objects that span files. This allows other files to avoid redundantly repeating common imports and encapsulates the locations of package imports; it is the only file that must be updated if paths change in the future. Using globals can make programs more difficult to understand in general (the source of some names is not as clear), but it is reasonable if all such names are collected in a single expected module such as this one, because there is only one place to search for unknown names.

Example 14-2. PP4EInternetEmailPyMailGuiSharedNames.py
"""
##############################################################################
objects shared by all window classes and main file: program-wide globals
##############################################################################
"""

# used in all window, icon titles
appname  = 'PyMailGUI 3.0'

# used for list save, open, delete; also for sent messages file
saveMailSeparator = 'PyMailGUI' + ('-'*60) + 'PyMailGUI
'

# currently viewed mail save files; also for sent-mail file
openSaveFiles = {}                     # 1 window per file,{name:win}

# standard library services
import sys, os, email.utils, email.message, webbrowser, mimetypes
from tkinter import *
from tkinter.simpledialog import askstring
from tkinter.filedialog   import SaveAs, Open, Directory
from tkinter.messagebox   import showinfo, showerror, askyesno

# reuse book examples
from PP4E.Gui.Tools      import windows      # window border, exit protocols
from PP4E.Gui.Tools      import threadtools  # thread callback queue checker
from PP4E.Internet.Email import mailtools    # load,send,parse,build utilities
from PP4E.Gui.TextEditor import textEditor   # component and pop up

# modules defined here
import mailconfig                            # user params: servers, fonts, etc.
import popuputil                             # help, busy, passwd pop-up windows
import wraplines                             # wrap long message lines
import messagecache                          # remember already loaded mail
import html2text                             # simplistic html->plaintext extract
import PyMailGuiHelp                         # user documentation

def printStack(exc_info):
    """
    debugging: show exception and stack traceback on stdout;
    3.0: change to print stack trace to a real log file if print
    to sys.stdout fails: it does when launched from another program
    on Windows;  without this workaround, PMailGUI aborts and exits
    altogether, as this is called from the main thread on spawned
    thread failures;  likely a Python 3.1 bug: it doesn't occur in
    2.5 or 2.6, and the traceback object works fine if print to file;
    oddly, the print() calls here work (but go nowhere) if spawned;
    """
    print(exc_info[0])
    print(exc_info[1])
    import traceback
    try:
        traceback.print_tb(exc_info[2], file=sys.stdout)   # ok unless spawned!
    except:
        log = open('_pymailerrlog.txt', 'a')               # use a real file
        log.write('-'*80)                                  # else gui may exit
        traceback.print_tb(exc_info[2], file=log)          # in 3.X, not 2.5/6

# thread busy counters for threads run by this GUI
# sendingBusy shared by all send windows, used by main window quit

loadingHdrsBusy = threadtools.ThreadCounter()   # only 1
deletingBusy    = threadtools.ThreadCounter()   # only 1
loadingMsgsBusy = threadtools.ThreadCounter()   # poss many
sendingBusy     = threadtools.ThreadCounter()   # poss many

ListWindows: Message List Windows

The code in Example 14-3 implements mail index list windows—for the server inbox window and for one or more local save-mail file windows. These two types of windows look and behave largely the same, and in fact share most of their code in common in a superclass. The window subclasses mostly just customize the superclass to map mail Load and Delete calls to the server or a local file.

List windows are created on program startup (the initial server window, and possible save-file windows for command-line options), as well as in response to Open button actions in existing list windows (for opening new save-file list windows). See the Open button’s callback in this example for initiation code.

Notice that the basic mail processing operations in the mailtools package from Chapter 13 are mixed into PyMailGUI in a variety of ways. The list window classes in Example 14-3 inherit from the mailtools mail parser class, but the server list window class embeds an instance of the message cache object, which in turn inherits from the mailtools mail fetcher. The mailtools mail sender class is inherited by message view write windows, not list windows; view windows also inherit from the mail parser.

This is a fairly large file; in principle it could be split into three files, one for each class, but these classes are so closely related that it is handy to have their code in a single file for edits. Really, this is one class, with two minor extensions.

Example 14-3. PP4EInternetEmailPyMailGuiListWindows.py
"""
###############################################################################
Implementation of mail-server and save-file message list main windows:
one class per kind.  Code is factored here for reuse: server and file
list windows are customized versions of the PyMailCommon list window class;
the server window maps actions to mail transferred from a server, and the
file window applies actions to a local file.

List windows create View, Write, Reply, and Forward windows on user actions.
The server list window is the main window opened on program startup by the
top-level file;  file list windows are opened on demand via server and file
list window "Open".  Msgnums may be temporarily out of sync with server if
POP inbox changes (triggers full reload here).

Changes here in 2.1:
-now checks on deletes and loads to see if msg nums in sync with server
-added up to N attachment direct-access buttons on view windows
-threaded save-mail file loads, to avoid N-second pause for big files
-also threads save-mail file deletes so file write doesn't pause GUI

TBD:
-save-mail file saves still not threaded: may pause GUI briefly, but
 uncommon - unlike load and delete, save/send only appends the local file.
-implementation of local save-mail files as text files with separators
 is mostly a prototype: it loads all full mails into memory, and so limits
 the practical size of these files; better alternative: use 2 DBM keyed
 access files for hdrs and fulltext, plus a list to map keys to position;
 in this scheme save-mail files become directories, no longer readable.
###############################################################################
"""

from SharedNames import *     # program-wide global objects
from ViewWindows import ViewWindow, WriteWindow, ReplyWindow, ForwardWindow


###############################################################################
# main frame - general structure for both file and server message lists
###############################################################################


class PyMailCommon(mailtools.MailParser):
    """
    an abstract widget package, with main mail listbox;
    mixed in with a Tk, Toplevel, or Frame by top-level window classes;
    must be customized in mode-specific subclass with actions() and other;
    creates view and write windows on demand: they serve as MailSenders;
    """
    # class attrs shared by all list windows
    threadLoopStarted = False                     # started by first window
    queueChecksPerSecond = 20                     # tweak if CPU use too high
    queueDelay = 1000 // queueChecksPerSecond     # min msecs between timer events
    queueBatch = 5                                # max callbacks per timer event

    # all windows use same dialogs: remember last dirs
    openDialog = Open(title=appname + ': Open Mail File')
    saveDialog = SaveAs(title=appname + ': Append Mail File')

    # 3.0: avoid downloading (fetching) same message in parallel
    beingFetched = set()

    def __init__(self):
        self.makeWidgets()                        # draw my contents: list,tools
        if not PyMailCommon.threadLoopStarted:

            # start thread exit check loop
            # a timer event loop that dispatches queued GUI callbacks;
            # just one loop for all windows: server,file,views can all thread;
            # self is a Tk, Toplevel,or Frame: any widget type will suffice;
            # 3.0/4E: added queue delay/batch for progress speedup: ~100x/sec;

            PyMailCommon.threadLoopStarted = True
            threadtools.threadChecker(self, self.queueDelay, self.queueBatch)

    def makeWidgets(self):
        # add all/none checkbtn at bottom
        tools = Frame(self, relief=SUNKEN, bd=2, cursor='hand2')    # 3.0: configs
        tools.pack(side=BOTTOM, fill=X)
        self.allModeVar = IntVar()
        chk = Checkbutton(tools, text="All")
        chk.config(variable=self.allModeVar, command=self.onCheckAll)
        chk.pack(side=RIGHT)

        # add main buttons at bottom toolbar
        for (title, callback) in self.actions():
            if not callback:
                sep = Label(tools, text=title)                # 3.0: separator
                sep.pack(side=LEFT, expand=YES, fill=BOTH)    # expands with window
            else:
                Button(tools, text=title, command=callback).pack(side=LEFT)

        # add multiselect listbox with scrollbars
        listwide = mailconfig.listWidth  or 74    # 3.0: config start size
        listhigh = mailconfig.listHeight or 15    # wide=chars, high=lines
        mails    = Frame(self)
        vscroll  = Scrollbar(mails)
        hscroll  = Scrollbar(mails, orient='horizontal')
        fontsz   = (sys.platform[:3] == 'win' and 8) or 10      # defaults
        listbg   = mailconfig.listbg   or 'white'
        listfg   = mailconfig.listfg   or 'black'
        listfont = mailconfig.listfont or ('courier', fontsz, 'normal')
        listbox  = Listbox(mails, bg=listbg, fg=listfg, font=listfont)
        listbox.config(selectmode=EXTENDED)
        listbox.config(width=listwide, height=listhigh) # 3.0: init wider
        listbox.bind('<Double-1>', (lambda event: self.onViewRawMail()))

        # crosslink listbox and scrollbars
        vscroll.config(command=listbox.yview, relief=SUNKEN)
        hscroll.config(command=listbox.xview, relief=SUNKEN)
        listbox.config(yscrollcommand=vscroll.set, relief=SUNKEN)
        listbox.config(xscrollcommand=hscroll.set)

        # pack last = clip first
        mails.pack(side=TOP, expand=YES, fill=BOTH)
        vscroll.pack(side=RIGHT,  fill=BOTH)
        hscroll.pack(side=BOTTOM, fill=BOTH)
        listbox.pack(side=LEFT, expand=YES, fill=BOTH)
        self.listBox = listbox

    #################
    # event handlers
    #################

    def onCheckAll(self):
        # all or none click
        if self.allModeVar.get():
            self.listBox.select_set(0, END)
        else:
            self.listBox.select_clear(0, END)

    def onViewRawMail(self):
        # possibly threaded: view selected messages - raw text headers, body
        msgnums = self.verifySelectedMsgs()
        if msgnums:
            self.getMessages(msgnums, after=lambda: self.contViewRaw(msgnums))

    def contViewRaw(self, msgnums, pyedit=True):     # do we need full TextEditor?
        for msgnum in msgnums:                       # could be a nested def
            fulltext = self.getMessage(msgnum)       # fulltext is Unicode decoded
            if not pyedit:
                # display in a scrolledtext
                from tkinter.scrolledtext import ScrolledText
                window  = windows.QuietPopupWindow(appname, 'raw message viewer')
                browser = ScrolledText(window)
                browser.insert('0.0', fulltext)
                browser.pack(expand=YES, fill=BOTH)
            else:
                # 3.0/4E: more useful PyEdit text editor
                wintitle = ' - raw message text'
                browser = textEditor.TextEditorMainPopup(self, winTitle=wintitle)
                browser.update()
                browser.setAllText(fulltext)
                browser.clearModified()

    def onViewFormatMail(self):
        """
        possibly threaded: view selected messages - pop up formatted display
        not threaded if in savefile list, or messages are already loaded
        the after action runs only if getMessages prefetch allowed and worked
        """
        msgnums = self.verifySelectedMsgs()
        if msgnums:
            self.getMessages(msgnums, after=lambda: self.contViewFmt(msgnums))

    def contViewFmt(self, msgnums):
        """
        finish View: extract main text, popup view window(s) to display;
        extracts plain text from html text if required, wraps text lines;
        html mails: show extracted text, then save in temp file and open
        in web browser;  part can also be opened manually from view window
        Split or part button;  if non-multipart, other: part must be opened
        manually with Split or part button;  verify html open per mailconfig;

        3.0: for html-only mails, main text is str here, but save its raw
        bytes in binary mode to finesse encodings;  worth the effort because
        many mails are just html today;  this first tried N encoding guesses
        (utf-8, latin-1, platform dflt), but now gets and saves raw bytes to
        minimize any fidelity loss;  if a part is later opened on demand, it
        is saved in a binary file as raw bytes in the same way;

        caveat: the spawned web browser won't have any original email headers:
        it may still have to guess or be told the encoding, unless the html
        already has its own encoding headers (these take the form of <meta>
        html tags within <head> sections if present; none are inserted in the
        html here, as some well-formed html parts have them);  IE seems to
        handle most html part files anyhow;  always encoding html parts to
        utf-8 may suffice too: this encoding can handle most types of text;
        """
        for msgnum in msgnums:
            fulltext = self.getMessage(msgnum)             # 3.0: str for parser
            message  = self.parseMessage(fulltext)
            type, content = self.findMainText(message)     # 3.0: Unicode decoded
            if type in ['text/html', 'text/xml']:          # 3.0: get plain text
                content = html2text.html2text(content)
            content  = wraplines.wrapText1(content, mailconfig.wrapsz)
            ViewWindow(headermap   = message,
                       showtext    = content,
                       origmessage = message)              # 3.0: decodes headers

            # non-multipart, content-type text/HTML (rude but true!)
            if type == 'text/html':
                if ((not mailconfig.verifyHTMLTextOpen) or
                    askyesno(appname, 'Open message text in browser?')):

                    # 3.0: get post mime decode, pre unicode decode bytes
                    type, asbytes = self.findMainText(message, asStr=False)
                    try:
                        from tempfile import gettempdir # or a Tk HTML viewer?
                        tempname = os.path.join(gettempdir(), 'pymailgui.html')
                        tmp = open(tempname, 'wb')      # already encoded
                        tmp.write(asbytes); tmp.close() # flush output now
                        webbrowser.open_new('file://' + tempname)
                    except:
                        showerror(appname, 'Cannot open in browser')

    def onWriteMail(self):
        """
        compose a new email from scratch, without fetching others;
        nothing to quote here, but adds sig, and prefills Bcc with the
        sender's address if this optional header enabled in mailconfig;
        From may be i18N encoded in mailconfig: view window will decode;
        """
        starttext = '
'                         # use auto signature text
        if mailconfig.mysignature:
            starttext += '%s
' % mailconfig.mysignature
        From  = mailconfig.myaddress
        WriteWindow(starttext = starttext,
                    headermap = dict(From=From, Bcc=From))    # 3.0: prefill bcc

    def onReplyMail(self):
        # possibly threaded: reply to selected emails
        msgnums = self.verifySelectedMsgs()
        if msgnums:
            self.getMessages(msgnums, after=lambda: self.contReply(msgnums))

    def contReply(self, msgnums):
        """
        finish Reply: drop attachments, quote with '>', add signature;
        presets initial to/from values from mail or config module;
        don't use original To for From: may be many or a listname;
        To keeps name+<addr> format even if ',' separator in name;
        Uses original From for To, ignores reply-to header is any;
        3.0: replies also copy to all original recipients by default;

        3.0: now uses getaddresses/parseaddr full parsing to separate
        addrs on commas, and handle any commas that appear nested in
        email name parts;  multiple addresses are separated by comma
        in GUI, we copy comma separators when displaying headers, and
        we use getaddresses to split addrs as needed;  ',' is required
        by servers for separator;  no longer uses parseaddr to get 1st
        name/addr pair of getaddresses result: use full From for To;

        3.0: we decode the Subject header here because we need its text,
        but the view window superclass of edit windows performs decoding
        on all displayed headers (the extra Subject decode is a no-op);
        on sends, all non-ASCII hdrs and hdr email names are in decoded
        form in the GUI, but are encoded within the mailtools package;
        quoteOrigText also decodes the initial headers it inserts into
        the quoted text block, and index lists decode for display;
        """
        for msgnum in msgnums:
            fulltext = self.getMessage(msgnum)
            message  = self.parseMessage(fulltext)         # may fail: error obj
            maintext = self.formatQuotedMainText(message)  # same as forward

            # from and to are decoded by view window
            From = mailconfig.myaddress                    # not original To
            To   = message.get('From', '')                 # 3.0: ',' sept
            Cc   = self.replyCopyTo(message)               # 3.0: cc all recipients?
            Subj = message.get('Subject', '(no subject)')
            Subj = self.decodeHeader(Subj)                 # deocde for str
            if Subj[:4].lower() != 're: ':                 # 3.0: unify case
                Subj = 'Re: ' + Subj
            ReplyWindow(starttext = maintext,
                        headermap =
                            dict(From=From, To=To, Cc=Cc, Subject=Subj, Bcc=From))

    def onFwdMail(self):
        # possibly threaded: forward selected emails
        msgnums = self.verifySelectedMsgs()
        if msgnums:
            self.getMessages(msgnums, after=lambda: self.contFwd(msgnums))

    def contFwd(self, msgnums):
        """
        finish Forward: drop attachments, quote with '>', add signature;
        see notes about headers decoding in the Reply action methods;
        view window superclass will decode the From header we pass here;
        """
        for msgnum in msgnums:
            fulltext = self.getMessage(msgnum)
            message  = self.parseMessage(fulltext)
            maintext = self.formatQuotedMainText(message)  # same as reply

            # initial From value from config, not mail
            From = mailconfig.myaddress                    # encoded or not
            Subj = message.get('Subject', '(no subject)')
            Subj = self.decodeHeader(Subj)                 # 3.0: send encodes
            if Subj[:5].lower() != 'fwd: ':                # 3.0: unify case
                Subj = 'Fwd: ' + Subj
            ForwardWindow(starttext = maintext,
                          headermap = dict(From=From, Subject=Subj, Bcc=From))

    def onSaveMailFile(self):
        """
        save selected emails to file for offline viewing;
        disabled if target file load/delete is in progress;
        disabled by getMessages if self is a busy file too;
        contSave not threaded: disables all other actions;
        """
        msgnums = self.selectedMsgs()
        if not msgnums:
            showerror(appname, 'No message selected')
        else:
            # caveat: dialog warns about replacing file
            filename = self.saveDialog.show()             # shared class attr
            if filename:                                  # don't verify num msgs
                filename = os.path.abspath(filename)      # normalize / to 
                self.getMessages(msgnums,
                        after=lambda: self.contSave(msgnums, filename))

    def contSave(self, msgnums, filename):
        # test busy now, after poss srvr msgs load
        if (filename in openSaveFiles.keys() and           # viewing this file?
            openSaveFiles[filename].openFileBusy):         # load/del occurring?
            showerror(appname, 'Target file busy - cannot save')
        else:
            try:                                           # caveat:not threaded
                fulltextlist = []                          # 3.0: use encoding
                mailfile = open(filename, 'a', encoding=mailconfig.fetchEncoding)
                for msgnum in msgnums:                     # < 1sec for N megs
                    fulltext = self.getMessage(msgnum)     # but poss many msgs
                    if fulltext[-1] != '
': fulltext += '
'
                    mailfile.write(saveMailSeparator)
                    mailfile.write(fulltext)
                    fulltextlist.append(fulltext)
                mailfile.close()
            except:
                showerror(appname, 'Error during save')
                printStack(sys.exc_info())
            else:                                          # why .keys(): EIBTI
                if filename in openSaveFiles.keys():       # viewing this file?
                    window = openSaveFiles[filename]       # update list, raise
                    window.addSavedMails(fulltextlist)     # avoid file reload
                    #window.loadMailFileThread()           # this was very slow

    def onOpenMailFile(self, filename=None):
        # process saved mail offline
        filename = filename or self.openDialog.show()      # shared class attr
        if filename:
            filename = os.path.abspath(filename)           # match on full name
            if filename in openSaveFiles.keys():           # only 1 win per file
                openSaveFiles[filename].lift()             # raise file's window
                showinfo(appname, 'File already open')     # else deletes odd
            else:
                from PyMailGui import PyMailFileWindow     # avoid duplicate win
                popup = PyMailFileWindow(filename)         # new list window
                openSaveFiles[filename] = popup            # removed in quit
                popup.loadMailFileThread()                 # try load in thread

    def onDeleteMail(self):
        # delete selected mails from server or file
        msgnums = self.selectedMsgs()                      # subclass: fillIndex
        if not msgnums:                                    # always verify here
            showerror(appname, 'No message selected')
        else:
            if askyesno(appname, 'Verify delete %d mails?' % len(msgnums)):
                self.doDelete(msgnums)

    ##################
    # utility methods
    ##################

    def selectedMsgs(self):
        # get messages selected in main listbox
        selections = self.listBox.curselection()  # tuple of digit strs, 0..N-1
        return [int(x)+1 for x in selections]     # convert to ints, make 1..N

    warningLimit = 15
    def verifySelectedMsgs(self):
        msgnums = self.selectedMsgs()
        if not msgnums:
            showerror(appname, 'No message selected')
        else:
            numselects = len(msgnums)
            if numselects > self.warningLimit:
                if not askyesno(appname, 'Open %d selections?' % numselects):
                    msgnums = []
        return msgnums

    def fillIndex(self, maxhdrsize=25):
        """
        fill all of main listbox from message header mappings;
        3.0: decode headers per email/mime/unicode here if encoded;
        3.0: caveat: large chinese characters can break '|' alignment;
        """
        hdrmaps  = self.headersMaps()                   # may be empty
        showhdrs = ('Subject', 'From', 'Date', 'To')    # default hdrs to show
        if hasattr(mailconfig, 'listheaders'):          # mailconfig customizes
            showhdrs = mailconfig.listheaders or showhdrs
        addrhdrs = ('From', 'To', 'Cc', 'Bcc')    # 3.0: decode i18n specially

        # compute max field sizes <= hdrsize
        maxsize = {}
        for key in showhdrs:
            allLens = []                                # too big for a list comp!
            for msg in hdrmaps:
                keyval = msg.get(key, ' ')
                if key not in addrhdrs:
                    allLens.append(len(self.decodeHeader(keyval)))
                else:
                    allLens.append(len(self.decodeAddrHeader(keyval)))
            if not allLens: allLens = [1]
            maxsize[key] = min(maxhdrsize, max(allLens))

        # populate listbox with fixed-width left-justified fields
        self.listBox.delete(0, END)                     # show multiparts with *
        for (ix, msg) in enumerate(hdrmaps):            # via content-type hdr
            msgtype = msg.get_content_maintype()        # no is_multipart yet
            msgline = (msgtype == 'multipart' and '*') or ' '
            msgline += '%03d' % (ix+1)
            for key in showhdrs:
                mysize  = maxsize[key]
                if key not in addrhdrs:
                    keytext = self.decodeHeader(msg.get(key, ' '))
                else:
                    keytext = self.decodeAddrHeader(msg.get(key, ' '))
                msgline += ' | %-*s' % (mysize, keytext[:mysize])
            msgline += '| %.1fK' % (self.mailSize(ix+1) / 1024)   # 3.0: .0 optional
            self.listBox.insert(END, msgline)
        self.listBox.see(END)         # show most recent mail=last line

    def replyCopyTo(self, message):
        """
        3.0: replies copy all original recipients, by prefilling
        Cc header with all addreses in original To and Cc after
        removing duplicates and new sender;  could decode i18n addrs
        here, but the view window will decode to display (and send
        will reencode) and the unique set filtering here will work
        either way, though a sender's i18n address is assumed to be
        in encoded form in mailconfig (else it is not removed here);
        empty To or Cc headers are okay: split returns empty lists;
        """
        if not mailconfig.repliesCopyToAll:
            # reply to sender only
            Cc = ''
        else:
            # copy all original recipients (3.0)
            allRecipients = (self.splitAddresses(message.get('To', '')) +
                             self.splitAddresses(message.get('Cc', '')))
            uniqueOthers  = set(allRecipients) - set([mailconfig.myaddress])
            Cc = ', '.join(uniqueOthers)
        return Cc or '?'

    def formatQuotedMainText(self, message):
        """
        3.0: factor out common code shared by Reply and Forward:
        fetch decoded text, extract text if html, line wrap, add > quote
        """
        type, maintext = self.findMainText(message)       # 3.0: decoded str
        if type in ['text/html', 'text/xml']:             # 3.0: get plain text
            maintext = html2text.html2text(maintext)
        maintext = wraplines.wrapText1(maintext, mailconfig.wrapsz-2) # 2 = '> '
        maintext = self.quoteOrigText(maintext, message)              # add hdrs, >
        if mailconfig.mysignature:
            maintext = ('
%s
' % mailconfig.mysignature) + maintext
        return maintext

    def quoteOrigText(self, maintext, message):
        """
        3.0: we need to decode any i18n (internationalizd) headers here too,
        or they show up in email+MIME encoded form in the quoted text block;
        decodeAddrHeader works on one addr or all in a comma-separated list;
        this may trigger full text encoding on sends, but the main text is
        also already in fully decoded form: could be in any Unicode scheme;
        """
        quoted = '
-----Original Message-----
'
        for hdr in ('From', 'To', 'Subject', 'Date'):
            rawhdr = message.get(hdr, '?')
            if hdr not in ('From', 'To'):
                dechdr = self.decodeHeader(rawhdr)       # full value
            else:
                dechdr = self.decodeAddrHeader(rawhdr)   # name parts only
            quoted += '%s: %s
' % (hdr, dechdr)
        quoted += '
' + maintext
        quoted  = '
' + quoted.replace('
', '
> ')
        return quoted

    ########################
    # subclass requirements
    ########################

    def getMessages(self, msgnums, after):        # used by view,save,reply,fwd
        after()                                   # redef if cache, thread test

    # plus okayToQuit?, any unique actions
    def getMessage(self, msgnum): assert False    # used by many: full mail text
    def headersMaps(self): assert False           # fillIndex: hdr mappings list
    def mailSize(self, msgnum): assert False      # fillIndex: size of msgnum
    def doDelete(self): assert False              # onDeleteMail: delete button


###############################################################################
# main window - when viewing messages in local save file (or sent-mail file)
###############################################################################


class PyMailFile(PyMailCommon):
    """
    customize PyMailCommon for viewing saved-mail file offline;
    mixed with a Tk, Toplevel, or Frame, adds main mail listbox;
    maps load, fetch, delete actions to local text file storage;
    file opens and deletes here run in threads for large files;

    save and send not threaded, because only append to file; save
    is disabled if source or target file busy with load/delete;
    save disables load, delete, save just because it is not run
    in a thread (blocks GUI);

    TBD: may need thread and O/S file locks if saves ever do run in
    threads: saves could disable other threads with openFileBusy, but
    file may not be open in GUI;  file locks not sufficient, because
    GUI updated too;  TBD: appends to sent-mail file may require O/S
    locks: as is, user gets error pop up if sent during load/del;

    3.0: mail save files are now Unicode text, encoded per an encoding
    name setting in the mailconfig module; this may not support worst
    case scenarios of unusual or mixed encodings, but most full mail
    text is ascii, and the Python 3.1 email package is partly broken;
    """
    def actions(self):
        return [ ('Open',   self.onOpenMailFile),
                 ('Write',  self.onWriteMail),
                 ('  ',     None),                           # 3.0:  separators
                 ('View',   self.onViewFormatMail),
                 ('Reply',  self.onReplyMail),
                 ('Fwd',    self.onFwdMail),
                 ('Save',   self.onSaveMailFile),
                 ('Delete', self.onDeleteMail),
                 ('  ',     None),
                 ('Quit',   self.quit) ]

    def __init__(self, filename):
        # caller: do loadMailFileThread next
        PyMailCommon.__init__(self)
        self.filename = filename
        self.openFileBusy = threadtools.ThreadCounter()      # one per window

    def loadMailFileThread(self):
        """
        load or reload file and update window index list;
        called on Open, startup, and possibly on Send if
        sent-mail file appended is currently open;  there
        is always a bogus first item after the text split;
        alt: [self.parseHeaders(m) for m in self.msglist];
        could pop up a busy dialog, but quick for small files;

        2.1: this is now threaded--else runs < 1sec for N meg
        files, but can pause GUI N seconds if very large file;
        Save now uses addSavedMails to append msg lists for
        speed, not this reload;  still called from Send just
        because msg text unavailable - requires refactoring;
        delete threaded too: prevent open and delete overlap;
        """
        if self.openFileBusy:
            # don't allow parallel open/delete changes
            errmsg = 'Cannot load, file is busy:
"%s"' % self.filename
            showerror(appname, errmsg)
        else:
            #self.listBox.insert(END, 'loading...')      # error if user clicks
            savetitle = self.title()                     # set by window class
            self.title(appname + ' - ' + 'Loading...')
            self.openFileBusy.incr()
            threadtools.startThread(
                action   = self.loadMailFile,
                args     = (),
                context  = (savetitle,),
                onExit   = self.onLoadMailFileExit,
                onFail   = self.onLoadMailFileFail)

    def loadMailFile(self):
        # run in a thread while GUI is active
        # open, read, parser may all raise excs: caught in thread utility
        file = open(self.filename, 'r', encoding=mailconfig.fetchEncoding)   # 3.0
        allmsgs = file.read()
        self.msglist  = allmsgs.split(saveMailSeparator)[1:]       # full text
        self.hdrlist  = list(map(self.parseHeaders, self.msglist)) # msg objects

    def onLoadMailFileExit(self, savetitle):
        # on thread success
        self.title(savetitle)         # reset window title to filename
        self.fillIndex()              # updates GUI: do in main thread
        self.lift()                   # raise my window
        self.openFileBusy.decr()

    def onLoadMailFileFail(self, exc_info, savetitle):
        # on thread exception
        showerror(appname, 'Error opening "%s"
%s
%s' %
                           ((self.filename,) +  exc_info[:2]))
        printStack(exc_info)
        self.destroy()                # always close my window?
        self.openFileBusy.decr()      # not needed if destroy

    def addSavedMails(self, fulltextlist):
        """
        optimization: extend loaded file lists for mails
        newly saved to this window's file; in past called
        loadMailThread to reload entire file on save - slow;
        must be called in main GUI thread only: updates GUI;
        sends still reloads sent file if open: no msg text;
        """
        self.msglist.extend(fulltextlist)
        self.hdrlist.extend(map(self.parseHeaders, fulltextlist))  # 3.x iter ok
        self.fillIndex()
        self.lift()

    def doDelete(self, msgnums):
        """
        simple-minded, but sufficient: rewrite all
        nondeleted mails to file; can't just delete
        from self.msglist in-place: changes item indexes;
        Py2.3 enumerate(L) same as zip(range(len(L)), L)
        2.1: now threaded, else N sec pause for large files
        """
        if self.openFileBusy:
            # dont allow parallel open/delete changes
            errmsg = 'Cannot delete, file is busy:
"%s"' % self.filename
            showerror(appname, errmsg)
        else:
            savetitle = self.title()
            self.title(appname + ' - ' + 'Deleting...')
            self.openFileBusy.incr()
            threadtools.startThread(
                action   = self.deleteMailFile,
                args     = (msgnums,),
                context  = (savetitle,),
                onExit   = self.onDeleteMailFileExit,
                onFail   = self.onDeleteMailFileFail)

    def deleteMailFile(self, msgnums):
        # run in a thread while GUI active
        indexed = enumerate(self.msglist)
        keepers = [msg for (ix, msg) in indexed if ix+1 not in msgnums]
        allmsgs = saveMailSeparator.join([''] + keepers)
        file = open(self.filename, 'w', encoding=mailconfig.fetchEncoding)   # 3.0
        file.write(allmsgs)
        self.msglist = keepers
        self.hdrlist = list(map(self.parseHeaders, self.msglist))

    def onDeleteMailFileExit(self, savetitle):
        self.title(savetitle)
        self.fillIndex()              # updates GUI: do in main thread
        self.lift()                   # reset my title, raise my window
        self.openFileBusy.decr()

    def onDeleteMailFileFail(self, exc_info, savetitle):
        showerror(appname, 'Error deleting "%s"
%s
%s' %
                           ((self.filename,) +  exc_info[:2]))
        printStack(exc_info)
        self.destroy()                # always close my window?
        self.openFileBusy.decr()      # not needed if destroy

    def getMessages(self, msgnums, after):
        """
        used by view,save,reply,fwd: file load and delete
        threads may change the msg and hdr lists, so disable
        all other operations that depend on them to be safe;
        this test is for self: saves also test target file;
        """
        if self.openFileBusy:
            errmsg = 'Cannot fetch, file is busy:
"%s"' % self.filename
            showerror(appname, errmsg)
        else:
            after()                      # mail already loaded

    def getMessage(self, msgnum):
        return self.msglist[msgnum-1]    # full text of 1 mail

    def headersMaps(self):
        return self.hdrlist              # email.message.Message objects

    def mailSize(self, msgnum):
        return len(self.msglist[msgnum-1])

    def quit(self):
        # don't destroy during update: fillIndex next
        if self.openFileBusy:
            showerror(appname, 'Cannot quit during load or delete')
        else:
            if askyesno(appname, 'Verify Quit Window?'):
                # delete file from open list
                del openSaveFiles[self.filename]
                Toplevel.destroy(self)


###############################################################################
# main window - when viewing messages on the mail server
###############################################################################


class PyMailServer(PyMailCommon):
    """
    customize PyMailCommon for viewing mail still on server;
    mixed with a Tk, Toplevel, or Frame, adds main mail listbox;
    maps load, fetch, delete actions to email server inbox;
    embeds a MessageCache, which is a mailtools MailFetcher;
    """
    def actions(self):
        return [ ('Load',   self.onLoadServer),
                 ('Open',   self.onOpenMailFile),
                 ('Write',  self.onWriteMail),
                 ('  ',     None),                           # 3.0:  separators
                 ('View',   self.onViewFormatMail),
                 ('Reply',  self.onReplyMail),
                 ('Fwd',    self.onFwdMail),
                 ('Save',   self.onSaveMailFile),
                 ('Delete', self.onDeleteMail),
                 ('  ',     None),
                 ('Quit',   self.quit) ]

    def __init__(self):
        PyMailCommon.__init__(self)
        self.cache = messagecache.GuiMessageCache()    # embedded, not inherited
       #self.listBox.insert(END, 'Press Load to fetch mail')

    def makeWidgets(self):                             # help bar: main win only
        self.addHelpBar()
        PyMailCommon.makeWidgets(self)

    def addHelpBar(self):
        msg = 'PyMailGUI - a Python/tkinter email client  (help)'
        title = Button(self, text=msg)
        title.config(bg='steelblue', fg='white', relief=RIDGE)
        title.config(command=self.onShowHelp)
        title.pack(fill=X)

    def onShowHelp(self):
        """
        load,show text block string
        3.0: now uses HTML and webbrowser module here too
        user setting in mailconfig selects text, HTML, or both
        always displays one or the other: html if both false
        """
        if mailconfig.showHelpAsText:
            from PyMailGuiHelp import helptext
            popuputil.HelpPopup(appname, helptext, showsource=self.onShowMySource)

        if mailconfig.showHelpAsHTML or (not mailconfig.showHelpAsText):
            from PyMailGuiHelp import showHtmlHelp
            showHtmlHelp()    # 3.0: HTML version without source file links

    def onShowMySource(self, showAsMail=False):
        """
        display my sourcecode file, plus imported modules here & elsewhere
        """
        import PyMailGui, ListWindows, ViewWindows, SharedNames, textConfig
        from PP4E.Internet.Email.mailtools import (    # mailtools now a pkg
             mailSender, mailFetcher, mailParser)      # can't use * in def
        mymods = (
            PyMailGui, ListWindows, ViewWindows, SharedNames,
            PyMailGuiHelp, popuputil, messagecache, wraplines, html2text,
            mailtools, mailFetcher, mailSender, mailParser,
            mailconfig, textConfig, threadtools, windows, textEditor)
        for mod in mymods:
            source = mod.__file__
            if source.endswith('.pyc'):
                source = source[:-4] + '.py'       # assume a .py in same dir
            if showAsMail:
                # this is a bit cheesey...
                code   = open(source).read()       # 3.0: platform encoding
                user   = mailconfig.myaddress
                hdrmap = {'From': appname, 'To': user, 'Subject': mod.__name__}
                ViewWindow(showtext=code,
                           headermap=hdrmap,
                           origmessage=email.message.Message())
            else:
                # more useful PyEdit text editor
                # 4E: assume in UTF8 Unicode encoding (else PeEdit may ask!)
                wintitle = ' - ' + mod.__name__
                textEditor.TextEditorMainPopup(self, source, wintitle, 'utf-8')

    def onLoadServer(self, forceReload=False):
        """
        threaded: load or reload mail headers list on request;
        Exit,Fail,Progress run by threadChecker after callback via queue;
        load may overlap with sends, but disables all but send;
        could overlap with loadingMsgs, but may change msg cache list;
        forceReload on delete/synch fail, else loads recent arrivals only;
        2.1: cache.loadHeaders may do quick check to see if msgnums
        in synch with server, if we are loading just newly arrived hdrs;
        """
        if loadingHdrsBusy or deletingBusy or loadingMsgsBusy:
            showerror(appname, 'Cannot load headers during load or delete')
        else:
            loadingHdrsBusy.incr()
            self.cache.setPopPassword(appname) # don't update GUI in the thread!
            popup = popuputil.BusyBoxNowait(appname, 'Loading message headers')
            threadtools.startThread(
                action     = self.cache.loadHeaders,
                args       = (forceReload,),
                context    = (popup,),
                onExit     = self.onLoadHdrsExit,
                onFail     = self.onLoadHdrsFail,
                onProgress = self.onLoadHdrsProgress)

    def onLoadHdrsExit(self, popup):
        self.fillIndex()
        popup.quit()
        self.lift()
        loadingHdrsBusy.decr()                     # allow other actions to run

    def onLoadHdrsFail(self, exc_info, popup):
        popup.quit()
        showerror(appname, 'Load failed: 
%s
%s' % exc_info[:2])
        printStack(exc_info)                       # send stack trace to stdout
        loadingHdrsBusy.decr()
        if exc_info[0] == mailtools.MessageSynchError:    # synch inbox/index
            self.onLoadServer(forceReload=True)           # new thread: reload
        else:
            self.cache.popPassword = None          # force re-input next time

    def onLoadHdrsProgress(self, i, n, popup):
        popup.changeText('%d of %d' % (i, n))

    def doDelete(self, msgnumlist):
        """
        threaded: delete from server now - changes msg nums;
        may overlap with sends only, disables all except sends;
        2.1: cache.deleteMessages now checks TOP result to see
        if headers match selected mails, in case msgnums out of
        synch with mail server: poss if mail deleted by other client,
        or server deletes inbox mail automatically - some ISPs may
        move a mail from inbox to undeliverable on load failure;
        """
        if loadingHdrsBusy or deletingBusy or loadingMsgsBusy:
            showerror(appname, 'Cannot delete during load or delete')
        else:
            deletingBusy.incr()
            popup = popuputil.BusyBoxNowait(appname, 'Deleting selected mails')
            threadtools.startThread(
                action     = self.cache.deleteMessages,
                args       = (msgnumlist,),
                context    = (popup,),
                onExit     = self.onDeleteExit,
                onFail     = self.onDeleteFail,
                onProgress = self.onDeleteProgress)

    def onDeleteExit(self, popup):
        self.fillIndex()                     # no need to reload from server
        popup.quit()                         # refill index with updated cache
        self.lift()                          # raise index window, release lock
        deletingBusy.decr()

    def onDeleteFail(self, exc_info, popup):
        popup.quit()
        showerror(appname, 'Delete failed: 
%s
%s' % exc_info[:2])
        printStack(exc_info)
        deletingBusy.decr()                  # delete or synch check failure
        self.onLoadServer(forceReload=True)  # new thread: some msgnums changed

    def onDeleteProgress(self, i, n, popup):
        popup.changeText('%d of %d' % (i, n))

    def getMessages(self, msgnums, after):
        """
        threaded: prefetch all selected messages into cache now;
        used by save, view, reply, and forward to prefill cache;
        may overlap with other loadmsgs and sends, disables delete,load;
        only runs "after" action if the fetch allowed and successful;
        2.1: cache.getMessages tests if index in synch with server,
        but we only test if we have to go to server, not if cached;

        3.0: see messagecache note: now avoids potential fetch of mail
        currently being fetched, if user clicks again while in progress;
        any message being fetched by any other request in progress must
        disable entire toLoad batch: else, need to wait for N other loads;
        fetches are still allowed to overlap in time, as long as disjoint;
        """
        if loadingHdrsBusy or deletingBusy:
            showerror(appname, 'Cannot fetch message during load or delete')
        else:
            toLoad = [num for num in msgnums if not self.cache.isLoaded(num)]
            if not toLoad:
                after()         # all already loaded
                return          # process now, no wait pop up
            else:
                if set(toLoad) & self.beingFetched:   # 3.0: any in progress?
                    showerror(appname, 'Cannot fetch any message being fetched')
                else:
                    self.beingFetched |= set(toLoad)
                    loadingMsgsBusy.incr()
                    from popuputil import BusyBoxNowait
                    popup = BusyBoxNowait(appname, 'Fetching message contents')
                    threadtools.startThread(
                        action     = self.cache.getMessages,
                        args       = (toLoad,),
                        context    = (after, popup, toLoad),
                        onExit     = self.onLoadMsgsExit,
                        onFail     = self.onLoadMsgsFail,
                        onProgress = self.onLoadMsgsProgress)

    def onLoadMsgsExit(self, after, popup, toLoad):
        self.beingFetched -= set(toLoad)
        popup.quit()
        after()
        loadingMsgsBusy.decr()    # allow other actions after onExit done

    def onLoadMsgsFail(self, exc_info, after, popup, toLoad):
        self.beingFetched -= set(toLoad)
        popup.quit()
        showerror(appname, 'Fetch failed: 
%s
%s' % exc_info[:2])
        printStack(exc_info)
        loadingMsgsBusy.decr()
        if exc_info[0] == mailtools.MessageSynchError:      # synch inbox/index
            self.onLoadServer(forceReload=True)             # new thread: reload

    def onLoadMsgsProgress(self, i, n, after, popup, toLoad):
        popup.changeText('%d of %d' % (i, n))

    def getMessage(self, msgnum):
        return self.cache.getMessage(msgnum)                # full mail text

    def headersMaps(self):
        # list of email.message.Message objects, 3.x requires list() if map()
        # return [self.parseHeaders(h) for h in self.cache.allHdrs()]
        return list(map(self.parseHeaders, self.cache.allHdrs()))

    def mailSize(self, msgnum):
        return self.cache.getSize(msgnum)

    def okayToQuit(self):
        # any threads still running?
        filesbusy = [win for win in openSaveFiles.values() if win.openFileBusy]
        busy = loadingHdrsBusy or deletingBusy or sendingBusy or loadingMsgsBusy
        busy = busy or filesbusy
        return not busy

ViewWindows: Message View Windows

Example 14-4 lists the implementation of mail view and edit windows. These windows are created in response to actions in list windows—View, Write, Reply, and Forward buttons. See the callbacks for these actions in the list window module of Example 14-3 for view window initiation calls.

As in the prior module (Example 14-3), this file is really one common class and a handful of customizations. The mail view window is nearly identical to the mail edit window, used for Write, Reply, and Forward requests. Consequently, this example defines the common appearance and behavior in the view window superclass, and extends it by subclassing for edit windows.

Replies and forwards are hardly different from the write window here, because their details (e.g., From and To addresses and quoted message text) are worked out in the list window implementation before an edit window is created.

Example 14-4. PP4EInternetEmailPyMailGuiViewWindows.py
"""
###############################################################################
Implementation of View, Write, Reply, Forward windows: one class per kind.
Code is factored here for reuse: a Write window is a customized View window,
and Reply and Forward are custom Write windows.  Windows defined in this
file are created by the list windows, in response to user actions.

Caveat:'split' pop ups for opening parts/attachments feel nonintuitive.
2.1: this caveat was addressed, by adding quick-access attachment buttons.
New in 3.0: platform-neutral grid() for mail headers, not packed col frames.
New in 3.0: supports Unicode encodings for main text + text attachments sent.
New in 3.0: PyEdit supports arbitrary Unicode for message parts viewed.
New in 3.0: supports Unicode/mail encodings for headers in  mails sent.

TBD: could avoid verifying quits unless text area modified (like PyEdit2.0),
but these windows are larger, and would not catch headers already changed.
TBD: should Open dialog in write windows be program-wide? (per-window now).
###############################################################################
"""

from SharedNames import *     # program-wide global objects


###############################################################################
# message view window - also a superclass of write, reply, forward
###############################################################################


class ViewWindow(windows.PopupWindow, mailtools.MailParser):
    """
    a Toplevel, with extra protocol and embedded TextEditor;
    inherits saveParts,partsList from mailtools.MailParser;
    mixes in custom subclass logic by direct inheritance here;
    """
    # class attributes
    modelabel       = 'View'                   # used in window titles
    from mailconfig import okayToOpenParts     # open any attachments at all?
    from mailconfig import verifyPartOpens     # ask before open each part?
    from mailconfig import maxPartButtons      # show up to this many + '...'
    from mailconfig import skipTextOnHtmlPart  # 3.0: just browser, not PyEdit?
    tempPartDir     = 'TempParts'              # where 1 selected part saved

    # all view windows use same dialog: remembers last dir
    partsDialog = Directory(title=appname + ': Select parts save directory')

    def __init__(self, headermap, showtext, origmessage=None):
        """
        header map is origmessage, or custom hdr dict for writing;
        showtext is main text part of the message: parsed or custom;
        origmessage is parsed email.message.Message for view mail windows
        """
        windows.PopupWindow.__init__(self, appname, self.modelabel)
        self.origMessage = origmessage
        self.makeWidgets(headermap, showtext)

    def makeWidgets(self, headermap, showtext):
        """
        add headers, actions, attachments, text editor
        3.0: showtext is assumed to be decoded Unicode str here;
        it will be encoded on sends and saves as directed/needed;
        """
        actionsframe = self.makeHeaders(headermap)
        if self.origMessage and self.okayToOpenParts:
            self.makePartButtons()
        self.editor  = textEditor.TextEditorComponentMinimal(self)
        myactions    = self.actionButtons()
        for (label, callback) in myactions:
            b = Button(actionsframe, text=label, command=callback)
            b.config(bg='beige', relief=RIDGE, bd=2)
            b.pack(side=TOP, expand=YES, fill=BOTH)

        # body text, pack last=clip first
        self.editor.pack(side=BOTTOM)               # may be multiple editors
        self.update()                               # 3.0: else may be @ line2
        self.editor.setAllText(showtext)            # each has own content
        lines = len(showtext.splitlines())
        lines = min(lines + 3, mailconfig.viewheight or 20)
        self.editor.setHeight(lines)                # else height=24, width=80
        self.editor.setWidth(80)                    # or from PyEdit textConfig
        if mailconfig.viewbg:
            self.editor.setBg(mailconfig.viewbg)    # colors, font in mailconfig
        if mailconfig.viewfg:
            self.editor.setFg(mailconfig.viewfg)
        if mailconfig.viewfont:                     # also via editor Tools menu
            self.editor.setFont(mailconfig.viewfont)

    def makeHeaders(self, headermap):
        """
        add header entry fields, return action buttons frame;
        3.0: uses grid for platform-neutral layout of label/entry rows;
        packed row frames with fixed-width labels would work well too;

        3.0: decoding of i18n headers (and email names in address headers)
        is performed here if still required as they are added to the GUI;
        some may have been decoded already for reply/forward windows that
        need to use decoded text, but the extra decode here is harmless for
        these, and is required for other headers and cases such as fetched
        mail views;  always, headers are in decoded form when displayed in
        the GUI, and will be encoded within mailtools on Sends if they are
        non-ASCII (see Write);  i18n header decoding also occurs in list
        window mail indexes, and for headers added to quoted mail text;
        text payloads in the mail body are also decoded for display and
        encoded for sends elsewhere in the system (list windows, Write);

        3.0: creators of edit windows prefill Bcc header with sender email
        address to be picked up here, as a convenience for common usages if
        this header is enabled in mailconfig;  Reply also now prefills the
        Cc header with all unique original recipients less From, if enabled;
        """
        top    = Frame(self); top.pack   (side=TOP,   fill=X)
        left   = Frame(top);  left.pack  (side=LEFT,  expand=NO,  fill=BOTH)
        middle = Frame(top);  middle.pack(side=LEFT,  expand=YES, fill=X)

        # headers set may be extended in mailconfig (Bcc, others?)
        self.userHdrs = ()
        showhdrs = ('From', 'To', 'Cc', 'Subject')
        if hasattr(mailconfig, 'viewheaders') and mailconfig.viewheaders:
            self.userHdrs = mailconfig.viewheaders
            showhdrs += self.userHdrs
        addrhdrs = ('From', 'To', 'Cc', 'Bcc')    # 3.0: decode i18n specially

        self.hdrFields = []
        for (i, header) in enumerate(showhdrs):
            lab = Label(middle, text=header+':', justify=LEFT)
            ent = Entry(middle)
            lab.grid(row=i, column=0, sticky=EW)
            ent.grid(row=i, column=1, sticky=EW)
            middle.rowconfigure(i, weight=1)
            hdrvalue = headermap.get(header, '?')    # might be empty
            # 3.0: if encoded, decode per email+mime+unicode
            if header not in addrhdrs:
                hdrvalue = self.decodeHeader(hdrvalue)
            else:
                hdrvalue = self.decodeAddrHeader(hdrvalue)
            ent.insert('0', hdrvalue)
            self.hdrFields.append(ent)               # order matters in onSend
        middle.columnconfigure(1, weight=1)
        return left

    def actionButtons(self):                         # must be method for self
        return [('Cancel', self.destroy),            # close view window silently
                ('Parts',  self.onParts),            # multiparts list or the body
                ('Split',  self.onSplit)]

    def makePartButtons(self):
        """
        add up to N buttons that open attachments/parts
        when clicked; alternative to Parts/Split (2.1);
        okay that temp dir is shared by all open messages:
        part file not saved till later selected and opened;
        partname=partname is required in lambda in Py2.4;
        caveat: we could try to skip the main text part;
        """
        def makeButton(parent, text, callback):
            link = Button(parent, text=text, command=callback, relief=SUNKEN)
            if mailconfig.partfg: link.config(fg=mailconfig.partfg)
            if mailconfig.partbg: link.config(bg=mailconfig.partbg)
            link.pack(side=LEFT, fill=X, expand=YES)

        parts = Frame(self)
        parts.pack(side=TOP, expand=NO, fill=X)
        for (count, partname) in enumerate(self.partsList(self.origMessage)):
            if count == self.maxPartButtons:
                makeButton(parts, '...', self.onSplit)
                break
            openpart = (lambda partname=partname: self.onOnePart(partname))
            makeButton(parts, partname, openpart)

    def onOnePart(self, partname):
        """
        locate selected part for button and save and open;
        okay if multiple mails open: resaves each time selected;
        we could probably just use web browser directly here;
        caveat: tempPartDir is relative to cwd - poss anywhere;
        caveat: tempPartDir is never cleaned up: might be large,
        could use tempfile module (just like the HTML main text
        part display code in onView of the list window class);
        """
        try:
            savedir  = self.tempPartDir
            message  = self.origMessage
            (contype, savepath) = self.saveOnePart(savedir, partname, message)
        except:
            showerror(appname, 'Error while writing part file')
            printStack(sys.exc_info())
        else:
            self.openParts([(contype, os.path.abspath(savepath))])   # reuse

    def onParts(self):
        """
        show message part/attachments in pop-up window;
        uses same file naming scheme as save on Split;
        if non-multipart, single part = full body text
        """
        partnames = self.partsList(self.origMessage)
        msg = '
'.join(['Message parts:
'] + partnames)
        showinfo(appname, msg)

    def onSplit(self):
        """
        pop up save dir dialog and save all parts/attachments there;
        if desired, pop up HTML and multimedia parts in web browser,
        text in TextEditor, and well-known doc types on windows;
        could show parts in View windows where embedded text editor
        would provide a save button, but most are not readable text;
        """
        savedir = self.partsDialog.show()          # class attr: at prior dir
        if savedir:                                # tk dir chooser, not file
            try:
                partfiles = self.saveParts(savedir, self.origMessage)
            except:
                showerror(appname, 'Error while writing part files')
                printStack(sys.exc_info())
            else:
                if self.okayToOpenParts: self.openParts(partfiles)

    def askOpen(self, appname, prompt):
        if not self.verifyPartOpens:
            return True
        else:
            return askyesno(appname, prompt)   # pop-up dialog

    def openParts(self, partfiles):
        """
        auto-open well known and safe file types, but only if verified
        by the user in a pop up; other types must be opened manually
        from save dir;  at this point, the named parts have been already
        MIME-decoded and saved as raw bytes in binary-mode files, but text
        parts may be in any Unicode encoding;  PyEdit needs to know the
        encoding to decode, webbrowsers may have to guess or be told;

        caveat: punts for type application/octet-stream even if it has
        safe filename extension such as .html; caveat: image/audio/video
        could be opened with the book's playfile.py; could also do that
        if text viewer fails: would start notepad on Windows via startfile;
        webbrowser may handle most cases here too, but specific is better;
        """

        def textPartEncoding(fullfilename):
            """
            3.0: map a text part filename back to charset param in content-type
            header of part's Message, so we can pass this on to the PyEdit
            constructor for proper text display;  we could return the charset
            along with content-type from mailtools for text parts, but fewer
            changes are needed if this is handled as a special case here;

            part content is saved in binary mode files by mailtools to avoid
            encoding issues, but here the original part Message is not directly
            available; we need this mapping step to extract a Unicode encoding
            name if present; 4E's PyEdit now allows an explicit encoding name for
            file opens, and resolves encoding on saves; see Chapter 11 for PyEdit
            policies: it may ask user for an encoding if charset absent or fails;
            caveat: move to mailtools.mailParser to reuse for <meta> in PyMailCGI?
            """
            partname = os.path.basename(fullfilename)
            for (filename, contype, part) in self.walkNamedParts(self.origMessage):
                if filename == partname:
                    return part.get_content_charset()     # None if not in header
            assert False, 'Text part not found'           # should never happen

        for (contype, fullfilename) in partfiles:
            maintype  = contype.split('/')[0]                      # left side
            extension = os.path.splitext(fullfilename)[1]          # not [-4:]
            basename  = os.path.basename(fullfilename)             # strip dir

            # HTML and XML text, web pages, some media
            if contype  in ['text/html', 'text/xml']:
                browserOpened = False
                if self.askOpen(appname, 'Open "%s" in browser?' % basename):
                    try:
                        webbrowser.open_new('file://' + fullfilename)
                        browserOpened = True
                    except:
                        showerror(appname, 'Browser failed: trying editor')
                if not browserOpened or not self.skipTextOnHtmlPart:
                    try:
                        # try PyEdit to see encoding name and effect
                        encoding = textPartEncoding(fullfilename)
                        textEditor.TextEditorMainPopup(parent=self,
                                   winTitle=' - %s email part' % (encoding or '?'),
                                   loadFirst=fullfilename, loadEncode=encoding)
                    except:
                        showerror(appname, 'Error opening text viewer')

            # text/plain, text/x-python, etc.; 4E: encoding, may fail
            elif maintype == 'text':
                if self.askOpen(appname, 'Open text part "%s"?' % basename):
                    try:
                        encoding = textPartEncoding(fullfilename)
                        textEditor.TextEditorMainPopup(parent=self,
                                   winTitle=' - %s email part' % (encoding or '?'),
                                   loadFirst=fullfilename, loadEncode=encoding)
                    except:
                        showerror(appname, 'Error opening text viewer')

            # multimedia types: Windows opens mediaplayer, imageviewer, etc.
            elif maintype in ['image', 'audio', 'video']:
                if self.askOpen(appname, 'Open media part "%s"?' % basename):
                    try:
                        webbrowser.open_new('file://' + fullfilename)
                    except:
                        showerror(appname, 'Error opening browser')

            # common Windows documents: Word, Excel, Adobe, archives, etc.
            elif (sys.platform[:3] == 'win' and
                  maintype == 'application' and                      # 3.0: +x types
                  extension in ['.doc', '.docx', '.xls', '.xlsx',    # generalize me
                                '.pdf', '.zip',  '.tar', '.wmv']):
                    if self.askOpen(appname, 'Open part "%s"?' % basename):
                        os.startfile(fullfilename)

            else:  # punt!
                msg = 'Cannot open part: "%s"
Open manually in: "%s"'
                msg = msg % (basename, os.path.dirname(fullfilename))
                showinfo(appname, msg)


###############################################################################
# message edit windows - write, reply, forward
###############################################################################


if mailconfig.smtpuser:                              # user set in mailconfig?
    MailSenderClass = mailtools.MailSenderAuth       # login/password required
else:
    MailSenderClass = mailtools.MailSender


class WriteWindow(ViewWindow, MailSenderClass):
    """
    customize view display for composing new mail
    inherits sendMessage from mailtools.MailSender
    """
    modelabel = 'Write'

    def __init__(self, headermap, starttext):
        ViewWindow.__init__(self, headermap, starttext)
        MailSenderClass.__init__(self)
        self.attaches   = []                     # each win has own open dialog
        self.openDialog = None                   # dialog remembers last dir

    def actionButtons(self):
        return [('Cancel', self.quit),           # need method to use self
                ('Parts',  self.onParts),        # PopupWindow verifies cancel
                ('Attach', self.onAttach),
                ('Send',   self.onSend)]         # 4E: don't pad: centered

    def onParts(self):
        # caveat: deletes not currently supported
        if not self.attaches:
            showinfo(appname, 'Nothing attached')
        else:
            msg = '
'.join(['Already attached:
'] + self.attaches)
            showinfo(appname, msg)

    def onAttach(self):
        """
        attach a file to the mail: name added here will be
        added as a part on Send, inside the mailtools pkg;
        4E: could ask Unicode type here instead of on send
        """
        if not self.openDialog:
            self.openDialog = Open(title=appname + ': Select Attachment File')
        filename = self.openDialog.show()        # remember prior dir
        if filename:
            self.attaches.append(filename)       # to be opened in send method

    def resolveUnicodeEncodings(self):
        """
        3.0/4E: to prepare for send, resolve Unicode encoding for text parts:
        both main text part, and any text part attachments;  the main text part
        may have had a known encoding if this is a reply or forward, but not for
        a write, and it may require a different encoding after editing anyhow;
        smtplib in 3.1 requires that full message text be encodable per ASCII
        when sent (if it's a str), so it's crucial to get this right here; else
        fails if reply/fwd to UTF8 text when config=ascii if any non-ascii chars;
        try user setting and reply but fall back on general UTF8 as a last resort;
        """

        def isTextKind(filename):
            contype, encoding = mimetypes.guess_type(filename)
            if contype is None or encoding is not None:    # 4E utility
                return False                               # no guess, compressed?
            maintype, subtype = contype.split('/', 1)      # check for text/?
            return maintype == 'text'

        # resolve many body text encoding
        bodytextEncoding = mailconfig.mainTextEncoding
        if bodytextEncoding == None:
            asknow = askstring('PyMailGUI', 'Enter main text Unicode encoding name')
            bodytextEncoding = asknow or 'latin-1'    # or sys.getdefaultencoding()?

        # last chance: use utf-8 if can't encode per prior selections
        if bodytextEncoding != 'utf-8':
            try:
                bodytext = self.editor.getAllText()
                bodytext.encode(bodytextEncoding)
            except (UnicodeError, LookupError):       # lookup: bad encoding name
                bodytextEncoding = 'utf-8'            # general code point scheme

        # resolve any text part attachment encodings
        attachesEncodings = []
        config = mailconfig.attachmentTextEncoding
        for filename in self.attaches:
            if not isTextKind(filename):
                attachesEncodings.append(None)        # skip non-text: don't ask
            elif config != None:
                attachesEncodings.append(config)      # for all text parts if set
            else:
                prompt = 'Enter Unicode encoding name for %' % filename
                asknow = askstring('PyMailGUI', prompt)
                attachesEncodings.append(asknow or 'latin-1')

            # last chance: use utf-8 if can't decode per prior selections
            choice = attachesEncodings[-1]
            if choice != None and choice != 'utf-8':
                try:
                    attachbytes = open(filename, 'rb').read()
                    attachbytes.decode(choice)
                except (UnicodeError, LookupError, IOError):
                    attachesEncodings[-1] = 'utf-8'
        return bodytextEncoding, attachesEncodings

    def onSend(self):
        """
        threaded: mail edit window Send button press;
        may overlap with any other thread, disables none but quit;
        Exit,Fail run by threadChecker via queue in after callback;
        caveat: no progress here, because send mail call is atomic;
        assumes multiple recipient addrs are separated with ',';
        mailtools module handles encodings, attachments, Date, etc;
        mailtools module also saves sent message text in a local file

        3.0: now fully parses To,Cc,Bcc (in mailtools) instead of
        splitting on the separator naively;  could also use multiline
        input widgets instead of simple entry;  Bcc added to envelope,
        not headers;

        3.0: Unicode encodings of text parts is resolved here, because
        it may require GUI prompts;  mailtools performs the actual
        encoding for parts as needed and requested;

        3.0: i18n headers are already decoded in the GUI fields here;
        encoding of any non-ASCII i18n headers is performed in mailtools,
        not here, because no GUI interaction is required;
        """

        # resolve Unicode encoding for text parts;
        bodytextEncoding, attachesEncodings = self.resolveUnicodeEncodings()

        # get components from GUI; 3.0: i18n headers are decoded
        fieldvalues = [entry.get() for entry in self.hdrFields]
        From, To, Cc, Subj = fieldvalues[:4]
        extraHdrs = [('Cc', Cc), ('X-Mailer', appname + ' (Python)')]
        extraHdrs += list(zip(self.userHdrs, fieldvalues[4:]))
        bodytext = self.editor.getAllText()

        # split multiple recipient lists on ',', fix empty fields
        Tos = self.splitAddresses(To)
        for (ix, (name, value)) in enumerate(extraHdrs):
            if value:                                           # ignored if ''
                if value == '?':                                # ? not replaced
                    extraHdrs[ix] = (name, '')
                elif name.lower() in ['cc', 'bcc']:             # split on ','
                    extraHdrs[ix] = (name, self.splitAddresses(value))

        # withdraw to disallow send during send
        # caveat: might not be foolproof - user may deiconify if icon visible
        self.withdraw()
        self.getPassword()      # if needed; don't run pop up in send thread!
        popup = popuputil.BusyBoxNowait(appname, 'Sending message')
        sendingBusy.incr()
        threadtools.startThread(
            action  = self.sendMessage,
            args    = (From, Tos, Subj, extraHdrs, bodytext, self.attaches,
                                         saveMailSeparator,
                                         bodytextEncoding,
                                         attachesEncodings),
            context = (popup,),
            onExit  = self.onSendExit,
            onFail  = self.onSendFail)

    def onSendExit(self, popup):
        """
        erase wait window, erase view window, decr send count;
        sendMessage call auto saves sent message in local file;
        can't use window.addSavedMails: mail text unavailable;
        """
        popup.quit()
        self.destroy()
        sendingBusy.decr()

        # poss  when opened, / in mailconfig
        sentname = os.path.abspath(mailconfig.sentmailfile)  # also expands '.'
        if sentname in openSaveFiles.keys():                 # sent file open?
            window = openSaveFiles[sentname]                 # update list,raise
            window.loadMailFileThread()

    def onSendFail(self, exc_info, popup):
        # pop-up error, keep msg window to save or retry, redraw actions frame
        popup.quit()
        self.deiconify()
        self.lift()
        showerror(appname, 'Send failed: 
%s
%s' % exc_info[:2])
        printStack(exc_info)
        MailSenderClass.smtpPassword = None        # try again; 3.0/4E: not on self
        sendingBusy.decr()

    def askSmtpPassword(self):
        """
        get password if needed from GUI here, in main thread;
        caveat: may try this again in thread if no input first
        time, so goes into a loop until input is provided; see
        pop paswd input logic for a nonlooping alternative
        """
        password = ''
        while not password:
            prompt = ('Password for %s on %s?' %
                     (self.smtpUser, self.smtpServerName))
            password = popuputil.askPasswordWindow(appname, prompt)
        return password


class ReplyWindow(WriteWindow):
    """
    customize write display for replying
    text and headers set up by list window
    """
    modelabel = 'Reply'


class ForwardWindow(WriteWindow):
    """
    customize reply display for forwarding
    text and headers set up by list window
    """
    modelabel = 'Forward'

messagecache: Message Cache Manager

The class in Example 14-5 implements a cache for already loaded messages. Its logic is split off into this file in order to avoid further complicating list window implementations. The server list window creates and embeds an instance of this class to interface with the mail server and to keep track of already loaded mail headers and full text. In this version, the server list window also keeps track of mail fetches in progress, to avoid attempting to load the same mail more than once in parallel. This task isn’t performed here, because it may require GUI operations.

Example 14-5. PP4EInternetEmailPyMailGuimessagecache.py
"""
##############################################################################
manage message and header loads and context, but not GUI;
a MailFetcher, with a list of already loaded headers and messages;
the caller must handle any required threading or GUI interfaces;

3.0 change: use full message text  Unicode encoding name in local
mailconfig module; decoding happens deep in mailtools, when a message
is fetched - mail text is always Unicode str from that point on;
this may change in a future Python/email: see Chapter 13 for details;

3.0 change: inherits the new mailconfig.fetchlimit feature of mailtools,
which can be used to limit the maximum number of most recent headers or
full mails (if no TOP) fetched on each load request; note that this
feature is independent of the loadfrom used here to limit loads to
newly-arrived mails only, though it is applied at the same time: at
most fetchlimit newly-arrived mails are loaded;

3.0 change: though unlikely, it's not impossible that a user may trigger a
new fetch of a message that is currently being fetched in a thread, simply
by clicking the same message again (msg fetches, but not full index loads,
may overlap with other fetches and sends); this seems to be thread safe here,
but can lead to redundant and possibly parallel downloads of the same mail
which are pointless and seem odd (selecting all mails and pressing View
twice downloads most messages twice!); fixed by keeping track of fetches in
progress in the main GUI thread so that this overlap is no longer possible:
a message being fetched disables any fetch request which it is part of, and
parallel fetches are still allowed as long as their targets do not intersect;
##############################################################################
"""

from PP4E.Internet.Email import mailtools
from popuputil import askPasswordWindow


class MessageInfo:
    """
    an item in the mail cache list
    """
    def __init__(self, hdrtext, size):
        self.hdrtext  = hdrtext            # fulltext is cached msg
        self.fullsize = size               # hdrtext is just the hdrs
        self.fulltext = None               # fulltext=hdrtext if no TOP


class MessageCache(mailtools.MailFetcher):
    """
    keep track of already loaded headers and messages;
    inherits server transfer methods from MailFetcher;
    useful in other apps: no GUI or thread assumptions;

    3.0: raw mail text bytes are decoded to str to be
    parsed with Py3.1's email pkg and saved to files;
    uses the local mailconfig module's encoding setting;
    decoding happens automatically in mailtools on fetch;
    """
    def __init__(self):
        mailtools.MailFetcher.__init__(self)   # 3.0: inherits fetchEncoding
        self.msglist = []                      # 3.0: inherits fetchlimit

    def loadHeaders(self, forceReloads, progress=None):
        """
        three cases to handle here: the initial full load,
        load newly arrived, and forced reload after delete;
        don't refetch viewed msgs if hdrs list same or extended;
        retains cached msgs after a delete unless delete fails;
        2.1: does quick check to see if msgnums still in sync
        3.0: this is now subject to mailconfig.fetchlimit max;
        """
        if forceReloads:
            loadfrom = 1
            self.msglist = []                         # msg nums have changed
        else:
            loadfrom = len(self.msglist)+1            # continue from last load

        # only if loading newly arrived
        if loadfrom != 1:
            self.checkSynchError(self.allHdrs())      # raises except if bad

        # get all or newly arrived msgs
        reply = self.downloadAllHeaders(progress, loadfrom)
        headersList, msgSizes, loadedFull = reply

        for (hdrs, size) in zip(headersList, msgSizes):
            newmsg = MessageInfo(hdrs, size)
            if loadedFull:                            # zip result may be empty
                newmsg.fulltext = hdrs                # got full msg if no 'top'
            self.msglist.append(newmsg)

    def getMessage(self, msgnum):                     # get raw msg text
        cacheobj = self.msglist[msgnum-1]             # add to cache if fetched
        if not cacheobj.fulltext:                     # harmless if threaded
            fulltext = self.downloadMessage(msgnum)   # 3.0: simpler coding
            cacheobj.fulltext = fulltext
        return cacheobj.fulltext

    def getMessages(self, msgnums, progress=None):
        """
        prefetch full raw text of multiple messages, in thread;
        2.1: does quick check to see if msgnums still in sync;
        we can't get here unless the index list already loaded;
        """
        self.checkSynchError(self.allHdrs())          # raises except if bad
        nummsgs = len(msgnums)                        # adds messages to cache
        for (ix, msgnum) in enumerate(msgnums):       # some poss already there
             if progress: progress(ix+1, nummsgs)     # only connects if needed
             self.getMessage(msgnum)                  # but may connect > once

    def getSize(self, msgnum):                        # encapsulate cache struct
        return self.msglist[msgnum-1].fullsize        # it changed once already!

    def isLoaded(self, msgnum):
        return self.msglist[msgnum-1].fulltext

    def allHdrs(self):
        return [msg.hdrtext for msg in self.msglist]

    def deleteMessages(self, msgnums, progress=None):
        """
        if delete of all msgnums works, remove deleted entries
        from mail cache, but don't reload either the headers list
        or already viewed mails text: cache list will reflect the
        changed msg nums on server;  if delete fails for any reason,
        caller should forceably reload all hdrs next, because _some_
        server msg nums may have changed, in unpredictable ways;
        2.1: this now checks msg hdrs to detect out of synch msg
        numbers, if TOP supported by mail server; runs in thread
        """
        try:
            self.deleteMessagesSafely(msgnums, self.allHdrs(), progress)
        except mailtools.TopNotSupported:
            mailtools.MailFetcher.deleteMessages(self, msgnums, progress)

        # no errors: update index list
        indexed = enumerate(self.msglist)
        self.msglist = [msg for (ix, msg) in indexed if ix+1 not in msgnums]


class GuiMessageCache(MessageCache):
    """
    add any GUI-specific calls here so cache usable in non-GUI apps
    """

    def setPopPassword(self, appname):
        """
        get password from GUI here, in main thread
        forceably called from GUI to avoid pop ups in threads
        """
        if not self.popPassword:
            prompt = 'Password for %s on %s?' % (self.popUser, self.popServer)
            self.popPassword = askPasswordWindow(appname, prompt)

    def askPopPassword(self):
        """
        but don't use GUI pop up here: I am run in a thread!
        when tried pop up in thread, caused GUI to hang;
        may be called by MailFetcher superclass, but only
        if passwd is still empty string due to dialog close
        """
        return self.popPassword

popuputil: General-Purpose GUI Pop Ups

Example 14-6 implements a handful of utility pop-up windows in a module, in case they ever prove useful in other programs. Note that the same windows utility module is imported here, to give a common look-and-feel to the pop ups (icons, titles, and so on).

Example 14-6. PP4EInternetEmailPyMailGuipopuputil.py
"""
#############################################################################
utility windows - may be useful in other programs
#############################################################################
"""

from tkinter import *
from PP4E.Gui.Tools.windows import PopupWindow


class HelpPopup(PopupWindow):
    """
    custom Toplevel that shows help text as scrolled text
    source button runs a passed-in callback handler
    3.0 alternative: use HTML file and webbrowser module
    """
    myfont = 'system'  # customizable
    mywidth = 78       # 3.0: start width

    def __init__(self, appname, helptext, iconfile=None, showsource=lambda:0):
        PopupWindow.__init__(self, appname, 'Help', iconfile)
        from tkinter.scrolledtext import ScrolledText    # a nonmodal dialog
        bar  = Frame(self)                               # pack first=clip last
        bar.pack(side=BOTTOM, fill=X)
        code = Button(bar, bg='beige', text="Source", command=showsource)
        quit = Button(bar, bg='beige', text="Cancel", command=self.destroy)
        code.pack(pady=1, side=LEFT)
        quit.pack(pady=1, side=LEFT)
        text = ScrolledText(self)                   # add Text + scrollbar
        text.config(font=self.myfont)
        text.config(width=self.mywidth)             # too big for showinfo
        text.config(bg='steelblue', fg='white')     # erase on btn or return
        text.insert('0.0', helptext)
        text.pack(expand=YES, fill=BOTH)
        self.bind("<Return>", (lambda event: self.destroy()))


def askPasswordWindow(appname, prompt):
    """
    modal dialog to input password string
    getpass.getpass uses stdin, not GUI
    tkSimpleDialog.askstring echos input
    """
    win = PopupWindow(appname, 'Prompt')               # a configured Toplevel
    Label(win, text=prompt).pack(side=LEFT)
    entvar = StringVar(win)
    ent = Entry(win, textvariable=entvar, show='*')    # display * for input
    ent.pack(side=RIGHT, expand=YES, fill=X)
    ent.bind('<Return>', lambda event: win.destroy())
    ent.focus_set(); win.grab_set(); win.wait_window()
    win.update()                                       # update forces redraw
    return entvar.get()                                # ent widget is now gone


class BusyBoxWait(PopupWindow):
    """
    pop up blocking wait message box: thread waits
    main GUI event thread stays alive during wait
    but GUI is inoperable during this wait state;
    uses quit redef here because lower, not leftmost;
    """
    def __init__(self, appname, message):
        PopupWindow.__init__(self, appname, 'Busy')
        self.protocol('WM_DELETE_WINDOW', lambda:0)        # ignore deletes
        label = Label(self, text=message + '...')          # win.quit() to erase
        label.config(height=10, width=40, cursor='watch')  # busy cursor
        label.pack()
        self.makeModal()
        self.message, self.label = message, label
    def makeModal(self):
        self.focus_set()                                   # grab application
        self.grab_set()                                    # wait for threadexit
    def changeText(self, newtext):
        self.label.config(text=self.message + ': ' + newtext)
    def quit(self):
        self.destroy()                                     # don't verify quit

class BusyBoxNowait(BusyBoxWait):
    """
    pop up nonblocking wait window
    call changeText to show progress, quit to close
    """
    def makeModal(self):
        pass

if __name__ == '__main__':
    HelpPopup('spam', 'See figure 1...
')
    print(askPasswordWindow('spam', 'enter password'))
    input('Enter to exit')  # pause if clicked

wraplines: Line Split Tools

The module in Example 14-7 implements general tools for wrapping long lines, at either a fixed column or the first delimiter at or before a fixed column. PyMailGUI uses this file’s wrapText1 function for text in view, reply, and forward windows, but this code is potentially useful in other programs. Run the file as a script to watch its self-test code at work, and study its functions to see its text-processing logic.

Example 14-7. PP4EInternetEmailPyMailGuiwraplines.py
"""
###############################################################################
split lines on fixed columns or at delimiters before a column;
see also: related but different textwrap standard library module (2.3+);
4E caveat: this assumes str; supporting bytes might help avoid some decodes;
###############################################################################
"""

defaultsize = 80

def wrapLinesSimple(lineslist, size=defaultsize):
    "split at fixed position size"
    wraplines = []
    for line in lineslist:
        while True:
            wraplines.append(line[:size])         # OK if len < size
            line = line[size:]                    # split without analysis
            if not line: break
    return wraplines

def wrapLinesSmart(lineslist, size=defaultsize, delimiters='.,:	 '):
    "wrap at first delimiter left of size"
    wraplines = []
    for line in lineslist:
        while True:
            if len(line) <= size:
                wraplines += [line]
                break
            else:
                for look in range(size-1, size // 2, −1):       # 3.0: need // not /
                    if line[look] in delimiters:
                        front, line = line[:look+1], line[look+1:]
                        break
                else:
                    front, line = line[:size], line[size:]
                wraplines += [front]
    return wraplines

###############################################################################
# common use case utilities
###############################################################################

def wrapText1(text, size=defaultsize):         # better for line-based txt: mail
    "when text read all at once"               # keeps original line brks struct
    lines = text.split('
')                   # split on newlines
    lines = wrapLinesSmart(lines, size)        # wrap lines on delimiters
    return '
'.join(lines)                    # put back together

def wrapText2(text, size=defaultsize):         # more uniform across lines
    "same, but treat as one long line"         # but loses original line struct
    text  = text.replace('
', ' ')            # drop newlines if any
    lines = wrapLinesSmart([text], size)       # wrap single line on delimiters
    return lines                               # caller puts back together

def wrapText3(text, size=defaultsize):
    "same, but put back together"
    lines = wrapText2(text, size)              # wrap as single long line
    return '
'.join(lines) + '
'             # make one string with newlines

def wrapLines1(lines, size=defaultsize):
    "when newline included at end"
    lines = [line[:-1] for line in lines]      # strip off newlines (or .rstrip)
    lines = wrapLinesSmart(lines, size)        # wrap on delimiters
    return [(line + '
') for line in lines]   # put them back

def wrapLines2(lines, size=defaultsize):       # more uniform across lines
    "same, but concat as one long line"        # but loses original structure
    text  = ''.join(lines)                     # put together as 1 line
    lines = wrapText2(text)                    # wrap on delimiters
    return [(line + '
') for line in lines]   # put newlines on ends

###############################################################################
# self-test
###############################################################################

if __name__ == '__main__':
    lines = ['spam ham ' * 20 + 'spam,ni' * 20,
             'spam ham ' * 20,
             'spam,ni'   * 20,
             'spam ham.ni' * 20,
             '',
             'spam'*80,
             ' ',
             'spam ham eggs']

    sep = '-' * 30
    print('all', sep)
    for line in lines: print(repr(line))
    print('simple', sep)
    for line in wrapLinesSimple(lines): print(repr(line))
    print('smart', sep)
    for line in wrapLinesSmart(lines): print(repr(line))

    print('single1', sep)
    for line in wrapLinesSimple([lines[0]], 60): print(repr(line))
    print('single2', sep)
    for line in wrapLinesSmart([lines[0]], 60): print(repr(line))
    print('combined text', sep)
    for line in wrapLines2(lines): print(repr(line))
    print('combined lines', sep)
    print(wrapText1('
'.join(lines)))

    assert ''.join(lines) == ''.join(wrapLinesSimple(lines, 60))
    assert ''.join(lines) == ''.join(wrapLinesSmart(lines, 60))
    print(len(''.join(lines)), end=' ')
    print(len(''.join(wrapLinesSimple(lines))), end=' ')
    print(len(''.join(wrapLinesSmart(lines))), end=' ')
    print(len(''.join(wrapLinesSmart(lines, 60))), end=' ')
    input('Press enter')   # pause if clicked

html2text: Extracting Text from HTML (Prototype, Preview)

Example 14-8 lists the code of the simple-minded HTML parser that PyMailGUI uses to extract plain text from mails whose main (or only) text part is in HTML form. This extracted text is used both for display and for the initial text in replies and forwards. Its original HTML form is also displayed in its full glory in a popped-up web browser as before.

This is a prototype. Because PyMailGUI is oriented toward plain text today, this parser is intended as a temporary workaround until a HTML viewer/editor widget solution is found. Because of that, this is at best a first cut which has not been polished to any significant extent. Robustly parsing HTML in its entirety is a task well beyond the scope of this chapter and book. When this parser fails to render good plain text (and it will!), users can still view and cut-and-paste the properly formatted text from the web browser.

This is also a preview. HTML parsing is not covered until Chapter 19 of this book, so you’ll have to take this on faith until we refer back to it in that later chapter. Unfortunately, this feature was added to PyMailGUI late in the book project, and avoiding this forward reference didn’t seem to justify omitting the improvement altogether. For more details on HTML parsing, stay tuned for (or flip head to) Chapter 19.

In short, the class here provides handler methods that receive callbacks from an HTML parser, as tags and their content is recognized; we use this model here to save text we’re interested in along the way. Besides the parser class, we could also use Python’s html.entities module to map more entity types than are hardcoded here—another tool we will meet in Chapter 19.

Despite its limitations, this example serves as a rough guide to help get you started, and any result it produces is certainly an improvement upon the prior edition’s display and quoting of raw HTML.

Example 14-8. PP4EInternetEmailPyMailGuihtml2text.py
"""
################################################################
*VERY* preliminary html-to-text extractor, for text to be
quoted in replies and forwards, and displayed in the main
text display component.  Only used when the main text part
is HTML (i.e., no alternative or other text parts to show).
We also need to know if this is HTML or not, but findMainText
already returns the main text's content type.

This is mostly provided as a first cut, to help get you started
on a more complete solution.  It hasn't been polished, because
any result is better than displaying raw HTML, and it's probably
a better idea to migrate to an HTML viewer/editor widget in the
future anyhow.  As is, PyMailGUI is still plain-text biased.

If (really, when) this parser fails to render well, users can
instead view and cut-and-paste from the web browser popped up
to display the HTML.  See Chapter 19 for more on HTML parsing.
################################################################
"""
from html.parser import HTMLParser     # Python std lib parser (sax-like model)

class Parser(HTMLParser):              # subclass parser, define callback methods
    def __init__(self):                # text assumed to be str, any encoding ok
        HTMLParser.__init__(self)
        self.text = '[Extracted HTML text]'
        self.save = 0
        self.last = ''

    def addtext(self, new):
        if self.save > 0:
            self.text += new
            self.last = new

    def addeoln(self, force=False):
        if force or self.last != '
':
            self.addtext('
')

    def handle_starttag(self, tag, attrs):    # + others imply content start?
        if tag in ('p', 'div', 'table', 'h1', 'h2', 'li'):
            self.save += 1
            self.addeoln()
        elif tag == 'td':
            self.addeoln()
        elif tag == 'style':                  # + others imply end of prior?
            self.save -= 1
        elif tag == 'br':
            self.addeoln(True)
        elif tag == 'a':
            alts = [pair for pair in attrs if pair[0] == 'alt']
            if alts:
                name, value = alts[0]
                self.addtext('[' + value.replace('
', '') + ']')

    def handle_endtag(self, tag):
        if tag in ('p', 'div', 'table', 'h1', 'h2', 'li'):
            self.save -= 1
            self.addeoln()
        elif tag == 'style':
            self.save += 1

    def handle_data(self, data):
        data = data.replace('
', '')          # what about <PRE>?
        data = data.replace('	', ' ')
        if data != ' ' * len(data):
            self.addtext(data)

    def handle_entityref(self, name):
        xlate = dict(lt='<', gt='>', amp='&', nbsp='').get(name, '?')
        if xlate:
            self.addtext(xlate)     # plus many others: show ? as is

def html2text(text):
    try:
        hp = Parser()
        hp.feed(text)
        return(hp.text)
    except:
        return text

if __name__ == '__main__':

    # to test me: html2text.py mediahtml2text-testhtmlmail1.html
    # parse file name in commandline, display result in tkinter Text
    # file assumed to be in Unicode platform default, but text need not be

    import sys, tkinter
    text = open(sys.argv[1], 'r').read()
    text = html2text(text)
    t = tkinter.Text()
    t.insert('1.0', text)
    t.pack()
    t.mainloop()

Note

After this example and chapter had been written and finalized, I did a search for HTML-to-text translators on the Web to try to find better options, and I discovered a Python-coded solution which is much more complete and robust than the simple prototype script here. Regrettably, I also discovered that this system is named the same as the script listed here!

This was unintentional and unforeseen (alas, developers are predisposed to think alike). For details on this more widely tested and much better alternative, search the Web for html2text. It’s open source, but follows the GPL license, and is available only for Python 2.X at this writing (e.g., it uses the 2.X sgmllib which has been removed in favor of the new html.parser in 3.X). Unfortunately, its GPL license may raise copyright concerns if shipped with PyMailGUI in this book’s example package or otherwise; worse, its 2.X status means it cannot be used at all with this book’s 3.X examples today.

There are additional plain-text extractor options on the Web worth checking out, including BeautifulSoup and yet another named html2text.py (no, really!). They also appear to be available for just 2.X today, though naturally, this story may change by the time you read this note. There’s no reason to reinvent the wheel, unless existing wheels don’t fit your cart!

mailconfig: User Configurations

In Example 14-9, PyMailGUI’s mailconfig user settings module is listed. This program has its own version of this module because many of its settings are unique for PyMailGUI. To use the program for reading your own email, set its initial variables to reflect your POP and SMTP server names and login parameters. The variables in this module also allow the user to tailor the appearance and operation of the program without finding and editing actual program logic.

As is, this is a single-account configuration. We could generalize this module’s code to allow for multiple email accounts, selected by input at the console when first imported; in an upcoming section we’ll see a different approach that allows this module to be extended externally.

Example 14-9. PP4EInternetEmailPyMailGuimailconfig.py
"""
################################################################################
PyMailGUI user configuration settings.

Email scripts get their server names and other email config options from
this module: change me to reflect your machine names, sig, and preferences.
This module also specifies some widget style preferences applied to the GUI,
as well as message Unicode encoding policy and more in version 3.0.  See
also: local textConfig.py, for customizing PyEdit pop-ups made by PyMailGUI.

Warning: PyMailGUI won't run without most variables here: make a backup copy!
Caveat: somewhere along the way this started using mixed case inconsistently...;
TBD: we could get some user settings from the command line too, and a configure
dialog GUI would be better, but this common module file suffices for now.
################################################################################
"""

#-------------------------------------------------------------------------------
# (required for load, delete) POP3 email server machine, user;
#-------------------------------------------------------------------------------

#popservername = '?Please set your mailconfig.py attributes?'

popservername = 'pop.secureserver.net'             # see altconfigs/ for others
popusername   = '[email protected]'

#-------------------------------------------------------------------------------
# (required for send) SMTP email server machine name;
# see Python smtpd module for a SMTP server class to run locally ('localhost'),
# note: your ISP may require that you be directly connected to their system:
# I once could email through Earthlink on dial up, but not via Comcast cable;
#-------------------------------------------------------------------------------

smtpservername = 'smtpout.secureserver.net'

#-------------------------------------------------------------------------------
# (optional) personal information used by PyMailGUI to fill in edit forms;
# if not set, does not fill in initial form values;
# signature  -- can be a triple-quoted block, ignored if empty string;
# address    -- used for initial value of "From" field if not empty,
# no longer tries to guess From for replies--varying success;
#-------------------------------------------------------------------------------

myaddress   = '[email protected]'
mysignature = ('Thanks,
'
               '--Mark Lutz  (http://learning-python.com, http://rmi.net/~lutz)')

#-------------------------------------------------------------------------------
# (may be required for send) SMTP user/password if authenticated;
# set user to None or '' if no login/authentication is required, and set
# pswd to name of a file holding your SMTP password, or an empty string to
# force programs to ask (in a console, or GUI)
#-------------------------------------------------------------------------------

smtpuser  = None                           # per your ISP
smtppasswdfile  = ''                       # set to '' to be asked

#smtpuser = popusername

#-------------------------------------------------------------------------------
# (optional) PyMailGUI: name of local one-line text file with your POP
# password; if empty or file cannot be read, pswd is requested when first
# connecting; pswd not encrypted: leave this empty on shared machines;
# PyMailCGI always asks for pswd (runs on a possibly remote server);
#-------------------------------------------------------------------------------

poppasswdfile  = r'c:	emppymailgui.txt'      # set to '' to be asked

#-------------------------------------------------------------------------------
# (required) local file where sent messages are always saved;
# PyMailGUI 'Open' button allows this file to be opened and viewed;
# don't use '.' form if may be run from another dir: e.g., pp4e demos
#-------------------------------------------------------------------------------

#sentmailfile = r'.sentmail.txt'             # . means in current working dir

#sourcedir    = r'C:...PP4EInternetEmailPyMailGui'
#sentmailfile = sourcedir + 'sentmail.txt'

# determine automatically from one of my source files
import wraplines, os
mysourcedir   = os.path.dirname(os.path.abspath(wraplines.__file__))
sentmailfile  = os.path.join(mysourcedir, 'sentmail.txt')

#-------------------------------------------------------------------------------
# (defunct) local file where pymail saves POP mail (full text);
# PyMailGUI instead asks for a name in GUI with a pop-up dialog;
# Also asks for Split directory, and part buttons save in ./TempParts;
#-------------------------------------------------------------------------------

#savemailfile = r'c:	empsavemail.txt'       # not used in PyMailGUI: dialog

#-------------------------------------------------------------------------------
# (optional) customize headers displayed in PyMailGUI list and view windows;
# listheaders replaces default, viewheaders extends it; both must be tuple of
# strings, or None to use default hdrs;
#-------------------------------------------------------------------------------

listheaders = ('Subject', 'From', 'Date', 'To', 'X-Mailer')
viewheaders = ('Bcc',)

#-------------------------------------------------------------------------------
# (optional) PyMailGUI fonts and colors for text server/file message list
# windows, message content view windows, and view window attachment buttons;
# use ('family', size, 'style') for font; 'colorname' or hexstr '#RRGGBB' for
# color (background, foreground);  None means use defaults;  font/color of
# view windows can also be set interactively with texteditor's Tools menu;
# see also the setcolor.py example in the GUI part (ch8) for custom colors;
#-------------------------------------------------------------------------------

listbg   = 'indianred'                  # None, 'white', '#RRGGBB'
listfg   = 'black'
listfont = ('courier', 9, 'bold')       # None, ('courier', 12, 'bold italic')
                                        # use fixed-width font for list columns
viewbg     = 'light blue'               # was '#dbbedc'
viewfg     = 'black'
viewfont   = ('courier', 10, 'bold')
viewheight = 18                         # max lines for height when opened (20)

partfg   = None
partbg   = None

# see Tk color names: aquamarine paleturquoise powderblue goldenrod burgundy ....
#listbg = listfg = listfont = None
#viewbg = viewfg = viewfont = viewheight = None      # to use defaults
#partbg = partfg = None

#-------------------------------------------------------------------------------
# (optional) column at which mail's original text should be wrapped for view,
# reply, and forward;  wraps at first delimiter to left of this position;
# composed text is not auto-wrapped: user or recipient's mail tool must wrap
# new text if desired; to disable wrapping, set this to a high value (1024?);
#-------------------------------------------------------------------------------

wrapsz = 90

#-------------------------------------------------------------------------------
# (optional) control how PyMailGUI opens mail parts in the GUI;
# for view window Split actions and attachment quick-access buttons;
# if not okayToOpenParts, quick-access part buttons will not appear in
# the GUI, and Split saves parts in a directory but does not open them;
# verifyPartOpens used by both Split action and quick-access buttons:
# all known-type parts open automatically on Split if this set to False;
# verifyHTMLTextOpen used by web browser open of HTML main text part:
#-------------------------------------------------------------------------------

okayToOpenParts    = True      # open any parts/attachments at all?
verifyPartOpens    = False     # ask permission before opening each part?
verifyHTMLTextOpen = False     # if main text part is HTML, ask before open?

#-------------------------------------------------------------------------------
# (optional) the maximum number of quick-access mail part buttons to show
# in the middle of view windows; after this many, a "..." button will be
# displayed, which runs the "Split" action to extract additional parts;
#-------------------------------------------------------------------------------

maxPartButtons = 8             # how many part buttons in view windows


# *** 3.0 additions follow ***
#-------------------------------------------------------------------------------
# (required, for fetch) the Unicode encoding used to decode fetched full message
# bytes, and to encode and decode message text stored in text-mode save files; see
# the book's Chapter 13 for details: this is a limited and temporary approach to
# Unicode encodings until a new bytes-friendly email package parser is provided
# which can handle Unicode encodings more accurately on a message-level basis;
# note: 'latin1' (an 8-bit encoding which is a superset of 7-bit ascii) was
# required to decode message in some old email save files I had, not 'utf8';
#-------------------------------------------------------------------------------

fetchEncoding = 'latin-1'    # how to decode and store full message text (ascii?)

#-------------------------------------------------------------------------------
# (optional, for send) Unicode encodings for composed mail's main text plus all
# text attachments; set these to None to be prompted for encodings on mail send,
# else uses values here across entire session; default='latin-1' if GUI Cancel;
# in all cases, falls back on UTF-8 if your encoding setting or input does not
# work for the text being sent (e.g., ascii chosen for reply to non-ascii text,
# or non-ascii attachments); the email package is pickier than Python about
# names: latin-1 is known (uses qp MIME), but latin1 isn't (uses base64 MIME);
# set these to sys.getdefaultencoding() result to choose the platform default;
# encodings of text parts of fetched email are automatic via message headers;
#-------------------------------------------------------------------------------

mainTextEncoding       = 'ascii'   # main mail body text part sent (None=ask)
attachmentTextEncoding = 'ascii'   # all text part attachments sent (utf-8, latin-1)

#-------------------------------------------------------------------------------
# (optional, for send) set this to a Unicode encoding name to be applied to
# non-ASCII headers, as well as non-ASCII names in email addresses in headers,
# in composed messages when they are sent;  None means use the UTF-8 default,
# which should work for most use cases; email names that fail to decode are
# dropped (the address part is used);  note that header decoding is performed
# automatically for display, according to header content, not user setting;
#-------------------------------------------------------------------------------

headersEncodeTo = None     # how to encode non-ASCII headers sent (None=UTF-8)

#-------------------------------------------------------------------------------
# (optional) select text, HTML, or both versions of the help document;
# always shows one or the other: displays HTML if both of these are turned off
#-------------------------------------------------------------------------------

showHelpAsText = True      # scrolled text, with button for opening source files
showHelpAsHTML = True      # HTML in a web browser, without source file links

#-------------------------------------------------------------------------------
# (optional) if True, don't show a selected HTML text message part in a PyEdit
# popup too if it is being displayed in a web browser; if False show both, to
# see Unicode encoding name and effect in a  text widget (browser may not know);
#-------------------------------------------------------------------------------

skipTextOnHtmlPart = False       # don't show html part in PyEdit popup too

#-------------------------------------------------------------------------------
# (optional) the maximum number of mail headers or messages that will be
# downloaded on each load request; given this setting N, PyMailGUI fetches at
# most N of the most recently arrived mails; older mails outside this set are
# not fetched from the server, but are displayed as empty/dummy emails; if this
# is assigned to None (or 0), loads will have no such limit; use this if you
# have very many mails in your inbox, and your Internet or mail server speed
# makes full loads too slow to be practical; PyMailGUI also loads only
# newly-arrived headers, but this setting is independent of that feature;
#-------------------------------------------------------------------------------

fetchlimit = 50            # maximum number headers/emails to fetch on loads

#-------------------------------------------------------------------------------
# (optional) initial width, height of mail index lists (chars x lines);  just
# a convenience, since the window can be resized/expanded freely once opened;
#-------------------------------------------------------------------------------

listWidth = None           # None = use default 74
listHeight = None          # None = use default 15

#-------------------------------------------------------------------------------
# (optional, for reply) if True, the Reply operation prefills the reply's Cc
# with all original mail recipients, after removing duplicates and the new sender;
# if False, no CC prefill occurs, and the reply is configured to reply to the
# original sender only; the Cc line may always be edited later, in either case.
#-------------------------------------------------------------------------------

repliesCopyToAll = True   # True=reply to sender + all recipients, else sender

#end

textConfig: Customizing Pop-Up PyEdit Windows

The prior section’s mailconfig module provides user settings for tailoring the PyEdit component used to view and edit main mail text, but PyMailGUI also uses PyEdit to display other kinds of pop-up text, including raw mail text, some text attachments, and source code in its help system. To customize display for these pop ups, PyMailGUI relies on PyEdit’s own utility, which attempts to load a module like that in Example 14-10 from the client application’s own directory. By contrast, PyEdit’s Unicode settings are loaded from the single textConfig module in its own package’s directory since they are not expected to vary across a platform (see Chapter 11 for more details).

Example 14-10. PP4EInternetEmailPyMailGui extConfig.py
"""
customize PyEdit pop-up windows other than the main mail text component;
this module (not its package) is assumed to be on the path for these settings;
PyEdit Unicode settings come from its own package's textConfig.py, not this;
"""

bg = 'beige'                        # absent=white; colorname or RGB hexstr
fg = 'black'                        # absent=black;  e.g., 'beige', '#690f96'

# etc -- see PP4EGuiTextEditor	extConfig.py
# font = ('courier', 9, 'normal')
# height = 20                       # Tk default: 24 lines
# width  = 80                       # Tk default: 80 characters

PyMailGUIHelp: User Help Text and Display

Finally, Example 14-11 lists the module that defines the text displayed in PyMailGUI’s help pop up as one triple-quoted string, as well as a function for displaying the HTML rendition of this text. The HTML version of help itself is in a separate file not listed in full here but included in the book’s examples package.

In fact, I’ve omitted most of the help text string, too, to conserve space here (it spanned 11 pages in the prior edition, and would be longer in this one!). For the full story, see this module in the examples package, or run PyMailGUI live and click the help bar at the top of its main server list window to learn more about how PyMailGUI’s interface operates. If fact, you probably should; the help display may explain some properties of PyMailGUI not introduced by the demo and other material earlier in this chapter.

The HTML rendition of help includes section links, and is popped up in a web browser. Because the text version also is able to pop up source files and minimizes external dependencies (HTML fails if no browser can be located), both the text and HTML versions are provided and selected by users in the mailconfig module. Other schemes are possible (e.g., converting HTML to text by parsing as a fallback option), but they are left as suggested improvements.

Example 14-11. PP4EInternetPyMailGuiPyMailGuiHelp.py (partial)
"""
##########################################################################
PyMailGUI help text string and HTML display function;

History: this display began as an info box pop up which had to be
narrow for Linux; it later grew to use scrolledtext with buttons
instead; it now also displays an HTML rendition in a web browser;

2.1/3E: the help string is stored in this separate module to avoid
distracting from executable code.  As coded, we throw up this text
in a simple scrollable text box; in the future, we might instead
use an HTML file opened with a browser (use webbrowser module, or
run a "browser help.html" or DOS "start help.html" with os.system);

3.0/4E: the help text is now also popped up in a web browser in HTML
form, with lists, section links, and separators; see the HTML file
PyMailGuiHelp.html in the examples package for the simple HTML
translation of the help text string here, popped up in a browser;
both the scrolled text widget and HTML browser forms are currently
supported: change mailconfig.py to use the flavor(s) you prefer;
##########################################################################
"""


# new HTML help for 3.0/4E
helpfile = 'PyMailGuiHelp.html'     # see book examples package

def showHtmlHelp(helpfile=helpfile):
    """
    3.0: popup HTML version of help file in a local web browser via webbrowser;
    this module is importable, but html file might not be in current working dir
    """
    import os, webbrowser
    mydir = os.path.dirname(__file__)       # dir of this module's filename
    mydir = os.path.abspath(mydir)          # make absolute: may be .., etc
    webbrowser.open_new('file://' + os.path.join(mydir, helpfile))


##########################################################################
# string for older text display: client responsible for GUI construction
##########################################################################

helptext = """PyMailGUI, version 3.0
May, 2010 (2.1 January, 2006)
Programming Python, 4th Edition
Mark Lutz, for O'Reilly Media, Inc.

PyMailGUI is a multiwindow interface for processing email, both online and
offline.  Its main interfaces include one list window for the mail server,
zero or more list windows for mail save files, and multiple view windows for
composing or viewing emails selected in a list window.  On startup, the main
(server) list window appears first, but no mail server connection is attempted
until a Load or message send request.  All PyMailGUI windows may be resized,
which is especially useful in list windows to see additional columns.

Note: To use PyMailGUI to read and write email of your own, you must change
the POP and SMTP server names and login details in the file mailconfig.py,
located in PyMailGUI's source-code directory.  See section 11 for details.

Contents:
0)  VERSION ENHANCEMENTS
1)  LIST WINDOW ACTIONS
2)  VIEW WINDOW ACTIONS
3)  OFFLINE PROCESSING
4)  VIEWING TEXT AND ATTACHMENTS
5)  SENDING TEXT AND ATTACHMENTS
6)  MAIL TRANSFER OVERLAP
7)  MAIL DELETION
8)  INBOX MESSAGE NUMBER SYNCHRONIZATION
9)  LOADING EMAIL
10) UNICODE AND INTERNATIONALIZATION SUPPORT
11) THE mailconfig CONFIGURATION MODULE
12) DEPENDENCIES
13) MISCELLANEOUS HINTS ("Cheat Sheet")

...rest of file omitted...

13) MISCELLANEOUS HINTS ("Cheat Sheet")

- Use ',' between multiple addresses in To, Cc, and Bcc headers.
- Addresses may be given in the full '"name" <addr>' form.
- Payloads and headers are decoded on fetches and encoded on sends.
- HTML mails show extracted plain text plus HTML in a web browser.
- To, Cc, and Bcc receive composed mail, but no Bcc header is sent.
- If enabled in mailconfig, Bcc is prefilled with sender address.

- Reply and Fwd automatically quote the original mail text.
- If enabled, replies prefill Cc with all original recipients.
- Attachments may be added for sends and are encoded as required.
- Attachments may be opened after View via Split or part buttons.
- Double-click a mail in the list index to view its raw text.
- Select multiple mails to process as a set: Ctrl|Shift + click, or All.

- Sent mails are saved to a file named in mailconfig: use Open to view.
- Save pops up a dialog for selecting a file to hold saved mails.
- Save always appends to the chosen save file, rather than erasing it.
- Split asks for a save directory; part buttons save in ./TempParts.
- Open and save dialogs always remember the prior directory.
- Use text editor's Save to save a draft of email text being composed.

- Passwords are requested if/when needed, and not stored by PyMailGUI.
- You may list your password in a file named in mailconfig.py.
- To print emails, "Save" to a text file and print with other tools.
- See the altconfigs directory for using with multiple email accounts.

- Emails are never deleted from the mail server automatically.
- Delete does not reload message headers, unless it fails.
- Delete checks your inbox to make sure it deletes the correct mail.
- Fetches detect inbox changes and may automatically reload the index.
- Any number of sends and disjoint fetches may overlap in time.

- Click this window's Source button to view PyMailGUI source-code files.
- Watch http://learning-python.com/books for updates and patches
- This is an Open Source system: change its code as you like.
"""

if __name__ == '__main__':
    print(helptext)                   # to stdout if run alone
    input('Press Enter key')          # pause in DOS console pop ups

See the examples package for the HTML help file, the first few lines of which are shown in Example 14-12; it’s a simple translation of the module’s help text string (adding a bit more pizzazz to this page is left in the suggested exercise column).

Example 14-12. PP4EInternetPyMailGuiPyMailGuiHelp.html (partial)
<HTML>
<TITLE>PyMailGUI 3.0 Help</TITLE>
<!-- TO DO: add pictures, screen shots, and such --!>
<BODY>

<H1 align=center>PyMailGUI, Version 3.0</H1>
<P align=center>
<B><I>May, 2010 (2.1 January, 2006)</I></B><BR>
<B><I>Programming Python, 4th Edition</I></B><BR>
<B><I>Mark Lutz, for O'Reilly Media, Inc.</I></B>

<P>
<I>PyMailGUI</I> is a multiwindow interface for processing email, both online and
...rest of file omitted...

altconfigs: Configuring for Multiple Accounts

Though not an “official” part of the system, I use a few additional short files to launch and test it. If you have multiple email accounts, it can be inconvenient to change a configuration file every time you want to open one in particular. Moreover, if you open multiple PyMailGUI sessions for your accounts at the same time, it would be better if they could use custom appearance and behavior schemes to make them distinct.

To address this, the altconfigs directory in the examples source directory provides a simple way to select an account and configurations for it at start-up time. It defines a new top-level script which tailors the module import search path, along with a mailconfig that prompts for and loads a custom configuration module whose suffix is named by console input. A launcher script is also provided to run without module search path configurations—from PyGadgets or a desktop shortcut, for example, without requiring PYTHONPATH settings for the PP4E root. Examples 14-13 through 14-17 list the files involved.

Example 14-13. PP4EInternetPyMailGuialtconfigsPyMailGui.py
import sys                             # ..PyMailGui.py or 'book' for book configs
sys.path.insert(1, '..')               # add visibility for real dir
exec(open('../PyMailGui.py').read())   # do this, but get mailconfig here
Example 14-14. PP4EInternetPyMailGuialtconfigsmailconfig.py
above = open('../mailconfig.py').read()       # copy version above here (hack?)
open('mailconfig_book.py', 'w').write(above)  # used for 'book' and as others' base
acct = input('Account name?')                 # book, rmi, train
exec('from mailconfig_%s import *' % acct)    # . is first on sys.path
Example 14-15. PP4EInternetPyMailGuialtconfigsmailconfig_rmi.py
from mailconfig_book import *                 # get base in . (copied from ..)
popservername = 'pop.rmi.net'                 # this is a big inbox: 4800 emails!
popusername   = 'lutz'
myaddress     = '[email protected]'
listbg = 'navy'
listfg = 'white'
listHeight = 20         # higher initially
viewbg = '#dbbedc'
viewfg = 'black'
wrapsz = 80             # wrap at 80 cols
fetchlimit = 300        # load more headers
Example 14-16. PP4EInternetPyMailGuialtconfigsmailconfig_train.py
from mailconfig_book import *                 # get base in . (copied from ..)
popusername = '[email protected]'
myaddress   = '[email protected]'
listbg = 'wheat'                              # goldenrod, dark green, beige
listfg = 'navy'                               # chocolate, brown,...
viewbg = 'aquamarine'
viewfg = 'black'
wrapsz = 80
viewheaders = None      # no Bcc
fetchlimit = 100        # load more headers
Example 14-17. PP4EInternetPyMailGuialtconfigslaunch_PyMailGui.py
# to run without PYTHONPATH setup (e.g., desktop)
import os                                         # Launcher.py is overkill
os.environ['PYTHONPATH'] = r'..........'      # hmm; generalize me
os.system('PyMailGui.py')                         # inherits path env var

Account files like those in Examples 14-15 and 14-16 can import the base “book” module (to extend it) or not (to replace it entirely). To use these alternative account configurations, run a command line like the following or run the self-configuring launcher script in Example 14-17 from any location. Either way, you can open these account’s windows to view the included saved mails, but be sure to change configurations for your own email accounts and preferences first if you wish to fetch or send mail from these clients:

C:...PP4EInternetEmailPyMailGuialtconfigs> PyMailGui.py
Account name?rmi

Add a “start” to the beginning of this command to keep your console alive on Windows so you can open multiple accounts (try a “&” at the end on Unix). Figure 14-45 earlier shows the scene with all three of my accounts open in PyMailGUI. I keep them open perpetually on my desktop, since a Load fetches just newly arrived headers no matter how long the GUI may have sat dormant, and a Send requires nothing to be loaded at all. While they’re open, the alternative color schemes make the accounts’ windows distinct. A desktop shortcut to the launcher script makes opening my accounts even easier.

As is, account names are only requested when this special PyMailGui.py file is run directly, and not when the original file is run directly or by program launchers (in which case there may be no stdin to read). Extending a module like mailconfig which might be imported in multiple places this way turns out to be an interesting task (which is largely why I don’t consider its quick solution here to be an official end-user feature). For instance, there are other ways to allow for multiple accounts, including:

  • Changing the single mailconfig module in-place

  • Importing alternative modules and storing them as key “mailconfig” in sys.modules

  • Copying alternative module variables to mailconfig attributes using __dict__ and setattr

  • Using a class for configuration to better support customization in subclasses

  • Issuing a pop-up in the GUI to prompt for an account name after or before the main window appears

And so on. The separate subdirectory scheme used here was chosen to minimize impacts on existing code in general; to avoid changes to the existing mailconfig module specifically (which works fine for the single account case); to avoid requiring extra user input of any kind in single account cases; and to allow for the fact that an “import module1 as module2” statement doesn’t prevent “module1” from being imported directly later. This last point is more fraught with peril than you might expect—importing a customized version of a module is not merely a matter of using the “as” renaming extension:

import m1 as m2      # custom import: load m1 as m2 alternative
print(m2.attr)       # prints attr in m1.py

import m2            # later imports: loads m2.py anyhow!
print(m2.attr)       # prints attr in m2.py

In other words, this is a quick-and-dirty solution that I originally wrote for testing purposes, and it seems a prime candidate for improvement—along with the other ideas in the next section’s chapter wrap up.

Ideas for Improvement

Although I use the 3.0 version of PyMailGUI as is on a regular basis for both personal and business communications, there is always room for improvement to software, and this system is no exception. If you wish to experiment with its code, here are a few suggested projects to close out this chapter:

Column sorts and list layout

Mail list windows could be sorted by columns on demand. This may require a more sophisticated list window structure which presents columns more distinctly. The current display of mail lists seems like the most obvious candidate for cosmetic upgrade in general, and any column sorting solution would likely address this as well. tkinter extensions such as the Tix HList widget may show promise here, and the third-party TkinterTreectrl supports multicolumn sortable listboxes, too, but is available only for Python 2.X today; consult the Web and other resources for pointers and details.

Mail save file (and sent file) size

The implementation of save-mail files limits their size by loading them into memory all at once; a DBM keyed-access implementation may work around this constraint. See the list windows module comments for ideas. This also applies to sent-mail save files, though the user can limit their sizes with periodic deletions; users might also benefit from a prompt for deletions if they grow too large.

Embedded links

Hyperlink URLs within messages could be highlighted visually and made to spawn a web browser automatically when clicked by using the launcher tools we met in the GUI and system parts of this book (tkinter’s text widget supports links directly).

Help text redundancy

In this version, the help text had grown so large that it is also implemented as HTML and displayed in a web browser using Python’s webbrowser module (instead of or in addition to text, per mailconfig settings). That means there are currently two copies of the basic help text: simple text and HTML. This is less than ideal from a maintenance perspective going forward.

We may want to either drop the simple text version altogether, or attempt to extract the simple text from the HTML with Python’s html.parser module to avoid redundant copies; see Chapter 19 for more on HTML parsing in general, and see PyMailGUI’s new html2text module for a plain-text extraction tool prototype. The HTML help version also does not include links to display source files; these could be inserted into the HTML automatically with string formatting, though it’s not clear what all browsers will do with Python source code (some may try to run it).

More threading contexts

Message Save and Split file writes could also be threaded for worst-case scenarios. For pointers on making Saves parallel, see the comments in the file class of ListWindows.py; there may be some subtle issues that require both thread locks and general file locking for potentially concurrent updates. List window index fills might also be threaded for pathologically large mailboxes and woefully slow machines (optimizing to avoid reparsing headers may help here, too).

Attachment list deletes

There is currently no way to delete an attachment once it has been added in compose windows. This might be supported by adding quick-access part buttons to compose windows, too, which could verify and delete the part when clicked.

Spam filtering

We could add an automatic spam filter for mails fetched, in addition to any provided at the email server or ISP. The Python-based SpamBayes might help. This is often better implemented by servers than clients, but not all ISPs filter spam.

Improve multiple account usage

Per the prior section, the current system selects one of multiple email accounts and uses its corresponding mail configuration module by running special code in the altconfigs subdirectory. This works for a book example, but it would be fairly straightforward to improve for broader audiences.

Increased visibility for sent file

We may want to add an explicit button for opening the sent-mails file. PyMailGUI already does save sent messages to a text file automatically, which may be opened currently with the list window’s Open button. Frankly, though, this feature may be a too-well-kept secret—I forgot about it myself when I revisited the program for this edition! It might also be useful to allow sent-mail saves to be disabled in mailconfig for users who might never delete from this file (it can grow large fairly quickly; see the earlier prompt-for-deletion suggestion as well).

Thread queue speed tuning

As mentioned when describing version 3.0 changes, the thread queue has been sped up by as much as a factor of 10 in this version to quicken initial header downloads. This is achieved both by running more than one callback per timer event and scheduling timer events to occur twice as often as before. Checking the queue too often, however, might increase CPU utilization beyond acceptable levels on some machines. On my Windows laptop, this overhead is negligible (the program’s CPU utilization is 0% when idle), but you may want to tune this if it’s significant on your platform.

See the list windows code for speed settings, and threadtools.py in Chapter 10 for the base code. In general, increasing the number of callbacks per event and decreasing timer frequency will decrease CPU drain without sacrificing responsiveness. (And if I had a nickel for every time I said that…)

Mailing lists

We could add support for mailing lists, allowing users to associate multiple email addresses with a saved list name. On sends to a list name, the mail would be sent to all on the list (the To addresses passed to smtplib), but the email list could be used for the email’s To header line. See Chapter 13’s SMTP coverage for mailing list–related examples.

HTML main text views and edits

PyMailGUI is still oriented toward supporting only plain text for the main text of a message, despite the fact that some mailers today are more HTML-biased in this regard. This partly stems from the fact that PyMailGUI uses a simple tkinter Text widget for main text composition. PyMailGUI can display such messages’ HTML in a popped-up web browser, and it attempts to extract text from the HTML for display per the next note, but it doesn’t come with its own HTML editor. Fully supporting HTML for main message text will likely require a tkinter extension (or, regrettably, a port to another GUI toolkit with working support for this feature).

HTML parser honing

On a related note, as described earlier, this version includes a simple-minded HTML parser, applied to extract text from HTML main (or only) text parts when they are displayed or quoted in replies and forwards. As also mentioned earlier, this parser is nowhere near complete or robust; for production-level quality, this would have to be improved by testing over a large set of HTML emails. Better yet, watch for a Python 3.X–compatible version of more robust and complete open source alternatives, such as the html2text.py same-named third-party utility described in this chapter’s earlier note. The open source BeautifulSoup system offers another lenient and forgiving HTML parser, but is based on SGMLParser tools available in 2.X only (removed in 3.X).

Text/HTML alternative mails

Also in the HTML department, there is presently no support for sending both text and HTML versions of a mail as a MIME multipart/alternative message—a popular scheme which supports both text- and HTML-based clients and allows users to choose which to use. Such messages can be viewed (both parts are offered in the GUI), but cannot be composed. Again, since there is no support for HTML editing anyhow, this is a moot point; if such an editor is ever added, we’d need to support this sort of mail structure in mailtools message object construction code and refactor parts of its current send logic so that it can be shared.

Internationalized headers throw list columns off

As is so often true in software, one feature added in this version broke another already present: the fonts used for display of some non-ASCII Unicode header fields is large enough to throw off the fixed-width columns in mail index list windows. They rely on the assumption that N characters is always the same width among all mails, and this is no longer true for some Chinese and other character set encodings.

This isn’t a showstopper—it only occurs when some i18n headers are displayed, and simply means that “|” column separators are askew for such mails only, but could still be addressed. The fix here is probably to move to a more sophisticated list display, and might be resolved as a side effect of allowing for the column sorts described earlier.

Address books

PyMailGUI has no notion of automatically filling in an email address from an address book, as many modern email clients do. Adding this would be an interesting extension; low-level keyboard event binding may allow matching as addresses are typed, and Python’s pickle and shelve modules of Chapters 1 and 17 might come in handy for data storage.

Spelling checker

There is currently no spelling checker of the sort most email programs have today. This could be added in PyMailGUI, but it would probably be more appropriate to add it in the PyEdit text edit component/program that it uses, so the spell-checking would be inherited by all PyEdit clients. A quick web search reveals a variety of options, including the interesting PyEnchant third-party package, none of which we have space to explore here.

Mail searches

Similarly, there is no support for searching emails’ content (headers or bodies) for a given string. It’s not clear how this should be provided given that the system fetches and caches just message headers until a mail is requested, but searching large inboxes can be convenient. As is, this can be performed manually by running a Save to store fetched mails in a text file and searching in that file externally.

Frozen binary distribution

As a desktop program, PyMailGUI seems an ideal candidate for packing as a self-contained frozen binary executable, using tools such as PyInstaller, Py2Exe, and others. When distributed this way, users need not install Python, since the Python runtime is embedded in the executable.

Selecting Reply versus Reply-All in the GUI

As described in the 3.0 changes overview earlier, in this version, Reply by default now copies all the original mail’s recipients by prefilling the Cc line, in addition to replying to the original sender. This Cc feature can be turned off in mailconfig because it may not be desirable in all cases. Ideally, though, this should be selectable in the GUI on a mail-by-mail basis, not per session. Adding another button to list windows for ReplyAll would suffice; since this feature was added too late in this project for GUI changes, though, this will have to be relegated to the domain of suggested exercise.

Propagating attachments?

When replying to or forwarding an email, PyMailGUI discards any attachments on the original message. This is by design, partly because there is currently no way to delete attached parts in the GUI prior to sending (you couldn’t remove selectively and couldn’t remove all), and partly because this system’s current sole user prefers to work this way.

Users can work around this by running a Split to save all parts in a directory, and then adding any desired attachments to the mail from there. Still, it might be better to allow the user to choose that this happen automatically for replies and forwards. Similarly, forwarding HTML mails well currently requires saving and attaching the HTML part to avoid quoting the text; this might be similarly addressed by parts propagation in general.

Disable editing for viewed mails?

Mail text is editable in message view windows, even though a new mail is not being composed. This is deliberate—users can annotate the message’s text and save it in a text file with the Save button at the bottom of the window, or simply cut-and-paste portions of it into other windows. This might be confusing, though, and is redundant (we can also edit and save by clicking on the main text’s quick-access part button). Removing edit tools would require extending PyEdit. Using PyEdit for display in general is a useful design—users also have access to all of PyEdit’s tools for the mail text, including save, find, goto, grep, replace, undo/redo, and so, though edits might be superfluous in this context.

Automatic periodic new mail check?

It would be straightforward to add the ability to automatically check for and fetch new incoming email periodically, by registering long-duration timer events with either the after widget method or the threading module’s timer object. I haven’t done so because I have a personal bias against being surprised by software, but your mileage may vary.

Reply and Forward buttons on view windows, too?

Minor potential ergonomic improvement: we could include Reply and Forward buttons on the message view windows, too, instead of requiring these operations to be selected in mail list windows only. As this system’s sole user, I prefer the uncluttered appearance and conceptual simplicity of the current latter approach; GUIs have a way of getting out of hand when persistent pop-up windows start nesting too deeply. It would be trivial to have Reply/Forward on view windows, too, though; they could probably fetch mail components straight from the GUI instead of reparsing a message.

Omit Bcc header in view windows?

Minor nit: mail view windows may be better off omitting the Bcc header even if it’s enabled in the configuration file. Since it shouldn’t be present once a mail is sent, it really needs to be included in composition windows only. It’s displayed as is anyhow, to verify that Bcc is omitted on sends (the prior edition did not), to maintain a uniform look for all mail windows, to avoid special-casing this in the code, and to avoid making such ergonomic decisions in the absence of actual user feedback.

Check for empty Subject lines?

Minor usability issue: it would be straightforward to add a check for an empty Subject field on sends and to pop up a verification dialog to give the user a second chance to fill the field in. A blank subject is probably unintended. We could do the same for the To field as well, though there may be valid use cases for omitting this from mail headers (the mail is still sent to Cc and Bcc recipients).

Removing duplicate recipients more accurately?

As is, the send operation attempts to remove duplicate recipients using set operations. This works, but it may be inaccurate if the same email address appears twice with a different name component (e.g., “name1 <eml>, name2 <eml>”). To do better, we could fully parse the recipient addresses to extract and compare just the address portion of the full email address. Arguably, though, it’s not clear what should be done if the same recipient address appears with different names. Could multiple people be using the same email account? If not, which name should we choose to use?

For now, end user or mail server intervention may be required in the rare cases where this might crop up. In most cases, other email clients will likely handle names in consistent ways that make this a moot point. On related notes, Reply removes duplicates in Cc prefills in the same simplistic way, and both sends and replies could use case-insensitive string comparisons when filtering for duplicates.

Handling newsgroup messages, too?

Because Internet newsgroup posts are similar in structure to emails (header lines plus body text; see the nntplib example in Chapter 13), this script could in principle be extended to display both email messages and news articles. Classifying such a mutation as clever generalization or diabolical hack is left as an exercise in itself.

SMTP sends may not work in some network configurations?

On my home/office network, SMTP works fine and as shown for sending emails, but I have occasionally seen sends have issues on public networks of the sort available in hotels and airports. In some cases, mail sends can fail with exceptions and error messages in the GUI; in worst cases, such sends might fail with no exception at all and without reporting an error in the GUI. The mail simply goes nowhere, which is obviously less than ideal if its content matters.

It’s not clear if these issues are related to limitations of the networks used, of Python’s smtplib, or of the ISP-provided SMTP server I use. Unfortunately, I ran out of time to recreate the problem and investigate further (again, a system with a single user also has just a single tester).

Resolving any such issues is left as an exercise for the reader, but as a caution: if you wish to use the system to send important emails, you should first test sends in a new network environment to ensure that they will be routed correctly. Sending an email to yourself and verifying receipt should suffice.

Performance tuning?

Almost all of the work done on this system to date has been related to its functionality. The system does allow some operation threads to run in parallel, and optimizes mail downloads by fetching just headers initially and caching already-fetched full mail text to avoid refetching. Apart from this, though, its performance in terms of CPU utilization and memory requirements has not been explored in any meaningful way at all. That’s for the best—in general we code for utility and clarity first in Python, and deal with performance later if and only if needed. Having said that, a broader audience for this program might mandate some performance analysis and improvement.

For example, although the full text of fetched mails is kept just once in a cache, each open view of a message retains a copy of the parsed mail in memory. For large mails, this may impact memory growth. Caching parsed mails as well might help decrease memory footprints, though these will still not be small for large mails, and the cache might hold onto memory longer than required if not intelligently designed. Storing messages or their parts in files (perhaps as pickled objects) instead of in memory might alleviate some growth, too, though that may also require a mechanism for reaping temporary files. As is, Python’s garbage collector should reclaim all such message space eventually as windows are closed, but this can depend upon how and where we retain object references. See also the gc standard library modules for possible pointers on finer-grained garbage collection control.

Unicode model tuning?

As discussed in brief at the start of this chapter and in full in Chapter 13, PyMailGUI’s support for Unicode encoding of message text and header components is broad, but not necessarily as general or universally applicable as it might be. Some Unicode limitations here stem from the limitations of the email package in Python 3.1 upon which PyMailGUI heavily depends. It may be difficult for Python-coded email clients to support some features better until Python’s libraries do, too.

Moreover, the Unicode support that is present in this program has been tested neither widely nor rigorously. Just like Chapter 11’s PyEdit, this is currently still a single-user system designed to work as a book example, not an open source project. Because of that, some of the current Unicode policies are partially heuristic in nature and may have to be honed with time and practice.

For example, it may prove better in the end to use UTF-8 encoding (or none at all) for sends in general, instead of supporting some of the many user options which are included in this book for illustration purposes. Since UTF-8 can represent most Unicode code points, it’s broadly applicable.

More subtly, we might also consider propagating the main text part’s Unicode encoding to the embedded PyEdit component in view and edit windows, so it can be used as a known encoding by the PyEdit Save button. As is, users can pop up the main text’s part in view windows to save with a known encoding automatically, but saves of drafts for mails being edited fall back on PyEdit’s own Unicode policies and GUI prompts. The ambiguous encoding for saved drafts may be unavoidable, though—users might enter characters from any character set, both while writing new mails from scratch and while editing the text of replies and forwards (just like headers in replies and forwards, the initial known encoding of the original main text part may no longer apply after arbitrary edits).

In addition, there is no support for non-ASCII encodings of full mail text, it’s not impossible that i18n encoded text might appear in other contexts in rare emails (e.g., in attachment filenames, whose undecoded form may or may not be valid on the receiving platform’s filesystem, and may require renaming if allowed at all), and although Internationalization is supported for mail content, the GUI itself still uses English for its buttons, labels, and titles—something that a truly location-neutral program may wish to address.

In other words, if this program were to ever take the leap to commercial-grade or broadly used software, its Unicode story would probably have to be revisited. Also discussed in Chapter 13, a future release of the email package may solve some Unicode issues automatically, though PyMailGUI may also require updates for the solutions, as well as for incompatibilities introduced by them. For now, this will have to stand as a useful object lesson in itself: for both better and worse, such changes will always be a fact of life in the constantly evolving world of software development.

And so on—because this software is open source, it is also necessarily open-ended. Ultimately, writing a complete email client is a substantial undertaking, and we’ve taken this example as far as we can in this book. To move PyMailGUI further along, we’d probably have to consider the suitability of both the underlying Python 3.1 email package, as well as the tkinter GUI toolkit. Both are fully sufficient for the utility we’ve implemented here, but they might limit further progress.

For example, the current lack of an HTML viewer widget in the base tkinter toolkit precludes HTML mail viewing and composition in the GUI itself. Moreover, although PyMailGUI broadly supports Internationalization today, it must rely on workarounds to get email to work at all. To be fair, some of the email package’s issues described in this book will likely be fixed by the time you read about them, and email in general is probably close to a worst case for Internationalization issues brought into the spotlight by Unicode prominence in Python 3.X. Still, such tool constraints might impede further system evolution.

On the other hand, despite any limitations in the tools it deploys, PyMailGUI does achieve all its goals—it’s an arguably full-featured and remarkably quick desktop email client, which works surprisingly well for my emails and preferences and performs admirably on the cases I’ve tested to date. It may not satisfy your tastes or constraints, but it is open to customization and imitation. Suggested exercises and further tweaking are therefore officially delegated to your imagination; this is Python, after all.

This concludes our tour of Python client-side protocols programming. In the next chapter, we’ll hop the fence to the other side of the Internet world and explore scripts that run on server machines. Such programs give rise to the grander notion of applications that live entirely on the Web and are launched by web browsers. As we take this leap in structure, keep in mind that the tools we met in this and the previous chapter are often sufficient to implement all the distributed processing that many applications require, and they can work in harmony with scripts that run on a server. To completely understand the Web world view, though, we need to explore the server realm, too.



[54] And remember: you would have to multiply these line counts by a factor of four or more to get the equivalent in a language like C or C++. If you’ve done much programming, you probably recognize that the fact that we can implement a fairly full-featured mail processing program in roughly 5,000 total lines of program code speaks volumes about the power of the Python language and its libraries. For comparison, the original 1.0 version of this program from the second edition of this book was just 745 total lines in 3 new modules, but it also was very limited—it did not support PyMailGUI 2.X’s attachments, thread overlap, local mail files, and so on, and did not have the Internationalization support or other features of this edition’s PyMailGUI 3.X.

[55] In fact, my ISP’s webmail send system went down the very day I had to submit the third edition of this book to my publisher! No worries—I fired up PyMailGUI and used it to send the book as attachment files through a different server. In a sense, this book submitted itself.

[56] Actually, the help display started life even less fancy: it originally displayed help text in a standard information pop up common dialog, generated by the tkinter showinfo call used earlier in the book. This worked fine on Windows (at least with a small amount of help text), but it failed on Linux because of a default line-length limit in information pop-up boxes; lines were broken so badly as to be illegible. Over the years, common dialogs were replaced by scrolled text, which has now been largely replaced by HTML; I suppose the next edition will require a holographic help interface…

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset