Chapter 21. Conclusion: Python and the Development Cycle
“That’s the End of the Book, Now Here’s the Meaning of Life”
Well, the meaning of Python, anyway. In the Preface of this book, I promised that we’d return to the issue of Python’s roles after seeing how it is used in practice. So in closing, here are some subjective comments on the broader implications of the language. Most of this conclusion remains unchanged since the first edition of this book was penned in 1995, but so are the factors that pushed Python into the development spotlight.
As I mentioned in the Preface sidebar, Python’s focus is on concepts such as quality, productivity, portability, and integration. I hope that this book has demonstrated some of the benefits of that focus in action. Along the way, we’ve seen Python applied to systems programming, GUI development, Internet scripting, database and text processing, and more. And we’ve witnessed firsthand the application of the language and its libraries to realistically scaled software development tasks, which go above and beyond what many classify as “scripting.” I hope you’ve also had some fun; that, too, is part of the Python story.
In this conclusion, I wish now to return to the forest after our long walk among the trees—to revisit Python’s roles in more concrete terms. In particular, Python’s role as a prototyping tool can profoundly affect the development cycle.
“Something’s Wrong with the Way We Program Computers”
This has to be one of the most overused lines in the business. Still, given time to ponder the big picture, most of us would probably agree that we’re not quite there yet. Over the last few decades, the computer software industry has made significant progress on streamlining the development task (anyone remember dropping punch cards?). But at the same time, the cost of developing potentially useful computer applications is often still high enough to make them impractical.
Moreover, systems built using modern tools and paradigms are often delivered far behind schedule. Software engineering remains largely defiant of the sort of quantitative measurements employed in other engineering fields. In the software world, it’s not uncommon to take one’s best time estimate for a new project and multiply by a factor of two or three to account for unforeseen overheads in the development task. This situation is clearly unsatisfactory for software managers, developers, and end users.
The “Gilligan Factor”
It has been suggested, tongue in cheek, that if there were a patron saint of software engineers, the honor would fall on none other than Gilligan, the character in the pervasively popular American television show of the 1960s, Gilligan’s Island. Gilligan is the enigmatic, sneaker-clad first mate, widely held to be responsible for the shipwreck that stranded the now-residents of the island.
To be sure, Gilligan’s situation seems oddly familiar. Stranded on a desert island with only the most meager of modern technological comforts, Gilligan and his cohorts must resort to scratching out a living using the resources naturally available. In episode after episode, we observe the Professor developing exquisitely intricate tools for doing the business of life on their remote island, only to be foiled in the implementation phase by the ever-bungling Gilligan.
But clearly it was never poor Gilligan’s fault. How could one possibly be expected to implement designs for such sophisticated applications as home appliances and telecommunications devices, given the rudimentary technologies available in such an environment? He simply lacked the proper tools. For all we know, Gilligan may have had the capacity for engineering on the grandest level. But you can’t get there with bananas and coconuts.
And pathologically, time after time, Gilligan wound up inadvertently sabotaging the best of the Professor’s plans: misusing, abusing, and eventually destroying his inventions. If he could just pedal his makeshift stationary bicycle faster and faster (he was led to believe), all would be well. But in the end, inevitably, the coconuts were sent hurling into the air, the palm branches came crashing down around his head, and poor Gilligan was blamed once again for the failure of the technology.
Dramatic though this image may be, some observers would consider it a striking metaphor for the software industry. Like Gilligan, we software engineers are often asked to perform tasks with arguably inappropriate tools. Like Gilligan, our intentions are sound, but technology can hold us back. And like poor Gilligan, we inevitably must bear the brunt of management’s wrath when our systems are delivered behind schedule. You can’t get there with bananas and coconuts…
Doing the Right Thing
Of course, the Gilligan factor is an exaggeration, added for comic effect. But few would argue that the bottleneck between ideas and working systems has disappeared completely. Even today, the cost of developing software far exceeds the cost of computer hardware. And when software is finally delivered, it often comes with failure rates that would be laughable in other engineering domains. Why must programming be so complex?
Let’s consider the situation carefully. By and large, the root of the complexity in developing software isn’t related to the role it’s supposed to perform—usually this is a well-defined, real-world process. Rather, it stems from the mapping of real-world tasks onto computer-executable models. And this mapping is performed in the context of programming languages and tools.
The path toward easing the software bottleneck must therefore lie, at least partially, in optimizing the act of programming itself by deploying the right tools. Given this realistic scope, there’s much that can be done now—there are a number of purely artificial overheads inherent in our current tools.
The Static Language Build Cycle
Using traditional static languages, there is an unavoidable overhead in moving from coded programs to working systems: compile and link steps add a built-in delay to the development process. In some environments, it’s common to spend many hours each week just waiting for a static language application’s build cycle to finish. Given that modern development practice involves an iterative process of building, testing, and rebuilding, such delays can be expensive and demoralizing (if not physically painful).
Of course, this varies from shop to shop, and in some domains the demand for performance justifies build-cycle delays. But I’ve worked in C++ environments where programmers joked about having to go to lunch whenever they recompiled their systems. Except they weren’t really joking.
Artificial Complexities
With many traditional programming tools, you can easily lose focus: the very act of programming becomes so complex that the real-world goal of the program is obscured. Traditional languages divert valuable attention to syntactic issues and development of bookkeeping code. Obviously, complexity isn’t an end in itself; it must be clearly warranted. Yet some of our current tools are so complex that the language itself makes the task harder and lengthens the development process.
One Language Does Not Fit All
Many traditional languages implicitly encourage homogeneous, single-language systems. By making integration complex, they impede the use of multiple-language tools. As a result, instead of being able to select the right tool for the task at hand, developers are often compelled to use the same language for every component of an application. Since no language is good at everything, this constraint inevitably sacrifices both product functionality and programmer productivity.
Until our machines are as clever at taking directions as we are (arguably, not the most rational of goals), the task of programming won’t go away. But for the time being, we can make substantial progress by making the mechanics of that task easier. This topic is what I want to talk about now.
Enter Python
If this book has achieved its goals, you should by now have a good understanding of why Python has been called a “next-generation scripting language.” Compared with similar tools, it has some critical distinctions that we’re finally in a position to summarize:
- Tcl
Like Tcl, Python can be used as an embedded extension language. Unlike Tcl, Python is also a full-featured programming language. For many, Python’s data structure tools and support for programming-in-the-large make it useful in more domains. Tcl demonstrated the utility of integrating interpreted languages with C modules. Python provides similar functionality plus a powerful, object-oriented language; it’s not just a command string processor.
- Perl
Like Perl, Python can be used for writing shell tools, making it easy to use for system services. Unlike Perl, Python has a simple, readable syntax and a remarkably coherent design. For some, this makes Python easier to use and a better choice for programs that must be reused or maintained by others. Without question, Perl is a powerful system administration tool. But once we move beyond processing text and files, Python’s features become attractive.
- Scheme/Lisp
Like Scheme (and Lisp), Python supports dynamic typing, incremental development, and metaprogramming; it exposes the interpreter’s state and supports runtime program construction. Unlike Lisp, Python has a procedural syntax that is familiar to users of mainstream languages such as C and Pascal. If extensions are to be coded by end users, this can be a major advantage.
- Smalltalk
Like Smalltalk, Python supports object-oriented programming (OOP) in the context of a highly dynamic language. Unlike Smalltalk, Python doesn’t extend the object system to include fundamental program control flow constructs. Users need not come to grips with the concept of
ifstatements as message-receiving objects to use Python—Python is more conventional.- Icon
Like Icon, Python supports a variety of high-level datatypes and operations such as lists, dictionaries, and slicing. In more recent times, Python’s notions of iteration and generators approach Icon’s backtracking in utility. Unlike Icon, Python is fundamentally simple. Programmers (and end users) don’t need to master esoteric concepts such as full-blown backtracking just to get started.
- BASIC
Like modern structured BASIC dialects, Python has an interpretive/interactive nature. Unlike most BASICs, Python includes standard support for advanced programming features such as classes, modules, exceptions, high-level datatypes, and general C integration. And unlike Visual Basic, Python provides a cross-platform solution, which is not controlled by a commercially vested company.
- Java
Like Java, Python is a general-purpose language; supports OOP, exceptions, and modular design; and compiles to a portable bytecode format. Unlike Java, Python’s simple syntax and built-in datatypes make development much more rapid. Python programs are typically one-third to one-fifth the size of the equivalent Java program.
- C/C++
Like C and C++, Python is a general-purpose language and can be used for long-term strategic system development tasks. Unlike compiled languages in general, Python also works well in tactical mode, as a rapid development language. Python programs are smaller, simpler, and more flexible than those written in compiled languages. For instance, because Python code does not constrain datatypes or sizes, it both is more concise and can be applied in a broader range of contexts.
All of these languages (and others) have merit and unique strengths of their own—in fact, Python borrowed most of its features from languages such as these. It’s not Python’s goal to replace every other language; different tasks require different tools, and mixed-language development is one of Python’s main ideas. But Python’s blend of advanced programming constructs and integration tools make it a natural choice for the problem domains we’ve talked about in this book, and many more.
But What About That Bottleneck?
Back to our original question: how can the act of writing software be made easier? At some level, Python is really “just another computer language.” It’s certainly true that Python the language doesn’t represent much that’s radically new from a theoretical point of view. So why should we be excited about Python when so many languages have been tried already?
What makes Python of interest, and what may be its larger contribution to the development world, is not its syntax or semantics, but its worldview: Python’s combination of tools makes rapid development a realistic goal. In a nutshell, Python fosters rapid development by providing features like these:
Fast build-cycle turnaround
A very high-level, object-oriented language
Integration facilities to enable mixed-language development
Specifically, Python attacks the software development bottleneck on four fronts, described in the following sections.
Python Provides Immediate Turnaround
Python’s development cycle is dramatically shorter than that of traditional tools. In Python, there are no compile or link steps—Python programs simply import modules at runtime and use the objects they contain. Because of this, Python programs run immediately after changes are made. And in cases where dynamic module reloading can be used, it’s even possible to change and reload parts of a running program without stopping it at all. Figure 21-1 shows Python’s impact on the development cycle.
Because Python is interpreted, there’s a rapid turnaround after program changes. And because Python’s parser is embedded in Python-based systems, it’s easy to modify programs at runtime. For example, we saw how GUI programs developed with Python allow developers to change the code that handles a button press while the GUI remains active; the effect of the code change may be observed immediately when the button is pressed again. There’s no need to stop and rebuild.
More generally, the entire development process in Python is an exercise in rapid prototyping. Python lends itself to experimental and interactive program development, and it encourages developing systems incrementally by testing components in isolation and putting them together later. In fact, we’ve seen that we can switch from testing components (unit tests) to testing whole systems (integration tests) arbitrarily, as illustrated in Figure 21-2.
Python Is “Executable Pseudocode”
Python’s very high-level nature means there’s less for us to program and manage. The lack of compile and link steps isn’t really enough to address the development cycle bottleneck by itself. For instance, a C or C++ interpreter might provide fast turnaround but would still be almost useless for rapid development: the language is too complex and low level.
But because Python is also a simple language, coding is dramatically faster, too. For example, its dynamic typing, built-in objects, and garbage collection eliminate much of the manual bookkeeping code required in lower-level languages such as C and C++. Since things such as type declarations, memory management, and common data structure implementations are conspicuously absent, Python programs are typically a fraction of the size of their C and C++ equivalents. There’s less to write and read, and there are fewer interactions among language components, and thus there is less opportunity for coding errors.
Because most bookkeeping code is missing, Python programs are easier to understand and more closely reflect the actual problem they’re intended to address. And Python’s high-level nature not only allows algorithms to be realized more quickly but also makes it easier to learn the language.
Python Is OOP Done Right
For OOP to be useful, it must be easy to apply. Python makes OOP a flexible tool by delivering it in a dynamic language. More importantly, its class mechanism is a simplified subset of C++’s; this simplification is what makes OOP useful in the context of a rapid-development tool. For instance, when we looked at data structure classes in this book, we saw that Python’s dynamic typing let us apply a single class to a variety of object types; we didn’t need to write variants for each supported type. In exchange for not constraining types, Python code becomes flexible and agile.
In fact, Python’s OOP is so easy to use that there’s really no reason not to apply it in most parts of an application. Python’s class model has features powerful enough for complex programs, yet because they’re provided in simple ways, they do not interfere with the problem we’re trying to solve.
Python Fosters Hybrid Applications
As we’ve seen earlier in this book, Python’s extending and embedding support makes it useful in mixed-language systems. Without good integration facilities, even the best rapid-development language is a “closed box” and is not generally useful in modern development environments. But Python’s integration tools make it usable in hybrid, multicomponent applications. As one consequence, systems can simultaneously utilize the strengths of Python for rapid development and of traditional languages such as C for rapid execution.
While it’s possible and common to use Python as a standalone tool, it doesn’t impose this mode. Instead, Python encourages an integrated approach to application development. By supporting arbitrary mixtures of Python and traditional languages, Python fosters a spectrum of development paradigms, ranging from pure prototyping to pure efficiency. Figure 21-3 shows the abstract case.
As we move to the left extreme of the spectrum, we optimize speed of development. Moving to the right side optimizes speed of execution. And somewhere in between is an optimum mix for any given project. With Python, not only can we pick the proper mix for our project, but we can also later move the RAD slider in the picture arbitrarily as our needs change:
- Going to the right
Projects can be started on the left end of the scale in Python and gradually moved toward the right, module by module, as needed to optimize performance for delivery.
- Going to the left
Similarly, we can move strategic parts of existing C or C++ applications on the right end of the scale to Python, to support end-user programming and customization on the left end of the scale.
This flexibility of development modes is crucial in realistic environments. Python is optimized for speed of development, but that alone isn’t always enough. By themselves, neither C nor Python is adequate to address the development bottleneck; together, they can do much more. As shown in Figure 21-4, for instance, apart from standalone use, one of Python’s most common roles splits systems into frontend components that can benefit from Python’s ease of use and backend modules that require the efficiency of static languages such as C, C++, or FORTRAN.
Whether we add Python frontend interfaces to existing systems or design them in early on, such a division of labor can open up a system to its users without exposing its internals.
When developing new systems, we also have the option of writing entirely in Python at first and then optimizing as needed for delivery by moving performance-critical components to compiled languages. And because Python and C modules look the same to clients, migration to compiled extensions is transparent.
Prototyping doesn’t make sense in every scenario. Sometimes splitting a system into a Python frontend and a C/C++ backend up front works best. And prototyping doesn’t help much when enhancing existing systems. But where it can be applied, early prototyping can be a major asset. By prototyping in Python first, we can show results more quickly. Perhaps more critically, end users can be closely involved in the early stages of the process, as sketched in Figure 21-5. The result is systems that more closely reflect their original requirements.
On Sinking the Titanic
In short, Python is really more than a language; it implies a development philosophy. The concepts of prototyping, rapid development, and hybrid applications certainly aren’t new. But while the benefits of such development modes are widely recognized, there has been a lack of tools that make them practical without sacrificing programming power. This is one of the main gaps that Python’s design fills: Python provides a simple but powerful rapid development language, along with the integration tools needed to apply it in realistic development environments.
This combination arguably makes Python unique among similar tools. For instance, Tcl is a good integration tool but not a full-blown language; Perl is a powerful system administration language but a weak integration tool. But Python’s marriage of a powerful dynamic language and integration opens the door to fundamentally faster development modes. With Python, it’s no longer necessary to choose between fast development and fast execution.
By now, it should be clear that a single programming language can’t satisfy all our development goals. Our needs are sometimes contradictory: the goals of efficiency and flexibility will probably always clash. Given the high cost of making software, the choice between development and execution speed is crucial. Although machine cycles are cheaper than programmers, we can’t yet ignore efficiency completely.
But with a tool like Python, we don’t need to decide between the two goals at all. Just as a carpenter wouldn’t drive a nail with a chainsaw, software engineers are now empowered to use the right tool for the task at hand: Python when speed of development matters, compiled languages when efficiency dominates, and combinations of the two when our goals are not absolute.
Moreover, we don’t have to sacrifice code reuse or rewrite exhaustively for delivery when applying rapid development with Python. We can have our rapid development cake and eat it too:
- Reusability
Because Python is a high-level, object-oriented language, it encourages writing reusable software and well-designed systems.
- Deliverability
Because Python is designed for use in mixed-language systems, we don’t have to move to more efficient languages all at once.
In many scenarios, a system’s frontend and infrastructure may be written in Python for ease of development and modification, but the kernel is still written in C or C++ for efficiency. Python has been called the tip of the iceberg in such systems—the part visible to end users of a package, as captured in Figure 21-6.
Such an architecture uses the best of both worlds: it can be extended by adding more Python code or by writing C extension modules, depending on performance requirements. But this is just one of many mixed-language development scenarios:
- System interfaces
Packaging libraries as Python extension modules makes them more accessible.
- End-user customization
Delegating logic to embedded Python code provides for onsite changes.
- Pure prototyping
Python prototypes can be moved to C all at once or piecemeal.
- Legacy code migration
Moving existing code from C to Python makes it simpler and more flexible.
- Standalone use
Of course, using Python all by itself leverages its existing library of tools.
Python’s design lets us apply it in whatever way makes sense for each project.
“So What’s Python?”: The Sequel
As we’ve seen in this book, Python is a multifaceted tool, useful in a wide variety of domains. What can we say about Python to sum up here? In terms of some of its best attributes, the Python language is:
General purpose
Object-oriented
Interpreted
Very high level
Openly designed
Widely portable
Freely available
Refreshingly coherent
Python is useful for both standalone development and extension work, and it is optimized to boost developer productivity on many fronts. But the real meaning of Python is really up to you, the reader. Since Python is a general-purpose tool, what it “is” depends on how you choose to use it.
In the Final Analysis…
I hope this book has taught you something about Python, both the language and its roles. Beyond this text, there is really no substitute for doing some original Python programming. Be sure to grab a reference source or two to help you along the way.
The task of programming computers will probably always be challenging. Perhaps happily, there will continue to be a need for intelligent software engineers, skilled at translating real-world tasks into computer-executable form, at least for the foreseeable future. After all, if it were too easy, none of us would get paid. No language or tool can obviate the hard work of full-scale programming entirely.
But current development practice and tools make our tasks unnecessarily difficult: many of the obstacles faced by software developers are purely artificial. We have come far in our quest to improve the speed of computers; the time has come to focus our attention on improving the speed of development. In an era of constantly shrinking schedules, productivity must be paramount.
Python, as a mixed-paradigm tool, has the potential to foster development modes that simultaneously leverage the benefits of rapid development and of traditional languages. While Python won’t solve all the problems of the software industry, it offers hope for making programming simpler, faster, and at least a little more enjoyable.
It may not get us off that island altogether, but it sure beats bananas and coconuts.