As a long time teacher, I’ve come to appreciate the above quote from the Irish playwright and Nobel laureate. However, I would add that no matter what practical experience you may have, you never really know a subject until you have to teach it. There are so many nuances to engineering endeavors that we don’t think about except at an intuitive level. Having to explain those nuances to someone else is a lot harder than it sounds.
I teach all the time, but my ten-year professorial career is the time in which I taught the most. Every day I was either teaching or preparing my course materials. It provided a lot of thinking time and in-class debate time to help me think through the whole job of testing.
I’ve included some of the more lively and obscure material in this chapter. Much of it is unpublished...the closest thing we had in the early 1990s to a blog. There is some insight here, but enough humor to make it palatable for what was then my primary constituency: the college student.
Reprinted with permission from Software Quality Engineering. See this and more of James’s work at www.stickyminds.com.
In 1996, I posted my version of the ten commandments of software testing on a website. I wrote these commandments in the middle of the night when I couldn’t sleep. I was thinking about what things I could do to try to repro an elusive bug. As most testers will attest, sleep can be hard to come by in such a circumstance! Somehow in my slightly warped tester brain, these “things” took the form of commandments. It was a very spiritual moment.
I took the time to write the commandments down, and then promptly went into a very deep sleep that made me late for work the next day. I was determined not to make my sleepless night—and late arrival—completely without benefit and decided to post the commandments to my website. (You can still find them at www.howtobreaksoftware.com/.) I wondered if anyone would notice.
Well, I can now say for certain that, indeed, they noticed. It took a while, but I started getting mail about them. At first, the mail trickled in. Then, months later, it began pouring in, at the rate of two or three a month. It steadily increased until I would be surprised if a week would go by without getting at least one note about my commandments.
The mail was neither complimentary nor derogatory. In typical testing fashion, it was inquisitive. Could I please interpret one of the commandments? Would I settle a bet between two colleagues about commandment number four (except I thought “settle” meant “pay,” and I told them “no”)? And the most frequently asked question: Why are there only nine commandments?
Well, after hundreds of private e-conversations (some of them months long in duration) with individual testers around the globe, I have finally decided to come clean, publicly, and get these commandments “off my chest” once and for all. But it will take three of these columns to accomplish the task. This column explains the first two. The second will explain numbers three through six, then seven through nine (and the lack of the tenth) in the third column. I hope you enjoy them, as I have enjoyed them over the years.
First, I’ll just list the commandments so that you can see them as original visitors to my old website saw them, listed but not explained:
- Thou shalt pummel thine app with multitudes of input.
- Thou shalt covet thy neighbor’s apps.
- Thou shalt seek thee out the wise oracle.
- Thou shalt not worship nonreproducible failures.
- Thou shalt honor thy model and automation.
- Thou shalt hold thy developers’ sins against them.
- Thou shalt revel in app murder (celebrate the BSOD!).
- Thou shalt keep holy the Sabbath (release).
- Thou shalt covet thy developers’ source code.1
1There is a reason there are nine instead of ten—more about that later.
And now, here are my interpretations of numbers one and two. I just hope I remember all the things I included in my earlier interpretations! Any of my old correspondents are welcome to chime in at the end of this column.
One of the first things that any tester learns is that the input domain of almost any nontrivial software application is infinite. Not only are there lots of individual inputs, but inputs can be combined and sequenced in so many different combinations that it is impossible to apply them all. One of the second things testers learn is that the trick is to apply the right set of inputs so that infinity doesn’t have to be faced head-on.
Well, of course I agree with this approach. My own writing and teaching is full of advice on how to select the right set of inputs. But I also counsel testers to buck up and face infinity anyway. The method of doing this: massive-scale random testing. It is a tool that should be in every tester’s toolkit and few, if any, testing projects should be without it.
Massive-scale random testing must be automated. Although it isn’t easy to do the first time, it does get progressively easier with each project and eventually becomes rote. It may not find a large number of bugs, but it is an excellent sanity check on the rest of your testing: If you are outperformed by random testing, you may have a problem on your hands. And I am always pleased with the high-quality (albeit few in number) bugs that random testing manages to find.
Another reason to apply massive-scale random testing is that setting up such tests requires a healthy knowledge about the input domain of the application under test. Testers must really get to know their inputs and the relationships that exist among the inputs. I almost always find bugs and get good testing ideas just from the act of planning massive-scale random testing.
This commandment sounds a bit perverted, but I can assure you it has a “G” rating. The idea I am trying to get across here is not to test your application in isolation. Otherwise, you might run into the nightmare scenario of “application compatibility,” or more specifically, lack thereof. Application compatibility, or “app compat” as it is widely abbreviated, means that one application does something to break another one. In case you are not sure, that is a bad thing.
One way to combat this problem is to keep a cache of apps (old ones, new ones, borrowed ones, blues ones—the more diverse the better) and make sure that you run your app at the same time each of these are running. Of course, we want to do this with operating systems as well. A user should not have to tell you that your application won’t run with a specific service pack installed; that is something you should find out in your own testing.
So covet those apps and those service packs: The more the merrier!
We all know that there are at least two parts to testing. First we apply; then we check. When we apply inputs we are testing whether the software did what it was supposed to do with those inputs. Without the ability to verify that this is indeed fact, testing is much less effective.
Testers call this the “oracle problem” in reference to the wise oracle that knows all the answers. Of course, the answer we are interested in is “did the app do what it was supposed to do when I applied some test?” This requires our oracle to intimately understand what the application is supposed to do given any specific combination of inputs and environmental conditions. Automating the oracle is very hard, but a worthwhile pursuit, not only as a valuable testing tool but also as an intellectual pursuit. Forcing yourself to think like such an oracle can often be more productive than anything else that you might choose to do, whether or not you ultimately succeed in automating it.
We’ve all been here, haven’t we? You see a bug, usually a good bug; then it won’t reproduce. The better the bug, the worse you feel about it. I have seen many good testers waste hours and even days trying to reproduce a bug that they saw only once.
The effort to reproduce such a bug is often valiant but without the proper tools, the effort can be a waste of time. But the problem I see is that the time is wasted anyhow, without the tester even realizing it. I had a tester spend an entire day trying to remember the reproduction steps of a crashing bug, with no success. I would have preferred that the particular tester spend his time in better ways than that. I understand the frustration as well as any tester, but the pursuit of such a bug is often time not well spent.
The moral of this commandment is twofold. First, try your best to be ever alert and remember (or record) the sequences of actions you are taking against the software. Remember also the application’s response. Second, consider using debugger-class tools that can track your actions and the state of the software. This takes much guesswork out of reproducing bugs and prevents otherwise good testers from breaking this commandment.
Commandment one was about the importance of random testing—emphasis on random. This commandment is about intelligent random testing—emphasis on intelligent. When intelligence meets automation, the result is called model-based testing. Get used to the term because it is the automation technology of the future.
Software models such as objects, black boxes, or structure diagrams help us to understand software. Testing models help us understand testing. A testing model is a blend of intelligence about what an application does (the model) and how it does it (the automation). Good models can make your automation smart enough to respond to errors and cover code that is out of reach of dumb automation. Modeling is an exercise that at the very least will make you more prepared to test, even if you don’t bother to automate it.
Development work is hard, very hard. Developers over the past few decades have had to solve the same problems over and over again, and in doing so, often make the same mistakes over and over again. We testers must remember those mistakes and design tests that ensure that lessons are being learned.
If one developer makes a mistake coding some module, then we should assume that other developers might make the same mistake on similar modules. If a particular developer is prone to coding infinite loops, then we need to make sure we test for such errors in every module that the developer writes. This is “learning from experience,” and we are here to make sure that is what our developers do: understand their patterns of mistakes so those mistakes can be eradicated.
I often make an analogy between testers and physicians. Physicians, I say in the story, treat their patients gingerly. They say, “Does it hurt when I touch you here?” And then they promptly stop touching that spot when you say “Yes!” If testers were physicians, the story would be somewhat different.
Test-physicians would also inquire, “Does it hurt when I touch it here?” But when the pain is confirmed, a test-physician would then poke, prod, and probe, until the pain became unbearable. Why? Well, it isn’t sadism; it’s our job. No bad deed should go unpunished.
You see, every bug is a proud moment for a tester, but no bug should go without further investigation. So you found a bug that causes ill-formatted data to be displayed on the screen. Great, but can you go further and make that same bug corrupt data that the application is storing internally? If so, you have a better bug. And, can you then make that corrupt data be used by the application in some internal computation? If so, you have now turned a simple little formatting bug into a severity-one bug that causes data to be corrupted and the application to crash.
And, of course, the Holy Grail would be to crash not only your application, but also to cause the entire operating system to hang. Ah, the blue screen of death. I remember my first like it was yesterday, I anticipate my next every time I apply a test.
The moral of this commandment is that behind every good bug, there may be a better bug. Never stop exploring until you’ve discovered just how deep the bug goes and just how damaging it can be.
Oh so many times I hear testers whine about release dates. Testers most often want to extend release dates and, more often than not, their reasoning for wanting to do so is right on the mark. But their reasoning sometimes doesn’t matter.
The fact is that there are more factors than just quality that go into determining when to release an application to users. Quality is important but market pressures, competition, strength of user demand, staffing and personnel issues, and many more nontesting issues must determine a suitable release date. As testers, we must simply get the most work done that we can in the amount of time allotted to us.
We should not complain about release dates. We should, instead, warn about consequences. That is the extent of our responsibility, and it should be the extent of our concern.
I am not much of a believer in white-box testing. I think that it is something developers should learn to do well so that we testers can concentrate on more important and complex behavioral tests. That said, however, don’t look a gift horse in the mouth. If you have access to the source code, use it.
But, use it as a tester should, not as a developer would. My interest in source code is many fold and too involved to discuss all the issues here. But I think there is much to be learned from reading the source. Top of my list is looking for error-handling code and the dialog boxes that will indicate to us that the error code is executing. Error handlers are the hardest code to see or get to from the user interface. Understanding what error handlers have been written and what inputs it takes to trigger them is time well spent.
Indeed, there are many such clues we can glean from the source that give us insight into tests that need to be performed. We should not be shy about asking for and using the source code.
So that’s the commandments. By the way, there is a reason there are nine instead of ten. We might assume that just because they are “commandments,” there have to be ten of them. Since we know the assumption to be true (because that’s the nature of an assumption), then we convince ourselves that there is no need to ever bother checking whether the assumption may become false.
Assumptions are a very bad thing for software testers. Assumptions can reduce productivity and undermine an otherwise good project. Assumptions can even undermine a career. Good testers can never assume anything. In fact, the reason we are called testers is that we test assumptions for a living. No assumption is true until we test and verify that it is true. No assumption is false until we test that it is false.
Any tester who assumes anything about anything should consider taking up development for a career. After all, what tester hasn’t heard a developer say, “Well, we assumed the user would never do that!” Assumptions must always be tested. I once heard a test consultant give the advice: “Expect the unexpected.” With this I disagree; instead, expect nothing; only then will you find what you seek.
Reprinted with permission from Software Quality Engineering. See this and more of James’s work at www.stickyminds.com.
There are two types of code that developers write. First, there is the code that gets the job done—which we’ll call this type of code functional code because it supplies the functionality that satisfies user requirements. Second, there is code that keeps the functional code from failing because of erroneous input (or some other unexpected environmental condition). We’ll call this type of code error code because it handles errors. For many programmers, this is the code that they are forced to write out of necessity, not because it is particularly enjoyable.
Writing both types of code simultaneously is problematic because there are context switches that must be made inside the head of a software developer between the two types of code. These context shifts are problematic; they require the developer to stop thinking about one type of code and start thinking about the other.
Consider Johnny, a hardworking hypothetical developer, writing a new application. Johnny begins by writing the functional code, maybe even going so far as using something like UML to fully understand the various user scenarios that Johnny must code. Good, Johnny. Indeed, good programmers like Johnny can find a wealth of information out there to help them write good functional code. The books all address it, the tutorials address it, and there are many useful published examples to work from.
But, what happens when Johnny realizes the need for error code? Perhaps he is in the middle of writing or specifying some code object when he decides that, say, an input needs to be bounds-checked. What does Johnny do? One choice for Johnny is to stop writing the functional code and write the error code instead. This requires a context shift inside Johnny’s developer-head. He must stop thinking about the user scenarios and the functional code that he is implementing, and start thinking about how to handle the error. Since handling errors can be complicated, this may take him some time.
Now, when Johnny returns to the task of writing the functional code, his mind has to recall what he was thinking about when he last put it down. This context shift is harder than the first, given the myriad design-decision details and minute technical details that go into writing any nontrivial program. You see the problem: Poor Johnny has had to endure two context switches to handle a single error. Imagine how many such context switches happen writing even a small application.
Another choice for Johnny would be to postpone writing the error code in order to avoid the context shift. Assuming Johnny remembers to eventually get around to writing the error code, he’s probably going to have to spend some time recalling the nature of the error event he’s trying to write a handler for. So now Johnny is writing the error code without the benefit of context. Writing error code is problematic no matter how you face it. And therefore, a ripe place for guys like me to look for bugs. So now let’s look at the testing perspective; how do we approach testing error code?
Forcing error messages to occur is the best way to get error code to execute. Software should either appropriately respond to bad input, or it should successfully prevent the input from ever getting to the software in the first place. The only way to know for sure is to test the application with a battery of bad inputs. There are many factors to consider when testing error code. Perhaps the most important is to understand how the application responds to erroneous input. I try to identify three different types of error handlers:
• Input filters can be used to prevent bad input from ever getting to the software under test. In effect, bad inputs are filtered by, for example, a graphical user interface, and only legal inputs are allowed past the interface.
• Input checking can be performed to ensure that the software will not execute using bad input. The simplest case is that every time an input enters the system, the developer inserts an IF statement to ensure that the input is legal before it is processed; that is, IF the input is legal, THEN process it, ELSE display an error message. During this first attack, it is our goal to ensure that we see all such error messages.
• Exception handlers are a last resort and are used to clean up after the software has failed as a result of processing bad input. In other words, bad inputs are allowed into the system, used in processing, and the system is allowed to fail. The exception handler is a routine that is called when the software fails. It usually contains code that resets internal variables, closes files, and restores the ability of the software to interact with its users. In general, some error message is also displayed.
Testers must consider each input that the software under test accepts and focus on erroneous values. The idea here is to enter values that are too big, too small, too long, too short—which values that are out of the acceptable range or values of the wrong data type. The major defect one will find with this approach is missing error cases—input data that the developer did not know was erroneous or individual cases that were overlooked. Missing cases almost always cause the software to hang or crash. One should also be on the lookout for misplaced error messages. Sometimes the developer gets the error message right but assigns it to the wrong input values. Thus, the message seems like nonsense for the particular input values submitted.
Finally, of pure nuisance value are uninformative error messages. Although such messages cause no direct harm to the user, they are sloppy and will cast doubt in a user’s mind on the credibility of the software producer. “Error 5—Unknown Data” might have seemed a good idea to some developer, but will cause frustration in the mind of the user who will have no idea what they did wrong. Whether one is testing an input field in a GUI panel or a parameter in an API call, one must consider properties of an input when conducting this attack. Some general properties to consider are
• Input type: Entering invalid types will often cause an error message. For example, if the input in question is an integer, then enter a real number or a character.
• Input length: For character (alphanumeric) inputs, entering a few too many characters will often elicit an error message.
• Boundary values: Every numeric data type has boundary values, and sometimes these values represent special cases. The integer zero for example is the boundary between positive and negative numbers.
Be prepared to find some spectacular bugs!
Reprinted with permission from Software Quality Engineering. See this and more of James’ work at www.stickyminds.com.
What is it that makes some companies havens for testing talent while other companies incite anger from their testing ranks? At every testing conference I attend, I hear the same laments:
• “Developers think they are better than us.”
• “Development is always late in delivering the code and it’s Test that gets blamed when the schedule slips. Everything is always our fault.”
• “Upper management treats us like second-class employees.”
• “How do we get the respect we deserve?”
And so on and so forth.
I’ve listened in on the conversations these folks have with the testing consultants in attendance. In general, the consultants are full of empathy, as well as suggestions about how to improve the situation. Most of the solutions I have overheard fall into two categories:
- You need to improve communication between test, development, and upper management. This will allow a dialog that will lead to a better understanding and appreciation of testers.
- The problem is that testers are not certified. Certification will legitimize testing as a field and help ensure adequate treatment.
Frankly, and with due respect to the test consulting community, the first solution sounds a lot like Dr. Phil giving marital advice, and the second sounds a lot like a labor union.
In my opinion, neither psychotherapy nor unionization will solve this problem. Respect is doled out in the technology sector only when it is deserved. That’s a good thing. Too many times we hear people in other industries complain that it doesn’t matter how talented you are, that merit has nothing to do with respect or advancement. Our goal is to get so good at what we do, that colleagues and management have no alternative but to respect us.
So I have been taking notes during the last year on a mission to understand this problem. I accomplished my mission by studying the organizations in which this problem does not occur, organizations where testers are respected by developers and management and are afforded pay and career paths equal to development.
The companies I studied have a large number of testers who take pride in their testing prowess. They are good, they know they are good, and they take great pride in demonstrating their excellence. I hear them speak about their bugs with as much pride as the developers talk about their code. They name their bugs and, if questioned, will recount their hunt, their technique, their insight, and every bit of minutiae that relates to isolating and reporting the problem they found. Personal pride in a job well done is not the exclusive domain of developers. To these testers I say, “Long live your stories and may your tests never lack targets!”
Lest you read item number one above and thought those testers arrogant and self-absorbed, my subjects in this study had one singular focus: maximum contribution to product success. Whereas the developers can look with understandable pride at what they put in a product, testers can feel equal pride for what they keep out of the product. For this, testers deserve our respect. They deserve our thanks. And forward-looking companies are generally ready to give it. To those companies who refuse to generously dole out this respect, perhaps they would be willing to re-insert those bugs and do without the services performed by their best and brightest testers.
I often get invited to teach one-day testing seminars onsite. These courses are full of testers who are eager to learn. I start each class with one simple theorem: “Anyone who thinks they can learn testing in a single day is a fool who has no business testing software.” I invite such people to leave my course and implore them to leave the discipline. By this rather harsh statement, I stand firm: Testing is not to be taken lightly.
Testing is a pursuit; testing begins, but it never ends. This is a fact of life: We can never be finished with our testing task. No matter how much code we cover, there is always more that remains uncovered. No matter how many input combinations we apply, there are so many more that we did not apply. No matter how much we think we excel at testing, there are more complexities and subtleties that elude our full understanding.
Testers must demand, and receive, the education they so desperately need. Companies that refuse continuing education benefits—conferences, onsite or remote courses, books, and (if you are lucky enough to find them) graduate college courses—should be banned from the practice of software development. Such companies are a hazard to software users everywhere. Testing is a challenging intellectual discipline. It not only requires training, but it also demands it. Companies that refuse to cover these expenses for testers are negligent.
Testing is not a clerical task. It is an engineering discipline and is best carried out by trained engineers. Obviously, one cannot overgeneralize here. There are some people who have majored in the arts who make fine testers. And we always need domain experts to test applications that require specialized skill. (Imagine testing a flight simulator without a pilot or two on your team.) But, in general, a computer science degree (or related major) is necessary. There is background knowledge about algorithms, computational complexity, graph theory, and data structures that are requisite skills, building blocks for a deep and well-rounded testing education.
Now, if we could get more universities to actually teach testing, we’d all be even better off. But testers need to understand development, even if they don’t practice it on a regular basis.
Begin by hiring college graduates who have majored in something technical and related to computer science (electrical engineering, computer engineering, physics, math, and so forth). The education level of your testers should be equivalent to or better than that of your developers.
Insist on continuing-education benefits. Start by showing that the bugs that managed to slip into the released product could have been found with more advanced testing techniques, and make a strong bid for training concerning those techniques. You must make the argument, convincingly, that more training will equate to better software.
Nurture a testing culture in your company that allows you to learn from the bugs that you find and the bugs that your customers find. Don’t apologize for the bugs that slip through. Make them a learning opportunity. Pick them apart and be sure every tester knows why that bug was overlooked. Bugs are corporate assets because they teach us about what we are doing wrong. Being proactive about correcting these mistakes will go a long way toward showing upper management that Test is a crucial aspect of product development that improves when it is taken seriously. You cannot expect upper management to take you seriously until you take yourselves and your discipline seriously. The more attention you pay to improving the performance of Test, the more respect you will gain.
Finally, I must note that the trend among software companies is moving in the right direction. More and more companies are taking Test seriously and recognizing its value. If you work for a company that treats testers as second-class employees, other opportunities are out there.
Your fate is in your own hands. Strive for individual excellence, recognize your importance to a project, celebrate the bugs that won’t ship because of you and your team, and demand the continuing education you deserve. Respect your discipline and you will gain the respect that your discipline deserves.
Long live the professional tester!
Reprinted with permission from Software Quality Engineering. See this and more of James’s work at www.stickyminds.com.
In the late 1970s the software quality problem was all too apparent. Software was hard to write, hard to maintain, and often failed to meet its requirements. Even at this early stage in the history of software engineering, a move was afoot to correct this problem. Beginning with structured programming, researchers studied better ways to write code. The push for formal methods began. If you’d only do things more formally, the plea went, your code would be better! A few even went so far as to make “zero defects” their Holy Grail. The search was on. Unfortunately, the search still continues.
Following close on the heels of the formal methods advocates were the tool vendors. If you’d only use the right tools, they cried, your code would be better! Many tools have greatly simplified the task of developing software. Development organizations spend hundreds of thousands of dollars on tools. Software is still buggy.
The latest entry into the fray has been the process improvement people. If you’d only work more carefully, they beg, your code would be better! Now our managers are as busy as our developers. Developers not only have software to develop but also make-work reports to write. Software is still buggy.
This article discusses these three “silver bullets” and exposes their Achilles heels. I suggest that the answer to the problem must, by definition, be technical—not managerial—in nature. I end by launching the search for a fourth proposal.
Formal methods are a great idea. Essentially, they equate programming a computer to solving a math problem. You can go about it using a mixture of creativity, intelligence, and lots of practice solving similar problems. However, there are a couple of problems with formal methods that can’t be overlooked.
First and foremost, software developers won’t use them. This puzzles formal methods advocates to no end. However, it’s fairly plain to the rest of us: No one can show developers just how to apply formal methods. The examples in books and papers are far too simple to scale the ideas to real development problems. Plus, once you get outside your comfort zone, the formal methods fall apart. You remember how you felt when you learned calculus after mastering algebra? Algebra problems were easy because you’d solved hundreds of them. But the methods you used didn’t seem to apply to calculus problems. All the rules had changed. It turns out this same situation plagues software problems, too. They can be as different as algebra and calculus. Why should we be surprised that what works on one problem is useless on another? Formal methods don’t scale in software because they don’t scale in mathematics either.
Second, one can use formal methods and still write buggy code. Formal methods don’t address anything but the algorithm. But we all know that an algorithm can be correct on paper but fail on a real computer. The problem is that real computers have space and time limitations, other applications, and very complex operating systems that must be dealt with by code that has nothing to do with the main algorithms of an application. Indeed, code to process inputs and handle errors is often much larger and more complex than the algorithm that actually gets the main work done. There are no formal methods for handling such code.
Formal methods are important, but they will only take you so far toward reliable software. Strike one.
Tools can make the software development task much less painful, but they cannot guarantee zero defects; in fact, they cannot even guarantee fewer bugs. Since the tools themselves can be buggy, they create one more unknown in a project. When a defect is found, is the product or the tool at fault?
Tools range from simple and indispensable text editors and compilers to more elaborate environments for analyses and design. Very few outrageous claims are made from the developers of editors and compilers—it’s the design tool vendors that corner that market. What’s more valuable to a project anyhow, a nicely drawn E-R diagram or a programmer who’s expert in the target implementation language? Would you buy $100K worth of tools or hire a person who intimately understands the problem domain in which you are involved? Tools are great when used properly, but they can only offer you steep learning curves and limited functionality. Plus, they bring along a whole new set of bugs to worry about: their own.
You see, if tools really were a silver bullet, they wouldn’t be buggy, would they? Strike two.
The latest attempt at getting the software quality problem under control has been made by the process improvement people. Obviously, controlling and improving the software development process is in everyone’s best interest. However, since software development is a technical problem and process improvement is a management problem, it simply cannot have a profound effect on quality. Good organizations can produce bad software. Bad organizations can produce good software.
Furthermore, process improvement initiatives are rarely met with enthusiasm from rank-and-file technical people. ISO certification is a pain. SEI assessment is demeaning. Both take away creativity and add management overhead. Heck, part of the joy of working in this field is not being micro-managed. Why would any developer in his or her right mind actually think this is a good idea?
Well, it is a good idea, but it won’t help appreciably with the quality problem. I was once a part of a partnership in which a consulting company that specialized in formal methods was training an SEI level three—on their way to level five—organization. An example code fragment was used extensively throughout the training. Neither the formal methods advocates who wrote the buggy code, nor the mature process organization that studied the code, noticed that it possessed a fatal flaw. Not even during the course of proving the code fragment correct was the bug ever discovered. Why? Because the formal methods people were concerned about math, and the process people were concerned about documentation. No one was looking at the code! Fortunately, the test organization was paying attention and the bug was caught.
Management solutions cannot solve technical problems. Strike three.
What we need is for someone to come up with silver bullet number four. Except this one shouldn’t be silver. In fact, I think it’s important that proposal number four should be bland-colored, say brown, so that no one really notices it. It shouldn’t be something so revolutionary (as a gold or platinum bullet might be) that it makes no sense and people avoid it. It should be so ordinary that developers can integrate it seamlessly into their normal pattern of design. It should be so straightforward that developers remark “this is simple, what’s the big deal?” Not only should it be these things, it must be in order for it to be used and have some positive industry impact. Otherwise, it’s just more ivory-tower nonsense that real practitioners don’t appreciate.
It turns out that parts of such technology exist, are readily understandable by capable developers, and will not fundamentally change the manner in which software development occurs in an organization. If you are a great developer now, then you’ll still be a great developer. (This is a fear of great developers that keeps them from being strong process advocates. They know they are not great at filling out forms.) If you are a mediocre developer, then perhaps you’ll become better. Either way, it is likely that the code you write will be more understandable and have fewer bugs than it did before. In future installments of this bimonthly column series, I will survey many of the techniques that will one day contribute to the fourth, this time brown, bullet and show how developers and testers can adjust—but not change—their development process to improve quality and maintainability.
It’s time for a new batter.
The first book2 on software testing set the tone for software testers and software testing careers. The title of that book The Art of Software Testing identified our discipline as a collection of artists applying their creativity to software quality. Practitioners of software testing and quality assurance have been sold short by such a label.
2Although several collections of testing papers were published as books before Myers’s The Art of Software Testing in 1979, his was the first book to be written from scratch as a software testing text.
Testing is not an art.
Software testing is a far cry from those endeavors that most people accept as art: painting, sculpture, music, literature, drama, and dance. In my mind, this is an unsatisfying comparison given that my training as a tester has been more engineering than art.
Certainly, I’ll agree that, like artists, software testers need to be creative, but art implies skill without training. Most virtuoso artists were born to the task, and those of us unlucky enough to have no artistic talent are unlikely to develop such skill despite a lifetime of practice.
I also understand that two authors attempted to copyright the title The Craft of Software Testing, acknowledging Myers’s title and also implying a growth of the discipline from art to craft. This, too, sells testers far short of the difficulty of their calling. Indeed, the idea of software testing as a craft is equally unsettling as calling it an art. Craftsmen are carpenters, plumbers, masons, and landscape designers. Crafts are exemplified by a lack of a real knowledge base. Most craftsmen learn on the job, and mastery of their craft is a given as long as they have the drive to practice. Crafts are two parts dexterity and only one part skill. Indeed, carpenters have no need to understand the biology of trees, only to skillfully mold wood into beautiful and useful things.
Testing as arts or crafts doesn’t begin to describe what we do; and I’ll start a fight with anyone who attempts to call it arts and crafts!
I suggest the most fitting title for a book on software testing would be The Discipline of Software Testing. I would argue that discipline better defines what we do as testers and provides us with a useful model on which to pattern our training and our careers. Indeed, this is the best reason to call it a discipline: By studying other disciplines, we gain more insight into testing than using the analogies of arts or crafts.
A discipline is a branch of knowledge or learning. Mastery of a discipline is achieved through training, not practice. Training is different than practice. Practice requires doing the same thing over and over again, the key being repetition. One can practice throwing a ball for example and even though “practice makes perfect,” simply throwing a ball will not make you a major league pitcher; becoming that good requires training.
Training is much more than just practice. Training means understanding every nuance of your discipline. A pitcher trains by building his muscles so that maximum force can be released when throwing a ball. A pitcher trains by studying the dynamics of the mound, where to land his foot for maximum effect on any given pitch and how to make use of his much stronger leg muscles to propel the ball faster. A pitcher trains by learning how to effectively use body language to intimidate batters and runners. A pitcher trains by learning to juggle, to dance, and to do yoga. A pitcher who trains to be at the top of his game does many things that have nothing to do with throwing a ball and everything to do with making himself a better ball thrower. This is why Hollywood’s “karate kid” waxed cars and balanced on fence posts; he wasn’t practicing to fight, he was training to be a better fighter.
Treating software testing as a discipline is a more useful analogy than treating it as an art or a craft. We are not artists whose brains are wired at birth to excel in quality assurance. We are not craftsmen who perfect their skill with on-the-job practice. If we are, then it is likely that full mastery of the discipline of software testing will elude us. We may become good, indeed quite good, but still fall short of achieving black belt—dare I say Jedi?—status. Mastery of software testing requires discipline and training.
A software testing training regime should promote understanding of fundamentals. I suggest three specific areas of pursuit to guide anyone’s training:
• First and foremost, master software testers should understand software. What can software do? What external resources does it use to do it? What are its major behaviors? How does it interact with its environment? The answers to these questions have nothing to do with practice and everything to do with training. One could practice for years and not gain such understanding.
Software works in a complex consisting of four major categories of software users (i.e., entities within an application’s environment that are capable of sending the application input or consuming its output). These are (1) the operating system, (2) the file system, (3) libraries/APIs (e.g., the network is reach through a library), and (4) humans who interact through a UI. It is interesting to note that of the four major categories of users, only one is visible to the human tester’s eye: the user interface. The interfaces to the operating system, the file system, and other libraries happen without scrutiny. Without understanding these interfaces, testers are taking into account only a very small percentage of the total inputs to their software. By paying attention only to the visible user interface, we are limiting what bugs we can find and what behaviors we can force.
Take as an example the scenario of a full hard drive. How do we test this situation? Inputs through the user interface will never force the code to handle the case of a full hard drive. This scenario can only be tested by controlling the file system interface. Specifically we need to force the files system to indicate to the application that the disk is full. Controlling the UI is only one part of the solution.
Understanding the environment in which your application works is a nontrivial endeavor that all the practice in the world will not help you accomplish. Understanding the interfaces that your application possesses and establishing the ability to test them requires discipline and training. This is not a task for artists and craftspeople.
• Second, master software testers should understand software faults. How do developers create faults? Are some coding practices or programming languages especially prone to certain types of faults? Are certain faults more likely for certain types of software behavior? How do specific faults manifest themselves as failures?
There are many different types of faults that testers must study, and this forum is too limited to describe them all. However, consider default values for data variables as an example. For every variable used in a program, the variable must be first declared and then given an initial value. If either of these steps is skipped, a fault exists for testers to look for. Failure to declare a variable (as is the case with languages that allow for implicit variable declaration) can cause a single value to be stored in multiple variables. Failure to initialize a variable means that when a variable is used, its value is unpredictable. In either case, the software will fail eventually. The trick for the tester is to be able to force the application to fail and then be able to recognize that it has failed.
• Third, master software testers should understand software failure. How and why does software fail? Are there symptoms of software failure that give us clues to the health of an application? Are some features systemically problematic? How does one drive certain features to failure?
And there is more, always more to learn. Discipline is a lifelong pursuit. If you trick yourself into thinking you have all the answers, mastery will elude you. But training builds knowledge, so the pursuit itself is worthwhile whether or not you ever reach the summit.
Fifty plus years of software development has resulted in one overwhelming truth: Our industry develops crappy applications. We’ve made insecurity and unreliability the norm.
It’s true and as an industry, we can no longer deny it. You can look at studies such as that performed at the National Institute of Standards and Technology (NIST) in 2002 that implicate defect removal as a major cost of software deployment. (The study is available at www.mel.nist.gov/msid/sima/sw_testing_rpt.pdf.) Or you could simply take a look at pop tech culture and observe that our buggy software is creating new dictionary entries: Spam, phishing, and pharming are only a sampling. Are bad apps so prevalent that we have resorted to assigning amusing monikers to our failure to protect our users? Is this a situation that any self-respecting software professional can be proud of?
The answers are clearly yes to the first and a resounding no! to the second. Investigating why this is and what we might do about it is one of the most worthwhile tasks that our industry can undertake. In fact, it may be the very thing that saves us from creating the next generation of security holes and quality problems to plague our users.
This article begins this investigation in the hopes that it will be joined by an enthusiastic army of quality-minded software developers.
Past attempts at writing secure and reliable code have been decidedly front-loaded. By this, I mean that the focus of software development practices has been on specification, architecture, and development: The early parts of the software development life cycle. The intuition being that we need to focus on preventing defects because “quality cannot be tested in.”
This concept was so intuitively pleasing that many software construction paradigms picked up on it beginning as early as the 1970s: structured analysis/structured design, clean room, OOA/OOD/OOP, and aspect-oriented programming are some examples.
Software defects continued and so did the process communities’ ill-fated attempts to squash them: design by contract, design patterns, RUP, and yes, oh yes, there were more.
Finally, we woke up and realized that such front-loaded processes simply don’t work. The idea that we can specify requirements and plan tests in advance when reality was changing too fast to predict hit our industry square in the face.
And we answered with more methodologies: Extreme (spell it whatever way you wish) programming and agile development took center stage. Progress? Hmm. Well, the jury is still out, but I am not holding out much hope. You see the problem with all of these methodologies is that they teach us the right way to do things.
Now granted, many industries have figured out the right way to do things. Artists study Picasso, Rembrandt, and the many other masters of their craft. Musicians have no lack of masters to study: Beethoven, Handel, Mozart, and Bach are only a few. Architects can study the pyramids, the Taj Mahal, and Frank Lloyd Wright for that matter. All these professions have existed for long enough that there are many, many examples of people getting it right so that those wishing to follow in their footsteps and master the craft have examples to study.
But it is our sad misfortune (and grand opportunity) to be in the software game so early that no such examples of perfection or inspiration exist. If they did, we’d be studying these “classic” programs so that the new generation of programmers could learn the discipline from those that went before them.
So is it even possible to construct a software development methodology without prior knowledge of how to do software right? I say no and the evidence I present is that software is getting no better. Indeed, I would argue that the complexity of the systems we build is far outpacing the small advances that any of the current menu of development methodologies offer our industry.
Throw them all away and face the fact that we have no idea how to build a high-quality software system of any reasonable size.
When pop tech culture stops naming our bugs and the other headaches we create for our users, that may be an indication that we are progressing. But until then, we need a better plan.
We cannot study success in an environment where only failure exists. So I propose, instead, that we study failure and build our development processes rear-loaded.
Let me explain what I mean by that: There is no more clear indication of what we are doing wrong than the bugs we write, fail to detect, and then ship in our products. But all of the past methodologies treat bugs as something to avoid, something to hush up.
This is unfortunate and I propose we stop treating bugs as a bad thing. I say we should embrace our bugs as the only sure way to guide them to extinction. There is no better way to improve than by studying the very thing that makes our industry the laughing stock of engineering disciplines.
We should be studying our bugs.
I propose starting with bugs and working backward toward a process that just might work. Here’s how I think we should proceed:
Step 1: Collect all the bugs that we ship to our customers (paying special attention to security vulnerabilities). Instead of treating them like snakes that might jump out and bite us, consider them corporate assets. After all, they are the surest indication of our broken processes, misdirected thinking, and mistakes that we have made. If we can’t learn from what we are doing wrong, shame on us. If we refuse to admit that we are doing wrong, then we have a bigger problem.
Step 2: Analyze each of these bugs so that we (1) stop writing them, (2) get better at finding them, and (3) understand how to recognize when they occur.
Step 3: Develop a culture in our organization in which every developer, tester, and technician understands every bug that we’ve ever written.
Step 4: Document the lessons learned. This becomes the basis for a body of knowledge about the bugs we write and the basis for a new set of methodologies that are aimed squarely at preventing our most egregious mistakes.
We can do this by questioning our bugs. I think the following three questions are a good start and will teach us a great deal about what we are doing wrong. For each bug we ship, we should ask ourselves:
1. What fault caused this bug in the first place?
The answer to this question will teach developers to better understand the mistakes they are making as they write code. When every developer understands their own mistakes and the mistakes of their colleagues, a body of knowledge will form inside our development groups that will reduce mistakes, help guide reviews and unit tests, and reduce the attack surface for testers.
The result will be better software entering test.
2. What were the failure symptoms that would alert us to the presence of this bug?
Remember that I am proposing to study bugs that shipped, so the assumption is that somehow the bug slipped by or was found and not fixed purposefully. In the former case, testers will create a body of knowledge and tools about how to better isolate buggy behaviors from correct behaviors, and in the latter the entire team will learn to agree on what an important bug really is.
The result will be better software shipping to our customers.
3. What testing technique would have found this bug?
For those bugs that were totally missed in test, we need to understand what test would have found the failure and helped us diagnose the fault. Now we are adding to the testing body of knowledge with tests that actually work to find important bugs.
The result will be more effective tests and a shorter test cycle.
What I am proposing here is that because we cannot possibly understand how to do software right, let’s understand how we’re doing it wrong and simply stop doing it that way. The resulting body of knowledge will not tell us what to do to develop software; it will tell us what not to do.
Perhaps we can follow this rear-loaded process using our existing front-loaded methodologies and meet somewhere in the middle.
Now that’s what I call progress toward a discipline we can all be proud of.
Let the celebration of bugs begin!