“If the dull substance of my flesh were thought, Injurious distance should not stop my way; For then, despite of space, I would be brought From limits far remote where thou dost stay No matter then although my foot did stand Upon the farthest earth removed from thee, For nimble thought can jump both sea and land As soon as think the place where he would be.”
William Shakespeare, Sonnet 44
The previous chapter introduced how Brownfield unites information in the Inventory. In this chapter, we show how combining formal Views that span business and IT results in a unique capability to create new levels of cooperation between the domains.
This chapter shows that recording the business and IT information in the kinds of Views introduced in the last chapter makes it possible for both business and IT professionals to reach a common understanding of the requirement and the solution.
This level of traceability and linkage between the requirement and the solution is generally associated with extremely heavyweight systems engineering methods. This chapter shows that such traceability and combined understanding of both business requirements and IT solutions is possible within a lightweight and fast agile development process.
The Brownfield approach lends itself to the creation of uniquely powerful perspectives that provide a new level of understanding between business and IT. We examine this perspective, which exploits the highway (or trunk road, if you know the equivalent UK term and want to extend the Elephant analogy a little further) to the brain, in this chapter.
As we have already discussed, the greatest source of IT project failure is not technology, but communication of complexity. Therefore, you might rightly expect that optimizing communication receives a great deal of attention. You might assume that the architects and technicians will do everything possible to ensure that the business understands their design and to provide a useful and full explanation of the problem. Surprisingly, however, achieving this level of communication with the business is rarely a priority.
Except for use cases, which usually create a good bridge between IT formality and business English, the IT industry has precious few other examples to be proud of.
Use cases combine standard English and some simple rules that cover their formatting and level of detail to ensure that both the business and the IT personnel can understand the requirement. However, Chapter 2, “The Confusion of Tongues,” showed that the English language is a poor medium for precise communication. In addition, language is one of the least direct ways of getting information into the brain. The immediate senses of seeing, hearing, touching, and smelling have far more immediate routes in. Many memory techniques work by converting information into visual scenes or mini mental plays, to make complex information much more memorable. In the IT industry, our ability to exploit these other routes, especially the visual cortex to achieve communication, is woeful.
To ensure that we have a good formal definition of our design processes, the IT industry loves to dream up new formal ways of portraying them. As we saw in Chapter 2, many of the improvements in communication in the IT industry have come down to a formalization of syntax (the structure of the language), which removes some structural ambiguity. The downside of such innovation is the creation of highly formal, structured, and private languages. Unless you’ve been trained in them, your chances of understanding them are slim—even when they are simple line drawings.
Such gaps in communication can be extremely costly. The “experiment” in communication in the previous anecdote (which had worked okay on smaller projects with less direct user involvement) was identified as a mistake. These diagrams needed to be translated into a formal tool so they could be maintained, tracked, and refined until they were useful. Unfortunately, the problems that the informal documents had created had already undermined customer confidence in the project as a whole—the project closed down at a huge cost.
Formal communication can be just as ineffective as informal communication, however. The use of formal representations with formal syntaxes such as UML, RDF/OWL, DDL, BNF, and a host of other three-letter acronyms can be even scarier for the uninitiated.
In addition, the amount of information that can be packed into these tools is worrying. Anyone who has worked on a big project will remember fondly the huge ANSI E-sized or A0-sized diagram that almost every such project has stuck to a prominent wall or notice board. Whether it’s the Component Model, which describes the internal structure of what is being built; or the Logical Data Model, which describes the data structures the system has to deal with; or a detailed Infrastructure diagram, they all have the power to terrify through complexity. In many projects, we have seen the architect or designer in charge finally manage to download the drivers for a particularly big plotter or printer, and then take great delight in printing out his or her masterpiece so that everyone could finally read the small print. Figure 4.1 caricatures this tendency of lead technical personnel to blind everyone else with science.
On the same project that involves walls papered by process flows, one of the lead architects was famous for answering any question about the system by presenting a slide extracted from the project’s similarly gargantuan Component Model. It did not matter if the question was about the user interface, the physical infrastructure supporting the system, or a detail in the business process. Of course, despite its size, the Component Model contained absolutely no useful information with regard to these topics. But this didn’t matter. With an astonishing ability to blind with science, the architect explained the answer in the context of a language that the architect was particularly happy with and the audience had little or no knowledge of. Almost always, they went away slightly confused but basically happy, much to the amusement of every other architect in the room. The barrier in communication that the private language created was, in this case, of some benefit to the architect—but, fortunately, the project was canceled before the joke wore too thin. | ||
| --R.H. | ||
The IT industry is quite happy to try to impose its own languages on the business. IT professionals often have sufficient hubris to believe: “They must conform to us, because we are precise and right.” The science fiction writer Robert Heinlein understood the foolishness of this attitude and expressed it in a famous quote illustrated in Figure 4.2. If the IT industry thought hard about it, it would realize that the privately formatted diagrams with hidden meanings are an exceptionally poor way to communicate anything outside of the IT domain.
This kind of experience is painful for everyone involved. So how do we actually manage to communicate these complex concepts and difficult information on projects? The answer is both universal and appallingly simple.
We use PowerPoint.
Oh, yes! The universal tool for communicating any complex concept is to use PowerPoint. Of course, this tends to be a lot of work. Clearly, you can’t just paste the complex information directly into a presentation and expect someone to follow it just because it’s now on a slide. No, you have to lose all that precision and formality that you’ve just spent a lot of time creating for your downstream consumers (the designers or the coders or the testers) and informally abstract it to the level of your audience.
If you’re following a formal method, you will probably have some intermediate summary- or architecture-level materials that will pad out your presentation and guide the listener to the right area. But to get the approval you’re seeking, chances are, you have to create a new level of abstraction to convey the complex and detailed concept you’re looking at.
If you’re lucky, this could be a diagram you can draw from the existing information you have in your design tool. (Rational® enables you to do this with UML diagrams, for example.) More often, though, such automated outputs retain the frightening aspects of the private language they were written in. This might include a lot of punctuation where your English teacher clearly told you it shouldn’t go (<>I_();) or loads of odd abbreviations or EnglishWordsStuckTogether like a version of Anglo-Deutsch that has a problem with the capital Shift key. You might have to bite the bullet and create the ultimate in dangerous project dead-ends: the pretty-picture PowerPoint communication deck.
Such decks are dead ends for a variety of reasons:
They are long and laborious to create in the first place.
They are likely inaccurate, incomplete, or, at best, imprecise abstractions of the truth.
The missing information makes them highly subjective (often one person’s view of how everything works). The hilltop or view used by one writer might be confusing or might omit key details that a consumer or fellow presenter needs. When multiple people from different perspectives (such as process and organization) prepare such decks, they’re almost always guaranteed to be inconsistent and to cause confusion in the audience.
Sadly, despite all this effort, nearly every presentation turns up someone in the audience who asks a question that requires referring to information that has been deliberately left out of the slides to make them easy to understand. Such are the dangers of trying to predict other people’s Views. After pointing out such discrepancies and inadequacies of the presentation, how many presentations have you attended at which the audience loses faith with the presenter or simply thinks, “You simply don’t understand this”?
Even if the communication at the initial presentation works and the audience walks away with copies of the presentation, what happens if they point out an inaccuracy during the presentation? Typically, the defect is captured in the room and the source data is updated with the new information—but will the presenter update the PowerPoint slides, too?
What happens if the original design or requirement materials change? Will the slide deck be changed then? Will it get reissued to everyone who attended the session?
Experience suggests that this won’t be the case unless the presenter has to use the slides again fairly soon.
As a result, the hundreds of pages of PowerPoint will begin to diverge from the source data. Pretty soon they will be effectively worthless. However, because the audience has often walked away with a copy of the presentation, those pages can quickly become superb sources of confusion. If you’re really unlucky, they might become a formal part of the baseline that you are asked to deliver to and will become a real millstone around your neck.
So how can an Elephant Eater communicate precisely without resorting to blinding science?
The previous chapter introduced the Elephant Eater, which absorbs details from the surrounding environment using formal Views. Each of these Views represents a particular viewpoint of the problem space being examined.
Of course, this massively holistic process has a drawback. Although each individual View is easy to maintain and understand, the combined Inventory is massively complex. Now, this should not be overly surprising. By definition, the Inventory consists of a lot of combined perspectives—only a superhero or an Elephant Eater could understand that complexity. A highly precise and high-capacity computer might be capable of storing and processing the Inventory as a whole, but no human could easily cram the combined information into his or her head.
If we did try to display the Inventory as a whole, it would be unimaginably complex. Imagine that each one of those short sentences that we called triples were drawn in two dimensions. The subject and object could be represented as a shape, such as a circle, and the relationship between them could be represented as a linking arrow (see Figure 4.3).
Now imagine an Inventory with thousands of such interrelated triples. The picture would not be dissimilar to Figure 4.1, but it would contain many more boxes and lines. If we tried to display the Inventory in two dimensions, lines would crisscross the paper like spaghetti. The time dimension of the Inventory also would mean that we might need a lot of copies to represent different time periods.
Each triple can have its own lifetime defined, so the number of these different time periods (often called time slices) could be very high. Three dimensions might provide a better representational space because different layers of the diagram could be used for different information, but the time dimension would require multiple copies of the three-dimensional model. If we wanted to see the whole Inventory in a single glance, we would probably have to use a multidimensional space with far more dimensions that we are used to exploring—more than the usual three plus time. We have something of a problem. Our Elephant Eater is actually a multidimensional monster of unimaginable complexity, impossible to visualize. Unfortunately, people don’t like dealing with things that they can’t see or understand. Lack of understanding breeds fear and lack of acceptance. We need to do something creative.
We have two problems here: one of too much information and one of how to represent complex interlinked data. If you had a lot of money to spend and a lot of time, you might decide to draw something by hand. But the more complex the graphic, the harder it is to maintain. Such an approach is another dead end because the Inventory changes daily; the picture could never keep up.
Perhaps there’s a way to present the Inventory so that it can be visualized and navigated?
One form of navigation would enable one subset of the Inventory to be seen at a time. The entire space could be explored by following the links between the atoms, as shown in Figure 4.4.
Fortunately, such tools do exist.[1][2] Many allow Inventory type data to be imported and then explored in this kind of dynamic way. The exploration can start at any particular atom. As atoms are selected, the surrounding data changes to depict the information surrounding that atom.
Navigating the Inventory in this way makes it possible to ask and answer simple queries. Does this solve our concern about a lack of understanding of the Inventory? Maybe, but it might be hard to find the information that we want if we don’t know where it is in the hierarchy. Plus, this isn’t the most intuitive representation. A designer or developer might feel at home, but that’s probably about it.
Alternatively, some tools even enable you to ask simple English-like questions about an Inventory. Such query languages, such as SPARQL, are relatively new but underline some of the power of the Inventory. Of course, because these semantically rich databases are relatively new, the tools that support them tend to be from research or early products—they are often not that user friendly, and their output is not easily understood.
Perhaps reading the Inventory directly is part of the problem. Might another way exist?
For once in this book, perhaps architecture is part of the solution. This time, however, we’re talking about building architecture, not the IT variety.
Fundamentally, building architects do things rather better than IT architects. Perhaps it’s because they’ve had several centuries’ head start, but whatever the reason, we should learn from them. For a building of any degree of complexity, architects tend to use Computer Aided Design (CAD) packages to put the building together. The various different layers of the building are available to peruse and can be displayed or hidden at will. Building architects know all about Views.
These CAD programs can output Artifacts specifically used for communication. Some Artifacts are traditional communications, such as blueprints and orthogonal Views. Sometimes the models are extracted as a different Artifact and reimported as a View into another tool that can turn the model into a full-color, three-dimensional representation augmented with trees, flowers, skies, and realistic materials.
Guided tours around the building can be given before the design is even approved—and, certainly, well before a single brick is laid or a girder is cast.
The lesson to take from this is that visualization and View maintenance and creation should be separate but automatically linked. The Views that are suitable for the building architect in designing and maintaining the design of a building are not suitable for showing to the client. The visualization Artifact that is used to give the client a guided tour would not work successfully as a View for designing the building. Having a Best of Breed in both areas in a single tool would be highly unusual. After all, this was one of the reasons that Brownfield supported multiple Views in the first place—no single optimal tool exists for maintaining all Views.
Note also that the marvelous odd-shaped buildings that today’s architects revel in often can’t be made physical without computers. The shapes that architects define within their CAD program are usually transferred to other programs so that they can be manufactured. Each strut and girder these days can be different and custom built. This technique, called Computer Aided Manufacturing (CAM), isn’t new. Somewhat surprisingly, the computer industry equivalent, Computer Aided Software Engineering (CASE), never really took off. Only relatively recently has the Model Driven Architecture movement rekindled the interest in building software directly from models. We discuss why this is the case in Chapter 7, “Evolution of the Elephant Eater.”
Ideally, we want a similar walkthrough of our Brownfield site and our new systems. If the CAD program can export to a 3D graphical rendering engine, why can’t we? Why can’t we export the Inventory data and visualize it?
Moving from today’s static two-dimensional representations to a dynamic three-dimensional space would give us two opportunities. First, we could represent more interlinked data in an understandable way because we would have an extra dimension to play with. Most systems are multilayered these days, so this seems a pretty sensible way to go. In addition, if the space we’re working in is dynamic, maybe we could show the dynamic nature of the problem and the solution. If we’re showing a process, perhaps we could show the effects of that process as it flows. If we’re showing a time series of Views, perhaps we could see them evolve. Furthermore, instead of using rather flat boxes and lines, perhaps each type of thing held in the Inventory could be given a different visual metaphor. We could then see how all these different types of things work together without needing to read all those labels and strange symbols. We’d be able to exploit the visual cortex, the highest-bandwidth route to the brain—the trunk road, no less.
Historically, such ideas have been expensive and difficult to achieve. For a fast three-dimensional display, you needed a dedicated and expensive CAD workstation. Even on expensive equipment, the rendering of a 3D walkthrough of a complex design could take many hours. In the last ten years, however, all that has changed. Even handheld computers can now process complex three-dimensional graphics in real time. As a result, feeding in complex information for real-time, three-dimensional animated visualization is pretty feasible. So we could do it that way, but why stop there?
One of the key problems we have continually identified in this book is communication. If we want to visualize our Inventory, why don’t we do it so that the understanding can be shared rather than experienced by one person alone? And with the IT industry being increasingly global, perhaps we could do it in a way to share the understanding around the globe, in real time.
Such thoughts would have been close to insane even a couple years ago. At the time of this writing, however, millions of people have logged on and tried virtual worlds. In a generation or less, surely their use will be ubiquitous. Other than for playing games, many have wondered about possible uses for virtual worlds—others think it’s the start of a 3D Internet.
These worlds provide the opportunity to share precise, three-dimensional, dynamic representations of information. You can also see other people interacting with the same information. If you know this information well, you could answer questions about it and share your understanding anywhere on the planet.
When this thought first occurred to us, the obvious choice of platform was Second Life, as shown in Figure 4.5.[3]
Second Life is an environment built entirely by the people who use it to meet and communicate. Second Life is not a game, but it has achieved world fame as a virtual world that is open for business. The difficulty of rendering Inventories in environments such as Second Life is twofold. First, the virtual world itself has no capability to store complex data, so the Inventory needs to be read in from outside the world. This can be solved by building a suitable interface.
The second is trickier. The Inventory has no “positional” data within it—the display algorithms must determine the shape the Inventory is displayed as. It’s as if you’ve been given a long list of Metro or Tube stations along with some track information that tells you which ones are connected to each other. Unfortunately, you’ve been given no information about where the stations actually are, and you’ve been asked to work out the best way to lay out the map. Now, this is a complex task, so it could be performed outside of the virtual world. However, this would result in Inventories that would essentially be static in the virtual world, which seems a poor use of a highly dynamic environment. The solution was to make the displays self-organizing. Figure 4.6 shows an early model.
This visualization shows a small part of the structure of a number of interfaces that are shared between three systems down to the attribute[4] level. Essentially, it is a single Inventory created by combining three overlapping Views. The atom and link structure of the Inventory is clearly visible.
Such visualizations are just the start, however.
Just as the Inventory can absorb technical information such as data models, component models, infrastructure, and interface models, it can also import business processes, use case models,[5] and classified business terms.[6] These kinds of information provide a user- and business-oriented perspective rather than technology-based perspective of the problem.
Formally linking this business information to the definition of the IT system within the Inventory makes it possible to start sophisticated analysis such as impact assessments. For example, if part of the system changes, we can use traceability between the business and IT spaces to understand which use cases should be retested. Alternatively, if a use case is significantly revised or dropped from the requirement, it is possible to analyze which areas of the system could be affected.
In recent years, tools have been created that define business processes in such a way that the information the tools hold can be exported for execution on a business process engine. We can use the same exports to import this dynamic perspective to the Inventory. The business processes thus defined enable us to link to other key business concerns, such as business data entities and resources (such as time, money, locations, and people).
Use cases are often identified within the business process as the boundary points between the primary and secondary actors (typically, the supporting systems). The full definition of these boundaries (along with the interface specifications) creates the systems context.
Formal use case data also often includes data entity manipulation data, such as which process updates which bit of business data. This information can form a bridge between the static logical data model and the dynamic business processes. This enables the Inventory to know what processes act on which data. Even if the use cases themselves are not sufficiently detailed to do this, the architects can create a process/data usage spreadsheet or database and use it to import the same data.
So if we can import all this business metadata,[7] IT metadata, and, most importantly, metadata that bridges the two perspectives as Views into the Inventory, can we build a visualization that provides new insight that traditional methods could never provide?
Figure 4.7 shows the conventional View of a combined business/IT picture that has been derived from a number of separate Views. To make the comparison that follows fair, we have used the autolayout feature of the tool to format the data for viewing.
It’s not wonderfully clear or intuitive, is it? Explaining this perspective in its entirety to a business end user would require one of those marathon PowerPoint decks. Although the Inventory would still offer significant benefit in terms of traceability and impact analysis, in its current form, the merged model would not improve communication.
A better approach would be to translate this model into a three-dimensional layered picture.
Hidden within the morass of detail in Figure 4.7 are some implied layers, which we can see if we create a model of our model (that is, a metamodel). In this form (Figure 4.8), we can see the hidden structure of the data we were looking at.
The data is organized into a number of different areas, as indicated by the shaded rectangles behind them. The top-left part of the diagram describes the business services, which are the capabilities provided by processes that the business executes. (There is no IT in this view.) These business services are underpinned by the business itself, which has a set of resources that it uses to deliver these business functions. Some of these resources are IT systems, but locations, people, and money all play their part.
The IT systems are underpinned by physical structures, shown in the bottom right (computers, networks, and the like), and logical structures, in the bottom left (data, components, and interfaces). Finally, the IT systems provide IT services (bottom, far left), which support the business services (returning to the top left, where we started the description).
This is a pretty powerful vocabulary—it’s not complete, by any means, but it gives a strong representation of the business-significant aspects of today’s business and IT systems. It is not a business View or an IT View—it is both.
We can use this implied structure to create a similarly layered three-dimensional representation of the original complex data, as shown in Figure 4.9.
You can visit the model yourself on the IBM islands in Second Life (at Second Life grid reference [IBM 1 140, 150, 60] or http://slurl.com/secondlife/IBM%201/140/150/60). Each of the tiers represents the perspectives (sometimes combined perspectives) of one or more sets of users.
Figure 4.9 shows this whole model. You can clearly see the layers and linkages between them, but the eagle-eyed might notice that three-dimensional metaphors have replaced the atoms and links of the earlier representations.
Perhaps only the chief architect will want to understand the whole building, but the business analyst or architect will certainly want a firm understanding of the top three floors. A business leader might only want to look at the business resources floor.
Let’s take a walk down through the entire building, starting at the top.
On the executive floor (see Figure 4.10), we can clearly see that this business offers a number of business services. These business services are supported by associated IT services. Currently, this level of description might be all we require. But you might notice that some of the Inventory links disappear down through the floor. If we follow these links, we can find out which part of the organization provides these services and what business data is used to support them. (As in many large organizations, there is a glass ceiling just below the executive floor...) This next floor down gives us a lot of information about the logical shape of the business (without specifying precisely what organizational elements or resources are involved in its execution).
On the floor below that, we are looking at the business from an operational perspective (see Figure 4.11). What resources are used to support the services we just looked at? On this floor, we can see that a number of different organizations, locations, systems, and databases support the business services and processes. The relationships between these resources are also clearly indicated on this floor. Many of these relationships are the existing business and IT constraints that we introduced in Chapter 1, “Eating Elephants Is Difficult.” This is a pretty unique perspective that addresses many of the kinds of questions we often find difficult to answer. This is because this floor displays information that is usually found in separate Views maintained by different users. It is highly unusual to see it brought together in a single space. You’ll also notice here that links go upward (to the business logical space) as well as downward.
Finally, in Figure 4.12, we make our way down into the machine room. All the systems and databases we identified on the floor above trace down to the physical machines (or clusters of machines) shown on this floor. Running between the machines are networks. From here, we can trace the impact of losing a machine or a location on the business that runs above us—a powerful perspective indeed.
Of course, you’ll notice that this isn’t all of the Inventory. The whole point of generating such a visualization is that you can choose to display the elements that make sense to your audience. Chapter 9, “Inside the Elephant Eater,” explains how to automatically extract high-level information from the detailed Inventory.
Indeed, the great beauty of this approach is that this visualization is just another Artifact generated from the Inventory. As the Inventory changes, the visualization will change when the Artifact is automatically regenerated. Such models take a trivial time to create because, unlike in PowerPoint, they draw themselves via self-organizing principles.
The capability to automatically harvest a series of Views from a site survey, combine those Views within an Inventory, and then use the Inventory to automatically generate an easy-to-understand 3D representation is a very powerful capability that makes the IT industry’s current Greenfield approaches (manually abstracting the existing environment so it can be understood, manually refining this description over a long period of time, and then manually reabstracting it to PowerPoint to explain it to the business people) look very primitive indeed.
The most encouraging element about these kinds of visualizations is the reactions they get. “Cool!” is a common reaction (or “Supercool!” according to Grady Booch), but some regard them as things of aesthetic quality and beauty. We need some more of that in our industry. The visualizations can be created from any point, from the site survey onward. Thus, they are an excellent mechanism for communicating the understanding gained from “eating” the complexity of the surrounding environment. They’re also an excellent way of presenting evolutionary options for any solution. We predict that within one generation, the IT industry will wonder how we built and collaborated on complex IT projects around the globe without them.
| 1. | Cooperative Association for Internet Data Analysis. “Walrus Tool.” www.caida.org/tools/visualization/walrus/gallery1/index.xml. |
| 2. | Munzner, Tamara. “H3 Viewer.” http://graphics.stanford.edu/~munzner/h3/. |
| 3. | Linden Lab. Second Life. http://secondlife.com/. |
| 4. | Attributes are the individual elements of information that make up more complex structures, such as interfaces and data entities. Thus, |
| 5. | Use case models capture the behavior of an end user as it interacts with a system to complete a goal or task. |
| 6. | Classified business terms are clear and unambiguous definitions of key elements of the business terminology that is specific to the problem. They define a common vocabulary for the project. |
| 7. | Metadata is information about information, so business metadata is information about business data, such as its structure or its allowed contents—not the business data itself. |