“The longest journey begins with a single step.” | ||
| --Confucius | ||
This book has explained the problems that face the industry today in successfully completing elephantine projects. The success rate of these projects is extremely poor. The methods and processes used in these very large projects were essentially designed for projects that are Greenfield in nature, and they have remained fundamentally unchanged over the last 30 years.
Unfortunately, very few opportunities exist for elephantine Greenfield developments. In addition, most enterprises have a vast infrastructure in which the developments must reside, with large amounts of data that must be migrated from the existing systems. Yet the industry persists with methods designed for Greenfield-type developments.
The Brownfield development approach suggested in this book is based on extending the latest best practices within the IT industry, together with the site survey currently used in the construction industry, combined with some further elements borrowed from architects who build real buildings. After reading this book, the benefits of using a Brownfield development approach on any complex project should be obvious—the problem is how and where to start....
The problem of where to start is complicated by the fact that the approach is still new and the skills required to deliver solutions in a Brownfield way differ from normal skills. The core technologies make extensive use of XML, XSLT, Eclipse, RSA, semantic engineering, RDF, and OWL. Although most developers are familiar with the first four of those, the remaining three skills are relatively rare (although the last two are not essential to the Brownfield approach).
At the time of writing, no good tools exist for natively maintaining combined business and IT architectures—this is why we’ve opted for models and metamodels in RSA with the dynamic aspects in WebSphere Business Modeler. This could change over time, but currently, “roll your own” is the main option. In addition, there’s no commercially available Inventory (yet). Some of the architecture tools out there have a subset of the Inventory capability, but if you want to do everything described in this book, you’ll have to build one yourself. Ideally, we recommend building on something that already exists—we don’t believe in Greenfield tool creation either.
In addition to having to roll your own Elephant Eater, you must choose a language for your Inventory. Formal ontologies (languages) expressed in triples for everything from business modeling to architecture and system designs have already been created. Many of them are published on the Internet. But these languages overlap, underlap, and are tricky to combine. No clear or single winner for an Inventory language has emerged yet. Until a great deal of further standardization occurs in this space via W3C, OMG, and ISO, the exchange of Inventories will remain complex.
More important than the skills and tools gap is the cultural investment in the Greenfield approaches that will take a lot of time to overcome. The IT industry has turned away from embracing complexity for many years. The idea of “abstraction first and getting to the detail later,” testing “V” models, and the pedestrian-crossing alternation of black box/white box are endemic and often regarded as best practice. Therefore, you’ll find an inherent conservatism in the IT industry.
The techniques we have described in this book are sufficiently novel that IBM has filed for patents to protect our particular implementation of Brownfield and the techniques we use to visualize the Inventory for testing purposes. Brownfield is not a research technology (it was created and is still maintained by field personnel), but it is new and novel.
When projects are very large and the risks are high, innovation tends to be introduced a little at a time. Brownfield came about only because of a persistent team of committed technical individuals were able to convince executives, colleagues, and clients that they had something that was worth investing in and exploiting. The next generation of Elephant Eaters will benefit from this book, new tooling, and further innovations, but change is always difficult.
However, even with these provisos, this kind of thought process is at the forefront of business and IT thinking. It will be some time before we have a new industry-accepted norm for business reengineering and modeling. But we need one!
Ultimately, the biggest challenge of implementing Brownfield is actually one of culture. You’ve probably noticed that all the Views used in Brownfield to generate a solution need to be formal. This is not simply a question of using standard Visio or PowerPoint notations when developing business process models or business organization charts; it involves ensuring that those charts are created in tools that can export (and preferably) import their data in a computer-readable format.
But even more than that is involved: The way in which diagrams are drawn must be standardized. The data that forms the intersection point between Views must be sufficiently formal and agreed upon by both parties within each View. This is neither necessarily hard nor difficult to achieve because both sides should understand the intersections well. However, this introduces rigor and collaboration into areas with relatively informal documentation, with previous handover processes that consisted of merely “chucking it over a wall.” Brownfield imposes an approach that mandates precision and collaboration. That in itself is not a bad thing, as long as the community that creates the Views still readily understands them.
On one large customer-management project that needed a complex scripted question set with more than 1,000 questions, the format of the questions also dictated the format of the forms to be sent to the customers. The scripted questions were modeled in a tool, and this was used to generate the Artifacts to create the appropriate forms. Business analysts expressed initial trepidation and frustration toward using the modeling tools to manage the scripting process because of the precision in which they had to capture the information. Although theoretically they had already been capturing the information in a similar-looking format, they had been using Office drawing tools to do so, and precision had not been enforced. Over time, this frustration turned to satisfaction as they realized that the diagrams and spreadsheets they were creating (the Views) directly dictated the output of what was produced. No one could interfere with or misinterpret the material they were creating. These business analysts were managing to directly generate solutions to customers’ problems by precisely documenting the requirements. | ||
| --K.J. | ||
Indeed, in our direct experience, after the initial trepidation of dealing with something new, business analysts and users love the idea that what they define gets put into the solution without change. No one can dilute or misinterpret their ideas or concepts with Brownfield—they are essentially engineering the solution. Isn’t that how it should be?
If this book has convinced you to use Brownfield as a means of developing complex projects for your enterprise, how do you take the first step? Tools and industry capabilities are still developing, so no off-the-shelf tools are currently available. Inevitably, some developing of tools will be required.
Recommending that someone abandon existing practices and wholeheartedly adopt the Brownfield approach seems a huge risk when the stakes are so high. Indeed, the technical and cultural challenges encountered with such a radical approach will almost certainly lead to failure.
As with any new method, a better way is to introduce the approaches in this book into some aspects of the development. Then, as the tooling develops, cultural problems slowly are resolved, and confidence in the tooling increases, you can extend the approach into more areas.
Taking this slow approach to Brownfield will still give the project some of the benefits of the complete approach, and it will minimize the risk from tooling and cultural issues.
If you’re not eating the whole elephant, what parts of the elephant are worthy of first nibbling? A suggested first step is the interfacing aspects of the project. In any part of today’s solutions, interfaces are increasingly a major part of the development—they are also a major source of problems in elephantine developments.
Few solutions today have no interfaces to other systems. In the elephantine projects that are being considered here, none have no interfaces. That might sound like a bold statement, but it is true: Virtually all projects of the scale being considered require many interfaces to other systems.
Brownfield and MDA offer many advantages when it comes to developing interfaces. Perhaps one of the main advantages of developing such an interface is best illustrated by an analogy.
Consider that you have been asked to write a single-page story on a word processor. After you have finished the story, imagine that the editor gives you some more information to include in your story. Let’s assume that your editor has asked you to state that the hero has a motorbike and that the heroine is a nurse. You include this by making some changes to a few sentences. The editor now asks that the story include the fact that the scene is a beach. You change a few more sentences.
Each time, you can include the new information by making a few small changes to a couple of sentences. At some point, as more changes are added, the main flow of the story will be lost and it will have to be rewritten. When code is written in a conventional manner, it is similar to the writing of this story. The code will need to be changed slightly to accommodate changes in requirements. It will often not be quite as good as if it was designed with those requirements from the start. As more changes to requirements are made, the structure of the code deteriorates. As with the story, at some point it should be rewritten, or it will be difficult to maintain.
This is a particular concern with elephantine developments. Often with interfaces, the full requirements of interfacing with a heritage system are not fully known or documented and must be discovered.
Naturally, when developing such an interface using Brownfield techniques, the reason for each test failure is investigated. The reason normally is a constraint (remember, in Brownfield, these are seen as requirements) that was not previously captured. This constraint can now be updated into a source View as a new requirement and imported into the Inventory. From the Inventory, the updated solution can then be generated. This new updated solution satisfies the new constraint and all the previous constraints that existed for that interface. As this is generated, it is new code. Therefore, it does not contain the contamination of its earlier development history.
VITA provides a means of accommodating the discovery of the “new” constraints or requirements found during the testing. In the example described earlier, in which only 12 percent of the requirements were known at the start of development, it was possible to perform two complete iterations per day of the solution generation, complete testing, and incorporate the new constraints discovered in the testing. These iterations were all completed in a change-controlled and -managed process. Each iteration of the solution was an improvement on the previous one. It would be exceptionally difficult to accomplish this using a conventional Greenfield technique.
Elephantine solutions require linking the existing heritage application to new applications. Software known as middleware is used to link these applications. The introduction of Service Oriented Architectures (SOAs) that allow your applications to talk to each other via services has led to the development of new types of middleware. The Enterprise Service Bus (ESB) was created as this new form of middleware to control and manage these services.
ESBs provide a means of communicating between systems in the form of services. Each “service” on the ESB effectively translates the data from the source system to a common form used within the ESB. The ESB then converts this common form to the form required to every system.
This technique allows each service to be built separately. New systems can then use that service by simply creating a transformation (or translation) between the common form and its own specific form. In some ways, this is much like the generic Brownfield approach that allows everyone in the solution to speak in their own language. The ESB is used as the “translator” between these different systems.
The ESB is seen as the current nirvana in the field of system integration because each system effectively talks in its own language. Each individual system does not have to worry about the precise language required for the other side of the interface. Translation within the ESB handles the complexity. However, the Brownfield Inventory contains all the semantics and syntax of the source system, the common form within the ESB and the target system. That information is used to generate each of the required transformations to be built into the ESB. In this way, Brownfield now supports an ESB that will have its transformation configuration generated automatically and, more important, consistently.
Via the VITA principle described in Chapter 3, “Big-Mouthed Superhero Required,” Brownfield allows each stage of the ESB process to be defined and captured in the Inventory. In Figure 10.1, this is the Systems A and B format together with the common form. To convert among these three formats, two Transforms within the ESB must be defined; Figure 10.1 also shows these.
After the definitions of the data interface between the two systems and the common format are defined within the Inventory, then the relationships, or transformations, between these forms can be defined and captured in the Inventory. The configurations required by the ESB to perform this service can then be generated from the Inventory.
Using a common form allows different systems to use the same service by just defining the relationships between its form and the standard form. This allows the enterprise to have little dependency on particular systems. If a system is changed within the enterprise, only the relationships to the common form need to be redefined; all other systems will remain untouched.
Although this is perhaps useful, it has automated only a couple stages in the process that were previously performed manually. It marks some progress but is hardly revolutionary. The process still doesn’t seem to be taking full advantage of the fact that the Inventory holds the complete story.
An ESB performs more than just a translation service. It enables the enterprise to build up business processes by combining and linking these services. The term used to describe this is choreography, an apt name because the business processes can be thought of as a dance, with the services as the dance steps. Just as you choreograph a dance in terms of dance steps, you can choreograph your business processes in terms of services. The ESB contains all the information required to accomplish this.
However, the Inventory contains all the information about these business processes and the IT services available. The information required to produce the choreography is all sitting inside the Inventory. If the Inventory contains the information, that information can be used to generate the Artifacts that the ESB requires to control the choreography.
Brownfield has now used the Inventory to generate the entire configuration that the ESB requires. The configuration has all been generated from a single source of truth. As such, the configuration produced is entirely consistent. Brownfield now has managed to produce an easier way to keep control of the ESB. Brownfield allows the inputs for all the translations to be generated, along with any configurations required for the choreography. As such, the integration of the systems can be generated from a single source. This offers considerably more than just the generation of the translation aspects of the interface.
Does Brownfield have more to offer to ESBs? The ESB enables you to keep control of your business processes within the ESB. Brownfield gives you control at a level above the ESB. This means that the ESB is effectively reduced to a collection of patterns and techniques to provide the integration required. The ESB controls all operations that pass through it in a generic way. If the Inventory is now used to generate the configuration, it could also be used to generate the complete solution required to replace an ESB.
This would seem of little use because it is just replacing something that a business could purchase, with no real gain. This goes against one of the Brownfield beliefs to buy rather than create. However, ESBs are designed to pass messages in a very generic manner, as a simple way to maintain control of the business processes within the enterprise. Now Brownfield provides the area of control and can generate the solution in an optimal manner. Unnecessary translations can be removed, and business processes can be optimized. For example, translations could be performed directly instead of in stages via the common form. This is similar to translating directly between English and German rather than accomplishing it indirectly via Esperanto. The whole ESB and middleware layer could be generated. If this happens, the ESB itself could have a limited life; it could be replaced by a set of utilities that the Brownfield-generated Artifacts could use to route and queue the services as required.
Could the use of Brownfield-generated Artifacts mean the end of middleware? Perhaps. Looking at it in another way, however, the middleware is still there but is provided within Brownfield-generated Artifacts. Middleware in the future could be sold as a set of add-ons for Artifact generation in Brownfield.
Many large organizations often have sought a complete metadata description of the entire enterprise, yet they have rarely achieved it—and even fewer have managed to maintain such a description. Such initiatives have historically been associated with a Greenfield reboot of core systems. (This technique was particularly popular with data models in the 1990s.)
We come across the remains of such initiatives on a regular basis. Most have failed (often spectacularly) because this involves much work and provides little gain until it is completed. The architects involved have often been accused of sitting in ivory towers (or even, in one case, a gilded ivory tower).
Brownfield reengineering provides a means of producing a formal enterprise architecture in an evolutionary way, producing immediate gains. We believe that VITA—which means “life” in Latin—really does mean the beginning of life for enterprise architectures. The Brownfield development approach and the VITA architecture provide a mechanism for the embodiment of enterprise-level specificity and precision for solution deployment and control—no more ivory towers, high-level building blocks, technology choices, and abstract edicts. Such a view is probably controversial but could well prove justified.
Using Brownfield to build interfaces starts the collection of such data. In fact, by the time Brownfield has been used to generate all the interface configurations on a major project, you will have attained a large part of the goal described earlier: a metadata description of much of the business’s important information. However, this will not be achieved by people behind closed doors who work for years to deliver little business benefit. This will emerge as a byproduct of your development process, not some nugatory work carried out by an isolated team. Cherish this data—do not discard it. This is a major asset to your enterprise.
Your business can use this information not just to develop your applications, but to manage and control your enterprise. You can use the information in the Inventory to assess changes to the enterprise. You can use this information to check the ripple effect of changes to your systems, which currently is very difficult to assess. You can also assess new systems or packages in a similar way to see if they are a good fit for the enterprise—not with some high-level study, but with a detailed analysis at the lowest level.
Ultimately, the information in the Inventory and the generation capabilities facilitate not just the initial creation of the deployable Artifacts, but the ongoing maintenance and changes required to those systems to be automatically generated. The enterprise can then truly gain the flexibility that it has always sought. Changes can be accommodated quickly, in a controlled engineered manner, for a known cost.
Your enterprise could change completely—you just have to recognize that it is a Brownfield site and take the first step....