But this analysis of complexity is somewhat putting the cart before the horse. Labels like “better” and “best” have meaning only within a specific context of measurement and interpretation of the resulting values. This context determines the specifics of how to compare two or more frameworks against each other.

For example, you could simply compare the number of Java methods associated with each API that you can glean from documentation on the assumption that the one with more functions is better.

We do not recommend this approach because it leaves you at the mercy of misleading marketing materials or incomplete documentation. Also, it is not clear that more methods make a framework better. However much people may disagree about the value of a given measurement or its interpretation, within that context there is no argument about which framework is better or best—and therein lies the value of objectivity.

The pentagon formed in the center of Figure 4.1 hints at much better context for comparison—if you consider it to represent the union of the features supported by the five options. You can think of each feature in the union as implying a “yes/no” question in a checklist, such as this:

A union of features is better because you are measuring against a relatively independent standard. An independent standard enables us to establish what an ideal framework should support, and compare what each mechanism actually supports against that ideal.

A helpful side effect is that an evaluation against a standard like a union of features is actually simpler than an exhaustive one—because you do not compare the features of every framework with every other. Instead, you merely compare the features of each mechanism to the checklist (making this an O(n) solution). And the comparison is still among “apples” if you consider the superset of features in the union as the “best” possible example of a persistence mechanism (sometimes called an archetype), as shown in Figure 4.2.

Figure 4.2. Using a union of features as an independent standard for comparison.

Developing a union of features is pretty straightforward. First pick an arbitrary framework and use its features as the starting point for the union. Then examine each other mechanism in turn, enumerating its features and adding any not already found in the current set.

For example, here are some introductory paragraphs about JDBC we found by browsing Wikipedia:

This cursory description leads to some good questions to ask about every framework that you might not have thought to consider, such as these:

Deeper digging into the materials shows that JDBC supports invoking arbitrary SQL statements and batching multiple statements into one request—providing full functionality and efficiency. Therefore, you should consider asking the following questions as well:

Turning next to iBATIS, its Wikipedia article begins with the following:

These features are important for ease of use and maintainability, and lead to even more questions to think about:

Hibernate, pureQuery, and JPA each add their own features to the union as well; but as you examine each framework in turn, you add fewer and fewer (because most of the features have already been encountered).

After you complete the union from the initial scan of each framework, the next step is to examine each again and determine whether it supports each of the features in the checklist. This second scan is important because it may be that a given mechanism supports a feature but never explicitly documented that fact. The total number of features supported usually increases. At the same time, you find out how many features a framework does not support.

It is possible to compare as you go while developing the union as long as you recheck each previously evaluated mechanism whenever a new feature is added. However, this approach is functionally equivalent to a second scan.

To illustrate, Table 4.1 summarizes the evaluation against the checklist questions we discussed so far.

It is reasonable to consider the framework with the most features supported, or the fewest not supported, or the largest percentage as “the best” within this context. Any of these three measurements used to summarize the table would serve as a more meaningful context for comparison than a simple count of advertised features.

Comparing to a union of features is not likely to be a satisfactory basis for your choice of mechanism because you are still at the mercy of what the framework documentation has to say about which features are important. Even though you have a collective group of opinions to consider, the basis for comparison is not truly independent.

An even more meaningful context for comparison is to create a list of your project’s requirements with respect to persistence features and then evaluate each framework against that checklist. This focus on your project requirements also ensures that the label “best” has a practical meaning—it is applied to the mechanism that does the best job of meeting your needs.

One benefit is that even less effort is required than that to compare against a union of features. You do not need to exhaustively research all the frameworks first to develop the union; you simply determine whether a given framework has the required features—which are likely to be a much smaller subset than the complete union, but still another form of “apple,” as Figure 4.3 shows.

Figure 4.3. Using your project requirements as a more practical standard for comparison.

Focusing on your immediate needs can further simplify an evaluation because you can stop considering an option that fails to meet a “must have” requirement. For example, project “A” may require a robust ORM that allows you to reject JDBC as the mechanism early in the evaluation process.

A more complete approach would be to weight each requirement based on its importance to your project, and then choose the framework with the highest score.

However, this continual comparison to project requirements, even though they are a smaller list than the union, is reminiscent of an exhaustive comparison (Figure 4.4 illustrates the similarity to Figure 4.1). In fact, the long-term costs are likely to be more than for an exhaustive comparison.

Figure 4.4. Long-term cost of using individual project requirements as a context for comparison.

But there are other issues with this “every man for himself (or herself)” approach to evaluation. One is that you are spending time evaluating frameworks when you should be implementing applications. Another is that an inexperienced team may not know the right questions to ask. And still another is that you may end up with inconsistent implementations across the enterprise even when the projects are similar.

However, you should consider the possibility that you need to choose two or even more frameworks for use within the enterprise—not because they end up with the same objective score in an evaluation, but because there are important underlying differences in the requirements of different types of projects.

So when you develop your checklist of enterprise requirements, create a weighted subset for each kind of project that you are likely to encounter, and optimize the choice of framework for each type.

For example, it is likely that highly tuned custom applications such as the order entry system for project “A” (we might call these “compiled” applications) would benefit from an object style programming model provided by Hibernate, or OpenJPA. But it may be that highly dynamic “metadata”-driven applications like the marketing analysis system developed by project “B” would require iBATIS or pureQuery (we might call these “interpreted” applications). Figure 4.6 graphically shows the relationship between these two project types and individual project requirements within the enterprise.

Figure 4.6. Relationship of project types to the enterprise and project requirements.

The possibility of choosing multiple frameworks implies that you should ensure that they can work together—adding the ability to integrate with other frameworks, such as JDBC, to your checklist of enterprise requirements to consider. Some requirements, like this one about integration, may be important for a number of project types—implying that a hierarchy of project types would be useful to minimize redundancy.

But the problem with checklists of features phrased as “yes/no” questions is that for most interesting aspects of persistence (and life in general), the answers are not really black and white. Most of the time there are shades of gray.

For example, consider the question about whether a framework is available under open source. The description about Hibernate found in Wikipedia provides an answer:

The answer was much more specific than just yes, and indicated that it was available under LGPL. We explore LGPL and other license agreements like GPL and Apache with respect to their impact on your ability to sell your applications in Chapter 2 “High-Level Requirements and Persistence.” In short, there are usually trade-offs to consider for each one, and most frameworks would answer yes to only one in the set. So rather than asking a number of related “yes/no” questions, like

it would be better to ask a single, more open-ended “what” question, such as the following:

This single “what” question not only minimizes the number of questions you have to ask, but also makes it clear that you are evaluating an important aspect of your enterprise requirements.

This transformation of many related questions into one that is somewhat open-ended also illustrates that most often, it is not a question of “if” a given framework supports a requirement, but one of “how.” In other words, the devil is in the details.

The underlying question we are after concerning the type of license agreements is this:

A “how” question like this is usually too open-ended for an apples-to-apples comparison—leading us to substitute one or more “what” questions like that given previously. For another example, consider the third sentence from the Hibernate description again (bold italics ours):

Certainly every framework would like to claim that it is easy to use, so asking a “yes/no” question is rather useless; a completely open-ended “how” question is nearly so:

One reason is that the notion of whether a framework is easy to use is a subjective evaluation that applies to the framework as a whole. A better “how” question is specific and objective:

Within this context, the framework with the fewest steps across all the tasks evaluated would be considered the “best” with respect to ease of use. The questionnaire to drive the data gathering might start with a “what” question around a specific feature, coupled with a very specific question about how that particular feature is used for that task. For example, support for isolation levels could be objectively evaluated with the following two questions:

The details provided by the “how” questions make it easy to objectively compare the complexity of the solution with that of others—as long as the same example is used for each framework to minimize the open-endedness.

An evaluation based on this amount of detail allows you to apply Occam’s Razor and choose according to its age-old best practice: “All things being equal, the simplest solution tends to be the best one.”

This section is most closely related to Chapter 1, “A Brief History of Object-Relational Mapping,” where we discuss the various frameworks that you might want to consider in your evaluation. Specifically, you want to be able to answer the following general questions:

This section is derived from the discussion about business drivers in Chapter 2 and is intended to capture some of the more intangible “costs” of moving to that particular mechanism. A reviewer should be able to answer the following questions:

The purpose of this section is to drill down to see how the framework can be used to support the basic usage patterns found in Java applications needing persistence discussed in Chapter 2.

The questions in this section are derived from Chapter 3, “Designing Persistent Object Services,” first focusing on the static definitional elements of the ORM problem. Specifically:

This section is derived from questions in Chapter 2 concerning quality of service IT requirements. You should be able to answer questions like the following:

The purpose of this section is to show the end-to-end steps required to develop the services shown in the common example outlined in Chapter 3. The overarching questions include the following:

To help offset the cost of customizing the questionnaire, we recommend that you reuse it as a starting point for your project requirements. This approach will ensure that your more inexperienced project teams do not overlook a crucial requirement.

Also, constant use of the questionnaire on actual projects gives you lots of opportunity to validate them and keep them current. If a new project represents a new type or comes up with some new requirements, you should first add them to your enterprise questionnaire. Then consider whether to evaluate each project in progress with respect to the new requirements.

Depending on the situation, you may need to revisit your choice of persistence frameworks being used in your production applications—especially if you found a problem during development and testing that led to the new requirements to consider in the future.

We have seen a questionnaire like this one used in other ways, too. In one project, we used it to review the project team’s choice of persistence mechanism. This questionnaire helped us to ask the right questions in a systematic fashion and get to the heart of the reasons for their choice.

Whether you use this questionnaire in practice to conduct your own formal evaluations or review the work of others, make sure to record your answers and choices so that you don’t forget them later and have to revisit.

A soft-copy version of the questionnaire in Microsoft Word is available on the download site, ibmpressbooks.com/title/9780131587564. You are welcome to use this template as-is or as a starting point for your own evaluation and review questionnaire.

We use this same template as a starting point for the next five chapters, which evaluate JDBC, iBATIS, Hibernate, OpenJPA, and pureQuery. Feel free to contact any of the authors if you would like to share new questions or reviews of other persistence mechanisms—whether exciting new ones or exciting ones still in common use. In this manner we can all learn from each other and avoid repeating ourselves.