Chapter 11. Managing dependencies
| 11.1 | Introducing Ivy | |
| 11.2 | Installing Ivy | |
| 11.3 | Resolving, reporting, and retrieving | |
| 11.4 | Working across projects with Ivy | |
| 11.5 | Other aspects of Ivy | |
| 11.6 | Summary |
One topic that this book has been avoiding is this: how to manage library dependencies in Ant. We’ve avoided it because it’s hard to get right. We’ve covered the basic mechanics of setting up a classpath, but not where those JAR files should come from and how to look after them. It’s time to face the challenge of managing JAR dependencies properly.
The first step is to define the problems we have. What do we need to know?
1. How to set up the build with the JAR files that are needed to compile, test, and run the application
2. How to pass a JAR file created in one project into other projects built on the same machine, or even to other developers in a team
3. How to build child projects that depend on each other in the right order
4. How to switch library versions on demand
A single product addresses these issues, but it isn’t built into Ant. It’s an extension Antlib called Ivy. Before we introduce it, we have to look at the underlying problem: managing JAR files.
How to add libraries to an Ant classpath
The simplest way to manage libraries is to have a directory containing all the JAR files. To add new JARs, just drop them in. The classpath includes every JAR file in the directory:
<path id="compile.classpath"> <fileset dir="lib" includes="**/*.jar" /> </path>
This is the main way of managing libraries in Ant. If the directory is kept under revision control, then all developers who check out the project get the JARs. They can roll back the library versions alongside the code, so old versions of an application stay synchronized with the libraries. This technique works for small to medium projects. However, it doesn’t work for projects with multiple child projects, or when the classpaths for different parts of the build (compile, test, deploy, embedded-use, or standalone) are all radically different. In other words, it doesn’t scale.
Now imagine if a server kept all the released versions of popular JAR files. You could have a task that took the name and version of a file and retrieved it. It could cache these files on the local disk, so laptops could still build when they were off the network. Finally, if this repository was writeable, team members could use it to share artifacts across projects.
Such a repository exists, as do the Ant tasks to work with them. The repository came from the Apache Maven project, whose developers wrote a build system for fixing what they saw as defects in Ant. One idea—the Maven repository—called for a centralized repository of open-source artifacts. Every artifact in the repository is accompanied by metadata in an XML file—the POM file. This file describes the artifact and even declares which versions of other libraries the JAR depends on. The result is that the build tool pulls down both the direct and the indirect dependencies of an application.
That’s what Apache Maven does, and it’s one of the selling points of Maven against Ant. Yet we aren’t going to advocate a switch to Maven. There’s certainly some appeal in its conformance-over-configuration vision: “Follow the rules of the Maven team exactly, and you don’t need to write build files.” But we have too many problems following their rules. They not only dictate how to lay out your source, but they have a strict notion of your development lifecycle. If your workflow is more complex, with activities such as creating Java source from XML files, making signed JARs of the test files, or even creating two JAR files from a single project, you end up fighting Maven’s rules all the way.
We shouldn’t have to switch to a different way of building, testing, and deploying applications just to get the classpaths in Ant set up. This is where Ivy comes into the picture. It manages dependencies and nothing else—and it does this job very well!
11.1. Introducing Ivy
Ivy is a tool to manage libraries in Ant and to coordinate Ant builds across related projects. It was created by Xavier Hanin, of the French company Jayasoft, in September 2004. In October 2006, it moved into the Apache Incubator as a way of joining the Apache organization. Once it has finished its incubation period, it should become a fully-fledged Apache project, possibly under the http://ant.apache.org site.
11.1.1. The core concepts of Ivy
The main concept behind Ivy is that Ivy files describe the dependencies of a Java library or application, something they term a module. Every module or child project in a big application has its own ivy.xml file. This file lists what configurations the module has, such as compile, test, standalone, or embedded. Each configuration lists the artifacts it depends on and the artifacts it generates. The dependencies can be used to set up the classpath for part of the project or to compile files for redistribution. The generated artifacts can be published to a repository, including the local file system. Other projects can declare a dependency on the published artifacts, listing the name of the project, the module, and the version required. They can also declare the configurations of a module on which they want to depend. A module not only exports its own artifacts, but it can also export those artifacts on which it depends. “Downstream” modules can get all of these dependencies too, if they want.
This is easier to demonstrate than explain. Let’s get Ivy to manage our project.
Our first ivy.xml file
The first action is to create the ivy.xml file for diary/core, our core diary library. Listing 11.1 shows this file, which goes into the diary/core directory alongside the build.xml file. It states that the library has two dependencies and five configurations.
Listing 11.1. The ivy.xml file for the diary-core module
This ivy.xml file states that the project depends on revision 1.2.13 of log4j from the log4j team and version 3.8.2 of junit. We don’t want to compile the main source against junit-3.8.2.jar, because only the test code should be using it. By adding a conf attribute to the <dependency> element, we can control which configurations add the JARs and their dependencies to the classpath. For Log4J, we say:
conf="compile->default;runtime->default"
This means that our compile configuration wants the default configuration of Log4J, as does the runtime configuration. Similarly, the conf="test->default" attribute for the JUnit library restricts JUnit to the test configuration. That configuration also needs Log4J on the classpath, but we haven’t asked for it. Why not? Because test extends the compile configuration 
. This tells Ivy that the test configuration wants to inherit every dependency of compile. Similarly, the default configuration extends both master and runtime 
. The configuration will contain all artifacts in both inherited configurations.
The master configuration has no dependencies. This configuration contains nothing but the module’s own artifact 
, which tells Ivy that we publish the diary-core JAR file in this configuration. We don’t need to bother stating the name of the artifact, as it defaults to that of the module,
diary-core, from the organization called org.antbook. (Note that Ivy usually spells “organisation” with an “s” [European style] in its XML files.)
That’s a lot of information in a few short lines. It’s enough for Ivy to pull down the JARs needed for compiling and testing the library, to know what artifacts to publish. When someone else declares a dependency on the diary-core JAR from the antbook.org organization, they can get the JAR file itself, they can get everything it needs at runtime, or they can get both. We aren’t just configuring Ivy for our own use, we’re providing information—metadata—about our JAR for downstream developers.
How Ivy resolves dependencies
The metadata in the ivy.xml file can be used by Ivy to determine what libraries each configuration of a module needs. It can then look across repositories to find and fetch those files. Before it downloads the artifacts—usually JAR files—it looks for metadata on the repository about the file. It can read ivy.xml files and most Maven POM files. Both file types describe the configurations and dependencies of the artifacts that are being retrieved. Ivy looks at the metadata, determines what new dependencies there are, and follows them, building up a complete graph of the dependencies.
The tool then validates the graph, looking for errors such as cycles or unresolvable dependencies. It detects when more than one version of the same artifact is on the graph, which constitutes a conflict, and hands off the problem to a conflict manager, which is an Ivy component that decides what to do. Conflict managers decide which version of a file to use, such as the latest or the one asked for explicitly.
After all dependencies are resolved and all conflicts are addressed, Ivy can perform a number of activities. The key ones are generating an HTML report showing the dependencies of every configuration and downloading the needed artifacts.
The HTML report is good for developers, and it will be the first thing we’ll work on. The second action, copying down the JAR files, is exactly what we need in order to set up the classpath. Ivy can copy over all the dependencies of a project, and then Ant can add them to its classpath through simple <fileset> declarations. Ivy does the dependency management, while Ant does the build.
Let’s get Ant to set up the classpath for our diary-core module. Step one: install Ivy.
11.2. Installing Ivy
In early 2007, Ivy was moving into Apache, and its source was moving into the Apache Subversion server. This means that the location of the Ivy downloads is changing, and the final URL is still unknown. Start off at http://www.jayasoft.org/ivy, from where a link to the latest version will always be found. This chapter was written against Ivy 1.4.1, which was the last pre-Apache release.
The file ivy-1.4.1-bin.zip contained the JAR ivy-1.4.1.jar, which should be added to ANT_HOME/lib or ${user.home}/.ant/lib so that Ant will pick it up.
The JAR file contains the Ivy code and an antlib.xml file which declares the tasks. To use Ivy in an Ant build file, all we need to do is declare the XML namespace, currently:
xmlns:ivy="antlib:fr.jayasoft.ivy.ant"
Once this is at the top of the file, we can use any of the Ivy tasks in table 11.1.
Table 11.1. Ant tasks provided by Ivy 1.4.1. Expect more tasks in later releases
|
Task |
Function |
|---|---|
| <ivy:artifactproperty> | Sets a set of properties to the full path of every loaded artifact, deriving the property names from a supplied pattern. |
| <ivy:artifactreport> | Creates an XML report of all the dependencies and configurations of a module. |
| <ivy:buildlist> | Creates an ordered list of targets to use (for <subant>). The root attribute declares the target for which everything needs to be built. |
| <ivy:buildnumber> | Calculates a build number from the repository, one more than the latest version. |
| <ivy:cachefileset> | Creates a fileset containing the cached dependencies of a configuration. |
| <ivy:cachepath> | Creates a path containing the cached dependencies of a configuration. |
| <ivy:configure> | Configures Ivy for the rest of the build. |
| <ivy:deliver> | Generates a resolved Ivy file, expanding all properties and hard coding all dependencies. |
| <ivy:findrevision> | Sets a property to the revision number of a resolved artifact. |
| <ivy:info> | Sets some Ant properties from an ivy.xml file, including the organization, module name, and revision. |
| <ivy:install> | Copies a module from one repository to another. |
| <ivy:listmodules> | Creates a list of modules matching the criteria, and sets an ant property for each one. |
| <ivy:publish> | Publishes the current module’s artifacts. |
| <ivy:report> | Creates an HTML report of all the dependencies and configurations of a module. |
| <ivy:repreport> | Generates a report across a big project. |
| <ivy:resolve> | Resolves artifacts to the local cache, print output. |
| <ivy:retrieve> | Copies resolved artifacts to a local location. |
We’ll focus on the core tasks to manage our dependencies and leave the rest for the online documentation. The first task, the one that must be called in every build, is called <ivy:configure>.
11.2.1. Configuring Ivy
Before it can download artifacts, Ivy needs to know where to find them and where to store them locally. This information comes from a configuration file, ivyconf.xml. Every module/child project has its own ivy.xml file, but ivyconf.xml should be shared between all projects of a big application. Listing 11.2 shows the diary’s configuration.
Listing 11.2. The diary’s Ivy configuration file, diary/ivyconf.xml
The full details of this file are beyond the scope of this book. What’s important is that it defines the different resolvers for retrieving and publishing artifacts. They are as follows:
|
|
A <filesystem> resolver team, which contains shared artifacts belonging to the team. This repository is under SCM; all developers get a copy. |
|
|
An <ibiblio> resolver called maven2 to download files from the Maven2 repository. The root of the repository is set to http://ibiblio.org/maven2. |
|
|
The local cache, whose definition is in the file ivyconf-local.xml. This resolver is created by Ivy itself and defines the standard layout and location for downloaded artifacts. |
 |
The default resolver is a chain resolver, a sequence of the local, team, and maven2 resolvers. Files are located by looking in the cache, the team repository, and, finally, in the Maven2 repository. |
 |
Another chain, internal, defines how to search the local repositories. |
The <modules> listing adds one extra feature: it declares that artifacts belonging to the org.antbook organization should be searched for on the internal resolver, which searches only the local repository. This avoids searching the network for the files that we create ourselves.
Getting an ivyconf.xml file right is quite tricky. Start with a working example and tweak it carefully. There’s an Apache Ivy-user mailing list where you can find help.
The ivyconf.xml file configures Ivy, but it needs to be passed to Ant. This is the role of the <configure> task. It sets up Ivy for the rest of the build to use the listed resolvers/repositories. It has to be called once per build, before any other Ivy task:
<target name="ivy-init">
<property name="ivy.lib.dir" location="build/ivy/lib"/>
<ivy:configure file="${root.dir}/ivyconf.xml"/>
</target>
This ivy-init target defines a directory for retrieved artifacts, and then calls <ivy:configure> against the shared ivyconf.xml file. Every build file that uses Ivy must have a target like this, and all targets that use other Ivy tasks must depend on it, so it gets called at startup:
ivy-init:
[ivy:configure] Loading jar:file:/C:/Java/Apache/Ant/lib/
ivy-1.4.1.jar!/fr/jayasoft/ivy/ivy.properties
[ivy:configure] :: Ivy 1.4.1 - 20061109165313 ::
http://ivy.jayasoft.org/ ::
[ivy:configure] :: configuring :: file = C:diaryivyconf.xml
BUILD SUCCESSFUL
If there’s an error in the configuration, the task will print a message and the build will fail. If everything is successful, all the other tasks are available, of which the most fundamental is <ivy:resolve>. This resolves all of the dependencies of a single module.
11.3. Resolving, reporting, and retrieving
When the dependencies of a module are resolved, that means that Ivy has determined the complete set of dependencies for all configurations of the module. It has managed to locate all the artifacts, locally or remotely, and any associated metadata. Resolution, then, is the single most important action Ivy can do for a project, and something on which most of the Ivy tasks depend. For the diary application, that means that the dependencies get analyzed and the relevant versions of the JUnit and Log4J libraries retrieved—along with any of their dependencies.
To resolve the ivy.xml file, we use the <resolve> task, which only works if Ivy has been already configured with the <configure> task:
<target name="ivy-resolve" depends="ivy-init"> <ivy:resolve/> </target>
This operation will trigger a search for the file ivy.xml in the local directory, a parse of it, and a recursive resolve and parse of all dependencies.
ivy-resolve: [ivy:resolve] :: resolving dependencies :: [ org.antbook | diary-core |working@Zermatt ] [ivy:resolve] confs: [compile, test, master, runtime, default] [ivy:resolve] found [ log4j | log4j | 1.2.13 ] in maven2 [ivy:resolve] found [ junit | junit | 3.8.2 ] in maven2 [ivy:resolve] :: resolution report :: ------------------------------------------------------------------- | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| ------------------------------------------------------------------- | compile | 1 | 0 | 0 | 0 || 1 | 0 | | test | 2 | 0 | 0 | 0 || 2 | 0 | | master | 0 | 0 | 0 | 0 || 0 | 0 | | runtime | 1 | 0 | 0 | 0 || 1 | 0 | | default | 1 | 0 | 0 | 0 || 1 | 0 | ------------------------------------------------------------------- BUILD SUCCESSFUL
The output is a brief summary of actions: what configurations there are, how many artifacts are depended upon, and how many files were downloaded. In this run everything was in the cache; if an artifact was not there, here’s where it would be downloaded.
Once an ivy.xml file has been resolved, tasks that act on the resolved files and metadata can be used. One of these tasks is invaluable when setting up a build: <report>, which creates a report on all the dependencies.
11.3.1. Creating a dependency report
When setting up a project’s dependency and configuration information in the ivy.xml file, you need to know what’s happening. You need to know which files are in which configuration and whether there were any conflicts. Rather than analyze the build file logs or look at retrieved artifacts, the tool to use is Ivy’s <report> task. This task creates an HTML report of the resolved system. There’s also the <artifactreport> task, which outputs pure XML for further processing. For most developers, the HTML report is the most useful.
<target name="ivy-report" depends="ivy-resolve">
<ivy:report todir="${build.dir}/ivy/report"/>
</target>
This target generates an HTML page for every configuration in the chosen output directory. Figure 11.1 shows the output for the default configuration, whose sole dependency is Log4J.
Figure 11.1. Ivy reports the dependencies for one of the configurations
Only after the report matches your expectations should you move on to retrieving artifacts.
11.3.2. Retrieving artifacts
After resolution, a module’s dependencies are all in the local Ivy cache, which is in ${user.home}/.ivy/cache by default. That’s good, but it isn’t directly where the project can fetch them. Three tasks make these files accessible. The <cachefileset> and <cachepath> tasks set up an Ant fileset or path to the list of dependencies for a configuration:
<ivy:cachepath pathid="compile.classpath" conf="compile" />
The generated Ant datatype can be passed straight to tasks that take paths or filesets, including <java> and <taskdef>.
For this chapter, we will use the <ivy:retrieve> task. It copies all of the libraries of the selected configurations into a directory tree:
<target name="ivy-retrieve" depends="ivy-resolve">
<ivy:retrieve
pattern="${ivy.lib.dir}/[conf]/[artifact]-[revision].[ext]"
sync="true"/>
</target>
This copies the files from the cache into separate subdirectories, one for each configuration:
ivy-retrieve: [ivy:retrieve] :: retrieving :: [ org.antbook | diary-core ] [sync] [ivy:retrieve] confs: [compile, test, master, runtime, default] [ivy:retrieve] 5 artifacts copied, 0 already retrieved
If different configurations use the same artifacts, they get their own private copy from the local cache. When the task is run again, it won’t copy the files if they already exist:
ivy-retrieve: [ivy:retrieve] :: retrieving :: [ org.antbook | diary-core ] [sync] [ivy:retrieve] confs: [compile, test, master, runtime, default] [ivy:retrieve] 0 artifacts copied, 5 already retrieved
We also can look into a directory to see what’s there, such as the test configuration:
>ls build/ivy/lib/test junit-3.8.2.jar log4j-1.2.13.jar
Clearly, the dependencies are all set up. Incidentally, the sync="true" attribute of the <retrieve> task hands over complete control of the directory to Ivy. Ivy will delete from the destination directories any files that aren’t currently part of the configuration’s dependencies. This action lets us avoid having to clean up the directories when changing the dependencies of configurations; Ivy does it for us.
Once the artifacts are retrieved into directories, Ant can use the files.
11.3.3. Setting up the classpaths with Ivy
To compile using the retrieved libraries, we can use Ant’s <path> datatype declaration to set up classpaths from the configuration directories:
<target name="classpaths" depends="ivy-retrieve">
<path id="compile.classpath">
<fileset dir="${ivy.lib.dir}/compile" includes="*.jar"/>
</path>
<path id="test.compile.classpath">
<fileset dir="${ivy.lib.dir}/test" includes="*.jar"/>
<pathelement location="${target.jar}"/>
</path>
<path id="test.classpath">
<path refid="test.compile.classpath"/>
<pathelement location="${test.classes.dir}"/>
</path>
</target>
To test that everything is set up, we run ant clean test to run all unit tests against a fresh rebuild of the classes. As they pass, we know that the build is working, which means that the classpaths are set up right for the compile and test targets.
That’s it! We’ve used Ivy to set up the classpaths for the different activities in our application: compiling and testing. One has Log4J; the other has Log4J and JUnit. Ivy resolves the dependencies for each configuration, including inheriting the dependencies of other configurations, downloads the files from repositories, adds the dependencies of those files to the configurations, and then creates status reports or copies the artifacts over for Ant to use. That’s a pretty impressive set of operations.
We could stop there; it’s powerful enough. But Ivy does more, much more. We can use it to glue together projects, feeding the artifacts of one project into another.
11.4. Working across projects with Ivy
Ivy can pull down artifacts from the local cache, a team repository, or a remote server; it can be used to pass metadata and artifacts from one Ant project to another, each of which is, in Ivy terminology, a separate module with its own ivy.xml file. Each project’s build needs to publish the JAR files and other artifacts it creates into a shared repository. The ivy.xml files of the other projects declare a dependency on the created artifacts, and Ivy will locate the files during resolution.
11.4.1. Sharing artifacts between projects
The <publish> task will copy the artifacts of a project to the repository identified by a named resolver, here the “local” resolver:
<target name="ivy-publish" depends="sign-jar,ivy-init">
<ivy:publish resolver="local" pubrevision="${project.version}"
overwrite="true"
artifactspattern="${dist.dir}/[artifact]-[revision].[ext]"/>
</target>
This resolver is bound to a repository in ${user.home}/.ivy/local. All builds by the same user have access to the repository, unless their ivyconf.xml files say otherwise. Because it’s shared by all of the user’s projects, all the user’s builds can retrieve the artifacts published to it.
The task requires a version number for the files, which is set with the pubrevision attribute. In our builds, we’ve been numbering all artifacts, but Ivy doesn’t require this. You can create artifacts with simple names and have their revision number tacked on when the file is published. This is useful when retrofitting Ivy to an existing project. Ivy’s <ivy:buildnumber> task can even determine a build number from a repository, by determining the version number of the latest artifact in the repository and setting an Ant property to that value plus one.
With our build files already using version numbers, we avoid that step. We do have to tell Ivy how to locate the files, which is where the artifactspattern attribute comes in. It contains a directory path and a pattern for finding artifacts. The pattern [artifact]-[revision].[ext] says that artifacts are created with the pattern of name-revision and the expected extension. Unnumbered artifacts would need the simpler pattern [artifact].[ext]. What we don’t do is list the actual files that are created. This seems surprising, but it’s because the ivy.xml file declared the artifact already in its <publications> element:
<publications> <artifact conf="master"/> </publications>
This element declares that an artifact is published in the “master” configuration only. The artifact’s name defaults to that of the module, and it also has the default extension/type of JAR. Projects can create artifacts with different names or extensions. In chapter 14, for example, we’ll create the EAR file diary.ear:
<artifact name="diary" conf="master" type="ear" />
Again, this is invaluable when adding Ivy to existing code. There’s no need to change the name of existing artifacts; you only need to add extra metadata and extend the build process with the <ivy:publish> task, a task which must be run as part of the local installation activities:
This trace of the publish process shows the actions Ivy took:
|
Ivy placed a copy of the ivy.xml file into the same directory in which the artifacts were created. All Ivy variables in this file are expanded, so all version numbers and other late-binding values are expanded. |
|
Ivy copied the JAR file to the specified repository. |
|
Ivy copied the XML file to the specified repository. |
Sometimes Ivy doesn’t seem to update the metadata before publishing the file. Deleting the entire dist directory fixed this. A clean build is always safest.
The <ivy:publish> operation placed the file in a user-specific repository. There’s nothing to stop the task from publishing the file to a team repository, such as one on a shared file store. This would work well if it takes a long time to build some of the project’s artifacts, and there’s no need for every developer to rebuild every part of the application. The team repository would share the files for everyone to use. This brings up the next part of the problem: using the published files.
11.4.2. Using published artifacts in other projects
Published artifacts can be pulled into other projects simply by listing them in the <dependencies> section of the other project’s ivy.xml files. The web application coming in chapter 12 uses the diary-core library, so the application’s ivy.xml file declares a dependency upon the diary-core artifact:
This declaration has three unusual features. Instead of a hard-coded revision number, or even a property-driven value, the
requested version is latest.integration . This is a way of asking for the latest version published on the repository. Ivy will look at the repositories and find the
latest version. As the ivyconf.xml file declares that org.antbook artifacts come from the local and team repositories only, Ivy doesn’t poll any remote server. As well as asking for the latest
version, we declare that the artifact and its metadata are changing . Normally Ivy doesn’t update metadata files once they’re in the cache; instead it saves the complete dependency graph of
every configuration to its own XML file nearby. For static dependencies, caching all the dependency information makes resolution
much faster. For changing files and metadata, that cached data can stop changes being picked up. We need the most current
copy from the repository, along with its metadata. Unless we used <ivy:buildnumber> or the current timestamp to create a new build number on every build—which is an option—we need to tell Ivy to poll for metadata
and artifact changes on every build. This is what changing ="true" does.
Declaring the configuration to depend on
The final detail is that in every configuration, we ask for the “master” configuration of the diary-core module only . That gives us nothing but diary-core.jar—not its dependencies. When Ivy publishes artifacts to the repository, it adds the module’s dependency data. If we had compiled
with the default configuration via a compile->default dependency, we would have had Log4J on the classpath.
Note
When you depend upon other modules, you get all the dependencies of the selected configuration, not just the published artifacts of the modules.
This is an important feature of dependency management tools. Their goal is to simplify your life by giving you all the files you need. There’s a price: sometimes you get more than you want. As we don’t need Log4J to compile the web application and we don’t want it at runtime, we must ask for a configuration that contains the diary-core artifact but not its dependencies.
The choice of public configurations is based on those of Maven2, because whenever Ivy pulls in metadata from the Maven repositories, it parses the metadata and maps it to Ivy configurations. Maven only supports the configurations of table 11.2.
Table 11.2. Maven2’s dependency configurations, as interpreted by Ivy
|
Configuration name |
Meaning |
|---|---|
| default | The artifact and its runtime dependencies |
| master | The artifact itself |
| runtime | All the runtime dependencies of the artifact |
| compile | Dependencies used to compile the artifact |
| provided | Compile time dependencies to be pre-installed on the classpath |
| system | Runtime dependencies to be pre-installed on the classpath |
Ivy doesn’t mandate any specific configurations. We normally start off with the Maven team’s configurations to be consistent with artifacts coming from the Maven repository. The master configuration has the artifacts with no dependencies, runtime has the dependencies with no artifact, and default has both. We use these standard configurations to make it possible to publish artifacts to the central repository, something that’s beyond the scope of this book. We can still add private configurations for internal use (such as setting up a <taskdef> classpath) or to create new public configurations with different dependencies, such as embedded, redist, or webapp.
Having declared the dependencies on the master configuration only, we can run the new build.
Resolving the new project
The complete ivy.xml file for the diary web application is much more complex than the diary-core module. Running its ivy-resolve target creates a long list of dependencies:
ivy-resolve:
[ivy:resolve] :: resolving dependencies ::
[ org.antbook | diary | working ]
[ivy:resolve] confs: [default, compile, test, master, runtime,
war, jing]
[ivy:resolve] found [ org.antbook | diary-core |
0.1alpha-SNAPSHOT ] in local
[ivy:resolve] [0.1alpha-SNAPSHOT]
[ org.antbook | diary-core | latest.integration ]
[ivy:resolve] found [ rome | rome | 0.8 ] in maven2
[ivy:resolve] found [ jdom | jdom | 1.0 ] in maven2
[ivy:resolve] found [ javax.servlet | servlet-api | 2.4 ]
in maven2
[ivy:resolve] found [ httpunit | httpunit | 1.6 ] in maven2
[ivy:resolve] found [ xerces | xmlParserAPIs | 2.2.1 ] in maven2
[ivy:resolve] found [ xerces | xercesImpl | 2.6.2 ] in maven2
[ivy:resolve] found [ nekohtml | nekohtml | 0.9.1 ] in maven2
[ivy:resolve] found [ xerces | xerces | 2.4.0 ] in maven2
[ivy:resolve] found [ junit | junit | 3.8.2 ] in maven2
[ivy:resolve] found [ rhino | js | 1.5R4.1 ] in maven2
[ivy:resolve] found [ jtidy | jtidy | 4aug2000r7-dev ] in maven2
[ivy:resolve] found [ thaiopensource | jing | 20030619 ]
in maven2
[ivy:resolve] downloading C:Documents and Settingsant.ivylocal
org.antbookdiary-core .1alpha-SNAPSHOTjarsdiary-core.jar
[ivy:resolve] ... (12kB)
[ivy:resolve] .. (0kB)
[ivy:resolve] [SUCCESSFUL ] [ org.antbook | diary-core |
0.1alpha-SNAPSHOT ] /diary-core.jar[jar] (100ms)
[ivy:resolve] :: resolution report ::
[ivy:resolve] :: evicted modules:
[ivy:resolve] [ junit | junit | 3.8.1 ]
by [[ junit | junit | 3.8.2 ]] in [test]
[ivy:resolve] [ javax.servlet | servlet-api | 2.3 ]
by [[ javax.servlet | servlet-api | 2.4 ]] in [test]
This build log shows that the diary-core module was downloaded from the local repository. This is the artifact published earlier with the <ivy:publish> task; it can now go on the classpath of the web application.
The other interesting log item is that there were two “evicted modules.” The artifact junit-3.8.1 was replaced by junit-3.8.2, and version 2.4 of the servlet API was used instead of version 2.3. The older dependencies came from HttpUnit, which is something we’ll be using for testing the web application. It was built against older versions of the two libraries, versions picked up by the Ivy resolver. As newer versions were in the web application’s ivy.xml, the older versions were evicted and the newer versions retained on the dependency graph. Eviction information is included in the HTML pages generated by <ivy:report>, so team members can see what versions are being used. Figure 11.2 shows the report. It highlights that the diary-core artifact was found on the local repository, and checked for changes (“searched”), while two artifacts were evicted.
Figure 11.2. In this configuration, two obsolete modules are evicted.
Now that we are feeding the output of one project into the next, we can chain builds together just by running them in the right order. We can use the <subant> task to do this:
<presetdef name="delegate">
<subant verbose="true">
<filelist >
<file name="core"/>
<file name="webapp"/>
</filelist>
</subant>
</presetdef>
This <delegate> presetdef can delegate targets down to the child components, building and perhaps publishing the core component before the web application.
Ordering the builds by hand works if the dependencies are simple and known by the author of the master build file, but doing so doesn’t scale. Once you have more than a few projects, projects with different team members maintaining the build.xml and ivy.xml files, soon you have dependencies where you don’t expect. If the list of build files to delegate to isn’t kept synchronized, projects start building with outdated artifacts and developers start wondering why changes don’t propagate, or worse: you ship old code.
11.4.3. Using Ivy to choreograph builds
If ordering builds by hand is unreliable, the solution is to have the machine work out the correct sequence. Every module’s ivy.xml file lists that module’s dependencies. If something were to look through all these files, it could see what modules were being built and which ones depended on others. It could then sort the list of modules in the order needed to ensure that all a module’s predecessors were built before the module itself was built.
The <buildlist> task does this, creating a filelist of all the build.xml files ordered so that projects are built in the right sequence:
<ivy:buildlist reference="child.projects" skipbuildwithoutivy="true"> <fileset dir="." includes="*/build.xml"/> </ivy:buildlist>
We can use this task to create a <presetdef> task to delegate work to the build files:
<presetdef name="delegate" description="Delegate the build">
<subant verbose="true">
<buildpath refid="child.projects"/>
<property file="master.properties"/>
</subant>
</presetdef>
The path can also be printed out, to show the execution order:
<target name="show-order">
<property name="child.projects.property"
refid="child.projects" />
<echo>The order to build the projects is
${child.projects.property}
</echo>
</target>
Running this target shows the build order of the projects:
> ant -f ch11-ivy-masterbuild.xml show-order
Buildfile: ch11-ivy-masterbuild.xml
[ivy:buildlist] :: Ivy 1.4.1 - 20061109165313 ::
http://ivy.jayasoft.org/ ::
[ivy:buildlist] :: configuring :: file = C:diaryivyconf.xml
show-order:
[echo] The order to build the projects is
[echo] C:diarycoreuild.xml;
C:diarypersistuild.xml;
C:diarypersist-webappuild.xml;
C:diarywebappuild.xml
This run shows that Ivy can not only manage the problem of dependencies between projects, it also can use that same information to order builds.
There are a couple of aspects of the task to be aware of: looping and indirection.
Loops in project dependencies
If there’s a loop—that is, if two or more modules create a cycle in the dependency—the order of those looped projects is undefined. Modules before the loop will be built in the right order, and modules after the loop will be ordered, but there’s no way to predict the order Ivy will choose for those in the loop. This means that it’s acceptable to have loops in a project, but Ivy will not order the dependencies correctly.
Loops are not unusual in Java library projects, especially if one configuration of a project depends upon the output of another, which itself depends on the first program. Often you can unroll the loop by creating a new module that depends on everything.
Indirect dependencies
Ivy doesn’t order builds correctly if two projects have an indirect dependency via a module that isn’t in the <ivy:buildlist> fileset. For example, the persist-webapp project coming in chapter 14 depends on diary-core-persist, which itself depends upon diary-core. If the diary-core-persist project was excluded from the <ivy:buildlist> fileset, the task wouldn’t care that persist-webapp depended indirectly on diary-core, and not build the components in the right order.
To create a correct ordering of the build files, the fileset inside the task must contain all related projects. This doesn’t mean that you have to build them all, because the root attribute can restrict the list to only those build files that matter for a chosen model:
<ivy:buildlist reference="root.projects"
skipbuildwithoutivy="true"
root="webapp" >
<fileset dir="." includes="*/build.xml"/>
</ivy:buildlist>
This task passes in all build files with matching ivy.xml files, but only selects those projects that the webapp module depends on. The task can be used to build subsets of a system, using Ivy to manage the selection and ordering of the subset.
The <ivy:buildlist> task is the core Ivy task for complex projects. There’s a reporting task <ivy:repreport>, which can create a report about all artifacts in a repository. If a project publishes all generated artifacts to a private repository, this reporting task will list everything that gets produced and show the dependencies between artifacts. This is a useful piece of documentation.
There are some other details about Ivy worth knowing about, which we’ll take a quick look at.
11.5. Other aspects of Ivy
There’s a lot more about Ivy that’s covered in its documentation, and there’s no substitute for experience. Here are some techniques that are worth pulling out because they’re so important.
11.5.1. Managing file versions through Ivy variables
Ivy files support Ivy variables. They are like Ant properties, but they’re truly variable: they can be changed. They are expanded in strings just like Ant properties, so ${ivy.default.ivy.user.dir} is bound to what in Ant would be ${user.home}/.ivy. We set some properties in our ivyconf.xml, such as this pattern:
<property name="maven2.pattern.ext"
value="${maven2.pattern}.[ext]" />
Ivy variables make configuration easier, especially as ivyconf.xml files can import other ivyconf.xml files, which may define properties as well as other parts of the system configuration. Where Ivy variables are invaluable is in setting properties inside an Ivy file from Ant, because all Ant properties are turned into Ivy variables. Whenever an Ivy task is invoked, it has access to all of Ant’s properties. This behavior lets us control library versions through Ant properties. The build file can load a properties file of all library versions, and Ivy files resolved later will pick up these values. At the beginning of the build, we load in three files—one an optional file for user-specific customizations, one a child project-specific set of libraries, and the final one a declaration of versions for all projects/modules in the diary application:
<property file="build.properties"/> <property file="libraries.properties"/> <property file="../libraries.properties"/>
The shared libraries.properties file defines the default artifact versions:
junit.version=3.8.2 log4j.version=1.2.13 mysql.version=3.1.13
These properties can be used just as Ant properties in an ivy.xml file:
<dependency org="log4j" name="log4j" rev="${log4j.version}"
conf="compile->default"/>
<dependency org="junit" name="junit" rev="${junit.version}"
conf="test->default"/>
During resolution, Ivy expands the properties, so the resolved Ivy files and generated reports show exactly which version was used. It’s essential to manage dependency versions this way, to provide a single place to upgrade all versions of a library, across all child projects. Doing so also lets developers experiment with new versions, such as a later release of JUnit:
ant clean release -Djunit.version=4.1
As it’s now trivial to change from one version of an artifact to another, the hard problem is finding out what the name of an artifact is, and what versions of it are available. There are web sites and search tools to help with this.
11.5.2. Finding artifacts on the central repository
The best way to locate artifacts on the central Maven repository is to use one of the search engines, such as mvnrepository, http://mvnrepository.com/, and Maven Repo Search at http://maven.ozacc.com/. These repository search engines will take a name of an artifact and show the full organization and artifact name, as well as a list of public versions. You also can point a web browser at http://ibiblio.org/maven2 and browse around.
Usually the releases available on the repository lag public releases by a few days to a few months, depending on the project. When new artifacts do come out, the metadata with them can be a bit unstable; it can take time before the dependency information is correct. If one machine is building projects with different artifacts from the others, delete the machine’s Ivy cache and rebuild everything; this will pull down the latest versions.
11.5.3. Excluding unwanted dependencies
The dependency report of figure 11.2 shows a problem. Two XML parsers are coming in: xerces/XercesImpl-2.6.2.jar and xerces/xerces-2.4.0.jar. Neither version is needed, as Java 1.5 ships with Xerces built in. Rather than switch to HttpUnit’s master configuration and explicitly ask for all its other dependencies, we can drop the XML parsers by excluding artifacts coming from xerces.
<dependency org="httpunit"
name="httpunit"
rev="${httpunit.version}"
conf="test->default">
<exclude org="xerces"/>
</dependency>
An <ivy:report> run after this change shows that the files have been dropped. Artifacts to exclude can be specified by organization, module name, artifact name, or artifact extension, all of which can be given a regular expression containing wildcards for easier matching. A conf attribute can restrict exclusions to specific configurations.
The <exclude> element is invaluable. The more dependencies a project has, the more likely it is that unwanted files will end up on its classpath. Being able to selectively exclude artifacts addresses the problem without requiring a private repository with private metadata files, which is the other solution.
11.5.4. Private repositories
The central Maven2 repository on ibiblio.org has some problems. There is inadequate security for any security-conscious organization. There’s the cost and bandwidth of connecting to the servers and the risk that they can be unavailable.
A private repository addresses these problems. It can contain all approved artifacts, with dependency metadata managed by the team, instead of the team having to rely on stability and availability of an external repository for successful builds. The private repository promises better security, as only audited artifacts could be placed on the repository, and it provides a place where private artifacts can be stored, including Sun artifacts that cannot be served on the public servers.
Most big Ivy projects end up creating a team repository because of the control it offers. One task, <ivy:install>, simplifies the process of setting up the repository, because it can copy artifacts and metadata from one repository (usually the remote one) to another repository. It can populate the repository with the base files and metadata, metadata that can then be hand-tuned to meet a team’s needs.
11.5.5. Moving to Ivy
Existing Ant projects can add Ivy support. How hard is it to do so? It takes a few days of careful work. Here’s a workflow for the migration that can be used as a starting point:
1. Start after a release or in a period of calm and relative code stability. Nobody should be editing the build files or changing dependencies.
2. Create a single ivyconf.xml file by copying a working example.
3. Define a common set of configurations for all projects, such as those of table 11.2.
A common set of configurations can be shared across ivy.xml files with the <include> element:
<configurations> <include file="../ivy/configurations.xml" /> <configuration name="package" extends="runtime" /> <configurations>
Including configurations from a shared file keeps maintenance effort down and makes it easy to add new configurations to all applications. Unfortunately, Ivy 1.4.1 has a bug here: it tries to resolve relative paths in the project running the task, including <ivy:buildlist> in a parent directory. It should be fixed in a later release.
To summarize: migration to Ivy is manageable, but you should practice on something small first.
11.6. Summary
At the beginning of this chapter, we listed four problems we wanted to address.
- How to set up the build with the JAR files that it needs.
- How to pass a JAR file created in one project into other projects.
- How to build multiple projects in the right order.
- How to be able to switch library versions on demand.
Together, Ant and Ivy can do all of these things.
- If projects list their dependencies in an ivy.xml file, Ivy can retrieve them. Different configurations can be set up for the different paths and filesets a project needs.
- JAR files and other artifacts can be published by using <ivy:publish>.
- The <buildlist> task determines the correct order in which to build subsidiary projects.
- If library versions are defined in a central properties file, a developer can switch library versions just by setting the appropriate version property to a different value.
We’ll be using Ivy throughout the remainder of the book, though we won’t go into quite so much detail about ivy.xml configurations. They are all available online, and we’ll mention some aspects of Ivy configuration when it encounters a new feature or problem.
Now that we can set up libraries and dependencies between modules, it’s time to use the techniques by writing a web application. This web application will use Ivy to pull in external dependencies and to add the diary-core JAR to the web application itself.