Using Ivy for Dependency Management

Adrian Sutton

At first glance, Ivy looks like a re-implementation of Maven’s dependency management that works nicely with ant, and to some degree it is, but it also adds some pretty significant improvements and some pretty significant complexity.

Maven Compatibility

Firstly, Ivy is compatible with Maven repositories, so if you think the way Maven manages dependencies is perfect, but don’t want to buy into the rest of Maven, Ivy provides a good answer. The configuration is a little bit different and you’ll have to learn a little bit about Ivy’s configurations which are both more powerful and more complex than Maven’s dependency “scope”, but you won’t have to go too far into them.

Ivy will use the Maven 2 repository by default so all the same libraries are available – complete with all the metadata problems.

Ivy Configurations

Configurations are probably the biggest difference between Ivy and Maven’s dependency management. At the simplest level they are roughly equivalent to setting the scope attribute in Maven. It lets you choose whether a library is required only for compilation, if it should be included in the packaged WAR/EAR or if it’s used only for testing. Since Ivy doesn’t actually build the project, what a configuration means is a lot more flexible than what I scope means in Maven. Each Ivy setup needs to setup the configurations it needs from scratch and it’s up to the Ant build process to ensure the libraries are used as intended.

If you import a project from the Maven repository, Ivy will convert the various scopes into the following configurations:

<th id="id_6274">
  Description
</th>

<th id="id_6275">
  Example Library
</th>

<td headers="id_6274">
  runtime dependencies and master artifact can be used with this conf
</td>

<td headers="id_6275">
</td>

<td headers="id_6274">
  contains only the artifact published by this module itself, with no transitive dependencies
</td>

<td headers="id_6275">
  The project’s jar itself
</td>

<td headers="id_6274">
  this is the default scope, used if none is specified. Compile dependencies are available in all classpaths
</td>

<td headers="id_6275">
  commons-lang
</td>

<td headers="id_6274">
  this is much like compile, but indicates you expect the JDK or a container to provide it. It is only available on the compilation classpath, and is not transitive
</td>

<td headers="id_6275">
  Servlet APIs
</td>

<td headers="id_6274">
  this scope indicates that the dependency is not required for compilation, but is for execution. It is in the runtime and test classpaths, but not the compile classpath
</td>

<td headers="id_6275">
  An AOP runtime library?
</td>

<td headers="id_6274">
  this scope indicates that the dependency is not required for normal use of the application, and is only available for the test compilation and execution phases
</td>

<td headers="id_6275">
  JUnit
</td>

<td headers="id_6274">
  this scope is similar to provided except that you have to provide the JAR which contains it explicitly. The artifact is always available and is not looked up in a repository
</td>

<td headers="id_6275">
  ??
</td>

<td headers="id_6274">
  this configuration contains the source artifact of this module, if any
</td>

<td headers="id_6275">
  Source for the project
</td>

<td headers="id_6274">
  this configuration contains the javadoc artifact of this module, if any
</td>

<td headers="id_6275">
  JavaDoc for the project
</td>

<td headers="id_6274">
  contains all optional dependencies
</td>

<td headers="id_6275">
  Anything optional
</td>
Name
default
master
compile
provided
runtime
test
system
sources
javadoc
optional

The three most commonly used configurations for dependencies are compile, provided and test – most projects will only ever need to add dependencies in those categories. The master, sources and javadoc configurations typically don’t contain any dependencies, they are used for publishing the results of the build into – the same configurations defined here are used by other projects that depend on this one. The default configuration is provided to make this easy, by including all the runtime dependencies and the project’s jar itself in one configuration.

It’s also worth nothing that configurations can extend each other. In the Maven import, runtime extends compile so any libraries you compile against are also available at run time. Default extends runtime and master. You could of course ignore the extends functionality and just include multiple lib dirs on the classpath within your normal ant script as you would if the jars were directly checked in to source control, but the functionality is reasonably simple and makes it very explicitly clear what jars are on what classpath1{#footlink1:1264424269169.footnote}.

My approach has been to stick with these configurations – they’re simple enough but cover every conceivable situation that I can currently imagine. The only real difference I’m not currently using the master configuration – the produced jars are just put directly into default. That means you can’t easily require the library without any dependencies, but it means that the defaults can be used for what most libraries produce (a single jar file with the same name as the project and in the default configuration).

Ivy Repositories

Ivy has it’s own concept of a repository that uses Ivy descriptor files instead of the Maven POM files and gives the full power, flexibility and complexity of Ivy’s configuration concepts. The configurations might be handy, but what I find most useful, is the fact that Ivy supports multiple different types of repositories. So instead of having to run specific software to provide the repository, it can be accessed via sftp, ssh, a shared drive or even subversion. This makes setting up your own repository significantly easier – it’s quite likely you already have a server that’s available via ssh somewhere. I also like the fact that the whole Ivy configuration can be included in the project itself, so once you check out the source code, you have all the config settings you need to be able to build it2{#footlink2:1264426193822.footnote}.

Importing Libraries

Ivy can pull libraries from the public maven repository and import them into your private one which makes it reasonably quick to spin up a private repository and get rid of the dependency on the public one altogether. However, it’s still a fairly slow process and takes up the bulk of the time required to get Ivy up and running. That said, it immediately solves a lot of headaches about incorrect meta-data or missing libraries in the maven repository.

What I found was missing however, was a simple tool to add new libraries that don’t exist in the Maven repository. For a single jar, it’s pretty easy to create the directory structure and an ivy.xml file for it, but when you have a large number like the GData APIs, it can take up a huge amount of time. I whipped up a simple bash script to work out most of the meta-data from a jar file (module name and version number), create the directory structure and a simple ivy.xml file then upload it to the repository. It doesn’t add the dependencies automatically but that’s easy once the directory structure and basic ivy.xml has been created.

Namespaces

One of the most common problems with the Maven repository is the number of Apache libraries that still use the Maven 1 naming scheme. For example Commons IO is available under both the org.apache.commons group and the commons-io group. To Maven and Ivy, the different groups mean it’s actually a different library so it can potentially wind up on the classpath twice. When you import libraries from the Maven repository, ivy lets you configure namespaces to avoid this problem. Essentially, Ivy will rewrite the group name to consistently use one or the other, resulting in a more consistent repository and no duplicate jars on the classpath. The rename is done both for the imported library and anything else you import that depends on it so things automatically point to the right place.

Actual Usage

Actually using Ivy is really quite straight forward. There are a lot of different ant tasks you can use, largely due to the huge amount of flexibility Ivy provides, but you can also keep it quite straight forward. I’ve got five basic uses at the moment:

  1. configure – I’m calling this outside of any target at the moment so it always happens before anything else. It sets up Ivy using the configuration file designed for this project. So it has the right configurations set up and points to the private repository instead of the default Maven one.
  2. retrieve – actually calculate the dependencies and link them into the specified directories. Each configuration gets the jar files placed in it’s own sub-directory. From then on, to use the dependencies in the ant script, just point at the appropriate directory. I also set this to use symlinks when possible so the files actually stay in Ivy’s cache directory and it just creates a symlink to save some time.
  3. Inline retrieves – the retrieve task can also be configured to retrieve a library that you specify directly via attributes instead of in an ivy.xml file. This is really handy for things that the build scripts itself want to use, like a scala compiler or cobertura. Otherwise each project would have to have the same dependencies.
  4. Import – I have an ant task set up specifically to make it easy to import libraries from the Maven repository. It uses a slightly different settings file (which includes the Maven repository) and prompts for the group, module and revision to import.
  5. buildlist – this is a handy little task for building sub-modules within a project. It looks at each project’s dependencies and calculates the correct build order which you can then pass in to subant. Very much like the Maven reactor.

The nice thing is that Ivy just plugs in to your existing ant file without many changes at all.

Tracking Versions

Ivy has some nice ideas for working with versions which I haven’t found in Maven before though they may exist. Firstly, it can generate a build number by looking at the latest version available in the repository and adding one which is much better than having a normal ant buildnumber file on a shared drive somewhere. Secondly, when Ivy published an artifact to the repository, it rewrites the ivy.xml to include the specific version of dependencies that it was built with, so anything that depends on it will get known good libraries, even if it was using a SNAPSHOT style build. In Ivy the SNAPSHOT version is equivalent to latest.integration, but it always generates some form of version number when the build is published, so you never get a latest.integration version in the repository. By default it uses a date stamp which actually works out really well.

Aside from latest.integration you can also specify a range of restrictions for acceptable versions which is handy but probably too complex to be worth the effort in most cases. Ivy already resolves most version conflicts automatically by evicting the older versions which works fine as long as the libraries maintain backwards compatibility3{#footlink3:1264429445570.footnote}. So when commons-httpclient requires commons-logging 1.0 but commons-io requires commons-logging 1.1, you wind up with just commons-logging 1.1 on the classpath and everything works.

Sharing Projects and Modules

Ultimately, the main reason I was looking to move away from jar files in source control was to make it easy to share modules between different projects. Right now, that would basically have to be done by either forking the code, occasionally building a jar and dropping it in or using svn:externals – none of which are particularly appealing. With Ivy, the buildlist can handle any submodule being dumped into the project directory and you can pull a version of the module from the repository regardless of whether or not you have the source code checked out for it. There are now two modes you can use sub-modules in:

  1. Just declare it as a dependency and Ivy will grab the version from the repository. Just like including a jar in source control but a little easier.
  2. Declare it as a dependency and check out a copy of the source as a submodule of the project you’re currently working on. This then gets picked up by the Ivy buildlist task and automatically inserted in the right place in the build process. Now you can easily make changes to the module and use them in your outer project without doing a full release. You only need to push a new version into the shared repository when you’ve got the bugfix or new API completely sorted out and ready for others to use. In the mean time, everything still builds with a single execution of ant.

Speed

The project I tested this out on has a lot of dependencies – it took ages to import them all into the private repository, many required manually fixing up metadata problems as well. With the jar files in subversion, the build server could check out the full source (including libraries) and build the project in 3-4 minutes (the build server and the subversion repository have a gigabit connection between them). The first build it did with Ivy took the build time up to about 25 minutes as it downloaded all the dependencies (the Ivy repository was in the US, the build server was in Australia), but because Ivy keeps a local cache of everything it downloaded the second build went back down to the usual 3-4 minutes.

Ivy can be a bit slow at calculating dependencies though I probably wouldn’t have noticed it if the project didn’t have as many submodules, each of which require Ivy to calculate the dependencies before building in. In the grand scheme of things though, the time that Ivy spends working out dependencies is small enough to be insignificant compared to whatever the rest of the build is doing. Since the jar files are just dropped in to a normal directory, it’s also easy to add a flag to completely skip Ivy and use the existing dependencies if you do have small tasks that are run really frequently. With a remote subversion server you can waste far more time updating, moving or deleting jar files.

Summary

I think Ivy is a pretty clear winner. It’s simple to set up a private repository and avoid all the common problems people hit with the Maven repository, what complexity Ivy adds can be isolated to the template build scripts so individual projects stay quite simple and with buildlist it’s now easy to share modules between projects which had been causing me a lot of headaches.

The downside: there’s a reasonable amount of learning that the team will have to take on and at first glance Ivy looks like it adds more complexity than benefit so getting buy-in isn’t a guarantee.

1 – once you “resolve” the dependencies you can simply look in the appropriate directory and see all the jar files for that configuration in one place↩

2 – I also put the Ivy jar files in subversion so that it’s available and doesn’t need to be installed separately. So the requirements to build are just a JDK, ant and a checked out source tree↩

3 – and if they don’t you’re in a lot of trouble as the project that wanted the newer version is unlikely to work with an older version anyway.↩