The P2 artifacts file provides a way to bind an installable unit to a downloadable file. Given a triplet of classifier, id, and version, the artifacts file allows an installable unit's URL to be calculated. It also provides some additional information such as what the expected download type of the file is, its size, and optionally an MD5 checksum.

A repository has a human-readable name, a type, and a version. Underneath it are three sections; a list of properties, a list of mappings, and finally a list of artifacts. The following XML shows an example of a repository called Update Site:

<?xml version='1.0' encoding='UTF-8'?>
<?artifactRepository version='1.1.0'?>
<repository name='Update Site' type='org.eclipse.equinox.p2.artifact.repository.simpleRepository' version='1'>
  <properties size='2'>
    <property name='p2.timestamp' value='1396184010474'/>
    <property name='p2.compressed' value='true'/>
  </properties>
  <mappings size='3'>
    <rule filter='(&amp; (classifier=osgi.bundle))'
     output='${repoUrl}/plugins/${id}_${version}.jar'/>
    <rule filter='(&amp; (classifier=binary))'
     output='${repoUrl}/binary/${id}_${version}'/>
    <rule filter='(&amp; (classifier=org.eclipse.update.feature))'
     output='${repoUrl}/features/${id}_${version}.jar'/>
  </mappings>
  <artifacts size='1'>
    <artifact classifier='org.eclipse.update.feature'
     id='Feature' version='1.0.0.201403301353'>
      <properties size='2'>
        <property name='download.contentType'
         value='application/zip'/>
        <property name='download.size' value='338'/>
      </properties>
    </artifact>
  </artifacts>
</repository>

The properties are used to provide additional information about the artifacts; in this case, the timestamp at which the content was last generated and whether the content of the repository should be compressed into an artifacts.jar file.

The properties can also contain a mirror reference with the p2.mirrorsURL. This allows a set of mirrors to be queried for an artifact instead of just the originating server; the Eclipse infrastructure uses this to share the load between mirror sites when new versions of Eclipse are released. When an artifact needs to be downloaded, the mirrors URL will be hit, and an XML file will be returned.

For Eclipse Luna, the mirror URL is http://www.eclipse.org/downloads/download.php?format=xml&file=/eclipse/updates/4.4/R-4.4-201406061215.

This is encoded in the artifacts.xml file as follows:

<property name='p2.mirrorsURL'
 value='http://www.eclipse.org/downloads/download.php?format=xml&amp;file=/eclipse/updates/4.4/R-4.4-201406061215'/>

Because XML files cannot contain ampersands (&) without escaping, the XML file has & to separate parameters. The returned XML file looks like the following:

<mirrors>
  <mirror
   url="http://www.mirrorservice.org/sites/download.eclipse.org/"
   label="[United Kingdom] UK Mirror Service (http)"/>
  <mirror url="http://ftp.snt.utwente.nl/pub/software/eclipse/"
   label="[Netherlands] SNT, University of Twente (http)"/>
  <mirror url="http://eclipse.mirror.triple-it.nl/"
   label="[Netherlands] Triple IT (http)"/>
  ...
</mirrors>

Although the Eclipse servers generate the request dynamically with a PHP script, this could be provided with a static XML file or another automatically generated mechanism.

The Eclipse servers send out both HTTP and FTP mirrors; for sites that have firewalls that don't support FTP, it's possible to add &protocol=http to get just a list of HTTP mirrors.

The contents file contains much more information and records properties extracted from the bundle, such as the license, the copyright, and vendor, and for OSGi bundles, what packages are imported and exported along with other generic require-capability elements. There is a one-to-one mapping between entries in the artifacts file with entries in the content file.

Along with JARs hosted on an update site, other binary content can be stored and served. There are two categories for non-JAR content:

Both of these can be stored either as top-level files in the binary directory or in a special .blobstore folder.

The blobstore is a means of allowing arbitrary content to be stored without the file extension causing problems for the update site server, by creating randomized filenames to distinguish between assets. Because the names of the files in the blobstore do not correspond to any well-known algorithm (such as md5 or sha1), the names of the files give no information about its content. This can be used for storing binary executables as well as packed JAR files.

The blobstore causes problems for some webservers that don't know or expect a particular type of data. As a result, it is often desirable for a P2 repository to store pack200 files next to the JARs instead of the blobstore. This is achieved by storing a property packFilesAsSiblings with the value true in the artifacts.xml file of the destination repository:

<property name='publishPackFilesAsSiblings'
 value='true'/>

This will ensure that the blobstore is not used for writing out the content of pack files, but they will be put next to the JAR files instead.

Although a direct file or rsync copy will allow a set of artifacts to be mirrored, it is possible to use built-in functionality in Eclipse to allow P2 repositories to be mirrored. This will ensure that the contents listed in the remote update site are transferred as expected, and the metadata files are updated correctly.

Mirroring is done separately for artifacts and metadata, but they both follow the same structure. One mirrors the artifacts.jar files, and the other mirrors the content.jar files.

To mirror Luna's artifacts, use the following command:

$ /path/to/eclipse -consoleLog -noSplash -application
   org.eclipse.equinox.p2.artifact.repository.mirrorApplication
 -source http://download.eclipse.org/releases/luna/
 -destination file:///path/to/luna-mirror
 -verbose
 -raw
 -ignoreErrors

The -verbose flag tells the mirroring process to print out what is being copied at each point and is an optional argument.

The -raw flag tells the mirroring process to copy as is without translating or rebuilding the metadata from the original bundle. This is faster, but it can sometimes cause problems when mirroring from old-style update sites. This only has an effect on artifact mirroring and is an optional argument.

When the -ignoreErrors flag is given, any errors seen during the mirroring operation are ignored, causing the mirroring to continue. If not specified, then any error will terminate the mirroring process. This only has an effect on artifact mirroring and is an optional argument.

To mirror Luna's metadata, use the following command:

$ /path/to/eclipse -consoleLog -noSplash -application
   org.eclipse.equinox.p2.metadata.repository.mirrorApplication
 -source http://download.eclipse.org/releases/luna/
 -destination file:///path/to/luna-mirror
 -verbose

The previous two commands are almost identical; the only difference is the application name.

While mirroring, P2 will take advantage of any mirrors found. If the remote site lists a set of mirrors, then P2 will consult the mirror lookup and download assets from mirrors in order to spread the load.

The mirroring process can also verify the MD5 signatures of the files when they are being mirrored. This can be disabled by passing the command-line arguments -vmargs -Declipse.p2.MD5Check=false.

Some old update sites do not have P2 metadata generated; instead, they just have a site.xml file as a classic update site. While this works in older versions of Eclipse, it may cause errors with a new Eclipse install (or a Tycho build) with a cryptic error message that reads Update site contains Partial IUs and cannot be used.

To generate P2 metadata from a folder that contains features/ and plugins/ directories, the following Eclipse command can be run. Given a directory DIR, the following command will generate the content.jar and artifacts.jar files:

$ /path/to/eclipse -consoleLog -noSplash -application
   org.eclipse.equinox.p2.publisher.FeaturesAndBundlesPublisher
 -source DIR
 -metadataRepository file:/DIR
 -artifactRepository file:/DIR
 -compress

Note that DIR must be a filesystem, but the -metadataRepository and -artifactRepository arguments require URLs with a file:/ prefix.

The -compress argument tells the P2 publisher to generate an artifacts.jar that contains the artifacts.xml file; similarly, the content.jar contains content.xml. While the standalone XML files are more human-readable, they are often between two and ten times smaller when compressed.

The previous command will generate the artifacts.jar and content.jar files in the same location as the features/ and bundles/ directories, which is often what is required. However, if the output directory is in a different location, then only the artifacts.jar and content.jar files will be generated. To copy the features and plug-ins from the old location to the new location, use the -publishArtifacts argument as well:

$ /path/to/eclipse -consoleLog -noSplash -application
   org.eclipse.equinox.p2.publisher.FeaturesAndBundlesPublisher
 -source INDIR
 -metadataRepository file:/OUTDIR
 -artifactRepository file:/OUTDIR
 -compress
 -publishArtifacts

This will copy the features and bundles from INDIR to OUTDIR as well as generating the P2 metadata.

Updates and installations in Eclipse are usually feature based. Features are logical groupings of plug-ins and other features; for example, the Java Development Tools core feature consists of 24 individual plug-ins. Instead of showing the 24 plug-ins separately in the update site, only the JDT feature is shown.

There are also some other grouping features that do not necessarily need to be shown to the user. For example, JDT depends on Platform; Platform depends on P2, Help, and RCP; and RCP depends on E4.

All of these features can be shown to the user when the Group items by category checkbox is unselected, which can be found under Help | Install New Software...:

Showing the user all of these options may be confusing. Instead, they can be categorized to provide one level of grouping if the Group items by category checkbox is selected:

These categories are published as P2 metadata and presented to the user. Typically, these are published in conjunction with the artifacts, but the Eclipse update mechanisms allow for composite update sites, including the ability to publish the category information to a separate site.

To generate category information for an update site, a category.xml or site.xml file can be used. These files have identical contents and tags, but the category.xml file is intended to be used solely as an input for generating P2 data and not consumed at installation time, whereas the site.xml file was used to publish and consume updates in pre-Eclipse 3.4 days.

PDE provides both a site.xml Update Site Map editor and a category.xml Category Definition editor. While both can define categories and referenced features, the latter can also refer to plug-ins directly. An example category file looks like the following:

<site>
  <category-def name="cat.id" label="Text name">
    <description>Description of the category</description>
  </category-def>
  <feature id="example.feature" version="1.2.3"
   url="features/example.feature_1.2.3.jar">
    <category name="cat.id"/>
  </feature>
</site>

With an appropriate category.xml (or site.xml) file, the CategoryPublisher application can be used to generate P2 metadata to display the groups in the Install New Software dialog. This allows additional or helper features (or those that just contain sources) to be hidden from the main list, but they can be exposed if the user wishes to install them directly. The following command will append the category data to the repository:

$ /path/to/eclipse -consoleLog -noSplash -application
   org.eclipse.equinox.p2.publisher.CategoryPublisher
 -metadataRepository file:/DIR
 -categoryDefinition file:/path/to/category.xml
 -compress

When the category publisher runs, it will attempt to resolve the features it finds via the URL; if it cannot find the referenced feature or plug-in, it will silently ignore the entry.

So far, the examples have all used a single repository for hosting data. This may be useful for building small or medium-sized sites, but the ability to aggregate many update sites is useful in a number of circumstances.

P2 provides a mechanism called composite update sites, which allows a set of update sites to be aggregated by the client when installing. This provides a means to aggregate the content together without having to duplicate the binary data between them.

Composite update sites can also be used to provide a consistent top-level site while aggregating the results of multiple release, milestone, nightly, or continuous integration builds. This technique is used in the Eclipse release process, where the point releases are represented as separate child locations. For example, the Kepler update site is http://download.eclipse.org/eclipse/updates/4.3/ and is a composite site.

Composite sites contain compositeArtifacts.jar and compositeContent.jar files, which contain compositeArtifacts.xml and compositeContent.xml files. These are almost identical, with the repository type being the only difference. Here are the Kepler SR2 compositeArtifacts.xml and compositeContent.xml files:

<?xml version='1.0' encoding='UTF-8'?>
<?compositeArtifactRepository version='1.0.0'?>
<repository name='The Eclipse Project repository'
 type='org.eclipse.equinox.internal.
  p2.artifact.repository.CompositeArtifactRepository'
 version='1.0.0'>
  <properties size='3'>
    <property name='p2.timestamp' value='1393595881853'/>
    <property name='p2.compressed' value='true'/>
    <property name='p2.atomic.composite.loading' value='true'/>
  </properties>
  <children size='3'>
    <child location='R-4.3-201306052000'/>
    <child location='R-4.3.1-201309111000'/>
    <child location='R-4.3.2-201402211700'/>
  </children>
</repository>

<?xml version='1.0' encoding='UTF-8'?>
<?compositeMetadataRepository version='1.0.0'?>
<repository name='The Eclipse Project repository'
 type='org.eclipse.equinox.internal.
  p2.metadata.repository.CompositeMetadataRepository'
 version='1.0.0'>
  <properties size='3'>
    <property name='p2.timestamp' value='1393595881941'/>
    <property name='p2.compressed' value='true'/>
    <property name='p2.atomic.composite.loading' value='true'/>
  </properties>
  <children size='4'>
    <child location='categoriesKepler'/>
    <child location='R-4.3-201306052000'/>
    <child location='R-4.3.1-201309111000'/>
    <child location='R-4.3.2-201402211700'/>
  </children>
</repository>

The location can be a relative URL (which is taken to be relative to the current location) or an absolute URL (such as to another site).

In Kepler's case, the composite repository consists of four child repositories:

When Eclipse looks at http://download.eclipse.org/eclipse/updates/4.3/ as an update URL, it will discover the child repositories and then subsequently hit:

It is possible to have composite sites pointing to composite sites, allowing for any number of update sites to be chained together.

Given that there are multiple different types of P2 repository available, as well as the fallback site.xml, what is the order of the network requests? The following steps are taken by P2 when downloading a site for the first time:

A similar thing happens for consulting the content files. Note that the p2.index file is repeatedly asked for, even if it hasn't changed, and so it should be as small as possible.

The content of a p2.index file for a composite update site should look like the following:

version=1
metadata.repository.factory.order=compositeContent.xml,!
artifact.repository.factory.order=compositeArtifacts.xml,!

The content of a p2.index file for a standalone update site should look like the following:

version=1
metadata.repository.factory.order=content.xml,!
artifact.repository.factory.order=artifacts.xml,!

When Eclipse was first created, the update manager was relatively simplistic. An update site would have a simple site.xml file that listed the available features, and the update manager would use that to determine if a newer feature was available.

The following is a snippet of the update site.xml file that was used for Eclipse 3.0:

<site>
  <description url="index.html">
    The Eclipse Update Site contains feature and
    plug-in versions for Eclipse project releases.
  </description>
  <feature url="features/org.eclipse.jdt_3.0.0.jar"
   patch="false" id="org.eclipse.jdt" version="3.0.0">
    <category name="3.0"/>
  </feature>
  … 
</site>

When a newer version of a feature was available, the update manager would download the feature.jar file and display information, including copyright notices, and determine which dependencies were required. The feature.jar file contains the feature.xml file, along with a small amount of other information such as the list of included plug-ins:

<feature id="org.eclipse.jdt" version="3.0.0"
 label="%featureName" provider-name="%providerName">
  <description>%description</description>
  <license url="%licenseURL">%license</license>
  <url>
    <update label="%updateSiteName"
     url="http://update.eclipse.org/updates/3.0"/>
    <discovery label="%updateSiteName"
     url="http://update.eclipse.org/updates/3.0"/>
  </url>
  <requires>
    <import plugin="org.eclipse.platform" version="3.0.0"
     match="compatible"/>
  </requires>
  <plugin id="org.eclipse.jdt" version="3.0.0"/>
  … 
</feature>

The feature.properties is used to substitute the percent values (such as %description) in the feature.xml file:

featureName=Eclipse Java Development Tools
providerName=Eclipse.org
description=Eclipse Java development tools.
updateSiteName=Eclipse.org update site

The feature.xml file thus forms a directed graph to a set of feature requirements and plug-in requirements. In the previous example, the JDT feature depended on the org.eclipse.platform plug-in, but import suggests that this plug-in is found in a different feature. On the other hand, plugin id=org.eclipse.jdt indicates the plug-in is part of this feature, and so it will be found on the same site as the JDT feature.

Although site.xml points to the feature by URL, there is no such URL referenced from the feature to the plug-in. Instead, the location of the plug-in is calculated as relative to the feature../plugins/id_version.jar.

When the old update manager ran, it would traverse the site.xml file of all the registered update sites. If changes were seen, it would download all the features, followed by all the necessary plug-ins.

There were several problems with the update manager, including having to download a lot of extra content in order to determine if there were any updates or incompatibilities. This led to a reduced user experience as failures would not occur until runtime. In addition, the update mechanism was only capable of updating the plug-ins, and not the other content (such as an embedded JRE or the eclipse.exe launcher).