Repositories consist of a set of installable units, which are built from a set of features and plug-ins, optionally organized into categories. In Eclipse, a special Update Site project can be used to represent the set of features in a development environment, or it can be used as the source for a Tycho eclipse-repository
build. An Update Site project can be created by navigating to File | New | Other | Plug-in Development | Update Site Project:
A feature can be built by clicking on the Build All or by highlighting the feature and clicking on Build:
The feature will be named features/name_version.timestamp.jar
, along with two files,
artifacts.jar and content.jar. These two files are generated by the export mechanism and contain the set of P2 data that is required for the content to be visible to P2 installers. A similar option is presented if the feature is exported on its own via the Generate P2 repository checkbox; for this, navigate to File | Export | Plug-in Development | Deployable features to view the option:
The contents of these files describe the individual downloadable files (artifacts.jar
) and the metadata of each file (content.jar
). The files themselves just consist of a single XML file, artifacts.xml
, and content.xml
.
The P2 artifacts file provides a way to bind an installable unit to a downloadable file. Given a triplet of classifier
, id
, and version
, the artifacts file allows an installable unit's URL to be calculated. It also provides some additional information such as what the expected download type of the file is, its size, and optionally an MD5 checksum.
A repository has a human-readable name, a type, and a version. Underneath it are three sections; a list of properties, a list of mappings, and finally a list of artifacts. The following XML shows an example of a repository called Update Site
:
<?xml version='1.0' encoding='UTF-8'?> <?artifactRepository version='1.1.0'?> <repository name='Update Site' type='org.eclipse.equinox.p2.artifact.repository.simpleRepository' version='1'> <properties size='2'> <property name='p2.timestamp' value='1396184010474'/> <property name='p2.compressed' value='true'/> </properties> <mappings size='3'> <rule filter='(& (classifier=osgi.bundle))' output='${repoUrl}/plugins/${id}_${version}.jar'/> <rule filter='(& (classifier=binary))' output='${repoUrl}/binary/${id}_${version}'/> <rule filter='(& (classifier=org.eclipse.update.feature))' output='${repoUrl}/features/${id}_${version}.jar'/> </mappings> <artifacts size='1'> <artifact classifier='org.eclipse.update.feature' id='Feature' version='1.0.0.201403301353'> <properties size='2'> <property name='download.contentType' value='application/zip'/> <property name='download.size' value='338'/> </properties> </artifact> </artifacts> </repository>
The properties are used to provide additional information about the artifacts; in this case, the timestamp at which the content was last generated and whether the content of the repository should be compressed into an artifacts.jar
file.
The properties can also contain a mirror reference with the p2.mirrorsURL
. This allows a set of mirrors to be queried for an artifact instead of just the originating server; the Eclipse infrastructure uses this to share the load between mirror sites when new versions of Eclipse are released. When an artifact needs to be downloaded, the mirrors URL will be hit, and an XML file will be returned.
For Eclipse Luna, the mirror URL is http://www.eclipse.org/downloads/download.php?format=xml&file=/eclipse/updates/4.4/R-4.4-201406061215.
This is encoded in the artifacts.xml
file as follows:
<property name='p2.mirrorsURL' value='http://www.eclipse.org/downloads/download.php?format=xml&file=/eclipse/updates/4.4/R-4.4-201406061215'/>
Because XML files cannot contain ampersands (&
) without escaping, the XML file has &
to separate parameters. The returned XML file looks like the following:
<mirrors> <mirror url="http://www.mirrorservice.org/sites/download.eclipse.org/" label="[United Kingdom] UK Mirror Service (http)"/> <mirror url="http://ftp.snt.utwente.nl/pub/software/eclipse/" label="[Netherlands] SNT, University of Twente (http)"/> <mirror url="http://eclipse.mirror.triple-it.nl/" label="[Netherlands] Triple IT (http)"/> ... </mirrors>
Although the Eclipse servers generate the request dynamically with a PHP script, this could be provided with a static XML file or another automatically generated mechanism.
The Eclipse servers send out both HTTP and FTP mirrors; for sites that have firewalls that don't support FTP, it's possible to add &protocol=http
to get just a list of HTTP mirrors.
The contents file contains much more information and records properties extracted from the bundle, such as the license, the copyright, and vendor, and for OSGi bundles, what packages are imported and exported along with other generic require-capability elements. There is a one-to-one mapping between entries in the artifacts file with entries in the content file.
Along with JARs hosted on an update site, other binary content can be stored and served. There are two categories for non-JAR content:
Both of these can be stored either as top-level files in the binary
directory or in a special .blobstore
folder.
The blobstore is a means of allowing arbitrary content to be stored without the file extension causing problems for the update site server, by creating randomized filenames to distinguish between assets. Because the names of the files in the blobstore do not correspond to any well-known algorithm (such as md5
or sha1
), the names of the files give no information about its content. This can be used for storing binary executables as well as packed JAR files.
The blobstore causes problems for some webservers that don't know or expect a particular type of data. As a result, it is often desirable for a P2 repository to store pack200
files next to the JARs instead of the blobstore. This is achieved by storing a property packFilesAsSiblings
with the value true
in the artifacts.xml
file of the destination repository:
<property name='publishPackFilesAsSiblings' value='true'/>
This will ensure that the blobstore is not used for writing out the content of pack files, but they will be put next to the JAR files instead.
Although a direct file or rsync copy will allow a set of artifacts to be mirrored, it is possible to use built-in functionality in Eclipse to allow P2 repositories to be mirrored. This will ensure that the contents listed in the remote update site are transferred as expected, and the metadata files are updated correctly.
Mirroring is done separately for artifacts and metadata, but they both follow the same structure. One mirrors the artifacts.jar
files, and the other mirrors the content.jar
files.
To mirror Luna's artifacts, use the following command:
$ /path/to/eclipse -consoleLog -noSplash -application org.eclipse.equinox.p2.artifact.repository.mirrorApplication -source http://download.eclipse.org/releases/luna/ -destination file:///path/to/luna-mirror -verbose -raw -ignoreErrors
The -verbose
flag tells the mirroring process to print out what is being copied at each point and is an optional argument.
The -raw
flag tells the mirroring process to copy as is without translating or rebuilding the metadata from the original bundle. This is faster, but it can sometimes cause problems when mirroring from old-style update sites. This only has an effect on artifact mirroring and is an optional argument.
When the -ignoreErrors
flag is given, any errors seen during the mirroring operation are ignored, causing the mirroring to continue. If not specified, then any error will terminate the mirroring process. This only has an effect on artifact mirroring and is an optional argument.
To mirror Luna's metadata, use the following command:
$ /path/to/eclipse -consoleLog -noSplash -application org.eclipse.equinox.p2.metadata.repository.mirrorApplication -source http://download.eclipse.org/releases/luna/ -destination file:///path/to/luna-mirror -verbose
The previous two commands are almost identical; the only difference is the application name.
While mirroring, P2 will take advantage of any mirrors found. If the remote site lists a set of mirrors, then P2 will consult the mirror lookup and download assets from mirrors in order to spread the load.
This can sometimes cause problems because mirrors are randomly switched between HTTP and FTP sites; if using a proxy that doesn't support FTP or are behind a misconfigured NAT router, then these connections will silently fail. To disable this, pass -vmargs -Declipse.p2.mirrors=false
as the last entry on the command line.
The mirroring process can also verify the MD5 signatures of the files when they are being mirrored. This can be disabled by passing the command-line arguments -vmargs -Declipse.p2.MD5Check=false
.
Some old update sites do not have P2 metadata generated; instead, they just have a site.xml
file as a classic update site. While this works in older versions of Eclipse, it may cause errors with a new Eclipse install (or a Tycho build) with a cryptic error message that reads Update site contains Partial IUs and cannot be used.
To generate P2 metadata from a folder that contains features/
and plugins/
directories, the following Eclipse command can be run. Given a directory DIR, the following command will generate the content.jar
and artifacts.jar
files:
$ /path/to/eclipse -consoleLog -noSplash -application org.eclipse.equinox.p2.publisher.FeaturesAndBundlesPublisher -source DIR -metadataRepository file:/DIR -artifactRepository file:/DIR -compress
Note that DIR must be a filesystem, but the -metadataRepository
and -artifactRepository
arguments require URLs with a file:/
prefix.
The -compress
argument tells the P2 publisher to generate an artifacts.jar
that contains the artifacts.xml
file; similarly, the content.jar
contains content.xml
. While the standalone XML files are more human-readable, they are often between two and ten times smaller when compressed.
The previous command will generate the artifacts.jar
and content.jar
files in the same location as the features/
and bundles/
directories, which is often what is required. However, if the output directory is in a different location, then only the artifacts.jar
and content.jar
files will be generated. To copy the features and plug-ins from the old location to the new location, use the -publishArtifacts
argument as well:
$ /path/to/eclipse -consoleLog -noSplash -application org.eclipse.equinox.p2.publisher.FeaturesAndBundlesPublisher -source INDIR -metadataRepository file:/OUTDIR -artifactRepository file:/OUTDIR -compress -publishArtifacts
This will copy the features and bundles from INDIR to OUTDIR as well as generating the P2 metadata.
Note that these commands will overwrite the existing metadata, and in the case of the -publishArtifacts
option, will overwrite existing features and plugins.
To add data to the existing repository instead of overwriting, use the -append
argument. This will allow multiple updates to be mirrored into a single location.
Updates and installations in Eclipse are usually feature based. Features are logical groupings of plug-ins and other features; for example, the Java Development Tools core feature consists of 24 individual plug-ins. Instead of showing the 24 plug-ins separately in the update site, only the JDT feature is shown.
There are also some other grouping features that do not necessarily need to be shown to the user. For example, JDT depends on Platform; Platform depends on P2, Help, and RCP; and RCP depends on E4.
All of these features can be shown to the user when the Group items by category checkbox is unselected, which can be found under Help | Install New Software...:
Showing the user all of these options may be confusing. Instead, they can be categorized to provide one level of grouping if the Group items by category checkbox is selected:
These categories are published as P2 metadata and presented to the user. Typically, these are published in conjunction with the artifacts, but the Eclipse update mechanisms allow for composite update sites, including the ability to publish the category information to a separate site.
To generate category information for an update site, a category.xml
or site.xml
file can be used. These files have identical contents and tags, but the category.xml
file is intended to be used solely as an input for generating P2 data and not consumed at installation time, whereas the site.xml
file was used to publish and consume updates in pre-Eclipse 3.4 days.
PDE provides both a site.xml
Update Site Map editor and a category.xml
Category Definition editor. While both can define categories and referenced features, the latter can also refer to plug-ins directly. An example category file looks like the following:
<site> <category-def name="cat.id" label="Text name"> <description>Description of the category</description> </category-def> <feature id="example.feature" version="1.2.3" url="features/example.feature_1.2.3.jar"> <category name="cat.id"/> </feature> </site>
With an appropriate category.xml
(or site.xml
) file, the CategoryPublisher
application can be used to generate P2 metadata to display the groups in the Install New Software dialog. This allows additional or helper features (or those that just contain sources) to be hidden from the main list, but they can be exposed if the user wishes to install them directly. The following command will append the category data to the repository:
$ /path/to/eclipse -consoleLog -noSplash -application org.eclipse.equinox.p2.publisher.CategoryPublisher -metadataRepository file:/DIR -categoryDefinition file:/path/to/category.xml -compress
When the category publisher runs, it will attempt to resolve the features it finds via the URL; if it cannot find the referenced feature or plug-in, it will silently ignore the entry.
The entries are written into the metadata with the ID defined in the category (in the example previously, it was cat.id
). To disambiguate different categories, an additional argument can be specified to prefix the category IDs with a value. By adding -categoryQualifier example.prefix
to the command, the category ID will become example.prefix.cat.id
in the P2 metadata.
So far, the examples have all used a single repository for hosting data. This may be useful for building small or medium-sized sites, but the ability to aggregate many update sites is useful in a number of circumstances.
P2 provides a mechanism called composite update sites, which allows a set of update sites to be aggregated by the client when installing. This provides a means to aggregate the content together without having to duplicate the binary data between them.
Composite update sites can also be used to provide a consistent top-level site while aggregating the results of multiple release, milestone, nightly, or continuous integration builds. This technique is used in the Eclipse release process, where the point releases are represented as separate child locations. For example, the Kepler update site is http://download.eclipse.org/eclipse/updates/4.3/ and is a composite site.
Composite sites contain compositeArtifacts.jar
and compositeContent.jar
files, which contain compositeArtifacts.xml
and compositeContent.xml
files. These are almost identical, with the repository type being the only difference. Here are the Kepler SR2 compositeArtifacts.xml
and compositeContent.xml
files:
<?xml version='1.0' encoding='UTF-8'?> <?compositeArtifactRepository version='1.0.0'?> <repository name='The Eclipse Project repository' type='org.eclipse.equinox.internal. p2.artifact.repository.CompositeArtifactRepository' version='1.0.0'> <properties size='3'> <property name='p2.timestamp' value='1393595881853'/> <property name='p2.compressed' value='true'/> <property name='p2.atomic.composite.loading' value='true'/> </properties> <children size='3'> <child location='R-4.3-201306052000'/> <child location='R-4.3.1-201309111000'/> <child location='R-4.3.2-201402211700'/> </children> </repository> <?xml version='1.0' encoding='UTF-8'?> <?compositeMetadataRepository version='1.0.0'?> <repository name='The Eclipse Project repository' type='org.eclipse.equinox.internal. p2.metadata.repository.CompositeMetadataRepository' version='1.0.0'> <properties size='3'> <property name='p2.timestamp' value='1393595881941'/> <property name='p2.compressed' value='true'/> <property name='p2.atomic.composite.loading' value='true'/> </properties> <children size='4'> <child location='categoriesKepler'/> <child location='R-4.3-201306052000'/> <child location='R-4.3.1-201309111000'/> <child location='R-4.3.2-201402211700'/> </children> </repository>
The location
can be a relative URL (which is taken to be relative to the current location) or an absolute URL (such as to another site).
It is possible to refer to a ZIP file as a child site using the standard Java jar:
protocol. For example, to install Drools from Maven Central, an absolute URL jar:https://repo1.maven.org/maven2/org/drools/org.drools.updatesite/6.0.0.Final/org.drools.updatesite-6.0.0.Final.zip!/
can be used as a child site.
Note the jar:
prefix as well as the !/
suffix of the URL. Using this is not generally recommended as it can be slow, but for small sites or places where an expanded file cannot be hosted, this approach may be useful.
In Kepler's case, the composite repository consists of four child repositories:
R-4.3-201306052000
R-4.3.1-201309111000
R-4.3.2-201402211700
categoriesKepler
(content metadata only)When Eclipse looks at http://download.eclipse.org/eclipse/updates/4.3/ as an update URL, it will discover the child repositories and then subsequently hit:
It is possible to have composite sites pointing to composite sites, allowing for any number of update sites to be chained together.
Given that there are multiple different types of P2 repository available, as well as the fallback site.xml
, what is the order of the network requests? The following steps are taken by P2 when downloading a site for the first time:
<url>/p2.index
(if available)p2.index
is available, look for the files directed such as compositeArtifacts.jar
or artifacts.jar
artifacts.jar
, then artifacts.xml
, then compositeArtifacts.jar
, then compositeArtifacts.xml
, and finally site.xml
A similar thing happens for consulting the content
files. Note that the p2.index
file is repeatedly asked for, even if it hasn't changed, and so it should be as small as possible.
The content of a p2.index
file for a composite update site should look like the following:
version=1 metadata.repository.factory.order=compositeContent.xml,! artifact.repository.factory.order=compositeArtifacts.xml,!
The content of a p2.index
file for a standalone update site should look like the following:
version=1 metadata.repository.factory.order=content.xml,! artifact.repository.factory.order=artifacts.xml,!
When Eclipse was first created, the update manager was relatively simplistic. An update site would have a simple site.xml file that listed the available features, and the update manager would use that to determine if a newer feature was available.
The classic update manager (org.eclipse.update.*
) was deprecated in Eclipse 3.4 and removed in Eclipse 4.2, as described in the release notes at http://www.eclipse.org/eclipse/development/porting/4.2/incompatibilities.html#update-manager.
The following is a snippet of the update site.xml
file that was used for Eclipse 3.0:
<site> <description url="index.html"> The Eclipse Update Site contains feature and plug-in versions for Eclipse project releases. </description> <feature url="features/org.eclipse.jdt_3.0.0.jar" patch="false" id="org.eclipse.jdt" version="3.0.0"> <category name="3.0"/> </feature> … </site>
When a newer version of a feature was available, the update manager would download the feature.jar
file and display information, including copyright notices, and determine which dependencies were required. The feature.jar
file contains the feature.xml
file, along with a small amount of other information such as the list of included plug-ins:
<feature id="org.eclipse.jdt" version="3.0.0" label="%featureName" provider-name="%providerName"> <description>%description</description> <license url="%licenseURL">%license</license> <url> <update label="%updateSiteName" url="http://update.eclipse.org/updates/3.0"/> <discovery label="%updateSiteName" url="http://update.eclipse.org/updates/3.0"/> </url> <requires> <import plugin="org.eclipse.platform" version="3.0.0" match="compatible"/> </requires> <plugin id="org.eclipse.jdt" version="3.0.0"/> … </feature>
The feature.properties
is used to substitute the percent values (such as %description
) in the feature.xml
file:
featureName=Eclipse Java Development Tools providerName=Eclipse.org description=Eclipse Java development tools. updateSiteName=Eclipse.org update site
The feature.xml
file thus forms a directed graph to a set of feature requirements and plug-in requirements. In the previous example, the JDT feature depended on the org.eclipse.platform
plug-in, but import
suggests that this plug-in is found in a different feature. On the other hand, plugin id=org.eclipse.jdt
indicates the plug-in is part of this feature, and so it will be found on the same site as the JDT feature.
Although site.xml
points to the feature by URL, there is no such URL referenced from the feature to the plug-in. Instead, the location of the plug-in is calculated as relative to the feature../plugins/id_version.jar
.
When the old update manager ran, it would traverse the site.xml
file of all the registered update sites. If changes were seen, it would download all the features, followed by all the necessary plug-ins.
There were several problems with the update manager, including having to download a lot of extra content in order to determine if there were any updates or incompatibilities. This led to a reduced user experience as failures would not occur until runtime. In addition, the update mechanism was only capable of updating the plug-ins, and not the other content (such as an embedded JRE or the eclipse.exe
launcher).