Most applications interact with Solr over HTTP. This can either be done using a typical HTTP client, or indirectly via a Solr integration API such as SolrJ or Sunspot. Such APIs are discussed in Chapter 9, Integrating Solr.

Even though an application will be communicating with Solr over HTTP, it does not have to include the documents to be indexed in the request. Solr supports what it calls remote streaming in which it's given a URL to the data. It might be an HTTP URL, but it's more likely to be a filesystem-based URL, applicable when the data is already on Solr's machine or a locally accessible filesystem. This option avoids the overhead of sending documents over HTTP. Another way to ask Solr to pull data is to use the DataImportHandler (DIH), which can pull data from a database and other sources. The DIH offers an extensible framework that can be adapted to custom data sources.

The following are various data formats for indexing data into Solr:

We'll demonstrate Solr's capability to import MusicBrainz data in XML, CSV, and from a database. Other examples will include rich document importing via the DIH and Solr Cell. Most likely, an application would use just one format.

Before these approaches are described, we'll discuss remote streaming—a foundational topic.

Solr receives commands and possibly documents through HTTP POST.

One way to send an HTTP POST is through the Unix command-line program curl (also available on Windows through Cygwin: http://www.cygwin.com). An alternative is cross-platform import tool that comes with Solr is post.jar (also known as SimplePostTool) located in Solr's example/solr directory. To get some basic guidance on how to use it, run the following command:

>> java –jar example/solr/post.jar -help

You'll see in a bit that you can post name-value pair options as HTML form data. However, post.jar doesn't support that, so you'll have to specify the URL and put the options in the query string.

For the next set of examples, we'll use the command-line program curl.

There are several ways to tell Solr to index data, and all of them are through HTTP POST:

Here is an example of the first choice. Let's say we have a Solr Update-XML file named artists.xml in the current directory. We can post it to Solr using the following command line:

>> curl http://localhost:8983/solr/mbartists/update -H 'Content-type:text/xml; charset=utf-8' --data-binary @artists.xml

If it succeeds, you'll have output that looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">
    <int name="status">0</int><int name="QTime">128</int>
</lst>
</response>

To use the stream.body feature for the preceding example, do the following:

curl http://localhost:8983/solr/mbartists/update -F [email protected]

In both cases, the @ character instructs curl to get the data from the file instead of being @artists.xml literally. If the XML is short, you can just as easily specify it, literally, on the command line:

curl http://localhost:8983/solr/mbartists/update -F stream.body=' <commit />'

Notice the leading space in the value. This was intentional. In this example, curl treats @ and < to mean things we don't want. In this case, it might be more appropriate to use form-string instead of -F. However, it requires more typing, and we're feeling lazy.

In the preceding examples, we've given Solr the data to index in the HTTP message. Alternatively, the POST request can give Solr a pointer to the data in the form of either a file path accessible to Solr or an HTTP URL to it.

Just as in the earlier case, the originating request does not return a response until Solr has finished processing it. If the file is of a decent size or is already at some known URL, then you may find remote streaming faster and/or more convenient, depending on your situation.

Here is an example of Solr accessing a local file:

curl http://localhost:8983/solr/mbartists/update -F stream.file=/tmp/artists.xml

To use a URL, the parameter would change to stream.url, and we'd specify a URL. We're passing a name-value parameter (stream.file and the path), not the actual data.