Understanding the Elasticsearch RESTful API

In a REST-like architecture, every request is directed to a concrete object indicated by the path of the address. For example, if /books/ is a reference to a list of books in our library, /books/1 is the reference to the book with the identifier 1. Note that these objects can be nested. The /books/1/chapter/6 reference denotes the sixth chapter of the first book in the library, and so on. We have a subject for our API call. What about an operation that we would like to execute, such as GET or POST? To indicate this, request types are used. The HTTP protocol gives us quite a long list of types that can be used as verbs in the API calls. Logical choices are GET in order to obtain the current state of the requested object, POST to change the object state, PUT to create an object, and DELETE to destroy objects. There is also a HEAD request that is only used to fetch the base information of an object.

If we look at the following examples of the operations discussed in the Shutting down Elasticsearch section, everything should make more sense:

We now know what REST means, at least in general (you can read more about REST at http://en.wikipedia.org/wiki/Representational_state_transfer). Now, we can proceed and learn how to use the Elasticsearch API to store, fetch, alter, and delete data.

Storing data in Elasticsearch

As we have already discussed, in Elasticsearch, every piece of data—each document—has a defined index and type. Each document can contain one or more fields that will hold your data. We will start by showing you how to index a simple document using Elasticsearch.

Creating a new document

Now, we will try to index some of the documents. For example, let's imagine that we are building some kind of CMS system for our blog. One of the entities in this blog is articles (surprise!).

Using the JSON notation, a document can be presented as shown in the following example:

{
  "id": "1",
  "title": "New version of Elasticsearch released!",
  "content": "Version 1.0 released today!",
  "priority": 10,
  "tags": ["announce", "elasticsearch", "release"]
}

As we can see, the JSON document contains a set of fields, where each field can have a different form. In our example, we have a number (priority), text (title), and an array of strings (tags). In the following examples, we will show you the other types. As mentioned earlier in this chapter, Elasticsearch can guess these types (because JSON is semi-typed; for example, the numbers are not in quotation marks) and automatically customize how this data will be stored in its internal structures.

Of course, we would like to index our example document and make it available for searching. We will use an index named blog and a type named article. In order to index our example document to this index under the given type and with the identifier of 1, we will execute the following command:

curl -XPUT http://localhost:9200/blog/article/1 -d '{"title": "New version of Elasticsearch released!", "content": "Version 1.0 released today!", "tags": ["announce", "elasticsearch", "release"] }'

Note a new option to the cURL command: the -d parameter. The value of this option is the text that will be used as a request payload—a request body. This way, we can send additional information such as document definition. Also, note that the unique identifier is placed in the URL and not in the body. If you omit this identifier (while using the HTTP PUT request), the indexing request will return the following error:

No handler found for uri [/blog/article/] and method [PUT]

If everything is correct, Elasticsearch will respond with a JSON response similar to the following output:

{
 "_index":"blog",
 "_type":"article",
 "_id":"1",
 "_version":1
}

In the preceding response, Elasticsearch includes the information about the status of the operation and shows where a new document was placed. There is information about the document's unique identifier and current version, which will be incremented automatically by Elasticsearch every time it is updated.

curl -XPOST http://localhost:9200/blog/article/ -d '{"title": "New version of Elasticsearch released!", "content": "Version 1.0 released today!", "tags": ["announce", "elasticsearch", "release"] }'

{
  "_index" : "blog",
  "_type" : "article",
 "_id" : "XQmdeSe_RVamFgRHMqcZQg",
  "_version" : 1
}

Retrieving documents

We already have documents stored in our instance. Now let's try to retrieve them by using their identifiers. We will start by executing the following command:

curl -XGET http://localhost:9200/blog/article/1

Elasticsearch will return a response similar to the following output:

{
 "_index" : "blog",
 "_type" : "article",
 "_id" : "1",
 "_version" : 1,
 "exists" : true, 
 "_source" : {
 "title": "New version of Elasticsearch released!", 
 "content": "Version 1.0 released today!", 
 "tags": ["announce", "elasticsearch", "release"] 
 }

In the preceding response, besides the index, type, identifier, and version, we can also see the information that says that the document was found (the exists property) and the source of this document (in the _source field). If document is not found, we get a reply as follows:

{
 "_index" : "blog",
 "_type" : "article",
 "_id" : "9999",
 "exists" : false
}

Of course, there is no information about the version and source because no document was found.

Updating documents

Updating documents in the index is a more complicated task. Internally, Elasticsearch must first fetch the document, take its data from the _source field, remove the old document, apply changes to the _source field, and then index it as a new document. It is so complicated because we can't update the information once it is stored in the Lucene inverted index. Elasticsearch implements this through a script given as an update request parameter. This allows us to do more a sophisticated document transformation than simple field changes. Let's see how it works in a simple case.

Please recall the example blog article that we've indexed previously. We will try to change its content field from the old one to new content. To do this, we will run the following command:

curl -XPOST http://localhost:9200/blog/article/1/_update -d '{
 "script": "ctx._source.content = \"new content\""
}'

Elasticsearch will reply with the following response:

{"_index":"blog","_type":"article","_id":"1","_version":2}

It seems that the update operation was executed successfully. To be sure, let's retrieve the document by using its identifier. To do this, we will run the following command:

curl -XGET http://localhost:9200/blog/article/1

The response from Elasticsearch should include the changed content field, and indeed, it includes the following information:

{
 "_index" : "blog",
 "_type" : "article",
 "_id" : "1",
 "_version" : 2,
 "exists" : true, 
 "_source" : {
 "title":"New version of Elasticsearch released!",
 "content":"new content",
 "tags":["announce","elasticsearch","release"]
 }

Elasticsearch changed the contents of our article and the version number for this document. Note that we didn't have to send the whole document, only the changed parts. However, remember that to use the update functionality, we need to use the _source field—we will describe how to use the _source field in the Extending your index structure with additional internal information section in Chapter 2, Indexing Your Data.

There is one more thing about document updates; if your script uses a field value from a document that is to be updated, you can set a value that will be used if the document doesn't have that value present. For example, if you want to increment the counter field of the document and it is not present, you can use the upsert section in your request to provide the default value that will be used. For example, look at the following lines of command:

curl -XPOST http://localhost:9200/blog/article/1/_update -d '{
 "script": "ctx._source.counter += 1",
 "upsert": {
 "counter" : 0
 }
}'

If you execute the preceding example, Elasticsearch will add the counter field with the value of 0 to our example document. This is because our document does not have the counter field present and we've specified the upsert section in the update request.

Deleting documents

We have already seen how to create (PUT) and retrieve (GET) documents. We also know how to update them. It is not difficult to guess that the process to remove a document is similar; we need to send a proper HTTP request using the DELETE request type. For example, to delete our example document, we will run the following command:

curl -XDELETE http://localhost:9200/blog/article/1

The response from Elasticsearch will be as follows:

{"found":true,"_index":"blog","_type":"article","_id":"1","_version":3}

This means that our document was found and it was deleted.

Now we can use the CRUD operations. This lets us create applications using Elasticsearch as a simple key-value store. But this is only the beginning!

Versioning

In the examples provided, you might have seen information about the version of the document, which looked like the following:

"_version" : 1

If you look carefully, you will notice that after updating the document with the same identifier, this version is incremented. By default, Elasticsearch increments the version when a document is added, changed, or deleted. In addition to informing us about the number of changes made to the document, it also allows us to implement optimistic locking (http://en.wikipedia.org/wiki/Optimistic_concurrency_control). This allows us to avoid issues when processing the same document in parallel. For example, we read the same document in two different applications, modify it differently, and then try to update the one in Elasticsearch. Without versioning the version, we will see the one sent for indexation as the last version. Using optimistic locking, Elasticsearch guards the data accuracy—every attempt to write the document that has been already changed will fail.

curl -XDELETE 'localhost:9200/library/book/1?version=1'

{
 "error": "VersionConflictEngineException[[library][4] [book][1]: version conflict, current [2], provided [1]]",
 "status": 409
}

curl -XPUT 'localhost:9200/library/book/1?version=123456' -d {...}