Installing Java

In order to set up Elasticsearch, the first step is to make sure that a Java SE environment is installed properly. Elasticsearch requires Java Version 6 or later to run. You can download it from http://www.oracle.com/technetwork/java/javase/downloads/index.html. You can also use OpenJDK (http://openjdk.java.net/) if you wish. You can, of course, use Java Version 6, but it is not supported with patches by default, so we suggest that you install Java 7.

Installing Elasticsearch

To install Elasticsearch, just download it from http://www.elasticsearch.org/download/ and unpack it. Choose the last stable version. That's it! The installation is complete.

The main interface to communicate with Elasticsearch is based on an HTTP protocol and REST. This means that you can even use a web browser for some basic queries and requests, but for anything more sophisticated, you'll need to use additional software such as the cURL command. If you use the Linux or OS X command, the curl package should already be available. If you use Windows, you can download it from http://curl.haxx.se/download.html.

Installing Elasticsearch from binary packages on Linux

The other way to install Elasticsearch is to use the provided binary packages—the RPM or DEB packages, depending on your Linux distribution. The mentioned binary packages can be found at the following URL address: http://www.elasticsearch.org/download/.

sudo yum elasticsearch-1.0.0.noarch.rpm

sudo dpkg -i elasticsearch-1.0.0.deb

The directory layout

Now, let's go to the newly created directory. We should see the following directory structure:

After Elasticsearch starts, it will create the following directories (if they don't exist):

Configuring Elasticsearch

One of the reasons—of course, not the only one—why Elasticsearch is gaining more and more popularity is that getting started with Elasticsearch is quite easy. Because of the reasonable default values and automatic settings for simple environments, we can skip the configuration and go straight to the next chapter without changing a single line in our configuration files. However, in order to truly understand Elasticsearch, it is worth understanding some of the available settings.

We will now explore the default directories and layout of the files provided with the Elasticsearch tar.gz archive. The whole configuration is located in the config directory. We can see two files there: elasticsearch.yml (or elasticsearch.json, which will be used if present) and logging.yml. The first file is responsible for setting the default configuration values for the server. This is important because some of these values can be changed at runtime and can be kept as a part of the cluster state, so the values in this file may not be accurate. The two values that we cannot change at runtime are cluster.name and node.name.

The cluster.name property is responsible for holding the name of our cluster. The cluster name separates different clusters from each other. Nodes configured with the same cluster name will try to form a cluster.

The second value is the instance (the node) name. We can leave this parameter undefined. In this case, Elasticsearch automatically chooses a unique name for itself. Note that this name is chosen during every startup, so the name can be different on each restart. Defining the name can help when referring to concrete instances by the API or when using monitoring tools to see what is happening to a node during long periods of time and between restarts. Think about giving descriptive names to your nodes.

Other parameters are well commented in the file, so we advise you to look through it; don't worry if you do not understand the explanation. We hope that everything will become clear after reading the next few chapters.

The second file (logging.yml) defines how much information is written to system logs, defines the logfiles, and creates new files periodically. Changes in this file are usually required only when you need to adapt to monitoring or backup solutions or during system debugging; however, if you want to have a more detailed logging, you need to adjust it accordingly.

Let's leave the configuration files for now. An important part of the configuration is tuning your operating system. During the indexing, especially when having many shards and replicas, Elasticsearch will create many files; so, the system cannot limit the open file descriptors to less than 32,000. For Linux servers, this can be usually changed in /etc/security/limits.conf and the current value can be displayed using the ulimit command. If you end up reaching the limit, Elasticsearch will not be able to create new files; so, merging will fail, indexing may fail, and new indices will not be created.

The next set of settings is connected to the Java Virtual Machine (JVM) heap memory limit for a single Elasticsearch instance. For small deployments, the default memory limit (1024 MB) will be sufficient, but for large ones, it will not be enough. If you spot entries that indicate the OutOfMemoryError exceptions in a logfile, set the ES_HEAP_SIZE variable to a value greater than 1024. When choosing the right amount of memory size to be given to the JVM, remember that, in general, no more than 50 percent of your total system memory should be given. However, as with all the rules, there are exceptions. We will discuss this in greater detail later, but you should always monitor your JVM heap usage and adjust it when needed.

Running Elasticsearch

Let's run our first instance that we just downloaded as the ZIP archive and unpacked. Go to the bin directory and run the following commands depending on the OS:

Congratulations! Now, we have our Elasticsearch instance up and running. During its work, the server usually uses two port numbers: the first one for communication with the REST API using the HTTP protocol, and the second one for the transport module used for communication in a cluster and in between the native Java client and the cluster. The default port used for the HTTP API is 9200, so we can check the search readiness by pointing the web browser to http://127.0.0.1:9200/. The browser should show a code snippet similar to the following:

{
  "status" : 200,
  "name" : "es_server",
  "version" : {
    "number" : "1.0.0",
    "build_hash" : "a46900e9c72c0a623d71b54016357d5f94c8ea32",
    "build_timestamp" : "2014-02-12T16:18:34Z",
    "build_snapshot" : false,
    "lucene_version" : "4.6"
  },
  "tagline" : "You Know, for Search"
}

The output is structured as a JSON (JavaScript Object Notation) object. If you are not familiar with JSON, please take a minute and read the article available at http://en.wikipedia.org/wiki/JSON.

[2013-11-16 11:56:12,101][INFO ][http] [Red Lotus] bound_address {inet[/0:0:0:0:0:0:0:0%0:9200]}, publish_address {inet[/192.168.1.101:9200]}

Now, we will use the cURL program. For example, to check cluster health, we will use the following command:

curl -XGET http://127.0.0.1:9200/_cluster/health?pretty

The -X parameter is a request method. The default value is GET (so, in this example, we can omit this parameter). Temporarily, do not worry about the GET value; we will describe it in more detail later in this chapter.

As a standard, the API returns information in a JSON object in which new line characters are omitted. The pretty parameter added to our requests forces Elasticsearch to add a new line character to the response, making the response more human friendly. You can try running the preceding query with and without the ?pretty parameter to see the difference.

Elasticsearch is useful in small- and medium-sized applications, but it has been built with large clusters in mind. So, now we will set up our big, two-node cluster. Unpack the Elasticsearch archive in a different directory and run the second instance. If we look at the log, we see what is shown as follows:

[2013-11-16 11:55:16,767][INFO ][cluster.service ] [Stane, Obadiah] detected_master [Martha Johansson] [vswsFRWTSjOa_fy7uPuOMA] [inet[/192.168.1.19:9300]], added {[Martha Johansson] [vswsFRWTSjOa_fy7uPuOMA] [inet[/192.168.1.19:9300]],}, reason: zen-disco-receive(from master [[Martha Johansson][vswsFRWTSjOa_fy7uPuOMA] [inet[/192.168.1.19:9300]]]) 

This means that our second instance (named Stane,Obadiah) discovered the previously running instance (named Martha Johansson). Here, Elasticsearch automatically formed a new, two-node cluster.

Shutting down Elasticsearch

Even though we expect our cluster (or node) to run flawlessly for a lifetime, we may need to restart it or shut it down properly (for example, for maintenance). The following are three ways in which we can shut down Elasticsearch:

We will focus on the last method now. It allows us to shut down the whole cluster by executing the following command:

curl -XPOST http://localhost:9200/_cluster/nodes/_shutdown

To shut down just a single node, for example, a node with the BlrmMvBdSKiCeYGsiHijdg identifier, we will execute the following command:

curl –XPOST http://localhost:9200/_cluster/nodes/BlrmMvBdSKiCeYGsiHijdg/_shutdown

The identifier of the node can be read either from the logs or using the _cluster/nodes API, with the following command:

curl -XGET http://localhost:9200/_cluster/nodes/

Running Elasticsearch as a system service

Elasticsearch 1.0 can run as a service both on Linux-based systems as well as on Windows-based ones.

curl -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz

sudo mv *servicewrapper*/service /usr/local/share/elasticsearch/bin/

rm -Rf *servicewrapper*

sudo /usr/local/share/elasticsearch/bin/service/elasticsearch install

sudo ln -s 'readlink -f /usr/local/share/elasticsearch/bin/service/elasticsearch' /usr/local/bin/rcelasticsearch

/etc/init.d/elasticsearch start

service.bat install

service.bat

service.bat start