How to Install and Configure Elastic Stack (Elasticsearch, Logstash and Kibana) on Ubuntu 17.04

As IT infrastructure is moving to cloud and Internet of Things is becoming popular, organizations and IT professionals are using public cloud services to a greater extent. As servers and services running on them are increasing, the amount of system generated logs are also increasing. Analysis of these logs is very important in an infrastructure for several reasons. This includes compliance with security policies and regulations, system troubleshooting, responding to a security related incident or to understand user behavior.

Three very popular open source applications named Elasticsearch, Logstash and Kibana combine together to create Elastic Stack or ELK Stack. Elastic Stack is a very powerful tool for searching, analyzing and visualizing logs and data. Elasticsearch is a distributed, real time, scalable and highly available application to store logs and search through them. Logstash gathers the logs sent by Beats, enhances it, and then sends it to Elasticsearch. Kibana is the web UI used to visualize the logs and actionable insights.

In this tutorial, we will install the latest version of Elasticsearch, Logstash and Kibana with X-Pack on Ubuntu 17.04.

Prerequisites

To follow this tutorial, you will need a Vultr 64-bit Ubuntu 17.04 server instance with at least 4 GB RAM. For a production environment, the hardware requirements increase with user and log count.

This tutorial is written from a sudo user perspective. To set up a sudo user follow How to Use Sudo on Debian guide.

You will also need a domain pointed towards your server to obtain certificates from Let’s Encrypt CA.

Step 1: Perform a system update

Before installing any packages on the Ubuntu server instance, it is recommended to update the system. Log in using the sudo user and run the following commands to update the system.

sudo apt update
sudo apt -y upgrade

Once the system has finished upgrading, proceed to the next step.

Step 2: Install Java

Elasticsearch requires Java 8 to work. It supports both Oracle Java and OpenJDK. This section of tutorial demonstrates installation of both Oracle Java and OpenJDK.

Make sure that you install any one of the following Java versions. Installation of Oracle Java is recommended for Elasticsearch. However, you may also choose to install OpenJDK according to your preference.

Installing Oracle Java

To install Oracle Java on your Ubuntu system, you will need to add the Oracle Java PPA by running:

sudo add-apt-repository ppa:webupd8team/java

Now update the repository information by running:

sudo apt update

Now you can easily install the latest stable version of Java 8 by running:

sudo apt -y install oracle-java8-installer

Accept the licence agreement when prompted. Once the installation is finished, you can verify the Java version by running:

java -version

You should see output similar to:

user@vultr:~$ java -version
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)

You can also set the JAVA_HOME and other defaults by installing oracle-java8-set-default. Run:

sudo apt -y install oracle-java8-set-default

You can now verify if the JAVA_HOME variable is set by running:

echo "$JAVA_HOME"

The output should resemble:

user@vultr:~$ echo "$JAVA_HOME"
/usr/lib/jvm/java-8-oracle

If you do not get the output shown above, you may need to logout and login to the shell again. Oracle Java is now installed on your server. You can now proceed to the Step 3 of the tutorial skipping installation of OpenJDK.

Installing OpenJDK

Installation of OpenJDK is pretty straightforward. Simply run the following command to install OpenJDK.

sudo apt -y install default-jdk

Once the installation is finished, you can verify the Java version by running:

java -version

You should see output similar to:

user@vultr:~$ java -version
openjdk version "1.8.0_131"
OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-2ubuntu1.17.04.2-b11)
OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode)

To set the JAVA_HOME variable, run the following command:

sudo echo "JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64" >> /etc/environment

Reload the environment file by running:

sudo source /etc/environment

You can now verify if the JAVA_HOME variable is set by running:

echo "$JAVA_HOME"

The output should resemble:

user@vultr:~$ echo "$JAVA_HOME"
/usr/lib/jvm/java-8-openjdk-amd64/

Step 3: Install Elasticsearch

Elasticsearch is a super-fast, distributed, highly available, RESTful search engine. Add the Elasticsearch APT repository by running:

echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

The above command creates a new repository file for Elasticsearch and adds the source entry into it. Now import the PGP key used to sign the packages.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Update the APT repository metadata by running:

sudo apt update

Install Elasticsearch by running the following command.

sudo apt -y install elasticsearch

The above command will install the latest version of Elasticsearch on your system. Once Elasticsearch has been installed, reload the Systemd service daemon by running:

sudo systemctl daemon-reload

Start Elasticsearch and enable it to automatically start at boot time.

sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

To stop Elasticsearch, you can run:

sudo systemctl stop elasticsearch

To check the status of the service you can run:

sudo systemctl status elasticsearch

Elasticsearch is now running on port 9200. You can verify if it is working and producing results by running the following command.

curl -XGET 'localhost:9200/?pretty'

A message similar to the following will be printed.

user@vultr:~$ curl -XGET 'localhost:9200/?pretty'
{
  "name" : "wDaVa1K",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "71drjJ8PTyCcbai33Esy3Q",
  "version" : {
    "number" : "5.5.1",
    "build_hash" : "19c13d0",
    "build_date" : "2017-07-18T20:44:24.823Z",
    "build_snapshot" : false,
    "lucene_version" : "6.6.0"
  },
  "tagline" : "You Know, for Search"
}

Install X-Pack for Elasticsearch

X-Pack is an Elastic Stack plug-in that provides many add on features such as security, alerting, monitoring, reporting, and graph capabilities. X-Pack also provides user authentication for Elasticsearch and Kibana, as well as monitoring of different nodes in Kibana. It is important that X-Pack and Elasticsearch are installed with the same version.

You can install X-Pack for Elasticsearch directly by running:

cd /usr/share/elasticsearch
sudo bin/elasticsearch-plugin install x-pack

To continue the installation, enter y when prompted. This command will install the X-Pack plugin to your system. When installed, X-Pack enables authentication for Elasticsearch. The default username is elastic and password is changeme. You can check if authentication is enabled by running the same command you ran to check if Elasticsearch is working.

curl -XGET 'localhost:9200/?pretty'

Now the output will say that authentication has failed.

user@vultr:~# curl -XGET 'localhost:9200/?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication token for REST request [/?pretty]",
        "header" : {
          "WWW-Authenticate" : "Basic realm=/"security/" charset=/"UTF-8/""
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication token for REST request [/?pretty]",
    "header" : {
      "WWW-Authenticate" : "Basic realm=/"security/" charset=/"UTF-8/""
    }
  },
  "status" : 401
}

Change the default password changeme by running the following command.

curl -XPUT -u elastic:changeme 'localhost:9200/_xpack/security/user/elastic/_password?pretty' -H 'Content-Type: application/json' -d'
{
  "password": "NewElasticPassword"
}
'

Replace NewPassword with the actual password you want to use. You can check if the new password is set and Elasticsearch is working by running the following command.

curl -XGET -u elastic:NewElasticPassword 'localhost:9200/?pretty'    

You will see output showing the successful execution of the query.

Further, edit Elasticsearch configuration file by running:

sudo nano /etc/elasticsearch/elasticsearch.yml

Find the following lines, uncomment the lines and change them according to instructions provided.

#cluster.name: my-application    #Provide the name of your cluster
#node.name: node-1               #Provide the name of your node
#network.host: 192.168.0.1

For network.host, provide the private IP address assigned to the system. Restart the Elasticsearch instance by running:

sudo systemctl restart elasticsearch

Now, instead of localhost, you will need to use the IP address to run the query using curl.

curl -XGET -u elastic:NewElasticPassword '192.168.0.1:9200/?pretty'

Replace 192.168.0.1 with the actual private IP address of the server. Now that we have installed Elasticsearch, proceed further to install Kibana.

Step 4: Install Kibana

Kibana is used to visualize the logs and actionable insights using a web interface. It can also be used to manage Elasticsearch. It is recommended to install the same version of Kibana as Elasticsearch.

As we have already added the Elasticsearch repository and PGP key, we can install Kibana directly by running:

sudo apt -y install kibana

The previous command will install the latest version of Kibana on your system. Once Kibana has been installed, reload the Systemd service daemon by running:

sudo systemctl daemon-reload

You can start Kibana and enable it to automatically start at boot time by running:

sudo systemctl enable kibana
sudo systemctl start kibana

Install X-Pack for Kibana

You can install X-Pack for Kibana directly by running:

cd /usr/share/kibana
sudo bin/kibana-plugin install x-pack

X-Pack for Kibana has Graph, Machine Learning and Monitoring enabled by default. X-Pack also enables authentication for Kibana. The default username is kibana and password is changeme. It is important to change the default password of Kibana user. Run the following command to change the password.

curl -XPUT -u elastic '192.168.0.1:9200/_xpack/security/user/kibana/_password?pretty' -H 'Content-Type: application/json' -d'
{
  "password": "NewKibanaPassword"
}
'

Replace 192.168.0.1 with the actual private IP address of the server and NewKibanaPassword with the new password for Kibana user.

Edit the Kibana configuration file by running:

sudo nano /etc/kibana/kibana.yml

Find the following lines and change the values according to instructions provided.

#elasticsearch.url: "http://localhost:9200"
#elasticsearch.username: "user"
#elasticsearch.password: "password"

Uncomment the lines above and, in elasticsearch.url provide the URL for Elasticsearch instance. The IP address must be the same IP that was used in elasticsearch.yml. Further, set the username from user to elastic and also provide the password of the elastic user which you have set earlier.

Restart the Kibana instance by running:

sudo systemctl restart kibana

Install Nginx as reverse proxy for Kibana

Since we are running Kibana on localhost at port 5601, it is recommended to setup a reverse proxy with Apache or Nginx to access Kibana from outside the local network. In this tutorial, we will setup Nginx as a reverse proxy for Kibana. We will also secure the Nginx instance with a Let’s Encrypt free SSL certificate.

Install Nginx by running:

sudo apt -y install nginx

Start and enable Nginx to automatically start at boot time.

sudo systemctl start nginx
sudo systemctl enable nginx

Now that the Nginx web server is installed and running, we can proceed to install Certbot, which is the official and automatic Let’s Encrypt certificate client. Add Certbot PPA to your system by running:

sudo add-apt-repository ppa:certbot/certbot

Update the repository meta information.

sudo apt update

Now you can easily install the latest version of Certbot by running:

sudo apt -y install python-certbot-nginx 

The previous command will resolve and install the required dependencies along with the Certbot package.

Now that we have Certbot installed, generate the certificates for your domain by running:

sudo certbot certonly --webroot -w /var/www/html/ -d kibana.example.com

Do not forget to change kibana.example.com with your actual domain name. The previous command will use the Certbot client. The certonly parameter tells the Certbot client to generate the certificates only. Using this option ensures that certificates are not automatically installed, and that Nginx configuration has not changed. Verification will be done by placing the challenge files in the specified webroot directory.

Certbot will ask you to provide your email address to send the renewal notice. You will also need to accept the license agreement.

To obtain certificates from Let’s Encrypt CA, you must ensure that the domain for which the certificates you wish to generate are pointed towards the server. If not, then make the necessary changes to the DNS records of your domain and wait for the DNS to propagate before making the certificate request again. Certbot checks the domain authority before providing the certificates.

The generated certificates are likely to be stored in the /etc/letsencrypt/live/kibana.example.com/ directory. The SSL certificate will be stored as fullchain.pem and the private key will be stored as privkey.pem.

Let’s Encrypt certificates are due to expire in 90 days, hence it is recommended to setup auto renewal for the certificates using cronjobs. Cron is a system service which is used to run periodic tasks.

Open the cron job file by running:

sudo crontab -e

Add the following line at the end of the file.

30 5 * * 1 /usr/bin/certbot renew -a nginx --quiet

The above cron job will run every Monday at 5:30 AM. If the certificate is due for expiry, it will automatically renew them.

Edit the default virtual host file for Nginx by running the following command.

sudo nano /etc/nginx/sites-available/default

Replace the existing content with the following content.

server {
    listen 80 default_server;
    server_name kibana.example.com
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 default_server ssl http2;

    server_name kibana.example.com;

    ssl_certificate           /etc/letsencrypt/live/kibana.example.com/fullchain.pem;
    ssl_certificate_key       /etc/letsencrypt/live/kibana.example.com/privkey.pem;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
    ssl_ecdh_curve secp384r1;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;
    ssl_stapling on;
    ssl_stapling_verify on;
    resolver 8.8.8.8 8.8.4.4 valid=300s;
    resolver_timeout 5s;
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
    add_header X-Frame-Options DENY;
    add_header X-Content-Type-Options nosniff;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
}

Make sure you update kibana.example.com with your actual domain name, also verify the path to the SSL certificate and private key.

Restart the Nginx web server by running:

sudo systemctl restart nginx

If everything has been configured correctly, you will see the Kibana login screen. Login using username kibana and the password which you have set. You should be able to successfully log in and see the Kibana dashboard. Leave the dashboard, for now, we will configure it later.

Install Logstash

Logstash can be also installed through the official Elasticsearch repository which we added earlier. Install Logstash by running:

sudo apt -y install logstash

The above command will install the latest version of Logstash on your system. Once Logstash has installed, reload the Systemd service daemon by running:

sudo systemctl daemon-reload

Start Logstash and enable it to automatically start at boot time.

sudo systemctl enable logstash
sudo systemctl start logstash

Install X-Pack for Logstash

You can install X-Pack for Logstash directly by running:

cd /usr/share/logstash
sudo bin/logstash-plugin install x-pack

X-Pack for Logstash comes with a default user logstash_system. You can reset the password by running:

curl -XPUT -u elastic '192.168.0.1:9200/_xpack/security/user/logstash_system/_password?pretty' -H 'Content-Type: application/json' -d'
{
  "password": "NewLogstashPassword"
}
'

Replace 192.168.0.1 with the actual private IP address of the server and NewLogstashPassword with the new password for Logstash user.

Now restart the Logstash service by running:

sudo systemctl restart logstash

Edit the Logstash configuration file by running:

sudo nano /etc/logstash/logstash.yml

Add the following lines at the end of the file to enable monitoring of the Logstash instance.

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: http://192.168.0.1:9200
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: NewLogstashPassword

Replace the Elasticsearch URL and Logstash password according to your setup.

You can now configure Logstash to receive data using different Beats. There are several types of Beats available: Packetbeat, Metricbeat, Filebeat, Winlogbeat and Heartbeat. You will need to install each Beat separately.

Conclusion

In this tutorial, we have installed Elastic Stack with X-Pack on Ubuntu 17.04. A basic ELK Stack is now installed on your server.