Modern Log Management | The Right Way with ELK Stack

User activity on our apps is incredibly valuable data. System failures and performance states provide rich information that feeds the quality of a product. However collecting every single action from both humans and machines is sometimes a complex task.

ELK is a technology stack that mixes Elastic Search, Logstash and Kibana to provide a comprehensive approach to consolidate, manage, and analyze logs from your apps, by providing real-time insights from consolidated data.

The Problem

Assuming all systems activity goes to a single point is incorrect but not impossible, modern solutions like distributed systems produce thousand and millions of entries per day from different sources in our network.

I few words it’s an accelerated data growth from different sources.

|            +------------+                +-------------+ 
| | | logs | |
| | web-1 <----------------> dashboard-1 |
| | | | |
| +------------+ +-------------+

And with big logs generation comes exponential disk usage.

The Solution

The most common way of getting graphs and business numbers from log files is to place all resources in a single disk/storage.

Centralized logs storage

|                   +------------+                                             
| | |
| | web-1 >
| | |\
| +------------+ \
| \logs
| \
| +------------+ \ +------------+ +-------------+
| | | logs \| | | |
| | db-1 >-------------> log-1 +---------+ dashboard-1 |
| | | /| | | |
| +------------+ / +------------+ +-------------+
| /
| /logs
| +------------+ /
| | |/
| | api-1 >
| | |
| +------------+

Pull/Push strategy

Data can flow in both directions, pulled from the central log server or pushed from every node. It depends on resources and strategy. Once all eggs are in one basket, parse and search are an easy tasks with the right tools. And finally track and graph will produce an intersting desition making tool.

Diving into ELK (Elasticsearch, Logstash, Kibana)

The ELK Stack is popular because it fulfills a specific need in the log management and log analysis space. In cloud-based infrastructures, consolidating log outputs to a central location from different sources like web servers, mail servers, database servers, network appliances can be particularly useful. This is especially true when trying to make better data-informed decisions. The ELK stack simplifies searching and analyzing data by providing insights in real-time from the log data.

It is common to run the full ELK stack, not each individual component separately. Each of these services play a important role and in order to perform under high demand it is more advantageous to deploy each service on its own server. Doing so introduces other problems related to multi-node deployments, networking, security, and management. For now, we will stick with the single stack deployment so we can learn the basics of ELK and use it for development and testing purposes. We will address multi-node deployment suitable for production environments further along in the process.

ELK components the big picture

| +-----------+
| | Kibana UI |
| +----+------+
| |
| +-------------+ +--------------------------------------+ |
| | datasource | | logstash | |
| | logs | | | |
| +------+------+ | +--------------------------------+ | +--------+--------+
| | | | plugins | | | |
| | | | | |------| ElasticSearch |
| +------+-----+ +----------+ | | +-----+ +-------+ +------+ | | | |
| | web-2 |--->|file-beat |--->| | |input| |filter | |output| | | +-----------------+
| +------------+ +----------+ | | +-----+ +-------+ +------+ | |
| | +--------------------------------+ |
| +--------------------------------------+


A distributed RESTful search engine built for the cloud that stores the logged messages.


Collects, processes, and forwards events and log messages. Receives logged messages and relays them to ElasticSearch.


An open-source, browser-based analytics and search dashboard. Provides visualization capabilities on top of the content indexed on an Elasticsearch cluster.


Client agent in charge of pushing any new file or modification from the datasource.

Implementing ELK the easy way (Docker)

This guide offers a step-by-step walk-through on performing the basic setup of an ELK Stack using container technology (Docker) and learning the basics of implementing logs into the stack and performing real time analysis of the data.

This option is particularly useful approach if you are not interested in getting into low level details of the ELK setup process and just want to leverage its benefits.

Setting up the ELK Container

The first thing that we need to do is download an ELK image. We will be using the following image available in DockerHub, which packages the latest available versions of the components:

Let’s pull the image by issuing the following command:

$ docker pull sebp/elk

Using default tag: latest

Trying to pull repository ... latest: Pulling from sebp/elk

The image is around 900+GB so depending on your internet connection it will take some time. Once the image has been downloaded you will see the following message:

Status: Downloaded newer image for <a class="external-link" href="" rel="nofollow"></a>

And you should be able to see the image executing docker images command:

$ docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE latest 3e07e86fb40d 10 days ago 975.5 MB

Now we can create a container from the image with the following command:

$ sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -p 5000:5000 -it --name elk sebp/elk

This command publishes the following ports, which are needed for proper operation of the ELK stack:

  • 5601 (Kibana web interface).
  • 9200 (Elasticsearch JSON interface).
  • 5044 (Logstash Beats interface, receives logs from Beats such as Filebeat – see the Forwarding logs with Filebeat section).
  • 5000 (Logstash Lumberjack interface, receives logs from Logstash forwarders – see theForwarding logs with Logstash forwarder section).
  • Access Kibana’s web interface by browsing to http://:5601, where is the hostname or IP address of the host Docker is running on, e.g. localhost (if running a local native version of Docker, or the IP address of the virtual machine if running a VM-hosted version of Docker).

As in Kibana version 4.0.0, you won’t be able to see anything (not even an empty dashboard) until something has been logged, so let’s jump right on to the next section on how to forward logs from regular applications, like Apache Web Server.

Logstash Configuration

Let’s first review how the process works:

  1. The Apache Web Server is running, and logging requests to configured path.
  2. Logstash, running as a persistent daemon, monitors the Apache logs for new lines and processes them.
  3. Logstash will send parsed logs in JSON document form to Elasticsearch for storage and the ability to perform analytics on them.
  4. Kibana uses Elasticsearch as a back-end for dashboarding and searching.

First thing to do is confirm where Apache is logging to. The default apache log file would be found here:

sudo tail /var/log/apache2/access.log

You should see one or more lines that look like this: - - [18/Jun/2016:09:10:02 +0000] "GET / HTTP/1.1" 200 11359 "-" "curl/7.38.0"

This means, Logstash can now ingest and parse these logs.

Next, create a simple logstash configuration file - create a file named apache.conf (the name is arbitrary) under /etc/logstash/conf.d/apache.conf: /etc/logstash/conf.d/apache.conf

input {
file {
path => "/var/log/apache2/access.log"
start_position => beginning
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
output {
elasticsearch { host => localhost }

If your log file is located at another non-traditional location, adjust the third line, path => ‘/var/log/apache2/access.log’, accordingly.

The config file simply watches the apache log file for events, parses them with a grok pattern (a simplified predefined regular expression) called COMBINEDAPACHELOG, and will print those events to standard output (the Logstash documentation has additional information.)

Visualising Data

To confirm the pipeline is working correctly, make some requests to your web server using your web browser. Then, look at an Elasticsearch API to see that a new index has been created (it will appear in the list of indices as logstash- followed by the date):

curl localhost:9200/_cat/indices

There should be an index with the current date appended with documents inserted by Logstash. Now, with Kibana running in the background from your systemd unit, browse to the machine’s address under port 5601 to start using Kibana’s interface:


Kibana makes an educated guess at your index and time field names, so selecting “Create” here will get you started. Click on the “Discover” tab at the top and you’ll see a timeline of Apache log events.


Implementing ELK the hard way

By hard we mean deployment of every individual component from scratch on a fresh cloud machine.

Setup elasticsearch repositories

wget -qO - | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list
sudo apt-get update

sudo apt-get install elasticsearch
vi /etc/elasticsearch/elasticsearch.yml
vi /etc/elasticsearch/jvm.options
sudo systemctl start elasticsearch.service

Easy way to monitor service activity

sudo journalctl -f
sudo journalctl --unit elasticsearch --since "2016-10-30 18:17:16"

sudo apt-get install logstash

vi /etc/logstash/logstash.yml
sudo systemctl start logstash.service

Test elasticsearch endpoint

curl -XGET 'localhost:9200/?pretty'

First test

root@ubuntu-xenial:/vagrant# systemctl start logstash.service
root@ubuntu-xenial:/vagrant# ps aux | grep logstash
logstash 27235 152 39.8 3444488 404764 ? SNsl 23:22 0:15 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
root 27270 0.0 0.0 14224 928 pts/0 S+ 23:22 0:00 grep --color=auto logstash
root@ubuntu-xenial:/vagrant# whereis logstash
logstash: /etc/logstash /usr/share/logstash
root@ubuntu-xenial:/vagrant# /usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout {} }'


Install filebeat on client machine

sudo apt-get install filebeat
vi /etc/filebeat/filebeat.yml
- input_type: log
- /var/log/syslog
hosts: ["localhost:5044"]
systemctl start filebeat.service

Create your first pipeline

vi /etc/logstash/conf.d/localhost_syslog.conf
input {
beats {
port => "5044"
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
geoip {
source => "clientip"
output {
stdout { codec => rubydebug }
systemctl restart logstash.service

Test config

root@ubuntu-xenial:/home/ubuntu# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/localhost_syslog.conf --config.test_and_exit
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/ Using default config which logs to console
Configuration OK
15:48:23.938 [LogStash::Runner] INFO logstash.runner - Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash

Fire up the agent service on the client side

systemctl restart filebeat.service

Deploy kibana

sudo apt-get install kibana    
whereis kibana
kibana: /etc/kibana /usr/share/kibana
vi /etc/kibana/kibana.yml
elasticsearch.url: "http://localhost:9200"
  • Set elasticsearch.url to point at your Elasticsearch instance config/kibana.ym

    systemctl start kibana.service

  • Run bin/kibana


Control main services

ubuntu@ubuntu-xenial:~$ sudo su
root@ubuntu-xenial:/home/ubuntu# systemctl start elasticsearch.service
root@ubuntu-xenial:/home/ubuntu# systemctl start logstash.service
root@ubuntu-xenial:/home/ubuntu# systemctl start filebeat.service
root@ubuntu-xenial:/home/ubuntu# systemctl start kibana.service

Monitor for startup activity

Apr 19 21:42:49 ubuntu-xenial su[1779]: pam_systemd(su:session): Cannot create session: Already running in a session
Apr 19 21:43:01 ubuntu-xenial systemd[1]: Starting Elasticsearch...
Apr 19 21:43:01 ubuntu-xenial systemd[1]: Started Elasticsearch.
Apr 19 21:43:08 ubuntu-xenial systemd[1]: Started logstash.
Apr 19 21:43:15 ubuntu-xenial systemd[1]: Started filebeat.
Apr 19 21:43:21 ubuntu-xenial systemd[1]: Started Kibana.

Check web instance is available

root@ubuntu-xenial:/home/ubuntu# curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size
yellow open .kibana Ru4w0KGQRLOa-Ogp5dHk7Q 1 1 2 0 23.4kb 23.4kb
yellow open filebeat-2017.04.20 bhSrqJLpQk2RTrJikBjTXw 5 1 1278749 0 152.8mb 152.8mb
yellow open logstash-2017.04.20 ez3TSPJZSniovcTPHtd92Q 5 1 586259 0 76.1mb 76.1mb
yellow open filebeat-2017.04.19 LvDgezqwSDaVVzg0utKIqg 5 1 3414161 0 407mb 407mb



Log management and analysis are key activities when it comes to maintaining a stable infrastructure. Having the possibility to consolidate logs from different sources and analyze them easily adds a lot of value to your DevOps practice and fosters a culture of continuous improvement within your organization.

Share this post

Table of Contents