INSTALL & CONFIGURE ELK STACK ON CENTOS7

“ELK” is the acronym for the three open source projects call Elasticsearch, Logstash and Kibana. ELK stack made easier to analyze logs to system administrators. ELK stack collect logs from clients using Beats protocol.

ELK STACK MAIN COMPONENTS:-

Elasticsearch is an open source, distributed, RESTful, JSON based search and analytic engine. Easy to use and flexible. Elasticsearch is the heart of ELK stack. Elasticsearch is a No-SQL database.

Logstash is a open source, server-side data processing pipeline that pull events data from multitude of sources simultaneously, transform it, and then sends it to Elasticsearch. Easily pull data from logs, metrics, web applications, data sources and various AWS services. Logstash dynamically transforms and prepares data regardless of format or complexity. Derive structure from unstructured data with grock.

Kibana provides GUI for users visualize data with charts and graphs real-time. It is a window into the Elastic Stack. Provides data exploration, visualization and dashboarding.

Beats is the platform for single-purpose data shippers. They install as lightweight agents and send data from numerous machines to Logstash of Elasticsearch.

FileBeat will send logs to Logstash, Logstash process incoming logs and stores into Elasticsearch, and then we can visualize through the Kibana web interface.

ELK STACK ARCHITECTURE

Software Versions I have used in this tutorial.

ELK Stack Server 192.168.100.10 (CentOS7) Beat Client 192.168.100.50 (CentOS7) Elasticsearch Version: 6.3.1 Logstash Version 6.3.1 Kibana Version 6.3.1

STEP 1 - INSTALL JAVA 8

Java is required for the Elastic stack deployment. Elasticsearch requires Java 8, it is recommended to use the Oracle JDK 1.8. I will install Java 8 from the official Oracle rpm package. ELK requires the Oracle Java JDK package has to be installed. The same JVM version should be installed on all Elasticsearch nodes and clients.

INSTALL JDK RPM

[root@elk-stack /]# rpm -ivh jdk-8u171-linux-x64.rpm

SET DEFAULT JAVA VERSION

[root@elk-stack /]# alternatives --config java

SET JAVAC AND JAR PATHS

[root@elk-stack /]# alternatives --set jar /usr/java/jdk1.8.0_171-amd64/bin/jar
[root@elk-stack /]# alternatives --set javac /usr/java/jdk1.8.0_171-amd64/bin/javac

CHECK JAVA VERSION

[root@elk-stack /]# java -version

SET JAVA ENVIRONMENT VARIABLES

[root@elk-stack /]# export JAVA_HOME=/usr/java/jdk1.8.0_171-amd64/
[root@elk-stack /]# export JRE_HOME=/usr/java/jdk1.8.0_171-amd64/jre/
[root@elk-stack /]# export PATH=$PATH:/usr/java/jdk1.8.0_171-amd64/bin/:/usr/java/jdk1.8.0_171-amd64/jre/bin/

Set Environment Variables on /etc/bashrc to Auto Set at System Boot

[root@elk-stack /]# vim ~/.bashrc

JAVA_HOME=/usr/java/jdk1.8.0_171-amd64/
JRE_HOME=/usr/java/jdk1.8.0_171-amd64/jre/
PATH=$PATH:/usr/java/jdk1.8.0_171-amd64/bin/:/usr/java/jdk1.8.0_171-amd64/jre/bin/

STEP 2 – INSTALL AND CONFIGURE ELASTICSEARCH

In this step, I will install and configure Elasticsearch version 6.3.1

A. IMPORT PUBLIC GPG KEY TO THE ELK-STACK SERVER

[root@elk-stack /]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

B. DOWNLOAD ELASTICSEARCH RPM PACKAGE

[root@elk-stack /]# wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.1.rpm

C. INSTALL ELASTICSEARCH RPM PACKAGE

[root@elk-stack /]# rpm -ivh elasticsearch-6.3.1.rpm

D. FIREWALL CONFIGURATION

Allow traffic through the TCP port 9200 in the firewall.

[root@elk-stack /]# firewall-cmd --permanent --add-port=9200/tcp
[root@elk-stack ~]# firewall-cmd --permanent --add-port=9300/tcp
[root@elk-stack /]# firewall-cmd –reload

E. CONFIGURE ELASTICSEARCH

Do the following changes

[root@elk-stack /]# vim /etc/elasticsearch/elasticsearch.yml

cluster.name: my-application
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 127.0.0.1
#network.host: 192.168.100.10
http.port: 9200

F. START & ENABLE ELASTICSEARCH AT SYSTEM BOOT

[root@elk-stack /]# systemctl start elasticsearch.service
[root@elk-stack /]# systemctl enable elasticsearch.service

G. TEST ELASTICSEARCH

Check Elasticsearch port “9200” state as “LISTEN”

[root@elk-stack /]# netstat -plntu

[root@elk-stack /]# curl -XGET '192.168.100.10:9200/?pretty'

Output from the external web browser

STEP 3 – INSTALL AND CONFIGURE LOGSTASH

In this step I will install Logstash version 6.3.1 and configure it as a central log server, receives logs from clients with Filebeat, then filter and transform the syslog data and move it into the stash (Elasticsearch)

A. Import Public GPG Key to the ELK-Stack Server

[root@elk-stack /]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

B. DOWNLOAD LOGSTASH RPM PACKAGE

[root@elk-stack /]# wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.1.rpm

C. INSTALL LOGSTASH RPM PACKAGE

[root@elk-stack /]# rpm -ivh logstash-6.3.1.rpm

D. FIREWALL CONFIGURATION

Allow traffic through the TCP port 5044 in the firewall.

[root@elk-stack /]# firewall-cmd --permanent --add-port=5044/tcp
[root@elk-stack /]# firewall-cmd --reload

E. GENERATE A NEW SSL CERTIFICATE

Create new ssl certificate for securing communication between Logstash & Filebeat (clients). SSL Certificate file use clients to identify the elastic server.

Do the following changes under the “[ V3_ca ]” section for the server identification.

[ v3_ca ]
#Server IP Address
subjectAltName = IP: 192.168.100.10

Generate the certificate file with the openssl command.

[root@elk-stack /]# openssl req -config /etc/pki/tls/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout /etc/pki/tls/private/logstash-forwarder.key -out /etc/pki/tls/certs/logstash-forwarder.crt

Once ssl certificate is ready, this certificate should be copied to all the clients using scp command.

F. CONFIGURE LOGSTASH

Create new configuration file named “logstash.conf” under the “/etc/logstash/conf.d” directory.

[root@elk-stack /]# vim /etc/logstash/conf.d/logstash.conf

There are three sections in “logstash.conf” file.

1. INPUT SECTION

This section make Logstash to listen on port 5044 for incoming logs & provides SSL certificate for secure connection.

# input section
input {
 beats {
   port => 5044
   ssl => true
   ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
   ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
   congestion_threshold => "40"
  }
}

This section parse the logs before sending them to Elasticsearch.

# Filter section
filter {
if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGLINE}" }
    }
    date {
match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

}
  }
}

3. OUTPUT SECTION

This section defines the storage for the logs to be stored.

# output section
output {
 elasticsearch {
  hosts => localhost
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
       }
stdout {
    codec => rubydebug
       }
}

G. ENABLE & START LOGSTASH SERVICE

[root@elk-stack /]# systemctl daemon-reload
[root@elk-stack /]# systemctl enable logstash.service 
[root@elk-stack /]# systemctl start logstash.service

STEP 4 – INSTALL AND CONFIGURE KIBANA

A. IMPORT PUBLIC GPG KEY TO THE ELK-STACK SERVER

[root@elk-stack /]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

B. DOWNLOAD LOGSTASH RPM PACKAGE

[root@elk-stack /]# rpm -ivh kibana-6.3.1-x86_64.rpm

C. FIREWALL CONFIGURATION

Allow traffic through the TCP port 5601 in the firewall.

[root@elk-stack /]# firewall-cmd --permanent --add-port=5601/tcp
[root@elk-stack /]# firewall-cmd –reload

D. CONFIGURE KIBANA

[root@elk-stack /]# vim /etc/kibana/kibana.yml

server.port: 5601
server.host: "127.0.0.1"
server.name: "Kibana Monitor"
elasticsearch.url: “http://localhost:9200”

E. ENABLE & START LOGSTASH SERVICE

[root@elk-stack /]# systemctl daemon-reload
[root@elk-stack /]# systemctl enable kibana.service 
[root@elk-stack /]# systemctl start kibana.service

STEP 5 – INSTALL AND CONFIGURE NGINX

A. INSATLL EPEL REPOSITORY

[root@elk-stack /]# yum -y install epel-release

B. INSTALL NGINX & HTTPD-TOOLS

[root@elk-stack /]# yum -y install nginx httpd-tools

C. CREATE USERNAME “ADMIN” AND PASSWORD “123456” FOR KIBANA WEB INTERFACE

[root@elk-stack /]# htpasswd -c /etc/nginx/htpasswd.users admin

D. CONFIGURE NGINX

Edit the Nginx configuration file and remove the ‘server { }’ block, so we can add a new virtual host configuration.

[root@elk-stack /]# vim /etc/nginx/nginx.conf

Create new virtual host configuration file named “kibana.conf” under the conf.d directory.

[root@elk-stack /]# vim /etc/nginx/conf.d/kibana.conf

server {
    listen 80;
 
    server_name elk-stack.co;
 
    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/.kibana-user;
 
    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
    }
   }

Check Nginx Configuration

E. ENABLE & START NGINX SERVICE

[root@elk-stack /]# systemctl enable nginx.service
[root@elk-stack /]# systemctl start nginx.service

F. FIREWALL CONFIGURATION

Allow traffic through the TCP port 80 in the firewall.

[root@elk-stack ~]# firewall-cmd --permanent --add-port=80/tcp
[root@elk-stack ~]# firewall-cmd --permanent --add-service=http
[root@elk-stack ~]# firewall-cmd --reload

G. SELINUX CONFGURATION

[root@elk-stack kibana]# setsebool -P httpd_can_network_connect 1

STEP 6 – INSTALL & CONFIGURE FILEBEAT ON THE CLIENT

A. COPY SSL CERTIFICATE FROM ELK SERVER TO CLIENT

[root@elk-stack ~]# scp /etc/pki/tls/certs/logstash-forwarder.crt root@192.168.100.50:/etc/pki/tls/certs/

B. IMPORT PUBLIC GPG KEY TO THE ELK-STACK SERVER

[root@elk-stack /]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

C. DOWNLOAD & INSTALL FILEBEAT RPM PACKAGE

[root@cl1 /]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.3.1-x86_64.rpm

[root@cl1 /]# rpm -ivh filebeat-6.3.1-x86_64.rpm

Configure File Beat

[root@cl1 /]#vim /etc/filebeat/filebeat.yml

In the paths: section paths:

/var/log/secure

/var/log/messages

/var/log/*.log

By default, Filebeat is using Elasticsearch as the output target by default. Disable Elasticsearch output by commenting the configuration.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["localhost:9200"]

Now add the Logstash as the new output configuration and do the changes as show below.

#----------------------------- Logstash output --------------------------------
output.logstash:
 # The Logstash hosts
 hosts: ["localhost:5044"]
 bulk_max_size: 1024
 ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
 template.name: "filebeat"
 template.path: "filebeat.template.json"
 template.overwrite: false

After that configurations will look like below.

filebeat.inputs:

- type: log
  enabled: false
  paths:
    - /var/log/*.log
    
output.logstash:
  hosts: ["localhost:5044"]
  bulk_max_size: 1024
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
  template.name: "filebeat"
  template.path: "filebeat.template.json"
  template.overwrite: false

E. ENABLE & START FILEBEAT SERVICE

[root@cl1 /]# systemctl enable filebeat.service 
[root@cl1 /]# systemctl start filebeat.service

STEP 7 – USING ELASTIC STACK

Now ELK Initial Steps are completed. We’ll discuss regarding Log Monitoring in a next tutorial.