episyche logo represented by a symbol of a brain

Episyche

nav bar menu for mobile contains the nav bar content

Search/Elasticsearch/

How to configure Elasticsearch and Kibana setup

Published on

How to configure Elasticsearch and Kibana setup
Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack. It provides near real-time search. Here, we will learn about configuring Elasticsearch and Kibana with detailed steps.

Introduction:

Elasticsearch :

Elasticsearch provides near real-time search and analytics for all types of data. Whether you have structured or unstructured text, numerical data, or geospatial data, Elasticsearch can efficiently store and index it in a way that supports fast searches. You can go far beyond simple data retrieval and aggregate information to discover trends and patterns in your data. And as your data and query volume grows, the distributed nature of Elasticsearch enables your deployment to grow seamlessly right along with it.

Kibana:

Kibana enables you to give shape to your data and navigate the Elastic Stack. With Kibana, you can:

  • Search, observe, and protect your data. From discovering documents to analyzing logs to finding security vulnerabilities, Kibana is your portal for accessing these capabilities and more.

  • Analyze your data. Search for hidden insights, visualize what you’ve found in charts, gauges, maps, graphs, and more, and combine them in a dashboard.

  • Manage, monitor, and secure the Elastic Stack. Manage your data, monitor the health of your Elastic Stack cluster, and control which users have access to which features

Flow diagram:


flow diagram

Prerequisites :

  • Ubuntu Server

Steps :

Step 1: Establish SSH Connection to ELK Server

Using Putty

  • Install putty

  • Open putty , In HostName : <your_server_ip_address>

  • To connect server you must have an SSH key file in .ppk format.

  • Go to Connection > SSH > Auth > Browse your SSH key in .ppk format


ssh 
ssh

  • Go to Session > Save and Open

  • It will prompt the connect alert, please accept it and login to the server with an SSH username (for example : ubuntu)

Using WSL/Git Bash/Open SSH Terminal:

ssh -i {ssh_key_in_pem_format} <ssh_username>@{your_server_ip_address}

For example:
ssh -i /home/user/elk_server_ssh_key.pem ubuntu@13.126.68.231

Step 2: Import the PGP key

Download and install the public signing key :

1wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg

Step 3: Installing from the APT repository

You may need to install the apt-transport-https package on Debian before proceeding:

1# installing the packages before proceeding 2sudo apt-get install apt-transport-https 3 4# Save the repository 5echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

Step 4: Installing Elasticsearch

  • To Install Elasticsearch, please run the following commands.

1sudo apt-get update && sudo apt-get install elasticsearch
:warning:

After executing the above command, elastic superuser password will be prompt on the screen. Please save the same in a text file, we need this password for further configurations.

An example screenshot of the prompt is given below.


superuser password

  • Start the elasticsearch service using following command

1systemctl start elasticsearch.service

  • Enable elasticsearch service. This configuration helps to bring the Elasticsearch service online, as soon as the server is started.

1systemctl enable elasticsearch.service

  • Verify the Elasticsearch health using following command.

:warning:

Identify the http_ca.crt file path in your server and use it to check Elasticsearch health using curl command.

Change your path to tmp directory . Then, execute the curl command

1cd /tmp 2curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200

After executing curl command, please enter the password saved in text file from above step.

If Elasticsearch is running properly, It will show you Elasticsearch health status


elastic health

Step 5: Configuring the Elasticsearch to access it publicly.

:warning:

As standard practice, try to the backup of any .yml files before updating them. In this case, copy the elasticsearch.yml as elasticsearch.yml_backup_<date>

For example:

cp elasticsearch.yml elasticsearch.yml_backup_23_08_2022

  • Navigate to Elasticsearch configuration directory.
    cd /etc/elasticsearch

  • Update the elasticsearch.yml with following network host and port configurations.

    network.host: 0.0.0.0
    http.port: 9200

A sample Elasticsearch configuration can be found in following git repository file.

  • Restart the Elasticsearch service.

systemctl restart elasticsearch.service

  • Try to access the Elasticsearch URL (i.e https://<server-ip:9200>) from your browser and check the health of the Elasticsearch. An example screenshot is given below for your reference.



elastic health

Step 6: Installing Kibana

  • To Install Kibana, please run the following commands.

1sudo apt-get update && sudo apt-get install kibana

  • Start and enable Kibana auto start after server reboot, using following commands.

1systemctl start kibana.service 2systemctl enable kibana.service

Step 7: Configuring the Kibana

  • Navigate to Kibana configuration directory.

cd/etc/kibana

  • Update the kibana.yml with following server host and port configurations.

server.host : "0.0.0.0"

server.port : 5601

A sample Kibana configuration can be found in following git repository file.

  • Restart the Kibana service.

systemctl restart kibana.service

  • Try to access the Kibana URL(i.e http://<server-ip:5601>)from your browser.

  • In order to establish connection with Elasticsearch, the Kibana will prompt with enrollment token request popup box.



token

  • You can then generate an enrollment token for Kibana with the elasticsearch-create-enrollment-token tool

  • Navigate to the directory where you installed Elasticsearch and run the elasticsearch-create-enrollment-token tool to generate an enrollment token

    1cd /usr/share/elasticsearch 2bin/elasticsearch-create-enrollment-token -s kibana

  • Copy the generated token and paste it into the browser and click configure elastic button.

  • After Kibana will prompt for Verification code.

  • To generate Verification code , navigate to Kibana installation directory and execute the following script.

1cd /usr/share/kibana 2bin/kibana-verification-code

  • Please enter the verification code obtained from the previous step in the Kibana console, as shown in the following screenshot.



verification code

Result:



result

Comments