Setting up Elasticsearch for the Elastic SIEM
Jun 12, 2020 • 10 Minute Read
Introduction
So often, I hear from security professionals who tell me that they are, "...thinking about setting up an elastic stack..." but they feel like it may be too difficult or they’re just not sure where to start. This guide is meant to migrate you from the, "I was thinking about doing that," phase of the agile board to the final state of, "I totally did that and it was super easy."
This software stack can run on all the popular OS distributions, including windows. However, the best performance will be on Ubuntu or CentOS/Redhat. The options to install from the elasticsearch repos using package managers are available as well as the .deb and .rpm options for installations, but this guide will be using the Linux distro agnostic download and installation methods. I am specifically choosing CentOS 7 because it's more secure by default. I will be walking through the setup for commands working on RPM-based systems, accordingly.
Technology
Setting up an Elastic Stack
First things first, you need to have the base install of the correct versions of elasticsearch and kibana. This is much easier than you may initially imagine and simply means setting up the database and the web portal GUI that overlays the database for you to import data and browse that data respectively. How this works exactly and why it is related to Elastic SIEM becomes clear very quickly. The general idea is that elasticsearch is the database, kibana is the graphical interface for the database, and you need to ship the information into the database for analysis. The SIEM is included as a tab in the Kibana interface and is a way but not the only way to view the information that you have stored in the elasticsearch backend.
Installing Elasticsearch 7.3
Choosing the options to download the tarball and install outside of a package manager so it will not be specific to an individual version of Linux. Move to your /opt folder; general linux file structure designates this folder for third party optional packages.
Either create or use an existing user other than 'root' that you will be using to run the processes on this device. Elasticsearch will not run as root or with root privileges.
Change the directory to the /opt folder or wherever you would like to have the elasticsearch program running from.
Download the compressed tarball, download the sha512 hash, and compare the two to ensure file integrity. There is no reason to accidentally install malware as a part of your security platform. That would be embarrassing.
user]$ sudo wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.3.1-linux-x86_64.tar.gz
This will download the package from the elasticsearch repo. It does require a working internet connection and should look like this when complete.
Download the file containing the SHA512 hash of the tarball.
user]$ sudo wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.3.0-linux-x86_64.tar.gz.sha512
[user]$ sha512sum -c elasticsearch-7.3.1-linux-x86_64.tar.gz.sha512
On a debian system you can use shasum -c 512 elasticsearch-7.3.1-linux-x86_64.tar.gz.sha512
If you see anything other than "ok" in the results below, DO NOT INSTALL THIS PACKAGE. Verify the preceding steps, especially the download source, and potentially browse from another device to download a copy that has trusted file integrity. If you have the same result on different devices and different networks, please notify elastic!
Untar the downloaded file and move into the unpacked elasticsearch directory.
user]$ sudo tar -xzf elasticsearch-7.3.1-linux-x86_64.tar.gz
[user]$ cd elasticsearch-7.3.1/
[user]$ ls -lsa
The contents of the folder should look like this:
Configuring Elasticsearch
There are a set of checks that are done when elasticsearch boots to ensure that your system settings are optimized to run the software properly. Bootstrap checks are enforced when elasticsearch is published on a non-local loopback address, which will be required to use this instance to accept data from outside sources. The whole idea of SIEM is that you will collect network traffic, logs, and alerts from various sources. You need to set up this node for production use with the following steps.
Edit the elasticsearch.yml config file.
user]$ nano /opt/elasticsearch-7.3.1/config/elasticsearch.yml
Uncomment the networt.host setting and change it from the default of "localhost" to 0.0.0.0.
Change this line:
To look like this line:
Before closing this config file, scroll to the "Discovery" section and uncomment the "discovery.seed_hosts" setting and change the value in the brackets to ["0.0.0.0"].
With the hosting address value has been changed to a non-local loopback value, the bootstrap checks will be strictly enforced and cause the service to fail unless you change a few more settings.
When installing from .deb or .rpm file or directly from the elasticsearch repo with a package manager, some of these changes are made automatically to your system. So, you can certainly skip to the running elasticsearch section, try this first, and then address any failures with the steps in this section.
The first setting that needs to be set is vm.max_map_count. Simply change the max map count to the desired 262144 minimum value with sysctl.
user]$ sudo sysctl vm.max_map_count=262144
Additionally, you will need to change the max file descriptors value. This value is hard set at 4096, by default. Add two lines to the /etc/security/limits.conf file for the specific non-root user you will be using to run the elasticsearch process.
user]$ sudo nano /etc/security/limits.conf
Once complete, you need to reload the settings for the session to reflect the changes.
To reload the sysct.conf settings, perform the following command.
user]$ sysctl -p
To ensure the limits.conf changes are reloaded, it requires a logout and login of the session.
Modify your firewall to allow connections to elasticsearch on the correct port, 9200, and reload.
user]$ sudo firewall-cmd --permanent --zone=public --add-port=9200/tcp
[user]$ sudo firewall-cmd --reload
Use ufw for debian platforms.
Finally, change the permission on the elasticsearch directory. In practice, you will assign rights only to the user for this process to execute all binaries required and read/write to the remaining files. That will be up to your organization’s security policies.
user]$ sudo chmod -R 777 /opt/elasticsearch-7.3.1
Running Elasticsearch
Now it is time to run elasticsearch for the first time! In this installation format, you can run and the background binary from the terminal. Long term, this service needs to be added and configured to start on system boot. I will walk you through running each binary with console output as part of the general engineering process, allowing you to quickly see if the services fail and why, or if they are successful and you are ready to move on to the next step.
As part of this process, the use of screen allows for the quick switching between terminals and the background processes that are occupying stdout.
If you do not already have screen installed, do so now.
[user]$ sudo yum install screen -f
Once installed, run screen to enter the first session in which you will start the elasticsearch service.
user]$ screen
Then run the elasticsearch binary as the user selected. Do not use sudo
user]$ bin/elasticsearch
Notice the output indicating the IP address on which the service is running and the final status of the clusters. If the process exits then there was an error; read the error and walk back through the previous steps to identify the issue and retry.
Set the current terminal session to background by pressing ctrl+a+d and then use curl in the new terminal session to send an HTTP GET request to the elastic node.
user]$ curl http://0.0.0.0:9200
If you changed the elasticsearch instance address to something other than 0.0.0.0 then enter that value instead.
The output should look like this:
Conclusion
Congratulations! You are now the proud owner of a brand new elasticsearch node. Continue on to my next guideSetting Up Kibana for Elastic SIEM
References
-
Elastic SIEM Guide - https://www.elastic.co/guide/en/siem/guide/current/index.html
-
JVM Configuration for Production - https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
