Installing and Configuring ELK

– Part one(1) – Install OS (OEL 7.2), Tuning
– Part two(2) – Configure KVM, Tuning
– Part three(3) – install elasticsearch, configuration
– Part four(4) – Install logstash, configuration
– Part five(5) – Install Kibana, configuration
– Part six(6) – General platform tips & tricks

This is Part three(3) – install elasticsearch, configuration

Configuring the Elasticsearch Data Nodes

Add manual 5Tb FC disk to the VM

virsh edit elkdm1

# Append the section below to the disk section (of course change to your disk path)
# Note: For pooled stoarge, just add stoage in vit-manager
fidk -l will show the new disk mpath

    
      
      
      
    

# To add on the fly (without a reboot)
virsh attach-disk elkdm2 /dev/mapper/mpatha vdb

To remove the disk live

virsh detach-disk elkdm1 vdb

Now Lets configure the file system used for elasticsearch data

Create Ext4 filesystem

# In the VM, create ext4 file system
# Note For SSD array use the below
mkfs.ext4 -O extent -b 4096 -E stride=128,stripe-width=128 -O ^has_journal /dev/mapper/mpathd

# enable writeback mode, this mode will typically provide the best ext4 performance
tune2fs -o journal_data_writeback /dev/mapper/mpathd

# Verify
dumpe2fs /dev/mapper/mpathd |grep 'Filesystem features'

#
/etc/fstab options
/dev/mapper/mpathd /data2 ext4 discard,noatime,nodiratime,defaults,data=writeback,noatime 0 0
# Source for SSD tuning
http://pof.eslack.org/2013/01/12/ssd-alignment-on-linux-with-ext4-and-lvm/

# HDD config
mkfs.ext4 /dev/vdb
mkdir /data1
mount /dev/vdb /data1
chown -R elasticsearch:elasticsearch /data1

To resize / Grow the ext4 file system

umount /data2
#Note only needed for fsck
e2fsck -f /dev/mapper/mpathd

# Grow the fs
resize2fs /dev/mapper/mpathd

Add to automount after reboot

# Add to /etc/fstab
# elk mount
/dev/vdb     /data1                       ext4    defaults        0 2

On all nodes, create necessary directory’s

mkdir /data1 /data1/data /data1/plugins /data1/log

Now lets install elasticsearch

Install elasticsearch on all nodes besides the logstash nodes.
Note: Make sure you have java 1.8 otherwise install as below

yum install java-1.8.0-openjdk.x86_64

Lets configure elasticsearch repo

rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch

cat /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-2.1]
name=Elasticsearch repository for 2.x packages
baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Now install elasticsearch

yum -y install elasticsearch

Note: An alternative is to get it from there web site and extract in /opt. the rest of this document is referring to the per-packged version

wget https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.1.1/elasticsearch-2.1.1.tar.gz

Configure elasticsearch nodes

Modify /etc/elasticsearch/elasticsearch.yml
Note: Please look at the end of this document for a full elasticsearch.yml
Below is only the different node types

# Master node (max 3 for quorum)
node.master: true 
node.data: false

# Data Node, scale as you go
node.master: false
node.data: true

# Client node, for client access
node.master: false
node.data: false

# Set per node local name
node.name: elkms1

# Set per host local ip
network.host: 10.10.3.22

Elasticsearch Memory tuning

Enable Mlockall (so no memory swapping)
Set as per each node in part one

cat /etc/sysconfig/elasticsearch |egrep -v "^$|^#"
ES_HEAP_SIZE=20g
ES_STARTUP_SLEEP_TIME=5
MAX_OPEN_FILES=65535
MAX_LOCKED_MEMORY=unlimited
MAX_MAP_COUNT=262144

# If using systemd
/usr/lib/systemd/system/elasticsearch.service
LimitMEMLOCK=infinity

# In some cases this is also needed
# Modify /etc/init.d/elasticsearch
# [..] snip
echo -n $"Starting $prog: "
# if not running, start it up here, usually something like "daemon $exec"
su $ES_USER --shell /bin/bash -c "ulimit -l unlimited"
daemon --user $ES_USER --pidfile $pidfile $exec -p $pidfile -d -Des.default.path.home=$ES_HOME -Des.default.path.logs=$LOG_DIR -Des.default.path.data=$DATA_DIR -Des.default.path.work=$WORK_DIR -Des.default.path.conf=$CONF_DIR

# For testing
# Manuel set & start - export ES_HEAP_SIZE=30g ; ./bin/elasticsearch -Xmx30g -Xms30g

Verify that mlockall is set to true

curl http://localhost:9200/_nodes/process?pretty|grep mlockall

Enable and start services

systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch

Elasticsearch Tips

Get config details

curl http://localhost:9200/_nodes/process?pretty

Get index list

curl 'elkms1.domain.com:9200/_cat/indices?v'

Delete index

curl -XDELETE 'http://elkdm1.domain.com:9200/logstash-2016.02.08/'

Shutdown cluster

curl -XPOST "http://elkms1:9200/_shutdown"

elasticsearch.yml

cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please see the documentation for further information on configuration options:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: ElkCluster1
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: elkms1
#
# Add custom attributes to the node:
#
# node.rack: r1
# ------------------------------------ Node role -------------------------------
#
node.master: true 
node.data: false
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data1/data
#
# Path to log files:
#
path.logs: /data1/log
#
# Path to plugins files:
path.plugins: /data1/plugins
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.mlockall: true
#
# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
# available on the system and that the owner of the process is allowed to use this limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 10.10.3.22
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["elkms1", "elkms2", "elkms3"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 2
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
gateway.recover_after_nodes: 2
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
action.destructive_requires_name: true
#
# ---------------------------------- Extra Tuning ------------------------------
# Enable compress intracluster transport
transport.tcp.compress: true

# Cache Sizes
# Free cache after 50%
indices.fielddata.cache.size: 50%
#indices.fielddata.cache.size: 15%
indices.memory.index_buffer_size: 50%
indices.fielddata.cache.expire: 6h
indices.cache.filter.size: 15%
indices.cache.filter.terms.size: 1024mb
indices.cache.filter.expire: 6h

# Indexing Settings for Writes
index.refresh_interval: 30s
index.translog.flush_threshold_ops: 50000

# Only needed when sharing same dir for two instances
#node.max_local_storage_nodes = 1

# Disable delete_all_indices
action.disable_delete_all_indices: true

# Optimizing Index Requests
indices.cluster.send_refresh_mapping: false

# Concurrent rebalance num os CPU's
cluster.routing.allocation.cluster_concurrent_rebalance: 3
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: .97
cluster.routing.allocation.disk.watermark.high: .99

# Recovery Properties Allow for Faster Restart Times
cluster.routing.allocation.node_concurrent_recoveries: 4
cluster.routing.allocation.node_initial_primaries_recoveries: 18
indices.recovery.concurrent_streams: 4
indices.recovery.max_bytes_per_sec: 40mb

# Threadpool Properties Prevent Data Loss
#threadpool.bulk.queue_size: 3000

#http://kufli.blogspot.com/2014/11/elasticsearch-advanced-settings-and.html
## Threadpool Settings ##
# Search pool
#threadpool:
#    search:
#        type: cached
#        size: 100
#        queue_size: 2000
#
#threadpool.search.type: fixed
#threadpool.search.size: 50
#threadpool.search.queue_size: 200
# Bulk pool
#threadpool.bulk.type: fixed
#threadpool.bulk.size: 10
#threadpool.bulk.queue_size: 100
# Index pool
#threadpool.index.type: fixed
#threadpool.index.size: 60
#threadpool.index.queue_size: 1000
# Indices settings
#indices.memory.index_buffer_size: 30%
#indices.memory.min_shard_index_buffer_size: 12mb
#indices.memory.min_index_buffer_size: 96mb

KVM disk help
http://ronaldevers.nl/2012/10/14/adding-a-physical-disk-kvm-libvirt.html
Current Elasticsearch
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html
How to enable mlockall

Elasticsearch tuning
Next Part four(4) – Install logstash, configuration