The previous blog entry lacked using https so all communication is in plain text, which makes using passwords less than ideal. This blog entry fixes this.

The full source code for the docker-compose.yml and haproxy.cfg file is available here.


version: '3'
    restart: always
    container_name: ek
    image: sebp/elk
      - TZ=Asia/Tokyo
      - 9200
      - 5601
      - 5044
      - "./data:/var/lib/elasticsearch"
    restart: always
    container_name: haproxy2
    image: haproxy:2.1
      - 9100:9100
      - 9990:9990
      - 5601:5601
      - 6080:6080
      - "./haproxy:/usr/local/etc/haproxy"

What’s this docker-compose file doing?

It starts 2 containers: the ELK container and the HAProxy container. ELK can receive traffic via port 9200 (ES) and 5601 (Kibana). HAProxy connects to the outer world via ports 9100, 9990, 5601 and 6080.

The data directory for ElasticSearch is in ./data. The HAProxy configuration incl. TLS certificate is in ./haproxy/

What’s HAProxy doing?

HAProxy is the TLS terminator and port forwarder to the ELK container. Here’s part of the HAProxy config file (in ./haproxy/haproxy.conf)

frontend kibana-http-in
    bind *:6080
    default_backend kibana

frontend kibana-https-in
    bind *:5601 ssl crt /usr/local/etc/haproxy/ssl/private/
    default_backend kibana

backend kibana
    balance roundrobin
    option httpclose
    option forwardfor
    server kibana elk:5601 maxconn 32
  • HAProxy listens on port 6080 for HTTP traffic. It forwards traffic to the backend.
  • HAProxy listens on port 5601 for HTTPS traffic. TLS connection is terminated here. It then forwards the unencrypted traffic to the backend.
  • The backend is on port 5601 on the elk container
  • Not displayed above, but the same happens for ElasticSearch traffic.

Thus you would connect via HTTPS to port 5601 for Kibana and port 9100 for ElasticSearch. You could use HTTP on port 6080 for Kibana and port 9990 for ElasticSearch.

Steps to get ELK with HTTPS running

While I like automation, I doubt I’ll configure this more than twice until the automation breaks due to new versions of ELK. So it’s “half-automated”.


  • A Linux server with min. 2GB RAM, Docker and docker-compose installed. I prefer Debian. Both version 9 and 10 works.
  • A TLS certificate (e.g. via Let’s Encrypt) in ./haproxy/ssl/private/DOMAIN.pem (full cert chain + private key, no passphrase)
  • Define some variables which we’ll refer to later:


Prepare your Docker host:

# ES refuses to start if vm.max_map_count is less
sudo echo "vm.max_map_count = 262144" > /etc/sysctl.d/10es.conf
sudo sysctl -f --system

# verify:
sysctl vm.max_map_count
# Output should be "vm.max_map_count = 262144"

# Directory for docker-compose files
mkdir elk
cd elk

# Data directory for ES
mkdir data
sudo chown 991:991 data

# Start docker-compose
docker-compose up -d

It takes about 1-2 minutes. You’ll see the CPU being suddenly less busy. Now you could connect to http://DOCKER_HOST:6080. No check for accounts or passwords yet.

Note: If you delete the container (e.g. via “docker-compose down”), you have to do the following steps again!

# Enter the ELK container
docker exec -it ek /bin/bash

Inside the container modify some settings (replace the variables PW_BOOTSTRAP and PW_KIBANA with the actual passwords):

cd /opt/elasticsearch
mkdir /etc/elasticsearch/certs
bin/elasticsearch-certutil cert -out /etc/elasticsearch/certs/elastic-certificates.p12 -pass ""

cat >> /etc/elasticsearch/elasticsearch.yml <<_EOF_
xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12


cd /opt/elasticsearch
echo "${PW_BOOTSTRAP}" | bin/elasticsearch-keystore add bootstrap.password

chown -R elasticsearch:elasticsearch /etc/elasticsearch

# edit /opt/kibana/config/kibana.yml with Kibana user name and password
sed 's/.*elasticsearch.username:.*/elasticsearch.username: "kibana"/;s/.*elasticsearch.password:.*/elasticsearch.password: "${PW_KIBANA}"/' < /opt/kibana/config/kibana.yml > /tmp/kibana.yml && cp /tmp/kibana.yml /opt/kibana/config/kibana.yml

cat >> /opt/kibana/config/kibana.yml <<_EOF_
xpack.security.encryptionKey: "Secret 32 char long string of chars"
xpack.security.secureCookies: true

chown kibana:kibana /opt/kibana/config/kibana.yml


Restart the containers:

docker-compose restart

Again wait about 1 minute. Then execute from anywhere (i.e. not the docker host):

function setPW() {
curl -uelastic:${PW_BOOTSTRAP} -XPUT -H "Content-Type:application/json" "http://${DOCKER_HOST}:${ES_HTTP_PORT}/_xpack/security/user/$1/_password" -d "{ \"password\":\"$2\" }"

# Since ES is running by now, set the passwords
setPW "kibana" "${PW_KIBANA}"
setPW "apm_system" "${PW_APM}"
setPW "logstash_system" "${PW_LOGSTASH}"
setPW "beats_system" "${PW_BEATS}"
setPW "remote_monitoring_user" "${PW_MONITORING}"
setPW "elastic" "${PW_ELASTIC}"

Now you should be able to log in to Kibana at https://DOCKER_HOST:5601, account is “elastic” with its password. And you can connect to ES via https://DOCKER_HOST:9100


For convenience I create an index, and a user and a role to access only that index:

# Create index logs
curl -uelastic:${PW_ELASTIC} -XPUT -H "Content-Type:application/json" \
"http://${DOCKER_HOST}:${ES_HTTP_PORT}/logs" -d '{"settings": { "number_of_shards": 1 }, \
"mappings": { "properties": { "timestamp": { "type": "date" }, "status": { "type": "integer" },\
 "channel": { "type": "text" }, "msg": { "type": "text" }}}}'

# Create a data item
curl -uelastic:${PW_ELASTIC} -XPOST -H "Content-Type:application/json" \
"http://${DOCKER_HOST}:${ES_HTTP_PORT}/logs/_doc" \
-d '{ "timestamp": "'$(date --iso-8601=seconds -u)'", \
"status": 200, "channel": "curl", "msg": "Initial test via curl"}'

# Just for verification: find your previous entry via this

curl -uelastic:${PW_ELASTIC} -XGET -H "Content-Type:application/json" \

# Create role and user to index data directly into ElasticSearch

curl -uelastic:${PW_ELASTIC} -XPOST -H "Content-Type:application/json" \
"http://${DOCKER_HOST}:${ES_HTTP_PORT}/_security/role/logwriter" \
-d '{"cluster":[],"indices":[{"names":["logs*"],"privileges":["create_doc"], \
"allow_restricted_indices":false}],"applications":[],"run_as":[], \

curl -uelastic:${PW_ELASTIC} -XPOST -H "Content-Type:application/json" \
"http://${DOCKER_HOST}:${ES_HTTP_PORT}/_security/user/logger" -d '{ "password": "PASSWORD", \
"roles": [ "logwriter" ], "full_name": "Logging user", "email": "my.email@gmail.com" }'

And finally I can log from Cloudflare Workers into ELK via HTTPS:

addEventListener('fetch', event => {

async function esLog(status, msg) {
  const esUrl='https://DOCKER_HOST:9100/logs/_doc';
  let headers = new Headers;
  let now = new Date();
  const body = {
    timestamp: now.toISOString(),
    status: status,
    channel: "log-to-es",
    msg: msg
  headers.append('Authorization', 'Basic ' + btoa('logger:PASSWORD'));
  headers.append('Content-Type', 'application/json');
  try {
    res = await fetch(esUrl, {
      method: 'POST',
      body:    JSON.stringify(body),
      headers: headers,
      return true;
  } catch(err) {
    return false;

 * Respond to the request
 * @param {Request} request
async function handleRequest(request) {

  let now1 = new Date();
  await esLog(201, "now1 is set...");
  let now2 = new Date();
  await esLog(202, `now2-now1=${now2-now1} ms`);

  return new Response("Check Kibana for 2 log entries", {status: 200})

and then you run this once, you’ll get 2 log entries in ES:


Logging via ElasticSearch

The Elastic Stack is a simple way to log “things” into ElasticSearch and make them nicely visible via Kibana. Since ELK can handle logs as well as time series data, I’ll use it for my own logging incl. performance logging.

For pure time series data I’d use the TIG stack: Telegraf, InfluxDB and Grafana.


sudo sysctl vm.max_map_count=262144
mkdir elk
cd elk
cat >docker-compose.yaml <<_EOF_
version: '3'
    restart: always
    container_name: elk
    image: sebp/elk
      - TZ=Asia/Tokyo
      - "5601:5601"
      - "9200:9200"
      - "5044:5044"
      - "./data:/var/lib/elasticsearch"
mkdir -m 0755 data
sudo chown 991:991 data
docker-compose up -d

Starting takes a minute. It’s up when the Kibana web interface is reachable.

Reference: elk-docker

Using Kibana

Connect to http://the.docker.host:5601/ to get the Kibana interface. Click on the Dev Tools icon to create a new index:

PUT /logs
  "settings": {
  "number_of_shards": 1
  "mappings": {
    "properties": {
      "timestamp": { "type": "date" },
      "status": { "type": "integer" },
      "msg": { "type": "text" }

To see that there’s something inside the index:

GET /logs/_search

To put something into the index:

POST /logs/_doc
  "timestamp": "2020-02-02T15:15:18+09:00",
  "status": 201,
  "msg": "Not so difficult, is it?"

And did you know you can use SQL statements for ElasticSearch too?

POST /_sql?format=txt
  "query": """
    SELECT * FROM logs WHERE timestamp IS NOT NULL ORDER BY timestamp DESC LIMIT 5

Sending Logs

Via curl:

curl -H "Content-Type: application/json" -X POST "http://the.docker.host:9200/logs/_doc" -d '{ "timestamp": "'$(date --iso-8601="sec")'", "status": 200, "msg": "Testing from curl" }'

Via Node.js:

const { Client } = require('@elastic/elasticsearch');
const client = new Client({ node: 'http://your.docker.host:9200' });

async function run () {
  let now = new Date();
  try {
    await client.index({
      index: 'logs',
      body: {
        timestamp: now.toISOString(),
        status: 200,
        msg: 'Testing from Node.js'
  } catch (err) {
    console.error("Error: ", err);



Until here there’s no security. Nothing at all. Anyone can connect to ElasticSearch and execute commands to add data or delete indices. Anyone can login to Kibana and look at data. Not good.

This describes how to set up security. Here you got the explanation for the permissions you can give to roles.

I created a role ‘logswriter’ which has the permissions for ‘create_doc’ for the index ‘logs*’. Then a user ‘logger’ with above role. Now I can log via that user. In Node.js this looks like:

const client = new Client({
  node: 'http://the.docker.host:9200',
  auth: {
    username: 'logger',
    password: 'PASSWORD'

the rest is unchanged. Via fetch API it’s simple too:

try {
  fetch('http://logger:PASSWORD@the.docker.host:9200/logs/_doc', {
    method: 'post',
    body: JSON.stringify({
      timestamp: now.toISOString(),
      status: 206,
      msg: 'Fetch did this')
    headers: { 'Content-Type': 'application/json' },
} catch(err) {
  console.error('Error: ', ${err});

Note that while the username and password can come from an environment variable so it’s not in in the source code, the transport protocol is still unencrypted http.

To be continued. To make the use of https necessary, I have to deploy this on the Internet first.