Altough originally this was supposed to be short post about setting up ELK stack for logging. However with every moment I have been working with this technology it got me really ‘insipired’ and I thought it would be worth to start and make it working the right way from the very beggining
Now since we are up for automating things we wil try to make use of docker compose which will allow us to setup whole stack in automated way. Docker compose is detailed in here
Compose in short allows you to describe how your services will look like and how do they interact with each other (volumes/ports/links).
In this post we will be using docker + docker-compose on Ubuntu host running in Azure. If you would be wondering why I just show my IP addresses all the time on the screenshots … because those are not load balanced static IP addresses. So every time I spin a host I get a new one 🙂
This post contains information which have been updated in post
However for gettign idea of how solution works I recommend just reading through 🙂
So the first thing we need to do is to install docker-compose. Since as we all now docker is under constant development it is easiest to give you link to gitHub release page rather than direct link which can be out of date
Once installed you can use the following command to make sure it is installed :
Preparing folder structure
Since we will be using config files and storing elasticsearch data on the host we will need to setup folder structure. I’m aware that this can be done better with variables 🙂 but ubuntu is still learning curve so I will leave it up to you to find better ways 🙂 In the meantime let’s run the following command
sudo mkdir -p /cDocker/elasticsearch/data
sudo mkdir -p /cDocker/logstash/conf
sudo mkdir -p /cDocker/logstash/agent
sudo mkdir -p /cDocker/logstash/central
sudo mkdir -p /cDocker/compose/elk_stack
Clone configuration files
Once you have the folder structure we will prepare our config files. To do this we will be cloning gitHub repository (gists ) which I have prepared in advance (and tested as well of course ) .
git clone https://gist.github.com/60c3d7ff1b383e34990a.git /cDocker/compose/elk_stack
git clone https://gist.github.com/6627a2bf05ff956a28a9.git /cDocker/logstash/central/
git clone https://gist.github.com/0cd6594672ebfe1205a5.git /cDocker/logstash/agent/
git clone https://gist.github.com/c897a35f955c9b1aa052.git /cDocker/elasticsearch/data/
Since I keep a bit different names on github (this might be subject to change in future ) we need to rename them a bit 🙂 For this you can run following commands
mv /cDocker/compose/elk_stack/docker-compose_elk_with_redis.yml /cDocker/compose/elk_stack/docker-compose.yml
mv /cDocker/elasticsearch/data/elasticsearch_sample_conf.yml /cDocker/elasticsearch/data/elasticsearch.yml
mv /cDocker/logstash/agent/logstash_config_agent_with_redis.conf /cDocker/logstash/conf/agent.conf
mv /cDocker/logstash/central/logstash_config_central.conf /cDocker/logstash/conf/central.conf
Docker compose file
If you look at the code file below you will notice that we define how our image will be build. What ports will be epxosed , what links will be created amongst containers. Thanks to that machines will be created in specific order and linked accordingly, And since we have already prepared configuration files the whole stack will be ready to go.
Now we have everything in place to set up our first run of orechestration. Our next step is just navigating to compose folder (where our docker-compose file is ) and running following command :
/cDocker/compose/elk_stack#: docker-compose up -d
This will execute pulling of all layers and in creating of services afterwards. Once completed you should see something similar to the following :
Well and thats it folks! We of course have much more potential to do much more (using variables / labels etc ) however we will do more funky stuff in next posts. Since Azure Files is finally in production we will use it as persistent storage in one of our future posts so stay tuned.
On subject of ready to use ELK stack we will be looking into managing input based on logstash plugins and we will see on our own eyes how this Docker ELK stack will empower our IoT automations!