End to end steps to set up InfluxDB & Grafana using Docker-Compose

This post details steps to create a monitoring environment with InfluxDB & Grafana on the Linux machine using Docker-Compose.

Using docker-compose to bring up containers gives a standardized network with a single command which of course saves time. Further to that, we will also verify the complete environment by adding data to InfluxDB and further verifying it through Grafana.

So let's get started.

To visit more such posts follow hashtag - #RealTimeMonitoring #InfluxDB #Docker #Grafana

PRE-MILESTONE STEPS

Step 01 - Connect to AWS Instance

Step 02 - Install Docker

Step 03 - Set up docker-compose

MILESTONE STEPS

Step 1 - Pre-requisite check and server preparation

Step 2 - Identify Docker Images

Step 3 - Strategize & Create Network

Step 4 - Strategize & Create Volumes

Step 5 - Create docker-compose.yml

Step 7 - Run Containers with Docker-Compose

Step 8 - Add Sample Data to InfluxDB

Step 9 - Create Grafana dashboard to view sample data

Complete Series: Real-Time Monitoring Using InfluxDB & Grafana

• Avail Linux system. Else follow -Connect to AWS Instance

• Make sure below ports are enabled for incoming traffic. In AWS EC2 enable ports by adding 3000 and 8086 ports in inbound rules.

• Docker installed. Else follow-Install Docker. Give below command to verify :

docker --version

• Docker-Compose installed. Else follow-Set up docker-compose. Give below command to verify :

docker-compose --version

• Below are the official docker images from docker hub for our requirement:

By default, container on run gets attached to default bridge network which is called docker 0 and containers can talk to each other using this default bridge network.

Refer below diagram which will help us to decide network over which Influxdb and Grafana containers would talk.

In this diagram, we see two containers. We will create a customized network for logical separation of monitoring network which is good from a security perspective as well. So when containers are run with -network option, they get attached to the user-defined bridge network and talk over this network.
We can give any name to the user-defined network. Let's name our user-defined network as monitor_network.

Let's create the network on the server where monitoring solution will be installed

docker network create monitor_network

Verify network creation:

docker network ls

---

Know The Concepts: Docker Container Networking

---

Also, we need to do port binding(using -p option) for InfluxDB container (default port 8086)& Grafana (default port 3000)container with the host port so that, they can be connected via HTTP. 

Port binding would be specified in docker-compose.yml file using ports parameter.

---

Know The Concepts: Docker Port Binding

---

If we wish to have persisting data once we bring down our containers, we need to specify the docker volumes using volume option for InfluxDB & Grafana container.

We can give any name to the volumes. Let's name them influxdb-volume and grafana-volume respectively.

Let's create volume on the server:

docker volume create influxdb-volume
docker volume create grafana-volume

Verify volume creation:

docker volume ls

---

Know The Concepts: Docker Volume

---

Now we have a user-defined network and docker volume in place, we can create docker-compose.yml.

Note in docker-compose: External:true
When docker-compose runs, it will not attempt to create the volumes specified & assumes that these volumes are already created and for some reason, if these volumes are not there(if you have not created volumes before running docker-compose), it will throw an error.

You can create this file on Linux machine directly or you can create the file on your local machine & transfer to the location using WinSCP. 
I created the file on my windows notepad and transferred using WinSCP tool to /home/ec2-users

Run containers by giving the command from the location where docker-compose.yml is available or use -f option to specify file location:

docker-compose up -d

With this, containers are up and running as shown above.

Here we are done with the setup. However, we will perform a couple of more steps to verify the environment & data persistence.

• Make sure 8086 & 3000 ports are open for incoming traffic.

• For InfluxDB access URL http://<server-publicip>:8086/query

• For Grafana access URL http://<server-publicip>:3000

• Access influxdb container using container name or id :

 docker exec -it influxdb_container sh

• Once you are inside the container, type the command to access influx CLI

influx

• Once inside CLI, create sampledb database. Connect to it as well.

create database sampledb
use sampledb

• Add measurement to this db by doing copy paste of below insert query:

INSERT cpu,host=serverA value=10
INSERT cpu,host=serverA value=8
INSERT cpu,host=serverA value=6
INSERT cpu,host=serverA value=3
INSERT cpu,host=serverA value=10
INSERT cpu,host=serverA value=10
INSERT cpu,host=serverA value=10
INSERT cpu,host=serverA value=10
INSERT cpu,host=serverA value=10
INSERT cpu,host=serverA value=8
INSERT cpu,host=serverA value=6
INSERT cpu,host=serverA value=3
INSERT cpu,host=serverA value=10
INSERT cpu,host=serverA value=8
INSERT cpu,host=serverA value=6
INSERT cpu,host=serverA value=3

• Login to Grafana and add influxdb Datasource with the sampledb

• Create a dashboard and a query to view data.

Congratulations we are all set with the environment. Since we have added volumes for data persistence, bringing up and down containers multiple times will not let us lose data. So you are all set to go for the next steps which may be adding telegraf database or your application database to be monitored.

---

Complete Series: Real-Time Monitoring Using InfluxDB & Grafana