This post details steps to create a monitoring environment with InfluxDB & Grafana on the Linux machine using Docker-Compose.
Using docker-compose to bring up containers gives a standardized network with a single command which of course saves time. Further to that, we will also verify the complete environment by adding data to InfluxDB and further verifying it through Grafana.
So let's get started.
Step 01 - Connect to AWS Instance
Step 02 - Install Docker
Step 03 - Set up docker-compose
Step 1 - Pre-requisite check and server preparation
Step 2 - Identify Docker Images
Step 3 - Strategize & Create Network
Step 4 - Strategize & Create Volumes
Step 5 - Create docker-compose.yml
Step 7 - Run Containers with Docker-Compose
Step 8 - Add Sample Data to InfluxDB
Step 9 - Create Grafana dashboard to view sample data
Avail Linux system. Else follow -Connect to AWS Instance
Make sure below ports are enabled for incoming traffic. In AWS EC2 enable ports by adding 3000 and 8086 ports in inbound rules.
Docker installed. Else follow-Install Docker. Give below command to verify :
Docker-Compose installed. Else follow-Set up docker-compose. Give below command to verify :
Below are the official docker images from docker hub for our requirement:
By default, container on run gets attached to default bridge network which is called docker 0 and containers can talk to each other using this default bridge network.
Refer below diagram which will help us to decide network over which Influxdb and Grafana containers would talk.
In this diagram, we see two containers. We will create a customized network for logical separation of monitoring network which is good from a security perspective as well. So when containers are run with -network option, they get attached to the user-defined bridge network and talk over this network. We can give any name to the user-defined network. Let's name our user-defined network as monitor_network.
Let's create the network on the server where monitoring solution will be installed
docker network create monitor_network
Verify network creation:
docker network ls
Also, we need to do port binding(using -p option) for InfluxDB container (default port 8086)& Grafana (default port 3000)container with the host port so that, they can be connected via HTTP.
Port binding would be specified in docker-compose.yml file using ports parameter.
If we wish to have persisting data once we bring down our containers, we need to specify the docker volumes using volume option for InfluxDB & Grafana container.
We can give any name to the volumes. Let's name them influxdb-volume and grafana-volume respectively.
Let's create volume on the server:
docker volume create influxdb-volume docker volume create grafana-volume
Verify volume creation:
docker volume ls
Now we have a user-defined network and docker volume in place, we can create docker-compose.yml.
Note in docker-compose: External:true When docker-compose runs, it will not attempt to create the volumes specified & assumes that these volumes are already created and for some reason, if these volumes are not there(if you have not created volumes before running docker-compose), it will throw an error.
You can create this file on Linux machine directly or you can create the file on your local machine & transfer to the location using WinSCP. I created the file on my windows notepad and transferred using WinSCP tool to /home/ec2-users
Run containers by giving the command from the location where docker-compose.yml is available or use -f option to specify file location:
docker-compose up -d
With this, containers are up and running as shown above.
Here we are done with the setup. However, we will perform a couple of more steps to verify the environment & data persistence.
Make sure 8086 & 3000 ports are open for incoming traffic.
For InfluxDB access URL http://<server-publicip>:8086/query
For Grafana access URL http://<server-publicip>:3000
Access influxdb container using container name or id :
docker exec -it influxdb_container sh
Once you are inside the container, type the command to access influx CLI
Once inside CLI, create sampledb database. Connect to it as well.
create database sampledb use sampledb
Add measurement to this db by doing copy paste of below insert query:
INSERT cpu,host=serverA value=10 INSERT cpu,host=serverA value=8 INSERT cpu,host=serverA value=6 INSERT cpu,host=serverA value=3 INSERT cpu,host=serverA value=10 INSERT cpu,host=serverA value=10 INSERT cpu,host=serverA value=10 INSERT cpu,host=serverA value=10 INSERT cpu,host=serverA value=10 INSERT cpu,host=serverA value=8 INSERT cpu,host=serverA value=6 INSERT cpu,host=serverA value=3 INSERT cpu,host=serverA value=10 INSERT cpu,host=serverA value=8 INSERT cpu,host=serverA value=6 INSERT cpu,host=serverA value=3
Login to Grafana and add influxdb Datasource with the sampledb
Create a dashboard and a query to view data.
Congratulations we are all set with the environment. Since we have added volumes for data persistence, bringing up and down containers multiple times will not let us lose data. So you are all set to go for the next steps which may be adding telegraf database or your application database to be monitored.