Performance Monitoring using InfluxDB, Python & Grafana

This post details steps to get started with InfluxDB client python library. We will use this python library to insert data into InfluxDB.

To visit more such posts follow hashtag - #RealTimeMonitoring #InfluxDB #Python

Telegraf has got a variety of plugins to monitor infrastructure. However, the case may that we need to send custom data to InfluxDB from an app. Similarly in performance testing, we may need to send request data to the time series database & monitor it in Grafana on a real-time basis. With below example, we can send JSON data to the DB using python library InfluxDB.

Also, the Python client library’s write_points function can take data in either JSON or InfluxDB Line Protocol. However, the point to note is that InfluxdB takes data in the format of command-line protocol format only. Hence, any data provided in JSON format gets eventually converted to Line Protocol anyway before being sent to InfluxDB.

PRE-MILESTONE STEPS

Step 01 - Install Python

Step 02 - Avail or Setup InfluxDB instance

Step 03 - Setup PyCharm or any Python Editor

MILESTONE STEPS

Step 1 - Install InfluxDB Python client library

Step 2 - Import InfluxDB Python client library

Step 3 - Create InfluxDB instance

Step 4 - Connect to your database

Step 5 - Get data to be inserted in JSON format

Step 6 - Use write_points API to insert JSON data

Step 7 - Query data from InfluxDB

Complete Series: Real-Time Monitoring Using InfluxDB & Grafana

• Open a command prompt and verify that python is installed & setup by giving the command:

python --version

• Give below command to install influxdb-python library

 pip install influxdb

• With this step influxdb module gets installed inside :

<python path>\Lib\site-packages

• Open a command prompt and go to python shell by typing python

• Import influxdb-python library

from influxdb import InfluxDBClient

• Give the name of host and port on which InfluxDB server is running as below to connect to the client. This creates an instance of DB:

client=InfluxDBClient(host="localhost",port="8086")

• Additional parameters can also be passed in same like username, password in case authentication is required

• Verify database you intend to connect is available in the list

client.get_list_database()

• Switch to database

client.switch_database('MyDatabase')

• Now, we are ready to insert data in the database

InfluxDB takes data in the form of command-line protocol format. We can either directly insert data using this format or we can give JSON format data and it internally gets converted to command line format. Here in this post, we intend to give data in JSON format.

In JSON format, we can give data using attributes measurement, tags, fields & time.

[
    {
        "measurement": "measurement_name",
        "tags": {
            "tag1": "tag1_name",
            "tag2": "tag2_name"
        },
       "time": "timeofsendingdata",
        "fields": {
            "field_1": field1_value,
            "field_2":field2_value
                    }
    }]

For example, below is sample request-response data which we want to send to InfluxdB in this example. Here we want to build a measurement table called "MyResponseData". Tags are the ones on which basis we can filter data and field is actual data.

Note this is hardcoded data and in the actual script would be replaced by variables storing dynamic data.

[
    {
        "measurement": "MyResponseData",
        "tags": {
            "requestName": "LaunchURL",
            "requestType": "GET"
        },
       
        "fields": {
            "responseduration": 127,
            "responseLength":7253
                    }
    }]

Now we have a database to which we have to write data and also sample data which is to be written, we will take help of write_points method to write sample data to the database

As per the syntax below, we can pass the argument called points to the method. Point is a list of dictionaries where each dictionary is a point( measurement to be written)

write_points(points,time_precision=None,database=None,retention_policy=None,tags=None,batch_size=None,protocol=u'json',consistency=None)

In our case, we have only one point in our dictionary:

[ { "measurement": "MyResponseData", "tags": { "requestName": "LaunchURL", "requestType": "GET" }, "fields": { "responseduration": 127, "responseLength":7253 } }

]

We can add one or more point to our existing data to show a list of points and we can define it in a variable

my_json_data=[ { "measurement": "MyResponseData", "tags": { "requestName": "LaunchURL", "requestType": "GET" }, "fields": { "responseduration": 127, "responseLength":7253 } }, { "measurement": "MyResponseData", "tags": { "requestName": "Login", "requestType": "POST" }, "fields": { "responseduration": 137, "responseLength": 7287 } } ]

To write this data, give below:

client.write_points(my_json_data)

This will write data to the database.

Note here in this data, we have not given timestamp, hence time column of measurement will get timestamps of the time of insertion of data and not the time data is generated

To specify timestamp, add time attribute to the data point. Here time is read using date-time module of python. It reads time in UTC. For same we need to import datetime python module:

import datetime

import pytz

{ "measurement": "MyResponseData", "tags": { "requestName": "LaunchURL", "requestType": "GET" },

"time":datetime.datetime.utcnow(), "fields": { "responseduration": 127, "responseLength":7253 } }

To query data, python way, use query object, and get data in ResultSet type of object:

results=client.query('SELECT * FROM MyResponseData')

Data can be filtered using tags like below:

login_points=list(results.get_points(measurement='NewResponse',tags={"requestName": "Login"}))

Below is the final sample script.

Please note, in real-time implementation json_body would not be hard-coded and fed from some source.

To view data in Grafana, set DB as a data source. Follow instructions with this post

---

Complete Series: Real-Time Monitoring Using InfluxDB & Grafana