This week I have had a few customers ask how to integrate Grafana with NetFlow to improve their network operations center. For those of you who don’t know, Grafana is an open source platform that can be used to visualize a wide variety of data types. With Scrutinizer’s API, NetFlow can be one of them!
Typically, a NOC or SOC “command console view” provides multiple screens that give high-level data and alerts that analysts can glance at for a health check of the network. Grafana works great for this. A company can have the ever-sought-after ”single pane of glass” while keeping best-of-breed solutions doing their work under the hood. The integration we nailed together accomplishes this; the customer was able to get high-level reports into their Grafana instance and still use the Scrutinizer UI for forensic investigation and more complex filtering as needed.
NetFlow to Grafana: Setting Up the Integration
To get started, we will need to make sure the Grafana instance has the simpleJSON data source. If you don’t have Grafana installed, follow this guide on how to install Grafana on CentOS and install it directly on your Scrutinizer server.
Next, we should take a quick look at the simpleJSON documentation, which explains three different routes we will use within our script:
- The ”/“ route is used to return a 200 status code to show simpleJSON that we are up and running.
- The ”/search” route will return an array of different report types that can be graphed in Grafana.
- The ”/query” route will return the JSON object from Scrutinizer’s API.
I have found that either Node.js with Express or Python with Flask do a great job of listening for these routes and formatting the data for simpleJSON.
Here is an example from Node. When a POST request from Grafana hits the ”/search” route, our application sends back an array of available reports to use for querying the data.
If we look at the request object passed from Grafana, there are some key fields that we should use to retrieve data from Scrutinizer—namely the “to” and “from” timeframes and the “target,” which in our case is the report we want to be graphed.
By parsing these elements from the Grafana request, our integration becomes a lot more dynamic. We can click and drag into the graph, add different report elements, and start to harness the power of Grafana.
Integrations like this are the reason I’m happy to work for a company whose solution has a robust API. Within an hour, the customer and I were able to get reports heading into Grafana and we both learned some cool stuff along the way.
If integration with Grafana or other third-party integrations are of interest to you, please contact our support team for help getting started.
Since working on the initial integration of getting NetFlow to Grafana, I have been lucky enough to work with customers to test things out. During this process, there were a couple of features customers asked for:
- The ability to specify which exporter to graph data for.
- The ability to specify a data interval for the graph.
Both of these are popular features that exist within Scrutinizer today, so it makes sense to have them available for Grafana. Implementing new features required a modification to the original simple-json-datasource, which lead to the creation of a Plixer Data Source.
To be clear, this isn’t a full-blown custom Data Source. I have plans for that later, depending on how popular this integration turns out to be, but it is a slightly modified simple-json-datasource that will need to be installed on your Grafana server.
To import a custom Data Source in Grafana it needs to be installed under /var/lib/grafana/plugins. Your Grafana service then needs to be restarted. When the services are restarted, Grafana looks in that directory to see if there is anything new and then we are off to the races.
Import the Plixer Data Source
To makes things easy, I’ve made the Plixer Data Source code available on our GitHub. All you need to do to is create a directory in /var/lib/grafana/plugins and then clone the data source into it.
- SSH to Grafana and navigate to /var/lib/grafana/plugins
- Make sure your server has git installed and run ‘sudo yum install git’
- Create a directory to put the Plixer Data Source in with ‘mkdir PlixerDataSource’
- Inside this new directory, clone the Data Source with ‘git clone https://github.com/plixer/Plixer-Grafana.git’
- Restart the Grafana Service with ‘service grafana-server restart’
The Plixer Data Source will now be available from the dropdown menu on your Grafana server.
Import the Plixer Back-End Server
Since all we’ve done is modify the original data source we used for this integration, we can salvage all the code used for the backend server. Once again, I have made this code available on GitHub to make the process a bit easier.
- SSH to Scrutinizer and navigate to /home/plixer/scrutinizer/files
- Create a directory to put the Plixer Back End in with ‘mkdir PlixerGrafana’
- Inside this new directory clone the program with ‘git clone https://github.com/plixer/Plixer-Grafana-Backend.git’
- Make sure you server has pip 2.7 installed. You can run ‘pip -V’ to check.
- If pip is not installed install it with ‘yum –y install python-pip’
- Run ‘pip install –r requirements.txt’ to install required Python packages.
You’ll need to modify the settings.json file in order to get started.
Once this is done, you can start the program by running the command ‘python grafanaapp.py &’
You will see a message print saying that the server is up and listening on port 5000.
Connect Grafana to the Plixer Back-End Server
From here, go to the Plixer Data Source in Grafana and fill out the required information. Click Save & Test and you should be good to go.
The final step is creating our first dashboard within Grafana. To do this, hit the + sign in Grafana. Select Create Dashboard and then select Graph.
Finally, we want to edit this panel and select what data we want visualized. Under the metrics tab select a Report, Exporter, and Interval to graph in and wait for the data to appear.
All of the code for this integration is open source and available on GitHub. I would love to see the community improve this integration and bring new ideas for how Scrutinizer and Grafana can work together. If you would like to help develop this plugin, please download the code to get started!