Companies today seem to be screaming for easy access to time series data, and there are a few rising stars in the space—notably Grafana and Elastic. Both of these companies can collect metrics; Grafana with its close ties to Prometheus and their new Loki project, and Elastic touting Logstash as the leading alternative to Splunk.
Although these solutions are excelling in the world of syslogs, they do not have an easy way of visualizing network metadata (NetFlow, IPFIX, sFlow, etc.). Flow data has traditionally been used by networking gurus to troubleshoot network bandwidth and help in capacity planning. But over the years, use cases for DevOps and network security have gained traction as the protocol has evolved.
Use cases for time series metrics
Think of it this way:
If I’m in DevOps and people are complaining about the performance of an application, there are all sorts of things I would want to know right off the bat.
- Is the web server sending any interesting error codes?
- Are any of my endpoints seeing high response latency?
- Are database logs signaling that it is overworked??
- Are the network links saturated or seeing latency between the user and the server?
As a network security engineer, I can visualize all the connections of every user and server across the network. If a user’s PC gets infected with malware, I might want to know:
- Who else has this user been communicating with over the last week or month?
- Has anyone else connected to the same command and control server as the infected host?
- Can I do any pattern recognition or machine learning on this data to help prevent future breaches?
In both of these use cases, collecting log information alone may help answer a few questions, but it will leave big questions about traffic on the actual network. This is where flow data comes into the picture, and having a nice integration between a best-in-class flow collection system and a leading time series aggregator seems to make sense, right?
Grafana & Scrutinizer integration
Two years ago, I hacked together an integration with Grafana using Python and their pre-built simple-json-datasource. It worked, the data was accurate, but it could be a pain to get set up and get running. At the time I was happy to have the project completed and figured if people really started using it, I could take a look at putting more time into it.
Well, since then, Grafana has absolutely exploded and I have been seeing all sorts of requests to get Scrutinizer linked up with the system. I figured it was time to re-write this integration and give our customers a much easier way to get up and running.
After taking some feedback from customers, I’m happy to deliver version 2.0 of the Grafana/Scrutinizer integration. I also took some time to put together a Read the Docs guide that will be helpful in the initial setup.
Please reach out to me directly if you would like help setting this integration up or if there are any report types or different types of data you would like to visualize within Grafana.
