When it comes to Network Traffic Monitoring, how the data is stored, how long the data is stored, and how the data is presented, ultimately makes the difference when you select the NetFlow and sFlow analysis tool that you use.
Scrutinizer processes the NetFlow data exported from the devices and stores it in a database for traffic analysis and reporting. The flexible data history settings allow you to customize the storage patterns in the database based on the granular NetFlow reporting requirements you have, and your available disk storage resources.
Scrutinizer stores two types of data, raw data and aggregated data.
Raw data is each and every flow exported from the monitored interfaces of the routers. All of the flows exported from the routers are stored in the database as raw data in 1 minute data tables. Since, the 1 minute tables include each and every flow from the routers, the 1 minute granularity provides the most detailed analysis of the traffic. But, they can consume a lot of disk space. The 1 minute storage is determined by the Data History settings. One minute, as well as all of the data storage time intervals, can be configured by clicking on Admin Tab–> Settings –> Data History.
Apart from the raw data storage, Scrutinizer stores aggregated data in the database at different time intervals. The aggregation mechanism (roll ups) happen simultaneously at the back-end along with the raw data storage. The aggregated data is stored based on the Max Conversation setting of the application which can also be configured by clicking on Admin Tab–> Settings –> Data History.
The aggregation of NetFlow or sFlow data collected is done to avoid high disk space usage without impacting reporting and performance. The aggregated data on is generally used for historical reporting, capacity planning and trend analysis.
Older data is repeatedly rolled up into less granular times (5 minute, 30 minute, 2 hour, 12 hour, 1 day and weekly). The Max Conversation value determines the top number of records for every flow template type found for each device, based on octet value, that are aggregated and rolled up to the next time interval.
We update 5 minute tables every five minutes. We select the conversations over the last five minutes from the 1 minute tables, aggregate them together, then roll up the top conversations based on the Max Conversation value. We update a 30 minute table every thirty minutes. In the same manner, when we update a 30 minute table, we select conversations that took place over the last 30 minutes from the latest 5 minute table, aggregate them together, then roll them up. This process is repeated through all of the time intervals.
Now that we know how the data is stored, let’s take a look at how the data is presented.
When creating a report from the Scrutinizer application, the time interval used when displaying the report is automatically determined based on the date/time range specified in the report filter.
Note the Auto setting in the Granularity Box of this 11 hour report filter:
Scrutinizer uses the following time windows to determine the interval to build the report with.
- Reports of 1 hour duration or less will display 1 minute intervals
- Reports between 1 hour and 5 hours will display 5 minute intervals
- Reports between 5 hours and 30 hours will display 30 minute intervals
- Reports between 30 hours and 5 days will display 2 hour intervals
- Reports between 5 days and 30 days will display 12 hours intervals
- Reports between 30 days and 60 days will display 1 day intervals
- Reports beyond 60 days in duration will display 1 week intervals
But if you need a more granular view, Scrutinizer gives you the option to select a value in the granularity box other than Auto.
Let’s take a look at the same report filtered on 5 minute intervals.
And with any report you generate in Scrutinizer, you have the option to move the mouse over any point in the graph and select intervals to display the report on even a deeper granular level. Using Scrutinizer’s Flow View, you have the ability to even drill in on conversations as deep as 1 second intervals.
To learn more about these reporting features or any of the other network analysis benefits that the latest version of Scrutinizer provides, give us a call – (207)324-8805.