We have reached the end of another Friday. First! A special shout out to all the Dads out there. I hope you all have a great weekend of pampering and special treatment.
I decided to write on this subject since the above question does seem to be coming up more and more. So let’s take a moment and discuss how NetFlow works.
NetFlow is quickly becoming the technology of choice when monitoring network performance. One of the coolest things about it is the ability to see granular data on a minute by minute basis. Knowing exactly what happened at 12:13PM yesterday afternoon is just cool.
Since people prefer that granular data, all our math is based on the router exporting flow every minute. So with every minute that goes by, Scrutinizer is populating tables with this 1 minute data. But here is where database management becomes critical…
When a person wants to see what happened all last week, it would be insanely inefficient for Scrutinizer to query the 1 min tables since there would be too many rows to merge and join, not to mention the amount of disk space this would use.
To combat this problem, we have a roll up system. With this roll up system, we are always aggregating the 1 min tables into larger tables to make the queries more efficient. With each roll up, we include the top 10,000 or 100,000 conversations from the last granular table type (this is configurable). So with 1 min, 5 min, 30 min, 2 hr, 12 hr, 1 day and 1 week tables, each roll up that occurs drops a bit more of the original conversations.
So when you try to view larger time frames, our NetFlow Collector chooses which tables would be the most efficient to query. For example, when you view a 24 hour time frame, you are looking at the results queried from the 30 min tables. Since the 30 minute tables are aggregated, you will see variance as compared to a SNMP based tool.
However! That’s why we introduce the drag and drill option on our graphs. If you look at a 24 hour graph, you are looking at aggregated 30 min tables (sample below). If you see a spike that interests you, click and drill in on the time frame. When you select that block, Scrutinizer will query the smaller 1 min or 5 min tables to give you more accurate data for that time frame.
The data is there, we just don’t want you sitting around for 20 minutes while we run ridiculously long queries. So think big and then work your way into the small.
I hope that makes sense. Let me know if you have questions.