How to avoid IPFIX or NetFlow Sampling Vs. sFlow

Posted in application performance monitoring, Netflow Billing, netflow sampling, sFlow on November 16th, 2012 by Steve
How to avoid IPFIX or NetFlow Sampling Vs. sFlow

Sampling traffic in an effort to represent the overall traffic pattern in theory is a sound idea.  In fact, for the most part, I agree that statistically, sampling can be as accurate as rolling a dice 1296 times and expecting exactly 216 matches for each of the 6 possible outcomes.  Well, I doubt it.   When counting massive quantities of data and accuracy matters, we need an alternate solution to sampling but, what is it?

A word on sFlow Vs. NetFlow

First lets digress a bit on sFlow.  Its misleading name craftily implies that it is somehow similar to NetFlow when in fact, sFlow is not a flow technology at all.  It is a packet sampling technology and a counter export utility which compares more to SNMP.  It is intended to aid in network monitoring routines however most admins use it for packet sampling.  NetFlow and IPFIX are the only true flow technologies. AppFlow, CascadeFlow, J-Flow, NetStream, etc. are all rebrands of NetFlow or IPFIX.  The popularity of true flow technologies is growing for several reasons:

  • NetFlow and IPFIX metering is being done in hardware (e.g. Enterasys and Cisco) or software
  • They both support packet and flow sampling : Cisco, Juniper, Alcatel/Lucent, Nortel, etc. have implemented them for sampling.
  • Application awareness through DPI -> packet samples can’t do this
  • Application Performance monitoring with metrics on jitter, latency, packet loss, etc.
  • The ability to export machine messages like syslogs
  • Threat detection by analyzing flow behaviors

Unfortunately, when people try to use sFlow to accurately represent the volume of traffic from a particular IP address they are met with a bit of frustration because sFlow usually understates what they are looking for.  Another misleading point that the advocates of sFlow try to make is that sFlow scales better than NetFlow and IPFIX and true to the name ‘sFlow’ this is also misleading.   There is a dated but, still accurate post on Networkworld.com titled “Closer look: sFlow better than NetFlow?” that is worth reading.

With the above being said and if you understand its limitations, sFlow is a great option when NetFlow and IPFIX are not available.  I like to compare NetFlow Vs. sFlow to eating ice cream.  My favorite ice cream is strawberry (NetFlow) but, when the only other flavor available is pistachio (sFlow), I cringe and eat it because I like ice cream.

Vendors continue to migrate to NetFlow and IPFIX

Today, NetFlow and its proposed standard IPFIX include all of the functionality of sFlow and then some.  Yes, even packet sampling has been done with NetFlow and IPFIX.  When talking with customers, most just don’t like the idea of sampling but, agree that at some point it seems inevitable.  Good News:  in this post I will demonstrate a flow export trick that will allow administrators to gain 100% accuracy by compromising on only a couple of elements (E.g. source and destination ports) that often go unused in flow reporting especially with the introduction of technologies like NBAR.

The ability to capture 100% of the packets for traffic analysis is certainly going to give more insight however, it just doesn’t scale and although sampling has its benefits, in many cases it misses the data that customers want to see.  By defining a Flexible NetFlow tuple that matches on fewer fields, customers can often avoid sampling.” Thomas Pore, Dir. Of Field Engineering – Plixer

NetFlow Matching Fields

The ‘flexible’ aggregation capabilities of true flow technologies allow the user to specify the key fields they want to receive. In lieu of sampling packets, hardware can create flows based on specified key fields.  Below is what many consider a traditional flow matching tuple:

  • Source interface
  • Source port
  • Source IP address
  • Destination port
  • Destination IP address
  • Protocol
  • ToS / DSCP

Below is an example of the amount of flows exported using all of the above match fields.

Collecting NetFlow Match Fields

Notice above that the pagination at the bottom of the screen shot says 1422.  If each page has 10 flows, that is 14,220 entries. How can we reduce the flows exported and still represent 100% of the data?

With Flow technology we can reduce the volume of flows exported if we simplify the matching key fields to:

  • Source interface
  • Source IP address
  • Destination IP address
  • Protocol

Below is an example of the amount of flows exported for exactly the same traffic with the above smaller tuple:

Flexible NetFlow Matching

Notice above that the pagination at the bottom dropped from 1422 to 81 which is nearly a 95% reduction in flow volume while still representing 100% of the data.  If you want to reduce it more for NetFlow billing purposes, we can reduce the key fields to:

  • Source Address
  • ToS / DSCP

Gartner Group on NetFlow

With the above flexibility of specifying key matching fields, we can avoid sampling data almost entirely. Hardware vendors recognize this and they are investing in NetFlow and IPFIX.  True flow abstraction is the technology customers want.  Gartner recently stated that flow analysis should be done 80% of  the time and that packet capture with probes should be done 20% of the time. Source  The experts agree; NetFlow and IPFIX make the best ice cream.

Steve

For a free 30 day trial of Scrutinizer, Download Now!

Sign up for Advanced NetFlow Training™ coming to a city near you!

If you enjoyed this post, please consider leaving a comment or subscribing to the RSS feed to have future articles delivered to your feed reader.
Tags: , ,

5 Responses to “How to avoid IPFIX or NetFlow Sampling Vs. sFlow”

  1. HP 9300 NetFlow Configuration - NetFlow & sFlow Network Monitoring - NetFlowKnights.com Says:

    [...] nexthop, and subnet mask.  By removing source and destination port from the aggregation, the volume of flows can often be reduced by well over [...]

  2. Phil Aymer Says:

    When dealing with a lot of traffic, does it also make sense to not only do flow sampling at the router level (1 of N packets considered for the flow table) but also flow sampling at the collector level (1 of N flows considered by the collector) ?

  3. mike@plixer.com Says:

    Interesting question Phil.

  4. Considering whether to implement sFlow Vs IPFIX? This post will help. - NetFlow & sFlow Network Monitoring - NetFlowKnights.com Says:

    [...] of my coworkers posted an excellent article on how to avoid NetFlow sampling.  In short, all you have to do is modify the matching tuple to dramatically reduce the amount of [...]

  5. Juniper SRX100H NetFlow Support - NetFlow & sFlow Network Monitoring - NetFlowKnights.com Says:

    […] every packet, providing the most detailed reporting and flow analytics possible.  Our blog on packet sampling will give you more insight into pros and cons of flow sampling […]

Leave a Reply