Is your company looking to export URL information using IPFIX or NetFlow?  First of all, do not use NetFlow for this type of export as there is no standard element for URL in NetFlow.  You need to use IPFIX as Citrix, nBox and SonicWALL have for exporting URL details.  IPFIX is the proposed standard for ‘Flow’ information.

Exporting URL details via IPFIX is something that our company has dealt with on three occasions. I’d like to share with you the pros and cons to a couple of different approaches when exporting this exciting information.

First Approach:

The first approach is used by the nBox which exports URL details by adding columns to the existing template. HTTP_RET_CODE (RET = Return) and the corresponding HTTP_URL as shown below
< CLICK ON IMAGE >:

Exporting URL using NetFlow

The above format works very well however, adding all of these new columns to the existing template can make the template larger and results in wasted disk space because many flows do not contain URL information.  If you also decide to export VoIP details on jitter, packet loss, caller id, codec, etc., using the same template, this will consume additional storage resources and often result in many flows not populating these columns. The nBox addressed this issue by allowing the administrator to export different templates per type of information desired.  For example, the nBox can be configured to export the following templates:

  • All TCP Traffic
  • All UDP Traffic
  • All ICMP Traffic

Once the IPFIX collector has data for the above, the reporting front end can prompt you for the template you want to use and then provide a list of corresponding reports that will work with the selected template.  At plixer we call this Intelligent Template Recognition(TM) or ITR. Notice in the 3 examples below that the Cisco router is exporting multiple templates.  When you select a specific template, the NetFlow Reporting tool only displays the reports that will work with that template:

Example 1:

Reports specific to NetFlow NBAR

Example 2:

Reports specific to TCP Flows

Example 3:

Reports specific to RTP Flows

Second Approach:


The second approach is used by SonicWALL which exports a single ‘traditional’ flow template with IDs that link to ‘meta’ information located in other templates. This is also a great strategy because you end up with less wasted space however, it to has its problems. Below is a screen capture of the URL meta information exported by the SonicWALL in a template that is separate from the traditional flows.  The url_time_id is a 32 bit integer that can repeat in the same second so we also use the url_flow_id which links to the corresponding time_stamp and flowId columns in the traditional flows template.  We named them differently because SonicWALL used different element IDs.  It sounds confusing but, it isn’t really.

SonicWALL IPFIX URL Report

The above approach produces much less flows than the nBox for the same traffic however, there are down sides:

  • There is no HTTP Return Code as provided by the nBox.  Many customers don’t care.
  • There is no byte count for each URL so the byte count from the flow is used which often results in inaccurate byte volumes for each URL.  Remember a single flow can involve multiple URLs!

Third Approach:


The Third approach used by Citrix AppFlow (aka IPFIX) is similar to the nBox strategy.  Consistent IDs are used to link between templates (i.e. similar to SonicWALL) which helps a great deal with reporting. I can’t digress much yet on the Citrix technology but, I hope to in the coming months.

Fourth Approach:


The fourth approach might be led by Cisco as they have plans to export URLs as well. What will be their strategy? Hmmm, I hope they read this.

More Collaboration Needed


The IPFIX industry needs more collaboration. There needs to be a standard way of joining data between templates.  I hope to see some of you at the IETF meeting in Quebec, Canada on July 24th.  Hopefully issues related to this will come up.  Until then, you can contact me for IPFIX consultation.

Mike Patterson author pic

Michael

Michael is one of the Co-founders and the former product manager for Scrutinizer. He enjoys many outdoor winter sports and often takes videos when he is snowmobiling, ice fishing or sledding with his kids. Cold weather and lots of snow make the best winters as far as he is concerned. Prior to starting Somix and Plixer, Mike worked in technical support at Cabletron Systems, acquired his Novell CNE and then moved to the training department for a few years. While in training he finished his Masters in Computer Information Systems from Southern New Hampshire University and then left technical training to pursue a new skill set in Professional Services. In 1998 he left the 'Tron' to start Somix which later became Plixer.

Related

Big Data

Sankey Flow Graph

One of the greatest benefits of NetFlow collection for traffic analysis, is we’re provided with the ability to visualize the…

Leave a Reply