Is your company looking to export URL information using IPFIX or NetFlow?  First of all, do not use NetFlow for this type of export as there is no standard element for URL in NetFlow.  You need to use IPFIX as Citrix, nBox and SonicWALL have for exporting URL details.  IPFIX is the proposed standard for ‘Flow’ information.

Exporting URL details via IPFIX is something that our company has dealt with on three occasions. I’d like to share with you the pros and cons to a couple of different approaches when exporting this exciting information.

First Approach:

The first approach is used by the nBox which exports URL details by adding columns to the existing template. HTTP_RET_CODE (RET = Return) and the corresponding HTTP_URL as shown below
< CLICK ON IMAGE >:

The above format works very well however, adding all of these new columns to the existing template can make the template larger and results in wasted disk space because many flows do not contain URL information.  If you also decide to export VoIP details on jitter, packet loss, caller id, codec, etc., using the same template, this will consume additional storage resources and often result in many flows not populating these columns. The nBox addressed this issue by allowing the administrator to export different templates per type of information desired.  For example, the nBox can be configured to export the following templates:

• All TCP Traffic
• All UDP Traffic
• All ICMP Traffic

Once the IPFIX collector has data for the above, the reporting front end can prompt you for the template you want to use and then provide a list of corresponding reports that will work with the selected template.  At plixer we call this Intelligent Template Recognition(TM) or ITR. Notice in the 3 examples below that the Cisco router is exporting multiple templates.  When you select a specific template, the NetFlow Reporting tool only displays the reports that will work with that template:

Example 1:

Example 2:

Example 3:

Second Approach:

The second approach is used by SonicWALL which exports a single ‘traditional’ flow template with IDs that link to ‘meta’ information located in other templates. This is also a great strategy because you end up with less wasted space however, it to has its problems. Below is a screen capture of the URL meta information exported by the SonicWALL in a template that is separate from the traditional flows.  The url_time_id is a 32 bit integer that can repeat in the same second so we also use the url_flow_id which links to the corresponding time_stamp and flowId columns in the traditional flows template.  We named them differently because SonicWALL used different element IDs.  It sounds confusing but, it isn’t really.

The above approach produces much less flows than the nBox for the same traffic however, there are down sides:

• There is no HTTP Return Code as provided by the nBox.  Many customers don’t care.
• There is no byte count for each URL so the byte count from the flow is used which often results in inaccurate byte volumes for each URL.  Remember a single flow can involve multiple URLs!

Third Approach:

The Third approach used by Citrix AppFlow (aka IPFIX) is similar to the nBox strategy.  Consistent IDs are used to link between templates (i.e. similar to SonicWALL) which helps a great deal with reporting. I can’t digress much yet on the Citrix technology but, I hope to in the coming months.  

Fourth Approach:

The fourth approach might be led by Cisco as they have plans to export URLs as well. What will be their strategy? Hmmm, I hope they read this.

More Collaboration Needed

The IPFIX industry needs more collaboration. There needs to be a standard way of joining data between templates.  I hope to see some of you at the IETF meeting in Quebec, Canada on July 24th.  Hopefully issues related to this will come up.  Until then, you can contact me for IPFIX consultation.

Michael Patterson
Scrutinizer Product Manager
Click to download Scrutinizer now!
Join NetFlow Developments on Linkedin.com

Tags: citrix appflow, ipfix, IPFIX collector, NetFlow reporting, url details, url in netflow

This entry was posted on Tuesday, May 3rd, 2011 at 11:49 am and is filed under IPFIX, NetFlow. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.