Most people collecting NetFlow use it in a very traditional fashion (i.e. NetFlow v5 with ingress flows). Ingress flow means that only inbound (i.e. received) traffic is collected and exported in NetFlow datagrams. This may sound like you won’t know what is going ‘out’ an interface, but have no fear. There is any easy way to calculate outbound traffic using ingress NetFlow.

Above, out bound utilization on interface 1 is determined by looking at the flows from interfaces 2,3 & 4 that are destined for interface 1. Since an ingress flow contains the source and destination interface (i.e. port of the router). Out bound traffic is determined by using ingress flows from the other interfaces. For this reason, it is important to enable NetFlow on all interfaces of the switch or router. This trick is common practice in all NetFlow reporting tools. But, what about NetFlow v9 and its support for ‘Egress’ NetFlow (i.e. traffic going out an interface)?

NetFlow v9 supports ingress and egress
In most installations, ingress flows enabled on all the interfaces of the switch or router will deliver on the information most of us need. Apparently there is some confusion on when and how to use Egress Flows. Here are several reasons to use Egress flows, here are a few:

• In WAN compression environments (e.g. Cisco WAAS, Riverbed, etc.), we need to see traffic after it was compressed. Using Ingress flows causes an over stated outbound utilization on the WAN interface. Egress flows are calculated after compression.

• In multicast environments, ingress multicast flows have a destination interface of 0 because the router doesn’t know what interface they will go out until after it processes the datagrams. Exporting egress flows delivers the destination interface and as a result multiple flows are exported if the flow is headed for multiple interfaces.
• When exporting NetFlow on only one interface of the router or switch. Enabling both on a single interface means that all traffic in and out is exported in NetFlow datagrams.

The Problem with Exported Egress and Ingress Flows
Lets say ingress NetFlow v9 is configured on interface 1 and egress NetFlow v9 is configured on interface 2. When a flow is exported on interface 1, the same flow is also exported on interface 2. And guess what, the two flows look exactly the same except for one field ‘direction’. Below is a screen capture from Wireshark:

Beware of Direction with Flexible NetFlow
The above is fine and good with traditional NetFlow v9 but, if you are not careful when setting up Flexible NetFlow with egress, there is no way to tell which flow was collected as it was received (i.e. ingress) Vs. as it was transmitted (i.e. egress). How come this happens? Because by default, Flexible NetFlow does not export the flow direction. Hence, if you forget to tell the router to export the direction, the two flows are received by the NetFlow collector and the reporting tool can’t figure out if the flow was collected ingress or egress. This is a huge problem because the same flow exists twice in the database as ingress (by default as this is how NetFlow v5 saved it) and as a result, utilization will be overstated both inbound on interface 1 and outbound on interface 2. How do we solve this issue?

Flexible NetFlow to the Rescue
Here is what we had to do. It is a bit of a pain because you cannot modify a flow record while it is in use. With Flexible NetFlow (FnF), the direction is not automatically exported as part of the datagram as it is in traditional NetFlow v9. The device needs to be told to do it. This is pretty much the case with anything in FnF.

The steps below are based on the configuration of FnF NBAR in the post on How-to configure Cisco Flexible NetFlow for NBAR exports. We didn’t specify the direction in this prior blog so, I thought we might as build on it in case someone wants to export ingress and egress FnF at the same time..

Steps:

1. Since Cisco IOS does not allow you to update and active flow record, first you have to remove the flow monitor from each interface it is on:

interface FastEthernet0/0
no ip flow monitor andrew-mon input

‘andrew-mon’ is the monitor that we need to update so we must remove it from each interface.
We must remove it first because we need to edit the record and it can’t be active.

!
!
interface FastEthernet0/1
 no ip flow monitor andrew-mon input

Remove it from here as well.

!
!
interface Serial0/0/0
 no ip flow monitor andrew-mon input

‘input’ means ingress.

no ip flow monitor andrew-mon output

‘output’ means egress.

flow monitor andrew-mon

Go back into the monitor ‘andrew-mon’ so that we can remove the record.

no record nbar-mon

Remove the record.

2. Then I had to add the new column to the flow record.

flow record nbar-mon

Go into the record ‘nbar-mon’ so that we can update it with the direction column that we need.

match flow direction

Specify to export the direction!

3. Then I had to add the record back to the monitor.

flow monitor andrew-mon

Back into ‘andrew-mon’ monitor so that we can apply the record.

record nbar-mon

Specify that ‘andrew-mon’ should be using the ‘nbar-mon’ record.

4. Then I added the monitor back to the interfaces the way they were before.

interface FastEthernet0/0
ip flow monitor andrew-mon input

Reapply the monitor to each interface.

!
!
interface FastEthernet0/1

 ip flow monitor andrew -mon input

!
!
interface Serial0/0/0
 ip flow monitor andrew -mon input
 ip flow monitor andrew -mon output

This might seem like overkill to change one thing, however, there is a change to the template, so another whole template ID must be created so the collector is aware of the change.

As soon as I made this change, I started seeing egress flows in Wireshark.

Omitting the Direction is not a bug
The fact that the direction is not on by default in Flexible NetFlow is not a bug. This is probably more of a philosophical issue as none of the NetFlow v9/IPFIX fields are mandatory in every template definition. The flow direction information element has been specified since v9. Whether any implementation actually exports it or not is a preference.

NetFlow v9
                                            Flow direction:
    DIRECTION                    61   1     0 - ingress flow
                                            1 - egress flow

IPFIX (IP Flow Information Export)
5.11.6.  flowDirection

    Description:
       The direction of the Flow observed at the Observation Point.
       There are only two values defined.

       0x00: ingress flow
       0x01: egress flow

    Abstract Data Type: unsigned8
    Data Type Semantics: identifier
    ElementId: 61
    Status: current

As a practical matter, however, if flowDirection isn’t specified it has to be either ingress or egress. Since v5 was all ingress that is what most of us assume.

Summary
Best at NetFlow reporting solutions need to assume that he router could be misconfigured for ingress and or egress and make adjustments in its reporting interface so that the user doesn’t become confused.

Jon Mills
Marketing & Public Relations Manager
Follow Me On Twitter Tags: Best at NetFlow, Cisco WAAS, Collecting netflow, exporting NetFlow, Flexible NetFlow, FnF, ingress or egress, NetFlow Collector, NetFlow reporting, NetFlow v5, NetFlow v9, Riverbed, WAN compression

This entry was posted on Monday, January 25th, 2010 at 9:23 pm and is filed under NetFlow, NetFlow Analyzer. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.