Blog :: Flexible NetFlow :: Network Operations :: Security Operations

Nexus 7000 and Missed Flow Sequence Numbers

I would like to talk about the Nexus 7000 and missed flow sequence numbers. Advanced NetFlow collectors keep track of the flow sequence numbers to monitor NetFlow packets that are being dropped by the network, by a NetFlow exporter or by the collector itself. Is this feature important? Yes, to ensure accuracy in your NetFlow Reporting. Why should a NetFlow monitoring application have this capability? The answer is, RELIABILITY and ACCOUNTABILITY.

I recently worked with a customer who was seeing an overstated amount of missed flows associated with a Nexus 7000 in his NetFlow traffic analyzer. He was able to escalate the issue to Cisco and fixing the problem turned out to be quite simple. I would like to talk about the problem and what the customer did to resolve it. First, let’s briefly go over what a sequence number is.

What is a flow sequence number?

It is an information element that is exported in NetFlow or IPFIX packets.  The RFC 3954 defines it as an “Incremental sequence counter of all Export Packets sent from the current Observation Domain by the Exporter. This value MUST be cumulative, and SHOULD be used by the Collector to identify whether any Export Packets have been missed.

Problem

The customer I was working with was getting a warning in his NetFlow collector Indicating that over 60,000 flows were being dropped per second, which ultimately would result in a misrepresentation of his traffic statistics.

Using wireshark we found that the Nexus was reporting flow sequence numbers incorrectly. The sequence numbers associated with one particular Source ID had significantly large gaps. In two subsequent NetFlow packets we found 2 flows, respectively, where the difference between sequence numbers is 5301329. Meaning that 5301329 flows were possibly missed in virtually zero seconds.

The customer contacted Cisco and was told to disable and re-enable the NetFlow feature and that fixed the problem. In case this issue is not unique to Flexible NetFlow on a Nexus 7000, I encourage you to share your experience if you have seen this on other types of NetFlow exporters.

Besides a malfunction in how flows are being collected, as previously mentioned, a collector can also drop packets. Often because it is overwhelmed by the flow volume, or the hardware it is running on does not have sufficient resources. It could also be that the collector is simply poorly designed. Our high speed collector can handle up to 100, 000 flows per second.

Give us a call if you are looking for a reliable and accountable NetFlow monitoring tool which has the ability to detect dropped flows.