NetFlow Generators: Enabling NetFlow Without NetFlow Support (Part #1)

Introducing NetFlow and IPFIX

This article covers the benefits and capabilities provided by a new class of network monitoring technology called a NetFlow generator. But before we get too far into NetFlow generation details, let’s do a quick review of NetFlow itself for those that are new to the topic.

NetFlow and IPFIX are network monitoring technologies providing deep visibility into network traffic. NetFlow was originally developed by Cisco and later standardized into IPFIX by RFC 5101. Traditionally, NetFlow was included as a feature of routers, switches, firewalls, and other network devices. It’s even found in virtualization platforms such as VMWare’s vSphere 5.0 and above. Any device that can generate NetFlow packets is called an exporter. As packets travel through the exporter the device records information about the flow of traffic. Data elements such as packet count, source and destination IP, MAC address, and much more are stored in a memory resident data structure within the exporter called a cache. As the flows time out they are placed into a UDP datagram and sent across the network to a NetFlow Collector. The diagram below illustrates the process.

Once enabled NetFlow is used for a variety of network operations and security tasks including:

• Network Performance Monitoring
• Network Security (DoS, APT, Data Loss, Policy Violation Detection, ec.)
• VoIP Monitoring
• Compliance (PCI, HIPPA, FERC)
• Bandwidth Bill-back and Accounting
• Capacity Planning
• Network Forensics and Incident Response

No NetFlow Support? No Problem. Introducing the NetFlow Generator

While NetFlow is available in almost any piece of network equipment you come across, there are situations when you just can’t enable it for one reason or another. You might have older equipment that doesn’t support NetFlow or perhaps you’re worried about the impact NetFlow export will have on CPU or memory. Regardless of the reason, there is an alternative to traditional network device-based NetFlow. It’s called a NetFlow generator.

A NetFlow generator is a network appliance or software daemon that passively captures raw network packets and translates them into NetFlow records. The setup looks like this:

NetFlow generators are deployed exactly like you would a traditional passive IDS or packet sniffer. They use a promiscuous “capture port” that connects to a network SPAN port or TAP. Most NetFlow generators run Linux and are based on standard off-the-shelf hardware. However some NetFlow generators such as the Endace NetFlow Generator. Experience shows that software-based NetFlow generators typically have more advanced features and are easier to extend in the future. Unless you have extreme speeds 10Gbps+ the software-based NetFlow generators will suffice.

NetFlow generators benefit directly from Moore’s Law and offer vast memory and high-speed CPUs that allow for extremely large cache sizes and advanced capabilities not available in traditional exporters.

NetFlow Generator Deployment Options and Scenarios

> Attached to devices that don’t support NetFlow export

The most common deployment of a NetFlow generator is in areas of the network where the device doesn’t support flow export. Many layer-2 switches lack NetFlow support. In some cases you have an older device that is working just fine but doesn’t support NetFlow. The only way to get NetFlow support would be to refresh the device to a newer model. It’s often much more cost-effective to deploy a NetFlow generator instead. And while some firewalls do support NetFlow, many still do not.

> Augment devices that only offer sampling (sFlow / sampled NetFlow)

There has been much debate recently about sampled vs. non-sampled flows. Security analysts and most network engineers agree that sampled flows lack the accuracy needed for proper analysis and detail. In situations where you only have sampled flows (such as sFlow-enabled devices) the NetFlow generator is brought in to generate non-sampled NetFlow.

This article on NetFlow vs. sFlow covers the controversy surrounding sampled flows in greater depth.

> Upgrade critical locations where you need deep packet inspection

Even if you’re already collecting NetFlow from a core switch or router, it’s often wise to deploy a NetFlow generator for greater detail into packet contents. Most NetFlow generators will allow the user run a packet capture to view raw packet details. Most will even allow saving the packet capture in pcap format for import into a packet analyzer such as Ethereal.

Many NetFlow generators support advanced features such as latency calculations, application awareness and HTTP URL export. While some traditional exporters such as Cisco’s ISR G2 support some of these advanced capabilities, few others do.

> To offload NetFlow processing from the router

Some older network devices such as Cisco’s Catalyst 6500 w/Sup2 or Sup1A can take a significant hit when NetFlow processing is enabled. Sometimes the network administrator is simply afraid to enable NetFlow for fear of added load. In either case, a NetFlow generator can be the answer. Instead of enabling the built-in NetFlow export features in the router, you simply set up a mirror or SPAN to the generator’s promiscuous port.

> Help overcome political and service provider problems

Occasionally political and interpersonal issues within the organization can inhibit NetFlow rollout:

• Especially at remote offices, the ISP will often control the CPE (“customer premise equipment”). Many service providers won’t allow enablement of NetFlow export citing potential issues with SLAs. Others will charge a one-time fee or monthly service charge. A NetFlow generator can be used to overcome these issues, bypassing the ISP altogether.
• Another political issue that comes up is departmental. Security staff are increasingly using NetFlow for network security and incident response. If a rift exists between the network and security departments the network team might not be willing to send NetFlow to the security team’s NetFlow collector. Since the NetFlow generator only requires a SPAN port (which the security team usually has access to) it’s often a great way for the security folks to go around the network team.

> Server and workstation NetFlow support

One application of NetFlow generation that doesn’t get as much press as it should is server and host-level NetFlow export. Using a software NetFlow generator such as nProbe, a sysadmin can turn his or her server into a NetFlow-capable exporter. nProbe installs directly into the server and captures packets as they enter and leave the server itself. This is especially powerful with cloud-based server instances where NetFlow is typically non-existant and visibility is limited.

—

In part #2 of this blog on NetFlow generators we’ll cover the various commercial and open-source NetFlow generators available. We’ll also discuss the other end of NetFlow export: the NetFlow collector. Continue to Part #2 >>