Determining NetFlow Direction is the process of trying to ascertain who initiated a flow. Although “NetFlow Direction” is the title of this post, the word ‘Direction’ in NetFlow means something entirely different. According to the IETF standard for flow technologies called IPFIX, direction has to do with the observation point of a flow. (i.e. ingress or egress). Put in simpler terms: was the flow collected as it came in an interface or as it went out an interface? This distinction will become more important as I progress through this post.
In some cases, knowing the initiator of a flow can help us figure out which host was the client versus server. It can also presumably be used to assume which host (i.e. the source) is potentially infected with malware but, maybe not. Let’s explore how flow direction or Flow Initiator is decided.
NetFlow Deduplication
Some vendors believe that one way to determine NetFlow Direction is to stitch the flows from A to B and for B back to A into a single bidirectional flow. Flow stitching can require the process of flow deduplication. NetFlow Deduplication is performed when multiple routers/switches export the same flows for two hosts communicating with one another. If a flow (e.g. A to B) with a matching tuple is collected by 3 routers, the collector will save only one instance of the flow with a few details.
During flow deduplication, some of details are either dropped or assumed (e.g. DSCP values, packet and byte counts, next hop, etc.). Because the process of deduplication results in lost details, the original flows are often saved as well. Hence, the belief that flow deduplication saves disk space is false unless of course you are willing to work without much of the juicy details that make NetFlow and IPFIX awesome. In truth, NetFlow deduplication often leads to greater disk space consumption.
NetFlow Stitching
NetFlow Stitching is the process of looking at certain protocols (e.g. TCP) and assuming that a return flow occurred. If the return flow is found (E.g. B back to A), the details are merged into the A to B flow making it a single bidirectional flow. I said “If the return flow is found” because the return path could be going through another router which is exporting flows to a different flow collector. In most multi-collector environments (except Scrutinizer), flows aren’t stitched across collectors.
After the flows are deduplicated and stitched within a single collector, logic is applied to determine the NetFlow Initiator. Often, the largest deciding factors on which host initiated a connection is based on the host transmitting on the highest source port. However, this strategy is not always reliable and if we are using flow data forensically, we need to be able to rely on the results else, false positives ensue.
Strategies used that attempt to determine flow direction but, aren’t entirely reliable include:
- Comparison of flow start times
- Lowest source port = server
- The relationship of A and B with other hosts on the network. A host with many connections to unique hosts ‘could’ be a server.
- Byte volumes: the host with the lower sent octet delta count is the client. This can be very misleading.
- Observation of TCP flags however, due to flow technologies aggregation methods, this strategy generally fails.
- Examining TTL metrics (if exported) – doesn’t provide a definitive answer.
- Consideration of how the flows were metered (I.e. ingress or egress) however, this is unreliable as some vendors only support egress. In other cases both are exported. Frankly, I never really understood why this was a good strategy. Perhaps someone could share their insight on this.
Generally, a combination of the above strategies are employed to determine the correct flow initiator. Although a high degree of success can be had, no solution is 100% reliable when trying to determine NetFlow Initiator. If a flow device exports both directions of a flow, in most cases the oldest time stamp can be used to determine the initiator. If the return path comes from a different direction (I.e. asymmetrical) this approach is also unreliable however, on most networks, at least one device in the path will export both flows.
NetFlow Direction or NetFlow Initiator Goal
What is the underlying goal of identifying the Flow Initiator? Answer: It’s to better understand which host is ‘probably’ the server. If on the other hand the goal is related to incident response or chasing malware, the value of knowing flow initiator is largely unimportant or moot.
If the connection was made via TCP, the routers will identify the client / servers machines for us but, more and more malware is using UDP and because of this the client / server relationship has become irrelevant! Check out how bad actors are encoding thefts inside DNS NXDOMAIN and TXT messages.
Don’t get me wrong, determining the NetFlow Initiator provides additional insight when TCP is the transport however, I don’t believe we should rely on it. Some would contend that simply looking at the host IP addresses or host names can often help determine which host is the server. In some cases, I would agree with this but, it doesn’t scale. If we restrict this discussion to TCP and the use of NetFlow or IPFIX, the only reliable approach to determining the client and server relationship is for the hardware to tell you.
Good news! Cisco AVC IPFIX exports provide these exact details. See below:
Who Commits the SYN?
How does Cisco determine the NetFlow direction? It’s simple. Because the router sees the actual packets between the end systems, it can observe the initial SYN packet. The client will be initiating the SYN packet.
Question: How does the NetFlow collector know which host is the client or server?
Answer: RFC 5103
RFC 5103 and standard IANA elements can be used to send flow records in the standard format as biflows. In a biflow, both directions of a flow are collapsed into a single flow record. In other words, the hardware stitches the flows. What a novel idea! Inside the flow, element ID 239 is used to identify the client/server relationship. To help explain this, lets simplify NetFlow/IPFIX for a moment down to a simple tuple just a few elements:
- source IP(element 0_8)
- destination IP(element 0_12)
- biflowDirection(element 0_239)
- bytes (element 0_1)
- bytes reverse (element 29305_1)
A very simple template example would be:
0_8 | 0_12 | 0_239 | 0_1| 29305_1
10.1.1.1 | 10.2.2.2 | 0x01 | 32 | 64
10.2.2.2 | 10.1.1.1 | 0x02 | 64 | 32
NOTE: Elements (e.g. 0_8) are PEN_IANA-ID except for the 29305 which is the reverse of any standard element per RFC 5103.
octetDeltaCount = 0_1, octetDeltaCount_reverse = 29305_1
packetDeltaCount = 0_2, packetDeltaCount_reverse = 29305_2
In the first row of the table above, 10.1.1.1 is the initiator (or client) because the biflowDirection is set to 0x01. The client -> server bytes are 32 and the server -> client bytes are 64. The second row displays exactly the same data because the biflowDirection was changed to 0x02 and the addresses and byte values were flopped.
The peroration of my point is that only the hardware can accurately identify the initiator of a TCP flow. A flow reporting solution which purports to accomplish this without with above data is at best: fallible.
If you need a IPFIX or NetFlow solution that accurately displays the semantics in the client/server relationship, our scalable NetFlow collector can provide you with an award winning solution. Download a copy of our white paper The Future of Advanced NetFlow.