The other day I was helping an engineer who was trying to provide some Juniper MX240 IPFIX support when we ran into some “flow direction” problems that lead to reporting issues. After setting up the export and not seeing anything in our NetFlow Collector, I decided to get out the packet analyzer and capture the datagrams. It was time to take a look at the contents within the flows at deeper level. What we learned: Juniper is not exporting flow direction correctly and this is causing problems.
The root of the issue is that Juniper is sending the “flowDirection” field in the IPFIX export (See page 61 in http://www.iana.org/assignments/ipfix/ipfix.xhtml). According to the standard in RFC5102, flowdirection must be “0” or “1” to denote how the flow is metered on the interface and Juniper is sending “255” as the value which is not valid. It appears that the MX240 is not aware of flow direction. Our NetFlow collector is actually collecting and storing this data, however, it doesn’t know how to display it due to the invalid direction. Displaying the data would lead to incorrect reports because we can’t properly deduplicate flows that are metered differently (I.e ingress and egress). I wrote an in depth article on this titled ipfix flow direction and packet counters some time ago. The engineers at Juniper would benefit from reading it.
When Juniper was approached about the ‘255’ value issue, they told us that:
- They export 255 to avoid reporting the wrong flow direction when a packet is sampled by both ingress and egress PFE
- This is a poor decision because when both ingress and egress flows are metered, two flows should be recorded and exported with different direction values (i.e. 0 and 1) else utilization will be overstated.
- When a packet is sampled on both the ingress and egress side by the same PFE and flow direction is not part of flow key
- This is another problem that I can digress on in another post.
When flow monitoring is implemented on an interface, input and output define observation points for packets traversing the interfaces. When the flow is eventually exported, the router includes the source and destination interface numbers. Here’s why this is helpful: If a flow is monitored ingress as it came in an interface, the flow reporter will infer the output bandwidth on another interface by observing the destination interface of flows. In other words, by looking at the destination interface of ingress metered flows, we can use these same flows to assume how much data went out a selected interface. The trick however, is that we need to enable ingress metering of flows on all active interfaces else, we could under state outbound utilization on an interface.
Now sometimes it is wise to export egress flows. This means the flow was metered as it left the destination interface. I covered why this is done in my post Best Practices in Egress NetFlow Reporting. READ THAT ARTICLE if you want to know why egress is sometimes necessary. The point I want to make here is that if metering both ingress and egress on the same flows, two flow records should be created. One with an observation point of 0 (ingress) and another one with an observation point of 1 (egress). Juniper is smashing them together and giving them an observation point of 255. This is not supported by the standard and demonstrates Junipers serious lack in the understanding of NetFlow and the IPFIX standard. If an exporter is not capable of determining a flow direction, it would be best to simply remove the field from the export. It is better to send no data than it is to send invalid data that will cause problems for a compliant collector that expects this field to tell it something about the flow.
If our reporting system sees both ingress and egress metered flows, it uses ingress metered flows to report on inbound traffic and egress metered flows to report on outbound traffic. If egress metered flows are absent and usually they are, we will report outbound traffic using ingress metered flows as explained above. This is how vendors should be doing it. If however the flow reporting vendor ignores flow direction and both ingress and egress metered flows are being exported on all interfaces, they will overstate both inbound and outbound utilization values because the packet and byte counters are repeated.
BTW: if you try to avoid the problem by exporting just ingress flows from the MX240, the flow direction will still be 255 and break the reporting unless the flow reporting solution ignores the flow direction element. We have a work around but, ideally Juniper really should support the standard.
