As mentioned in Scott’s blog,  “Getting the most from your NetFlow and sFlow Analysis Tool“, disk fragmentation can be the primary cause for slow performance in running NetFlow reports.

Due to the large volume of data stored when collecting NetFlow packets, disk I/O may already be pushed to the limits on your server.  Add to that a highly fragmented disk drive and you might as well go hang out at the water cooler while you wait for your report to run.

Here’s an example of an extremely fragmented disk:

As Scott mentioned in his blog, “With hard drives, blue is a good thing, red is bad. Ideally we would want to see mostly blue and white.”

But, on the other hand,  if you don’t have anything better to do with your time, if using Scrutinizer has so streamlined your network monitoring that you need to slow your day down a bit, then please, leave your disk fully fragmented and take a break!

Otherwise, if you prefer your Netflow reporting to complete in your lifetime, then defrag!

And in the spare time that you now have to kill, you can monitor excessive Facebook traffic and other odd traffic patterns on your network, or read our blogs to learn how to enable Flexible NetFlow, or give us a call to find out what else our NetFlow solution can do for you.

- Joanne

The Need for NetFlow
“Commercial NetFlow Applications” explains that today’s networks often operate at multigigabit speeds that can often overwhelm traditional packet-based data capture tools and data analysis methods.  As a solution to this modern problem, Cisco created NetFlow.  Originally released in 1996, it has had multiple updates since.  NetFlow v5 is still widely utilized, but v9 is the most recent release.

But what is NetFlow?
According to the book, NetFlow likely already exists in your network infrastructure in supported devices (routers, etc.).  This technology collects and categorizes IP traffic as it passes through the device interfaces.  As these packets arrive, NetFlow scans them to determine the appropriate traffic flow.  Focusing on flows rather than packet captures allows NetFlow to keep up with the increasing speeds of business networks. 

What’s the point?
Identifying suspicious traffic for future investigation is much simpler through analyzing flows than it is through packet capture.  Ultimately, the desired result of flow analytics is to understand and safeguard your network.

How is it generated?
NetFlow can be generated when traffic enters an interface.  For holistic results, NetFlow should be enabled on all interfaces of all supported devices that contain traffic you are interested in analyzing.  This should be done because outbound utilization is calculated by using ingress flows from other interfaces.  Otherwise, traffic coming in from one interface destined for another interface will be missing from NetFlow calculation.

Once NetFlow is enabled through a few straightforward and simple commands, the router will write records for every conversation going through it and will then export them to a NetFlow collector.

It is noted that the emerging standard for NetFlow called Internet Protocol Flow Information eXport (IPFIX) is largely based on NetFlow v9, and this should not be confused with the packet sampling technology called sFlow, although NetFlow can also perform it.

The chapter explains that NetFlow v5 only supports ingress flows, while v9 supports ingress and egress.  Generally speaking, ingress flows enabled on all the interfaces of the switch or router will deliver the information needed for an investigation.  However, there are multiple reasons you may be required to enable egress NetFlow in addition to ingress NetFlow.

More to Come
The information Mike Patterson provided in Digital Forensics for Network, Internet, and Cloud Computing elaborates further on NetFlow v9 benefits and uses.  He discusses Flexible NetFlow, sFlow, and how Scrutinizer handles both.  Check out Digital Forensics for more information.

~Angela
Follow Us on Twitter!
Find Us on Facebook!

Scrutinizer Data Archiving System

The following blogs explain how it works:

1. NetFlow trends seems understated – Why?

2. The Most Granular NetFlow and sFlow Reporting.

Assuming we read the blogs I will now move on to another subtopic.

If you are using our NetFlow Analyzer, you may have seen the page illustrated in the following screen capture.

Seeing this page when trying to generate a report could mean that there is no data that fits the specified time frame. Both of the blogs I recommended reading give a clear description of DB tables that are created as a result of roll ups. There are 1 minute, 5 minute, 30 minute, 1 week and 1 day  tables. Because of this organization, when a time frame is selected, Scrutinizer will choose tables that best fit the select time frame. For example, if you chose to view data from the last hour, it makes sense that Scrutinizer first tries to retrieve data from the 5 minutes and 1 minutes tables because they are smaller intervals.

What if the flow collector server has just started collecting, and the 1 min tables have not been rolled up into 5 min tables yet? In this case, If Scrutinizer first tried to retrieve data from the 5 min tables and did not find any data, it will show you a page such as the above screen capture. However, on this page “1m” will be a link to an alternative report that will be generated from 1 min tables.

Moreover, the appearing of this page could mean that the  data in the Scrutinizer database is missing certain information that is necessary to constructing the requested report ; usually because a device was not configured to send such information. In our traffic analyzer for instance, in order to generate reports such as “Application NBAR” or “Conversation NBAR”, you will need Flexible NetFlow configured for NBAR export on your devices so that NBAR information is sent out as part of the flows.

To be continued in part 2

Such a dilemma, when it comes to Autonomous System NetFlow exports, which do you prefer: peer-as or origin-as?  If you don’t care about Autonomous System reports, you still just might find this post interesting.  I’ll try to keep you captivated!

Autonomous System
First of all, what is an Autonomous System? Within the Internet, an Autonomous System (AS) is a collection of connected Internet Protocol (IP) routing prefixes under the control of one or more network operators that presents a common, clearly defined routing policy to the Internet. A single ISP can support multiple Autonomous Systems Numbers (ASN). The ASNs supported by the ISP are advertised via their Internet router using the BGP Protocol. So what is BGP?

Border Gateway Protocol (BGP)
The primary function of a BGP speaking system (e.g. router) is to exchange network reachability information with other BGP systems. This network reachability information includes information on the list of Autonomous Systems (ASs) that reachability information traverses. Isn’t that a mouth full!

Configuring BGP
To enable BGP routing and establish a BGP routing process, use the following commands beginning in global configuration mode:

If you want adjacent routers to be able to export Autonomous System information as well, you need to tell the router to share the ASNs it knows about with its BGP neighbors.

Configuring BGP Neighbors
Like other EGPs, BGP must completely understand the relationships it has with its neighbors. Therefore, this task is required. BGP supports two kinds of neighbors: internal and external. Internal neighbors are in the same autonomous system; external neighbors are in different autonomous systems. Normally, external neighbors are adjacent to each other and share a subnet, while internal neighbors may be anywhere in the same autonomous system.

To configure BGP neighbors, use the following command in router configuration mode:

Now that we have the routers sharing the ASNs they know about, we have to tell the router to include the ASN information in the flows they are exporting in NetFlow v9 or v5. Preferably, you are using Flexible NetFlow.

Peer Vs. Origin
Now we configure the router to export the AS information in the NetFlow exports. Before we do this, Autonomous System information exported by NetFlow comes in one of two flavors:

• The origin-as keyword specifies that export statistics include the origin autonomous system (AS) for the source and destination. In my opinion, this is basically where it originated before it started hopping through routers.
• The peer-as keyword specifies that export statistics include the peer AS for the source and destination. In my opinion, this is sort of like next hop.

Before we get into the commands that export the data off to the NetFlow collector or NetFlow traffic analyzer, lets review the two bullets above.

Exporting from a Peer or Origin AS
I found the information below in this great Autonomous System document posted on Cisco’s web site.

If your router uses BGP protocol, you can configure AS to be included in exports with command:

router(config)# ip flow-export version 5 [peer-as | origin-as]

The following configuration example shows how to configure export from a peer AS using the Version 5 record format:

Router(config-if)# ip route-cache flow
Router(config)# ip flow-export destination 172.17.246.225 9996
Router(config)# ip flow-export version 5 peer-as
Router(config)# ip flow-export source loopback 0
Router(config)# ip flow-cache timeout

In this example, you configure export from a peer AS using the ip flow-export version 5 peer-as command. The AS source is AS2, and the AS destination is AS4.

You can also configure export from an origin AS using the ip flow-export version 5 origin-as command. The AS source is AS1, and the AS destination is AS5.

Autonomous System Reporting
Once the ASN information is being exported in the flows, the NetFlow Traffic Analyzer will display the information in the Autonomous System Report. See the below example:

Summary
I hope this post has helped someone understand why and how to export ASN information using NetFlow. Perhaps you could leave some constructive criticism or a kind message.

The adventure started as a birthday celebration, also known as “Stevefest 2010″, where we decided we would leave our electronics at home (because there is no service that far up Maine) and take on the great Maine outdoors. Like most of us in the group, it was my first time going whitewater rafting and I didn’t know what to expect.

Our crew met up at the Lake Moxie Camps where we camped out for the night in preparation for the rapids. This is where I heard a rumor from the natives that the secret ingredient in the famous Moxie soda is a nice large scoop of mystery mix from the bottom of Moxie Lake. Mmm, no wonder why it tastes so good.

Once morning hit we all got geared up with our life jackets, helmets, and paddles, then jumped on a bus that took us to the top of the river. It only took a few minutes of rafting, and learning how to paddle, before we were in class 3 and 4 rapids getting drenched and almost thrown overboard.

While no one in our group was thrown off the rafts, there were some very close calls that kept us, literally, on the edge of our seats.

The trip was such a blast that we’re all planning on stepping it up next year and taking on Maine’s premier whitewater rafting on the Penobscot River, which contains numerous class 5 rapids.

I highly suggest, if you’re interested in Maine whitewater rafting, to give Phil from Moxie Rafting a call and experience the trip for yourself. Just make sure to write down the directions because you don’t want to get lost with no cell service and a GPS that keeps cutting out.

Happy rafting!

I had a customer the other day show me how he was using Scrutinizer to catch DNS pirates.

Let’s take a look at how he setup the report filter to do this.

First, filter for the NetFlow capable switch or router and then add a second filter for the DNS request protocol (67 UDP Bootps).

The NetFlow trend will update as follows:

Next, we excluded our local DNS servers (10.1.1.132 and 10.1.4.1) Exclude as SRC or DST. Notice below that they are preceded by red squares instead of the include green square:

The report will update again.  You will notice that the remaining data are DNS requests going to or from IP addresses on the local network that are not your DNS server. We can then set an inbound threshold of 1Kb and get an alarm when the pirating occurs:

Notice above that the size of the packets is very small.  If your NetFlow Analyzer is not saving all the flows then you will probably will never find these hosts.

You can also use the NetFlow Matrix report to detect who the pirate is and who is talking to them:

Cool stuff and only with the Scrutinizer NetFlow and IPFIX collector.

Contact support if you have questions on this – (207)324-8805

-Scott

If you’re not familiar with this NetFlow Forwarder application and you have the need for exporting NetFlow packets to multiple (unlimited!) collectors, then you must read his blog.

With switches or routers that do not support NetFlow export to more than one NetFlow collector, or if you have the need to export to more than the typical two collectors, the samplicator is an ideal solution.

Configuration is quick and easy and, if using the config file to list source (exporters) and destinations (collectors), extremely scalable.

For example, in the configuration displayed below, we have 18 exporters forwarding to 9 different collectors in varying combinations.  Several of the exporters only forward to one collector, whereas the remainder forward to either 7 or 8 collectors.

The flexibility of configuring NetFlow duplication is limitless using the config file.

But reading the list of source ips and destination ips in this config file can be very confusing, and our manager, like so many, prefers to see a graphical display.

Graphical view

So we created a quick graph (using GraphViz) of the exporter and collector ip addresses with arrows of who forwards to who.

The exporters are all displayed around the outer perimeter of the graphs and the collectors are on the inside with the arrows pointing to them.  Gives you a simple display of the complexity that the configuration file can provide.

Using this NetFlow replicator and the config file, you can expand your NetFlow reporting capabilities to multiple NetFlow collectors, including my favorite, Scrutinizer NetFlow and sFlow Analyzer.  And don’t forget, since the samplicator forwards UDP packets, you can also forward sFlow and IPFIX packets, and also SNMP Traps or Syslogs.

- Joanne

Update:

To run the samplicator with a config file, use the following command syntax:

samplicate -p2002 -f -S -c /home/plixer/sample.cfg

See the sample.cfg file for an example.

Here are the steps I used in the study:

• I started WireShark
• I started iaxLite
• I made a call
• The other end picked up
• I hung up
• I closed iaxLite
• I stopped WireShark
• 1 Ingress Flow represents 1364 UDP packets
• 1 Egress Flow represents 1364 UDP packets

Wireshark Packet Trace
First I looked at the traffic from my PC to the PBX using Wireshark.  This very short call created 1364 packets!  I wonder how many flows will be exported? 

Click on the image below to expand.

We then looked at this traffic in our NetFlow, sFlow and IPFIX reporting tool ‘Scrutinizer’.  I setup a filter for 10.1.7.5 to 66.186.184.194, you can see it on the left hand side of the screen capture. Notice that the NetFlow collector only received 1 flow from the Cisco router and the packetDeltaCount is exactly the same as Wireshark 1364.  If the flow had gone through multiple routers and the netflow collection engine had performed deduplication, I wonder if the number would be exact? I doubt it. 

A word on NetFlow Deduplication
We perform deduplication within our Flow Analytics engine.  This allows us to avoid triggering multiple alarms as the exact same packets traversed several netflow exporting routers. We also use deduplication to show the top hosts, conversations, protocols, domains, countries, etc. across multiple routers and switches so there are practical applications for it.  However, it is all limited to our Flow Analytics module.  In the above scenario, the averaging of NetFlow in deduplication would not have been appropriate as sometimes network traffic analysis requires precise data. Also, many service providers feel that NetFlow billing requires accurate data as well.  Don’t get me wrong, averaging is good it just isn’t accurate. Lets go back to wireshark and see what the PBX was transmitting.

After configuring my filter, I noticed that the PBX had sent exactly the same amount of packets back to my pc: 1364.  Interesting. 

Now lets take a look at this using the NetFlow collector. In Scrutinizer I was able to leave the filter on the left and just toggle to Outbound at the top.  Exactly the same number: 1364. 

Summary
In this lab on VoIP – NetFlow analyzer, we saw tremendous aggregation on the part of NetFlow.  I think this final lab enforces what was stated in the summary of lab 1:

• Packet trace analysis : verbose, all the details, less high level information
• NetFlow traffic analysis : aggregated, summary details, more high level information

Although the above could change, only time will tell.  Keep your eye on the horizon by paying attentiog to Cisco Flexible NetFlow (FnF) and the evolving standard IPFIX.

Scott wrote a blog earlier that made a valid point: “A Network Administrator’s abilities are only as good as his awareness of what happens on his network.” In harmony with that statement, it’s beneficial to have useful tools to be able to collect that traffic information.

Recently, I learned that J-Flow is supported for the Juniper SRX series Gateways. I thought this might be good information for people who want to start monitoring flows on this type of device, especially our NetFlow and sFlow Analyzer users, since it can also process J-Flow packets. Below are some sample commands taken from Juniper’s Knowledge Base which walks you through your J-Flow configuration.

1. Enable sampling on desired interface(s) and directions:

set interfaces ge-0/0/0 unit 0 family inet sampling input
set interfaces ge-0/0/0 unit 0 family inet sampling output

2. Specify sampling rate and where to send the J-Flow data:

#Specify sampling rate.  Caution: Activation of Flow collection can have a significant impact on the performance of the SRX device. The smaller the sample rate, the bigger the impact .  A sampling input rate of 1 is not recommended.
set forwarding-options sampling input rate 100

#Specify UDP port number of host collecting cflowd packets
set forwarding-options sampling family inet output flow-server 192.168.1.5 port 9996

#Specify version format: 5, 8 or 500 (ASN 500)
set forwarding-options sampling family inet output flow-server 192.168.1.5 version 5

For a sample configuration, please check our FAQ archive for enabling flow export on a Juniper router.

If you are not familiar with NetFlow BGP Next Hop the Cisco documentation says it best:

The NetFlow Border Gateway Protocol (BGP) Next Hop Support feature lets you measure network traffic on a per BGP next hop basis. Without the NetFlow BGP Next Hop Support feature, NetFlow exports only IP next hop information (which provides only the next router); this feature adds BGP next hop information to the data export.

The NetFlow BGP Next Hop Support feature lets you track which service provider the traffic is going through. This functionality is useful if you have arrangements with several other service providers for fault-protected delivery of traffic. The feature lets you charge customers more per packet when traffic has a more costly destination—you can pass on some of the cost associated with expensive trans-oceanic links or charge more when traffic is sent to another ISP with which you have an expensive charge agreement.

This feature uses only the NetFlow Version 9 export format for its data export.

Commands to enable BGP NetFlow exports:

The command to enable it looks like this:

Router> enable
Password:
Router# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)# ip flow-export Version 9 origin-as bgp-nexthop
Router(config)# end
Router#
3w1d:%SYS-5-CONFIG_I: Configured from console by console
Router# exit

A word on Reporting
Just because you are exporting BGP information to your NetFlow collector doesn’t mean that your NetFlow reporting front end will allow you to report on the data. This of course is the case unless you are using a best at NetFlow Analyzer.

If you can help us out by sending over a capture, that would be great. Any information on what you want in the report would be very useful as well. I know there are some talented NetFlow experts out there with some insightful information to share.