How can I load balance my NetFlow traffic across multiple collectors?

Do you find that you are constantly modifying NetFlow configurations to balance collector workloads?  The latest release of Scrutinizer introduces an option to load balance the collector workload across multiple, distributed-collector clusters by automatically modifying Plixer Replicator profiles based on the number of exporters and flow volume processed at each collector.

What is a Replicator?

A flow replicator is a system that receives UDP streams from hundreds or even thousands of devices, duplicates them, then replaces the destination IP address of each frame with a new one, and forwards them off to the new destinations without modifying the original source IP address.

Some benefits gained from a replicator solution are that they:

• Reduce the amount of traffic on the network
• Reduce the load on routers and switches as they only have to send UDP messages to one location
• Lessen the configuration workload when all, or many of your network devices suddenly need to send NetFlow, sFlow, IPFIX, or syslogs to a different IP address
• Provide management station redundancy by sending logs to multiple destinations simultaneously
• Allow both network and security administrators to receive the same logs while maintaining separate systems

And now flow collection in a distributed-collector cluster can be load balanced automatically.

Here is how it is done

Step 1—configure the API connection between Scrutinizer and Replicator.

In the Scrutinizer web interface, navigate to admin tab>settings>Plixer Replicator

1. Check the Enable check box
2. Enter the Replicators web interface admin user password
3. Enter the receive/input UDP port that the Replicator is receiving UDP flow packets on
4. Enter the IP address of the Replicator
5. The Seed Profile name is pre-set
6. Enter the Sent/Output UDP port that the Replicator will forward to the collectors on
7. Save

Step 2—SSH into the Scrutinizer server and login as the ‘root’ user.

From the command prompt type:  scrut_util –autoreplicate

The first pass creates an autoreplicate.conf file in the /home/plixer/scrutinizer/files directory. 

The file will be pre-populated with JSON entries representing each collector, with the exporter count and flow rate threshold values

The first pass also creates a Scrutinizer Autobalance Seed profile on the Replicator.

Step 3—Login to the Replicator web interface and add a policy(s) to the seed profile that includes all subnets of incoming exporters (usually policy 0.0.0.0/0)

Step 4—Pivoting back to the SSH access to Scrutinizer.

Edit the autoreplicate.conf file adjusting the desired exporter count and flow volume for each collector.

{
   "collector_capacities" : {
      "10.1.73.74" : {
         "exporters" : 20,
         "flow_rate" : 5000
      },
      "10.1.73.75" : {
         "exporters" : 22,
         "flow_rate" : 5000
      },
      "10.1.73.76" : {
         "exporters" : 20,
         "flow_rate" : 5000
      },
      "10.1.73.77" : {
         "exporters" : 29,
         "flow_rate" : 5000
      }
   },
   "debug" : 1,
   "per_port_profile_name" : 1
}

In this case, I modified the exporter counts on each collector and set the flow volume on each collector to 5000 fps.

** Flow volume is based on average flow volume at each collector over the last 24 hours

Step 5—Re-Run the scrut_util –autoreplicate command

At this point, you will seed an automatically generated profile for each collector on the Replicator profile tab. Each profile will show an exporter count the is reflective of your configuration in the autoreplicate configuration file.

Step 6—You will need to set up a cron job to call the scrut_util –autoreplicate command once a day. 

In the crontab file (/etc/crontab), I have added the task to perform the autoreplicate at 9 am, every day.

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# For details see man 4 crontabs
# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed
0 9 * * * root /home/plixer/scrutinizer/bin/scrut_util --autoreplicate

Scrutinizer generates a Collector Message Alarm that notifies that the autobalance function ran including a URL that can be copied and processed to show the results.

Reach out to us if you want to learn more or need help with setting this configuration up and running.