Blog

Sending Syslogs as IPFIX or NetFlow

Why aren’t more companies exporting their syslog messages as IPFIX or NetFlow?  With all of the benefits that IPFIX offers, some people (i.e. namely me) think it is a no brainer.  In all fairness, I think the reason for the slow adoption of IPFIX is lack of awareness and experience.  With a bit of patience and a little help however, the transition is actually quite easy.

First of all, exporting syslog messages in NetFlow isn’t a wise idea.  Why?  It’s because NetFlow is owned by Cisco and to use elements within this technology outside of what it was intended for could lead to all of the hard work being trumped by Cisco.  If you want to export something new that hasn’t been defined in NetFlow yet, you better use IPFIX which is the proposed standard for NetFlow and open to all vendors.  Also, IPFIX allows for variable length strings, NetFlow doesn’t and variable length is important when discussing syslogs.

Why Export Messages in IPFIX instead of Syslogs?

The reason companies want to export messages in IPFIX instead of syslogs is for a few reasons:

  • Scalability: Syslogs have a 1:1 relationship with datagrams.  IPFIX support a 1:Many relationship.
  • Less network overhead: fewer datagrams means less impact on network utilization.
  • Ease of correlation: by breaking down message components into elements (i.e. different fields), messages can be more easily cross referenced with other messages
  • IPFIX saved data is faster to query.  This is because IPFIX data is structured and syslogs are unstructured

Sending Syslogs in IPFIX

If I was to export syslogs via IPFIX I would start by making sure I understand the syslog format. Notice the references to Facility levels and Severity levels.  In RFC 5424 we begin to see how these ‘levels’ can be broken out.  In fact, these levels fit nicely into IPFIX elements.  What are IPFIX elements?

What are IPFIX Elements?

IPFIX elements are basically similar to columns in a database.  If the elements are unique, they and can be used to tie data between different templates.  What are templates?  Templates are used by IPFIX to tell the collector how to decipher the different types of data records (i.e. groups of elements) that are being sent.  A typical IPFIX template maps out that a record being sent with a specific ID contains for example the source and destination IP addresses, source and destination ports, source and destination interfaces, protocol, etc.  Each of these items is an element and the collection of all items (i.e. elements) are called a record.  Templates tell the IPFIX collector the format of the elements within the records.  With syslogs, a template isn’t used rather,  they contain a header which includes details about the payload and it’s all in the same datagram which is sort of inefficient.  The components of a syslog message need to be broken out into individual IPFIX elements.

Example Syslog to IPFIX

Now that we understand that here are two parts to IPFIX: Templates + Data, we can create an example which I took from Jeroen Massar.

<<< BEGIN Jeroen Massar EXAMPLE >>>

The template describes how the data looks like, for instance, lets take an Apache CLF log entry:

66.249.66.174 – – [16/Feb/2011:10:48:11 +0100] “GET /robots.txt
HTTP/1.1” 200 2629 “-” “Googlebot-Image/0”

We can make an IPFIX template for that:

[
{4, IPv4_SRC },
{4, TIMESTAMP},
{4, HTTP_METHOD},
{v, URL},
{v, HTTP_PROTOCOL},
{2, HTTP_RESULT},
{8, OCTETS},
{v, HTTP_REFER},
{v, HTTP_USERAGENT},
]

The ‘v’ markers indicate variable field lengths, the others indicates the number of bytes such a field takes. The data is then just encoded in the above format, presto.

learn more about Jeroen Massar
Jeroen Massar

<<< END Jeroen Massar EXAMPLE >>>

What Can I do with The Message in IPFIX?

Now that the message is in IPFIX I can cross reference the above IPv4_SRC (66.249.66.174) with the NetFlow or IPFIX flows that I am receiving from my switch, firewall or router!  This correlation allows me to identify potential security issues across platforms without parsing logs.  This saves time, overhead on the exporting hardware and the collector.  It makes queries faster.  It even makes ice cream taste yummier.

Give Me an Example Please

Let’s say an apache server exports a message that 66.249.66.174 failed to login.  Let’s say 3 other servers export a similar IPFIX message complaining about the same IP address.  Let’s say the firewall denies flows to this host and exports the message in IPFIX.  Perhaps the NetFlow and IPFIX collector detects that 66.249.66.174 is a known internet threat with a poor host reputation.  The culmination of 5 different sources all reporting problems regarding the same element:  IPv4_SRC = 66.249.66.174 would be escalated through a simplified correlation process.

Syslogs Vs. IPFIX Summary

Get ready for an industry that is going to be exporting messages in IPFIX as well as Syslogs in the near future.  Don’t expect a simple syslog dump to IPFIX script to do the trick.  It isn’t that easy.  The different fields exported in the message as pointed out in the example provided by Jeroen Massar must be broken down into different elements.  Different groups of elements may trigger different templates.  A simple script trying to parse syslogs into IPFIX is likely to be inefficient and require frequent updates to deal with all the different possible log formats.  If you want something very simple, than define an IPFIX element as “SYSLOG_MESSAGE” and dump the entire message in one big glob.  The problem is that this will be nearly unusable to correlation engines and reporting tools.  The right thing to do is to get the vendor to support IPFIX as well as syslog and a great way to work in this direction is to consider CEE-enhanced syslog which is supported by Rainer Gerhards. Contact Plixer (www.plixer.com) if your company needs help getting started.