Book on NetFlow and IPFIX

  • Title: Unleashing the Power of NetFlow and IPFIX

    Unleashing the Power of NetFlow and IPFIX

    This tell all flow encyclopedia explains in detail how flow technologies are impacting businesses and improving the way IT organizations around the world are able to more easily manage voluminous amounts of information easier, faster and with more knowledge into the transactions utilizing the network infrastructure. Using nearly 150 color illustrations, the book covers what has been accomplished to date with flow technologies such as NetFlow, sFlow and IPFIX. It does this by including examples from different vendors and clearly outlines where this message export is going in the next 5 years.

    At the first level, this book will satisfy those managers, professionals, and engineers who need conceptual and practical answers to the questions:

    • What are NetFlow and IPFIX (IP Flow Information eXport)?
    • How do they work? Where do they fit in the world of network troubleshooting and traffic monitoring?
    • And why do we need or not need them?

    All this is mostly overview to moderately experienced NetFlow-aware IT professionals. It includes explanations on the fog of acronyms and special terminology that seems especially dense in the vicinity of NetFlow, including Flexible NetFlow, AppFlow, J-Flow, NetStream and sFlow. It also includes information about some of the many new performance and security applications emerging from Cisco, SonicWALL, Palo Alto, Plixer and other vendors. Additionally, it explains what these new technologies are for, and if they are really needed.

    At the second level, this book does not spend time deciphering and comparing NetFlow v1, v5, v6, v7, v8, v9 formats to one another or even to IPFIX. There is plenty of documentation on the Internet that covers this topic and frankly a chapter on this subject is not in line with the goal of this book.

    It is assumed that the reader has had some exposure to NetFlow and how to configure traditional NetFlow v5 and v9. This book, therefore, is intended to open the eyes of IT professionals to what is possible with NetFlow v9 (Flexible NetFlow) and IPFIX, how to properly implement them (and avoid the pitfalls), and why these flow exports can allow businesses to be far more competitive. As of the date of this book’s publication, these are considered to be the most cutting edge flow technologies.

    Performance issues are discussed and how the right hardware can dramatically improve exporting, collection, reporting and threat detection speeds. Basic guidelines are provided on where these technologies should be deployed, along with practical reasons why certain technologies can bring value to business applications. The troubleshooting potential of these technologies is also explored.

    At the third and final level, some sections are more technical and can be skipped by some readers without missing out on important concepts. These clearly indicated paragraphs are meant to help product managers and software engineers understand concepts such as how flow templates should be implemented, when meta-data and option templates are useful, and how to structure them to ensure speedy reporting. Suggestions on the creation of new elements and the reuse of existing IDs are also outlined. Examples are given on the pitfalls to avoid but again, these technical sections can easily be skipped without diminishing the value of this book.

  • CHAPTER 1: What is NetFlow 13

    • IPFIX, the Emerging Standard 13
    • Flows are Aggregated Based on a Tuple 14
    • Bidirectional Flows 15
    • Pick your Poison: Detailed or Generic Tuple 15
    • The Sky’s the Limit: Export it as Flow Data 17
    • The Value of Flow Information 18
    • Exporting Flows 22
    CHAPTER 2: Exporting NetFlow or IPFIX 23
    • Choosing NetFlow or IPFIX 23
    • Private Enterprise Numbers 23
    • A Word about Private Enterprise Numbers (PEN) 24
    • Differences between NetFlow v9 and IPFIX 25
    • A Final Warning: NetFlow v9 or IPFIX 27
    • Keep Marketing out of NetFlow and IPFIX 28
    • Vendors Supporting NetFlow and or IPFIX 28
    • Call it NetFlow or IPFIX 29
    • Building a Flow Cache 30
    • Defining the Tuple Case Study 30
    • Implement RFC 5610 34
    • What is IPFIXify™ 35
    • Key and Non-Key fields 38
    • Match and Collect 39
    • What is a Flexible NetFlow Key Field? 40
    • What is a Flexible NetFlow Non-Key Field? 41
    • The Four Steps to Configure Flexible NetFlow 42
    • When are Flow Cache Entries Exported? 45
    • Export using UDP or SCTP 46
    • Flow Sequence Number 47
    • Are You Missing NetFlow 47
    • What Does an Increase in MFSN Tell Us? 49
    • NetFlow v9 vs. v5 50
    • NetFlow Collection without Flow Sequence Number 50
    • CPU Impact from Exporting Flows 50
    • Exporting and Reporting of Flows 51
    • Metering 53
    • Observation Point 53
    • Flow Direction 54
    • Observation Domain 54
    • Common Flow Exporting Mistakes 55
    • SNMP Indexes 55
    • Active Timeout 55
    • octetDeltaCount 56
    • Flow Direction 56
    • Multiple Templates 56
    • Template Timeout 56

    CHAPTER 3: Collecting Flows 58

    • Measuring Flow Collection 58
    • Calculating Flow Storage Needs 61
    • Not All Flow Storage Solutions Were Created Equal 62
    • Flow Sampling 63
    • What is sFlow 63
    • NetFlow Sampling 67
    • Packet Sampling (PSAMP) 69
    • Deduplication and Stitching Flows 70
    • Dealing with Different Time Zones 72
    • Regulatory Compliance 72

    CHAPTER 4: Reporting on Flows 74

    • Limitations of NetFlow v5 74
    • NetFlow v5 record format 74
    • NetFlow v9 Introduces Templates 76
    • Ingress vs. Egress 76
    • NetFlow Reporting and SNMP 79
    • BGP and Autonomous Systems 80
    • Cisco Performance Monitoring or Medianet 81
    • DSCP Vs. ToS 84
    • MPLS Reporting and MPLS Tags 85
    • BYOD and Smart Phone Reporting 86
    • Social Networking Sites 88
    • Virtual Servers 90
    • Cross Template Reporting 91
    • Multicast Traffic 92
    • Billing and 95th Percentile 93
    • Service Providers Think Beyond the Network 99
    • IPv6 Support 103
    • Displaying Applications 103
    • Dashboards 108
    • Securing Access to the Data: Service Providers 110
    CHAPTER 5: Detecting and Mitigating Threats 111
    • TCP Flags 114
    • ICMP Type and Code Fields 115
    • Flow Behaviors 115
    • Custom Flow Behavior Monitors 116
    • Behavior Baselines 116
    • IP Host Reputation 117
    • Firewalls Exporting Flow Data 118
    • Cisco Smart Logging Telemetry 122
    • SonicWALL IPFIX Security Details 124
    • Identity Awareness 125
    • Country and Geo IP Details 126
    • Alarm Correlation 129
    • Threat Mitigation 131
    • Advanced Persistent Threats 133
    • Flow Sampling and Security 134
    • The Bottom Line 134
    CHAPTER 6: Troubleshooting and Examining Flow Data 137
    • Isolating Slowness 137
    • Detecting Malware 141
    • Find the User 144
    • Flexible Filtering and Reporting 145
    • Summary 147
    • Glossary of Terms 149
  • Aamer Akhter

    “As the IPFIX (based on cisco’s NetFlow v9) data export standard becomes well adopted across the networking industry, the experiences of Plixer and Mike Patterson become highly relevant for network operators, vendors but also other IPFIX/NetFlow management systems. ‘Unleashing the Power of NetFlow and IPFIX’ provides a basic background on the technology, the bleeding edge of where it is today, but most importantly provides the implementer’s experiences that are missing from the specifications.

    Michael Patterson has loaded the book up with clearly written examples of what is possible with NetFlow and IPFIX. Flow technology is spreading beyond basic packet counting on routers into virtual environments, threat detection, data inspection but also application analysis as well. With the increasing demand to gain greater network traffic insight on elements such as jitter, packet loss and URLs, the volume of Flow exports is exploding. Because of this, scalability is a concern and the content in these pages explores important aspects of NetFlow export and processing such as distributed collection, flow deduplication and stitching. If you are a developer or a product manager, leverage this book as an implementation guide. Its contents are designed to help with perspective on what you may want to achieve, the exports that match your goals and how to go about achieving them. The next step of course is figuring out how to best implement NetFlow and IPFIX, the problems to avoid and how to take your exports to the next level to stay competitive.

    For the last 6 years I have been working with cisco’s NetFlow engineering team, customers as well as many network management system vendors. I have had the pleasure of working with the Plixer team in my various roles at cisco and this book further reinforces by belief that Plixer is one of the industry’s premier thought leaders. It is clear to me that Michael and the team at Plixer are passionate when it comes to anything NetFlow and IPFIX related. This is a company that is on the bleeding edge of NetFlow/IPFIX."

    Aamer Akhter

    Technical Leader & Architect for Network Management Solutions
    Cisco Systems

    Cisco Logo

  • “This book makes it clear beyond any other resource that flow technology provides value far beyond its traditional implementations. It’s full of great examples from companies who have actually implemented the latest and greatest exports. I’m very proud to say that I was able to be a part of it and I hope you enjoy it.”

    Erik Peterson, Dir. of Engineering – Plixer


    “It’s clear to me that flow technology has entered the areas of Application Performance Management and message logging. IPFIX will without question continue to play a major roll in most new security appliances being developed today. The definable nature of IPFIX makes its exports ideal for forensic analysis. This book explains how to do it.“

    Thomas Pore, Dir. of Technical Services – Plixer


    "The value in this small volume is the practicality of its guidance. The author clearly knows his stuff and shares his insight with the reader.”

    William Flanagan


    “As a research analysts and professional editor, I was able to appreciate Mike’s ability to effectively outline the business value of the new exports available in NetFlow and IPFIX. He has clearly put in the time to research where these technologies are going and how they will provide value to businesses trying to maintain a competitive edge while controlling IT spending."

    Drew Robb, Owner – Robb Editorial


    “For me, this has been a great resource for trying to learn more about Cisco’s Flexible NetFlow capabilities. Initially I just wanted to learn more about NBAR and Performance Monitoring for Medianets. Because of the additional background the book provides, I learned about several other new exports from Cisco such as Performance Routing and Smart Logging Telemetry. The color illustrations provide actual proof of these cutting edge implementations that some of our hardware already supports. This book contains information we can take advantage of right now.”

    Dave – State College in New York


    “When Mike asked me to review his book, I had no hesitation. Yes! I’ve read dozens of his blogs and true to my expectations, it was well worth my time. I learned about packet capture with NetFlow, how this technology is very different from sFlow and where these technologies complement one another. If you are going to read an IT book in this year, make [Unleashing the Power of NetFlow and IPFIX] one of your top picks. It will open your mind to the power of flow technologies and how they can help you make better and quicker decisions in your daily IT routines. Well done Mike!”

    Matthew – University of Maine


    “I had no idea that NetFlow could be used for so much more beyond what our team does with it. I thought NetFlow was about top talkers, applications, protocols and the like. Wow, this book made me realize that I need to read books like this more often. By just changing the configuration of my NetFlow export, I can get VoIP details on jitter, packet loss and even caller ID. And, if we decide to migrate to cloud services next year, I can measure the round trip time of the employees making connections during the day. I had no idea I could do this with NetFlow. Great book!”

    Ken – Chicago, IL

  • Michael Patterson

    Michael Allen Patterson is the founder and CEO of Plixer International, a company that focuses on developing flow collection, reporting and analysis tools. Prior to starting the company, when he returned home from the University of Maine at Orono, Mike started his career in technical support at Cabletron Systems, acquired his Novell CNE and moved to the training department for a few years where he taught Cisco Routing, Ethernet, Token Ring, FDDI and ATM. While in training he finished his Masters in Computer Information Systems from Southern New Hampshire University and then ended technical training to pursue a new skill set in Professional Services. In 1998 he left Cabletron to start a successful web-based monitoring company named Somix before starting Plixer. Mike has written nearly 600 blogs, case studies and articles. He likes to participate as the product manager for Scrutinizer NetFlow and sFlow Analyzer as well as Flow Analytics. He enjoys many outdoor winter sports and often takes videos when he is snowmobiling, ice fishing or sledding with his kids. Cold weather and lots of snow make the best winters as far as he is concerned.

    Feel free to email him or follow him on twitter: @netflowpm

    He can be reached most hours of the day between work and home. If you want to stay current with netflow, join NetFlow Developments on linkedin where Mike is one of the mediators.

    Additional Contributors to this Book

    • Marc Bilodeau, Plixer International
    • Erik Peterson, Plixer International
    • Andrew Feren, Plixer International
    • Drew Robb, Robb Editorial
    • Adam Powers, formerly of Lancope
    • Pete Cruz, formerly of SevOne
    • Aamer Akhter of Cisco Systems

    “I hope you enjoy this book. I’ve poured just about every ounce of what I have learned over the last 7 years about NetFlow, sFlow and IPFIX into it. I could never have done it without the contributions from the above individuals. Behind just about every great resource is a team of knowledgeable people willing to share what they know. I am very fortunate to be one of the people they were so willing to help. Thanks Guys!” - Michael Patterson

  • Get started with Unleashing the Power of NetFlow and IPFIX by downloading and reading Chapter 1 titled “What is NetFlow”. Find out why this book starts off disparately from any other resource on flow technology. Free Download.

    Book on Netflow and IPFIX

    To view the chapter you will need Acrobat Reader.

  • 100% of all proceeds after publication and shipping costs are being donated to Grahamtastic Connection.

    Children's Cancer Foundation
    NetFlow Knights

    "We thank the Grahamtastic Connection for the support they have given to children during prolonged hospitalization. Although thousands of charities need donations, this organizations mission seems to tug the most at my heart. Please send a donation and a prayer to these children.”

    Michael Patterson, NetFlow Knight: CEO, Plixer

    Become a NetFlow Knight™
    A NetFlow Knight™ (NK) is an individual who believes that flow technology is one of the ideal technologies available today for gaining deeper insight into IT related issues. They demonstrate chivalry by fatefully serving their family, community, company and set good examples in leadership and character.

    At Plixer, our NetFlow Knights serve our customers and promote the technology in our blog and Advanced NetFlow Training class.