Virtual server traffic monitoring can sometimes require thinking out side of the box. First we need to understand where the application resides, what we want insight into and then spend some time learning how it behaves.  To do this, it is important to identify the traffic that belongs to an application and be able to see all of it regardless of where it came from.

“The complexity introduced when deploying them in a virtual environment can make it more difficult to pin point problem sources. In the virtual world, the application could be hosted on the Internet or within the confines of a local data center.” Michael Patterson, Founder – Plixer.com “Even then, the exact location of the application can change dynamically based on resources.  Traditional troubleshooting involved knowing where the application resided. This was a big part of the normal routine, but no longer a simple process in a virtual world.” Source

Since where the application resides is a moving target, we need to leverage a technology that provides visibility into all areas of the network without deploying probes.  That solution is NetFlow because the information it contains can be aggregated across collection points into a single report. The problem with this technology is that not all flow exports were created equal.

VMware switched from NetFlow to IPFIX

Early on, VMware decided on NetFlow to provide this capability.  Recently they switched to IPFIX support which is the IETF standard. IPFIX allows developers to export just about anything they want inside flow datagrams.  Because they made the switch, we anticipate richer traffic details about the connections being made to the virtual servers.  Since the change or at least as of today, they are still only exporting basic flow information such as those details available in NetFlow v5.

Critical for Monitoring Traffic in Virtual Environments

Here are 3 areas where virtual environments need improved insight:

  1. Performance: Knowing how many bytes were in a connection is great but, we need more details.  For example, what about the round trip time between the hosts?  What about packet loss, retransmits, the URL involved?  How about the TCP flags or TCP window size.  These are all details that Cisco AVC is exporting in IPFIX.  Hopefully companies like VMware will join the band wagon soon.
  2. Logs: vendors need to move away from their proprietary syslog structure and band together by exporting a consistent format. IPFIX allows for this. Below is an example of the Flow Replicator exporting VMware syslogs as IPFIX.  Flow Replicators can forward any machine log including Microsoft event logs as IPFIX.  esx syslog reporting ipfix
  3. Contextual Details: The ability of a NetFlow reporting solution to correlate flows to the actual usernames is paramount if we are to determine the source of a traffic pattern quickly.

How the VMware Application Behaves

To gain insight into the performance of an application, you have to know what you’re looking for.  What ports does it use?  What domains or HTTP hosts does it reach out to.  Sometimes we can lock in on a single or range of IP addresses but, as Patterson pointed out, the server and physical location of the application can dynamically change in a virtual environment. If for example you are thinking about Monitoring VMware Virtual Desktop Infrastructure Traffic there are several ports being used.

  • JMS – 4001
  • HTTP – 80
  • HTTPS – 443
  • RDP – 3389
  • SOAP – 80 or 443

Since the IP address associated with these ports could be moot, sometimes it’s easier to exclude hosts on the network that might be sharing these ports for other applications.

If the application is cloud based, some flow exports provide details on URL or HTTP host.  These details often allow us to avoid filters on port ranges which is important because some applications use random ports which are undoubtedly shared by many other applications (e.g. port 80 and 443).

Virtual Server Traffic Monitoring

In summary, know what you are looking for and use NetFlow or IPFIX to observe it.  If you are not getting the details you need, reach out to our team and we’ll show you a few short cuts that will get you what you need to monitor your virtual environment.

 

Thomas

Thomas

Thomas Pore is the Director of IT and Field Engineering at Plixer. He developed and leads, the Malware Incident Response and Advanced NetFlow Training programs which are being offered in cities across the USA. He is also an adjunct professor at the local community college and teaches ethical hacking. Thomas travels the globe meeting with customers and trying improve the Scrutinizer network incident response system. He helps clients optimize threat detection strategies and aids in the configuration of custom incident response solutions. He has a Bachelor of Science in Computer Science from Dickinson College.

Related