Good placement in the Gartner Magic Quadrant for Application Performance Monitoring is highly sought after by many vendors. The rich monitoring capabilities needed by a solution to participate in the quadrant is way beyond what most people consider possible with traditional flow exports. Today however, this has changed.
Several weeks ago I read a post from Jonah Kowall at Gartner that read “The movement of Application Aware Network Performance Monitoring (AA-NPM) away from Application Performance Monitoring (APM), as a unique market with different technologies and buyers. [sic]” Although I agree that it is important to clarify the differences, I believe the two will continue to work together for an ideal overall window into the performance of business applications. Separating the two might make it easier to manage the market leaders, but vendors will continue to provide a complete solution which encompasses all of AA-NPM and APM.
The differences between APM vs. AA-NPM
- APM: tends to be more focused on specific transaction monitoring within an application. It strives to answer the question, how could the applications code be optimized to provide faster response times and what additional hardware resources on the server could improve speed.
- AA-NPM: tends to be more focused on the impact the network has on an application. It strives to answer the question, where on the network is the connection time being degraded. Is the flow suffering from poor round trip times, packet loss, retransmits, jitter or out of order packets, etc.
Certainly the two can be clearly separated, but I’m sure most would agree that the two provide greater value when their measurements are brought together. A year ago, Gartner considered the leaders of APM to be IBM, CA and Compuware. These vendors demonstrated a clear leadership role in the 5 elements of the APM process as discussed below.
Source: Application Performance Management
The Gartner Magic Quadrant for Application Performance Monitoring defines APM as a process with five elements, most of which can be tracked with NetFlow or IPFIX:
- It tracks in real time the execution of the algorithms that constitute an application. Most operating systems allow their algorithms/processes to be monitored and their corresponding statuses can be routinely exported in IPFIX.
- It measures and reports on the scarce hardware and software resources that are consumed as the algorithms execute. Again, using IPFIX, we have exported the resource consumption and the hardware impact of each process involved with an application.
- It determines whether the application has executed successfully. Further explanation should be made here but, I’ll assume that this means some sort of synthetic transaction is executed from a robot standing in as a typical user and the response value and time is compared to a threshold.
- It records the latencies associated with some of the execution-step sequences. I’d like to learn more about what this means.
- If each execution step is a unique flow, technologies such as Cisco Application Visibility and Control flow exports or the details exported in IPFIX from the nBox can give us plenty of network traffic insight here. Even latency on a hop by hop basis can be measured. In many cases, the distributed nature of flow collection makes it ideal over packet capture. However, packet capture can definitely be handy when trying to help the application vendor trouble shoot their own software.
- If we are talking about performance issues deep within the application, generally this type of performance monitoring is done by the vendor who developed the software. In some cases, they are brought in when the application has to be troubleshot at this level.
- It determines why an application has failed to execute successfully or why resource consumption and latency levels have departed from expectations. This is a tough one to deliver on. Although vendors claim proficiency in pin pointing the root cause of a network performance issue, I think the reality is that human computation of the data is still necessary to isolate the issue.
APM using Robots and Packet Capture
Gartner Analyst, Will Cappelli and Jonah Kowall, had this to say about robots: “The oldest commercialized approach to capturing end-user experience data, this technology requires the placement of software robots at various locations near which users might access an application, then the robots run scripts against an application in production, and record response time and availability results. This technology has been frequently criticized because it does not deliver information about real user experiences” They went on to say this about packet analyzers: “packet capture and analysis appliances are, in whatever number of protocols they may have fluency, data-center-bound. Hence, they’re blind to the increasing number of sources of latency and other performance problems that originate in cloud services that lie along an application’s execution path”
Flow Technologies help with APM
The release of Cisco Application Visibility and Control flow exports allow NetFlow and IPFIX to play a larger role in AA-NPM and APM solutions. The application metrics include details on latency, packet loss, jitter, retransmits and more. These measurements differ from robots executing synthetic transactions because, AVC measures the performance of the flows involved with the end users application experience!
Gartner stated last year that flow analysis should be done 80% of the time and that packet capture with probes should be done 20% of the time. Source. With the release of AVC, perhaps Gartner will increase the ratio from 80/20 to 90/10. If you would like to read more about Gartners position on Application Performance Monitoring, the 29 page Gartner Magic Quadrant for Application Performance Monitoring document can be purchased from their web site.
UPDATE March 3rd 2014: This post has a follow up blog titled Gartner Application Performance Monitoring.