I was having a conversation with a customer the other day about Amazon AWS monitoring. He had some interesting insight on his company’s overall migration to Amazon Web Service (AWS). He started with, “Here’s the core of it, cloud based deployment isn’t going away for us. Though there are no directives, by this time next year I’m expecting all but two of our public-facing applications to be sitting outside of our buildings.” He even went on to say, “There’s a very real possibility that within 2-3 years we decommission half of our computer rooms”. Needless to say, any application that they use for network visibility and incident response needs to support Amazon AWS monitoring. I loved his input and because of it I decided to dig deeper.
Consequently, the need for visibility into cloud based conversation, like AWS, isn’t a new concept. Way back in 2014 we saw rumblings of this with CompTIA’s Fifth Annual Trends in Cloud Computing Study. CompTia went out and polled 400 decision making IT professionals, business professionals and executives from U.S. IT firms to obtain their data. They found out that among companies that have progressed from the experimental stage to a non-critical use stage, 28 percent rated the transition as requiring significant effort. One of the issues they were having was being able to see the traffic inside their cloud environment. In a January 2015 Cloud Computing Trends report we see that many business are moving from using the cloud for non-critical functions like marketing or CRM and are starting to rely on this technology for storing more valuable data.
Up till now we have been able to gain visibility into encrypted traffic (e.g. Amazon AWS and Akamai) via Fully Qualified Domain Name (FQDN) for the sites that a user visited instead of seeing only encrypted or CDN traffic URLs. As I suggested before, the volume of encrypted traffic has tripled in less than a year and the number of connections to “cloud based” computing sites like Amazon AWS has grown at nearly the same rate. With so many connections on port 443 headed to content delivery networks, technologies like traditional NetFlow have lost much of their ability to provide insight. This made me ask the question, “So how do we get Amazon AWS monitoring into Scrutinizer?
Amazon AWS monitoring and NetFlow
The good news is that this was a priority for our development team. I spent some time reading through the “new feature” tickets and talking to the developers. After a few Red Bulls I finally had a better idea of what they have been doing. Basically, from the AWS API engine they were able to construct flow elements similar to NetFlow v5 that could be used by Scrutinizer.
Here is a basic idea of the elements we are receiving.
Since these elements resemble v5 data we should be able to determine a conversations path, interface, port and the amount of data being sent inside that cloud environment. This also means that we will be able to provide conversation pair reports, applications via Well Known Ports, alerting and more.
In addition to that, this also means that any AWS instance won’t be an isolated device since you should be able to trace the conversation from the cloud to your internal network. At that point customers can uses features like IP to Username reporting. Most importantly, with this information they can identify who or what was responsible for that conversation. As a result, this adds another layer of visibility to your Amazon AWS monitoring.
If you are interested in knowing more about how we can help you gain better visibility into your AWS traffic make sure to contact us for an evaluation.