Blog :: Network Operations :: Security Operations

Incident Response Pocket Guide

Becoming a NetFlow private eye is easy if you’re using the right Incident Response System (IRS). Our team is lucky to be able to experience and work through the scenarios that keep most security professionals up at night. After reading Jake’s blog  a few weeks ago, we decided to infect our lab with a virus that would try to move around the network and pull info from the servers and send them out to a command and control (C&C) server out on the Internet. I’ll share with you what we did.  Although this was done in a lab, the point will still be clear: advanced persistent threats (APT)/malware will go undetected for long periods of time if you don’t have the right system in place to help with incident response.

The scenario: It’s Monday morning and you are sipping your coffee, getting ready for the day. Your phone rings and it’s the FBI (maybe this is not exactly how that happens, but you get the picture). They tell you that something in your network is communicating with a known C&C server. I’m sure you can think of a few things that start to run through your head. Now what do you do? If you have the right incident response system in place, the details are only a few clicks away. A couple things your IRS should give you are:

  • Confirm the breach and when it started
  • Determine the Scope of the Breach
  • Identify how far the infection has spread

APT Data Breach:

We know that something on the network has been communicating with a specific C&C server’s IP address so we will search by host to find out who was communicating first to confirm the breach and find out when it started. The figure below shows how we will filter on only that IP address. We can see that the C&C server is communicating with employee1 It looks like employee1 has been reaching out to the C&C server presumably to not only send data, but to possibly get new marching orders as well.  Check it out:  starting at about 2am July 25, 2014.

APT Data Breach

Incident Response Scope:

Let’s drill in a little further. We can change the report above to show a two hour timeframe at ten minute data. From here we can now see that the malware has an open connection from employee1 to exfiltrate stolen data from the network to the C&C server at a regular rate! What have we lost?! We can also see that there is a very regular download from the C&C server that could be malware updates or additional instructions on what to steal from you next.

Incident Response Scope

Network infection containment:

Now we are going to switch gears and check to see what other assets employee1 has been communicating with other than the C&C server. We will build a report filtering on the IP address of employee1 as the source. We can see that employee1 has been communicating with other hosts in the network. I want to focus on the red and orange trends in the first graph below. The red shows the communication going to the C&C server and the orange is going to employee2. It is interesting that the communication with employee2 is very similar in timing to the downloads to employee1 from the C&C server. Looks to be that employee1 could be trying to spread the infection over to employee2. We can also see there are four other IP addresses that are contributing to the data breach in this scenario. The second image below shows employee1 as the destination of traffic and shows that the other hosts in the network are sending data to employee1 possibly for transmission to the C&C server.

Network infection containment

Exfiltrating Data


With only a few clicks and keystrokes, and using the right Incident Response System, you have; confirmed the APT data breach and when it started, identified the scope of the breach and what internal assets have been contributing to the exfiltration of data, and revealed a possible secondary infection that could have gone unnoticed and become “live” to steal more data if you didn’t know it had any communication with an infected host. You now have all the ammunition you need to effectively take down this malicious virus and avoid data theft and corporate embarrassment. The only question left to ask, Can your Incidence Response System  do that?