It was a cold day in March, colder than usual for this time of year. The phone rang and it was Jon telling me that his router wasn’t performing well and was having issues. They all have an issue in this city. Some are big and some are small, but they all have issues. As for Jon, his issue was big and that’s why he called me… I’m Jimmy D, the Cisco NetFlow Detective.
His story was the same old song; everyday around a specific time, his network would slow down and the CPU on is his router would peg at 90%. He needed to know why, and fast. His company was getting ready to release a hot new piece of software and they needed the bandwidth to support it.
He had taken the first step; he was already monitoring his bandwidth with Scrutinizer. But Jon needed more. He needed to know what times his CPU utilization was high and what traffic patterns occurred during that time. If this was a perfect world, he would also be alerted when it happened.
“Let’s go get a cup of coffee.” I said.
“Jon, we can trend your CPU utilization via SNMP with Denika. We can also set up alarms and alerts in both Scrutinizer and Denika. We can also capture syslogs from the router with Logalot. All this information can be tied together to give us a better picture and possibly point out a pattern.”
“Awesome, that’s what I was looking for! Can you help me?” he replied.
“Sure Jon, I’m the NetFlow detective, that’s what I do.”
Later that day, we took some time to set up both products. I explained how the process worked and what we were looking for. I let him know that although we can store this data forever, We were specifically interested in the next 24 hours. I was positive that our culprit would strike again.
He let me know that he would call me the next day.
“Jimmy, I just got an alert!” said Jon.
“Lets look at what it said.” I asked.
It was 5:01 p.m. and I wasn’t surprised. Nasty things, like rats and bad packets, show up quickly. After a few minutes of searching, I could see a pattern and it wasn’t pretty.
“I believe that I found your issues Jon.”
I looked at the time of the CPU spikes in Denika’s SNMP reports. We then looked at the Layer 3 traffic reports within Scrutinizer. I compared the timeframes and quickly saw the traffic matched.
“We now know it is a user. So now let’s find out who it is. To do so, we can drill down through the IP addresses in Scrutinizer and find out what IP is causing the traffic. Here you go Jon, are you ready to see who is hogging your bandwidth and causing the high CPU utilization?”
In one click, I quickly resolved the top talkers and saw that it was jenny.abcorp.com.
“Oh no, that’s my girlfriend!” said Jon, “Can we tell who she was talking to?”
We quickly changed to the conversations destination and could see that she was uploading 6 gigs of information to cbacorp.com at 5 p.m. every day. Jon knew the rest of the story because it was a common one. Geek programmer meets cute Russian model who thinks he is Superman, but soon finds out that he had been taken by a pretty face. She was uploading the latest builds of their hot new software to the competitors. She was a spy.
“Thank you Jimmy, you saved our company.” said Jon.
“Don’t sweat it kid. My job is to shed some light in a dark world…”
Most of these names and happenings in this story are true. Some have been changed to protect the innocent.