Today, I am going to cover five new features available in Scrutinizer v7, as Part 4 of 5 in our “Whats new in Scrutinizer v7″ series. If you will recall, Nathan started this series off by covering encryption exclusions, more flows, collector improvements, group permissions for users, and proxy server configuration. Jon continued with part 2, covering overriding report intervals, Google Map connections, host and application quick search, user profiles, and alarm category filters. Last week, in part 3, Joanne blogged about applications being defined by combination of ports and IP addresses, emailed reports on demand or scheduled for regular time intervals, stacked trend graphs on all reports, LDAP and Active Directory support, and extensive flexibility for VoIP reports. This week I have five features that you’ll use time and time again.

Read more »

Are you looking for an inexpensive solution to gain visibility on your network? Traffic-Flow is a feature available on RouterOS by MikroTik. Traffic-Flow is comparable to Cisco’s NetFlow technology, providing statistical information about packets passing through the router. Traffic-Flow supports NetFlow formats: v1 (not recommend) , v5 (BGP, AS, and flow sequence support), and v9 (extend-able field and record type support); therefore, most NetFlow collectors, including Scrutinizer and similar, will listen for these flows.

RouterOS can be purchased by itself to run on a PC with two network interfaces, or you can purchase a RouterBoard, as I did, which will come with RouterOS loaded. You can run RouterOS in transparent bridge mode or as a router. If you run in bridge mode, all traffic exported will show as coming through one interface (the pass-through bridge), whereas, if run in router mode, you will get the different source and destination interface indexes and descriptions.

I bought the RB433AH and configured it to send flows to a Scrutinizer demo box. I have configured our RouterBoard as a bridge exporting Traffic-Flow v5 and placed this in-line between our firewall and core switch. As you can see in the screen capture below, the bridge information allows me to see traffic to and from our network. We are looking at the top 10 conversations for the last 5 minutes.

If you are currently running a network with devices that don’t support Cisco NetFlow, a RouterBoard for $145 is an inexpensive solution to give you the visibility you’ve been looking for.

Monitoring with CBQoS (Class-Based Quality of Service) in addition to NetFlow and IPSLA on a network can provide additional insight into an organization’s critical applications utilization. As more companies deploy VoIP and video conferencing CBQoS can be utilized to show how these applications are delivered across an infrastructure with defined policies. CBQoS is a Cisco feature set that is part of IOS 12.4(4)T and above, and is available at no additional cost. The QoS statistics provided by CBQoS are made available via SNMP polling and give detailed information about the defined QoS policies applied to interfaces and class-based traffic patterns.

A typical network with no QoS policies defined runs with a best-effort delivery, with all traffic having equal priority and an equal chance of being delivered. When these networks become congested, all traffic also has an equal chance of being dropped. The definition of policies (class-maps defined in policy-maps) allows specific traffic to be prioritized according to its relative importance and uses congestion-avoidance to ensure its delivery. CBQoS can also be used to limit the amount of bandwidth for certain network traffic making network performance more predictable and utilization more efficient.

CBQoS can be used to ensure network applications such as VoIP and video conferencing receive the highest priority. It also provides an in-depth look at the amount of traffic used, both before it is filtered through a policy (pre-policy) and after it has been filtered (post-policy). If any of the traffic was dropped during congestion because of the rules defined in a policy, CBQoS reports the amount of traffic that was dropped.

In the following example, I have configured one of Plixer’s lab routers (sent home with an employee) to prioritize both IAX and SIP traffic for VoIP calls along with a priority for webmail and file transfer. This policy, attached to a WAN interface, will ensure that the employee will be able to effectively work from home. After the policy is in place, I enabled CBQoS to monitor the pre-policy, post-policy, and dropped traffic from my defined class-maps.

In my example, I have classified the VoIP traffic for support protocol of Plixer’s PBX. When an employee is at home, the VoIP traffic will be given 70% (priority) of the interface, only when congested. I have classified PlixerVoIP traffic with an access-list.

lab(config)#ip access-list extended PlixerVoIP
lab(config-ext-nacl)#permit udp any any eq 4569
lab(config-ext-nacl)#permit udp any any eq 5060

This PlixerVoIP class references out access-list, defined above.
lab(config)#class-map match-any PlixerVoIP
lab(config-cmap)#match access-group name PlixerVoIP

The “Web_Email” class includes regular web taffic like HTTP, HTTPS, FTP, SMTP, and POP3. In other words, this includes web browsing, file transfer, and email traffic.

lab(config)#class-map match-any Web_Email
lab(config-cmap)#match protocol http
lab(config-cmap)#match protocol secure-http
lab(config-cmap)#match protocol ftp
lab(config-cmap)#match protocol smtp
lab(config-cmap)#match protocol pop3

The policy-map is used to match the classes, above, with the policy you define for that type of traffic.
In my examples, we are giving PlixerVoIP traffic 70% priority, if that traffic is present. We are also setting the DSCP EF bit to notify routers down the line that this traffic is important PlixerVoIP traffic. We are giving Web_Email web browing traffic 75% of the remaining bandwidth. For all traffic that was not defined (the class-default), it will just be fairly queued.

lab(config)#policy-map TP-Pol-FastEthernet0/1
lab(config-pmap)#class PlixerVoIP
lab(config-pmap-c)#priority percent 70
lab(config-pmap-c)#set dscp ef
lab(config-pmap)#class Web_Email
lab(config-pmap-c)#bandwidth remaining percent 75
lab(config-pmap-c)#exit
lab(config-pmap)#class class-default
lab(config-pmap-c)#fair-queue

On the WAN interface (your connection to the Internet, in this example it is FastEthernet0/1).
The NBAR protocol-discovery command must be applied for NBAR to recognize traffic.
The service-policy command applies your QoS policy to the Interface.

lab(config)#interface FastEthernet 0/1
lab(config-if)#ip nbar protocol-discovery
lab(config-if)#service-policy output TP-Pol-FastEthernet0/1

Now that our policy has been applied to the interface, let’s enable CBQoS.

lab(config)#enable
lab(config)#configure terminal
lab(config)#snmp-server ifindex persist
lab(config)#snmp mib persist cbqos
lab(config)#end
lab(config)#write mib-data
or
lab(config)#write

To verify our CBQoS commands, we can show them in the running-config:

lab(config)#show running-config | include cbqos
snmp mib persist cbqos
lab(config)#show running-config | include persist
snmp-server ifindex persist
snmp mib persist cbqos

We can monitor the CBQoS statistics with the Denika add-on to Scrutinizer.

As you can see below, there isn’t enough traffic to cause dropped packets, so the pre-policy traffic is the same as the post-policy traffic. The dropped bytes trending graphs indicate that we have not had any problems with our business specific applications.

Oftentimes, when I’m running around the country setting up Flow Analytics, I don’t see Null Scans pop up. However, recently I’ve visited high profile customers that are big targets for malicious behavior. As we configure Cisco NetFlow on their routers and ASA firewalls, I’ve noticed FA alerting on these packets with no flags set.

The Null Scan is a type of TCP scan that hackers — both ethical and malicious — use to identify listening TCP ports. In the right hands, a Null Scan can help identify potential holes for server hardening, but in the wrong hands, it is a reconnaissance tool. It is a pre-attack probe.

A Null Scan is a series of TCP packets that contain a sequence number of 0 and no set flags. In a production environment, there will never be a TCP packet that doesn’t contain a flag. Because the Null Scan does not contain any set flags, it can sometimes penetrate firewalls and edge routers that filter incoming packets with particular flags.

The expected result of a Null Scan on an open port is no response. Since there are no flags set, the target will not know how to handle the request. It will discard the packet and no reply will be sent. If the port is closed, the target will send an RST packet in response.

Information about which ports are open can be useful to hackers, as it will identify active devices and their TCP-based application-layer protocol.

Cisco NetFlow packets contain a summary of the packets flowing through an interface including TCP flags, or in this case, not set. Cisco NetFlow coupled with a behavior analysis tool can help identify when Null Scans are occurring on your network.

For the past couple of years, I’ve attained the reputation for playing practical jokes on my fellow employees. Last year, I pulled an epic support call prank, so this year I thought it’d be better to chill out. Since I let my guard down, I fell victim to this year’s prank, and I believe it was revenge for last year.

Last April 1st, I decided that I was going to have a member of our support team call a zoo and ask to speak to the animals. I came into work early and took 20 minutes calling every major zoo in the country starting on the east coast. I was listening for the best automated prompt to catch someone off guard.

“Hello, you have reached the San Diego Zoo. If you know your party’s extension, please dial it now. Otherwise press 0 to speak to the operator.”

Perfect. It’s quick and to the point. It gives my victim an out, since the message declares that it is in fact a zoo. The research for my prank is done. All I needed to do was to wait three hours for the operator to show up for work.

Around noon (9 a.m. PST), I took a walk around the office to see if anyone had gone to lunch. Raul was missing. I quickly scribbled a note, left it on his desk and disappeared.

The note read:

“Mr. Lyon called… if not available ask for Mr. Behr 619-231-1515″

Now, I had given some huge clues that this might be a prank, even though I disguised the clues with “lyon” and “behr”.

When I returned from lunch, I immediately checked on the status of my prank. It was a success. Raul had placed the call and the operator eventually pointed that it might be a prank, since he just called a zoo and asked to speak to the animals.

This was a harmless prank and everyone got a laugh. Yesterday, Raul got the last laugh.

I had originally decided I was going to work from home, but at the last minute I remembered I needed to be in the office. So I drove in and parked my car, backing into the parking space (something I don’t normally do). This worked to Raul’s advantage.

At the end of the day, I drove off, thinking I had gone through the day without being pranked. I hadn’t even left the parking lot before I heard the chime from my gas tank. “Right, I need gas.” Luckily there is a gas station in my sight, so I pulled over to fuel up. As I’m pumping gas, I noticed something hanging from the back of the Jeep, a Pepsi can attached to the frame with some duct tape and CAT5 cable.

I think this is a declaration of an office prank war. Please post any new ideas as comments. I will post updates to this as the pranks unfold.

As soon as football season ends, everyone looks for the next great excitement. Some take a warm vacation mid-February to break up the winter, or maybe look forward to Mardi Gras. Whatever that great excitement is, it is inevitable that many will find March Madness. March Madness is one of the most exciting times of the year. Whether you fill out a bracket and join an office pool or catch a few minutes in a crowded bar, this annual event will affect you in some way.

For those of us in the corporate world, it may affect us more than desired. It will distract us as fellow employees discuss the games and talk about their brackets. It may also put a pinch on the network, hampering our ability to do our jobs. For the past few years it is estimated that the cost of lost worker productivity due to employees watching games, filling out brackets, and talking trash, is in the billions of dollars. Last year research firm Challenger, Gray & Christmas estimated that the 2008 tournament would contribute a productivity loss of $1.7 billion. This year CEO John Challenger of Challenger, Gray & Christmas stated that there will undoubtedly be an impact of productivity loss, but has determined that effects are immeasurable.

Akamai Technologies provides a distributed solution that helps many media companies meet the demands of their client base. Back in August 2005, Akamai began keeping track of online media consumption around the world and correlated significant news events to highest peak bandwidth consumption. Five of the top 15 events are March Madness-related, with the third-highest being Day One of the 2008 tournament. With these types of trends, Day One of March Madness 2009 will likely top this list as well.

With the U.S. economy as frail as ever, can it endure losing billions in wasted dollars to this event? Probably not! With more and more layoffs occurring every day, employees should know they need to be focused on their work or they could be the next out the door. Maybe it’s time to help employees become more focused at work by updating or creating an Internet usage policy.

As a field engineer I’ve visited dozens of clients and many do not have a Web filter in place. They simply can’t afford the cost to deploy a filter or haven’t prioritized the issue of Internet misuse on their network. With March Madness minutes away, it may be time to start watching Internet use to save money on bandwidth and lost worker productivity.

As network engineers around the globe are experiencing an ever-shrinking budget and no time to deploy a Web filter, what are their options? They are most likely thinking, “I can revise my Internet usage policy, but how am I going to enforce it? How can I see what my users are doing without sniffing the traffic?” One very feasible option that is available on almost every corporate network is Cisco NetFlow.

NetFlow has been around for over a decade and is available on most Cisco routers. NetFlow will give you 100% visibility into the traffic on your network and is very easy to set up. There are many NetFlow vendors out there, but only a few that can provide a free solution.

There is plenty of documentation available for enabling NetFlow, along with free software solutions for analysis for use on your existing networking equipment. There is little excuse for companies to not put together a stronger usage policy in hopes of putting a stop to, or at least a dent in, the massive productivity loss during this annual event.

NetFlow analysis can be a great tool in helping define an Internet usage policy. It can show you that there may be streaming media on your network. It can help show how much YouTube traffic is on your network as well . Check out a sample Internet usage policy that has been shaped using NetFlow analysis.

Last Wednesday started like most days, tackling the project list. While multi-tasking I began putting final touches on my documentation for 3rd Party integration with Zenoss. At the same time I was testing the new web server I recently built which would serve as a replacement for our current server. It was just about noon when I needed to test reboot functionality and I issued a reboot command to our current web server (accidently). I figured it would take 5 or so minutes to reboot, so I waited around and set a “ping -t www.plixer.com”…

The server wasn’t coming up…

This was about the time my heart started racing. The corporate site was down and I didn’t know why. I grabbed my cell phone and numbers for the NOC at the Time Warner Data Center, where our server is hosted. Suddenly remembering that I dropped my Jeep off that morning to have new speakers installed. I borrow a friend’s car and drive up to Time Warner. By the time I get there 45 minutes have passed and I have a bad feeling things aren’t going to go smoothly.The stress builds.

I hook the server up to a KVM and sure enough, my server has crashed with a fatal kernel error. The server wasn’t going back online any time soon. Luckily I have a backup server, for just this purpose, with most of the websites up and running. I call in a quick IP change and www.plixer.com is back up. This blog, however, was running on a slightly outdated backup and needs updating immediately. I race back to the office with my old dead server in hand. There is a kernel error and the server just won’t boot. Since our websites are up, I just need to pull the latest copy off of the dead server. This is where the live linux distro comes into play.

I have always walked around with a live CD in my bag whether it’s Knoppix, Ubuntu, or openSuSE. In the past I’ve used a live CD to save files from a corrupt windows servers and workstations, but never a Linux server. This was the first time, and I stumbled my way through it. With the live distro I was able to mount the disk, tar up needed directories, and sftp them up to the new server. Phew! All is well.

Since I struggled through this I might as well lay out exactly what I did to get the job done in high hopes that it will help someone else down the road.

First of all you need a live CD. Download a live distro and burn to CD; in this example I used openSuSE 11.0 Live CD.

Boot the server to the live CD. In my experience it will automatically pick up an IP address from our DHCP server, this is great!

I start looking around to see if the hard disk mounted automatically, it did not. I try a few commands to mount sda, sda1, hda, etc… they all fail. I don’t know the name of the physical disk to mount. In an attempt to find out what the physical disk is called, I launch Yast and click on Hardware.

Now click on Hardware Information to find disk information, it will scan your server looking for all installed hardware. Once the inventory is returned, expand out Disk and you should see any physical disks you have. Found it, /dev/sda2

Now I need to mount the drive. To do this, launch xterm and run

su root

This will give you privilege to mount the drive. Now run:

mount /dev/sda2 /mnt

This command mounts the physical disk to /mnt.

At this point you can cd to /mnt and run ls to see your file system.

Now that the physical disk has been mounted and access is granted you will need to tar up any directories you want to save. It is very useful to tar a directory because all files and subdirectories  will tar recursively unless you specify otherwise. The nice part about tarring a directory is that it is untarred as a directory rather than as individual files.

I wanted to grab the latest copy of our blog. To tar up the blog directory I ran:

tar cvzf blog.tgz www.plixer.com/blog/

use sftp to upload or move your tar file and to extract the contents of your tarball run:

tar xvzf blog.tgz

With the site extracted, everything is back up and running. My live linux distro saves the day again.

Hi Everyone,

After coming off almost 2 weeks vacation with plenty of rest  and relaxation it’s time to dive back into everything NetFlow, the blood that runs through this company. Before getting immersed in too much work check out this video of me and others celebrating a new years tradition of  ice swimming, otherwise known as the Petit Polar Plunge. The air temperature was 10 degrees without the windchill, brrrr.