This may be caused by an internal Hash overflow issue with the Nortel hardware.

1. Connect Scrutinizer directly to the ERS node.
2. DO NOT directly connect Scrutinizer to that PR card and use a VLAN IP as the exporter IP. This will likely cause accuracy problems.
3. DO NOT use an out of band connection.
4. If you are not using many of the ports on the card, try spreading the links in 3 lanes, because each lane has a hash table and spreading the links may be helpful for reducing the hash overflow.

The issue may be due to a hash overflow. To check if you do have a hash overflow, do the following:

- show ip ipfix hash-stats

If the value in the second column for “Hash Overflows” is greater than 0, or it keeps increasing every time you issue the “show ip ipfix hash-stats” command, it means you have a hash overflow.

A hash overflow means although the packet was sampled, there is no space to record it, so the system can only drop it. To possibly improve collection and output to Scrutinizer, change the hash-key mask from 1 to 2 (where 1 is the default). Output as seen in Scrutinizer may improve.

NOTE: The Nortel Design Team noticed the following:

-Hash Key 1 ? Use all bits of source IP (lower 20-bits), destination IP (lower 19-bits), Protocol (1-5 bits), Source Port (lower 10 bits), and Destination Port (lower 10 bits).
-Hash Key 2 ? Use lower 20-bits of source IP, lower 19-bits of Destination IP, and lower 24 bits of protocol (8-bits), source port (16-bits) and destination port (16-bits)
-Hash Mask 3 ? Use source IP (32-bits) and destination IP (32-bits)
-Hash Mask 4 ? Use source IP

Hash Key 2 mask will use the second set of hash key building algorithm.

Hash Key 2 : Cascaded Hash – Use all bits from 5-tuple -
hash1 = hash(SIP,DIP)
hash2 = hash(hash1(lower 24 bits,protocol(8 bits),
srcport(16 bits),dstport(16 bits))

It will build a better hash key to avoid hash overflow, and after doing this Nortel found that hash overflow disappeared.

****If you are using the default value 1 for the Hash Key mask, please try changing it to 2 with the following command and see if the results improve:

config ip ipfix port hash-key 2 save config

Contact Nortel support with further questions.

- Joanne

See my next blog, Utilization Understated on Nortel IPFIX capable equipment, part ii, for more information on this subject.

Tags: hash overflow, ipfix, nortel

This entry was posted on Tuesday, January 13th, 2009 at 5:25 pm and is filed under General, Network Health Report, Network Problem Resolution, Scrutinizer. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.