Cisco MAC Address Flapping Causing High CPU Utilization
A MAC flap is caused when a switch receives packets from two different physical/logical interfaces with the same source MAC address. The switch then learns where the MAC address is and puts in to a table. This table has the physical/logical interface and the MAC address. When flapping occurs it causes this table to be updated whenever a packet is sent/received. The more data flowing through the interface that is flapping the higher your CPU Utilization is going to be which can have serious potential negative side effects. Such as dropped packets, laggy terminal session and complete drop of network connectivity.
The following will give you the commands you need to help identify MAC Address Flapping and High CPU Utilization on Cisco Catalyst series switch. This was performed to troubleshoot CPU utilization issues on a Cisco Catalyst 4500 series switch but the same commands should be available to other Cisco switches which run the IOS firmware.
cisco4500#show processes cpu CPU utilization for five seconds: 38%/1%; one minute: 32%; five minutes: 32% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 27 524 250268 2 0.00% 0.00% 0.00% 0 TTY Background 28 816 254843 3 0.00% 0.00% 0.00% 0 Per-Second Jobs 29 101100 5053 20007 0.00% 0.01% 0.00% 0 Per-minute Jobs 30 26057260 26720902 975 12.07% 11.41% 11.36% 0 Cat4k Mgmt HiPri 31 19482908 29413060 662 24.07% 19.32% 19.20% 0 Cat4k Mgmt LoPri 32 4468 162748 27 0.00% 0.00% 0.00% 0 Galios Reschedul
The following will give you a Target CPU percent and the Actual Percent. Look for percents that greatly exceed the Target CPU percent. This will help identify what is eating your processing power on your device. This is used to troubleshoot other items than MAC Address Flapping such as Routing Loops and other bad things that can bring your network to a halt.
cisco4500#show platform health %CPU %CPU RunTimeMax Priority Average %CPU Total Target Actual Target Actual Fg Bg 5Sec Min Hour CPU Protocol-aging-revie 0.20 0.00 2 0 100 500 0 0 0 0:01 Acl-Flattener 1.00 0.00 10 5 100 500 0 0 0 0:04 KxAclPathMan create/ 1.00 0.00 10 5 100 500 0 0 0 0:21 KxAclPathMan update 2.00 0.00 10 6 100 500 0 0 0 0:05 KxAclPathMan reprogr 1.00 0.00 2 1 100 500 0 0 0 0:00 TagMan-InformMtegRev 1.00 0.00 5 0 100 500 0 0 0 0:00 TagMan-RecreateMtegR 1.00 0.00 10 14 100 500 0 0 0 0:18 K2CpuMan Review 30.00 91.31 30 92 100 500 128 119 84 13039:02 K2AccelPacketMan: Tx 10.00 2.30 20 0 100 500 2 2 2 1345:30 K2AccelPacketMan: Au 0.10 0.00 0 0 100 500 0 0 0 0:00
First enter enabled mode then configure terminal mode. Issue the following command to ensure there is logging for mac-move which will identify MAC Address Flapping.
cisco4500(config)#mac address-table notification mac-move
After a period of time, view the log to identify the MAC address that is flapping.
cisco4500(config)#do show log Syslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled) ... *Oct 3 08:51:28.149: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (10.10.10.236) *Oct 3 09:43:46.437: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/40 to port Gi2/30 *Oct 3 09:43:48.629: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/30 to port Gi2/40 *Oct 3 09:43:48.717: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/40 to port Gi2/30 *Oct 3 09:43:49.581: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/30 to port Gi2/40
Furthermore, issue the following command at random periods of time to illustrate the MAC address bouncing between two different physical ports.
cisco4500#sh mac address-table address 00:60:48:1B:01:15 Unicast Entries vlan mac address type protocols port -------+---------------+--------+---------------------+-------------------- 400 0060.481b.0115 dynamic ip GigabitEthernet2/30 cisco4500#sh mac address-table address 00:60:48:1B:01:15 Unicast Entries vlan mac address type protocols port -------+---------------+--------+---------------------+-------------------- 400 0060.481b.0115 dynamic ip GigabitEthernet2/40
It depends on how your configuration is but it is generally a good idea to disable one of the two interfaces or fix an issue with a logical interface such as EtherChannel/LACP.
For furhter information and Cisco official documentation, http://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/65591-cat4500-high-cpu.html
Leave a Reply