Cisco MAC Address Flapping Causing High CPU Utilization

A MAC flap is caused when a switch receives packets from two different physical/logical interfaces with the same source MAC address. The switch then learns where the MAC address is and puts in to a table. This table has the physical/logical interface and the MAC address. When flapping occurs it causes this table to be updated whenever a packet is sent/received. The more data flowing through the interface that is flapping the higher your CPU Utilization is going to be which can have serious potential negative side effects. Such as dropped packets, laggy terminal session and complete drop of network connectivity.

The following will give you the commands you need to help identify MAC Address Flapping and High CPU Utilization on Cisco Catalyst series switch. This was performed to troubleshoot CPU utilization issues on a Cisco Catalyst 4500 series switch but the same commands should be available to other Cisco switches which run the IOS firmware.

cisco4500#show processes cpu
CPU utilization for five seconds: 38%/1%; one minute: 32%; five minutes: 32%
PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
27         524    250268          2  0.00%  0.00%  0.00%   0 TTY Background
28         816    254843          3  0.00%  0.00%  0.00%   0 Per-Second Jobs
29      101100      5053      20007  0.00%  0.01%  0.00%   0 Per-minute Jobs
30    26057260  26720902        975 12.07% 11.41% 11.36%   0 Cat4k Mgmt HiPri
31    19482908  29413060        662 24.07% 19.32% 19.20%   0 Cat4k Mgmt LoPri
32        4468    162748         27  0.00%  0.00%  0.00%   0 Galios Reschedul

The following will give you a Target CPU percent and the Actual Percent. Look for percents that greatly exceed the Target CPU percent. This will help identify what is eating your processing power on your device. This is used to troubleshoot other items than MAC Address Flapping such as Routing Loops and other bad things that can bring your network to a halt.

cisco4500#show platform health
%CPU   %CPU    RunTimeMax   Priority  Average %CPU  Total
Target Actual Target Actual   Fg   Bg 5Sec Min Hour  CPU
Protocol-aging-revie   0.20   0.00      2      0  100  500    0   0    0  0:01
Acl-Flattener          1.00   0.00     10      5  100  500    0   0    0  0:04
KxAclPathMan create/   1.00   0.00     10      5  100  500    0   0    0  0:21
KxAclPathMan update    2.00   0.00     10      6  100  500    0   0    0  0:05
KxAclPathMan reprogr   1.00   0.00      2      1  100  500    0   0    0  0:00
TagMan-InformMtegRev   1.00   0.00      5      0  100  500    0   0    0  0:00
TagMan-RecreateMtegR   1.00   0.00     10     14  100  500    0   0    0  0:18
K2CpuMan Review       30.00  91.31     30     92  100  500  128 119   84  13039:02
K2AccelPacketMan: Tx  10.00   2.30     20      0  100  500    2   2    2  1345:30
K2AccelPacketMan: Au   0.10   0.00      0      0  100  500    0   0    0  0:00

First enter enabled mode then configure terminal mode. Issue the following command to ensure there is logging for mac-move which will identify MAC Address Flapping.

cisco4500(config)#mac address-table notification mac-move

After a period of time, view the log to identify the MAC address that is flapping.

cisco4500(config)#do show log
Syslog logging: enabled (0 messages dropped, 1 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)
...
*Oct  3 08:51:28.149: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (10.10.10.236)
*Oct  3 09:43:46.437: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/40 to port Gi2/30
*Oct  3 09:43:48.629: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/30 to port Gi2/40
*Oct  3 09:43:48.717: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/40 to port Gi2/30
*Oct  3 09:43:49.581: %C4K_EBM-4-HOSTFLAPPING: Host 00:60:48:1B:01:15 in vlan 400 is moving from port Gi2/30 to port Gi2/40

Furthermore, issue the following command at random periods of time to illustrate the MAC address bouncing between two different physical ports.

cisco4500#sh mac address-table address 00:60:48:1B:01:15

Unicast Entries
vlan   mac address     type        protocols               port
-------+---------------+--------+---------------------+--------------------
400    0060.481b.0115   dynamic ip                    GigabitEthernet2/30

cisco4500#sh mac address-table address 00:60:48:1B:01:15
Unicast Entries
vlan   mac address     type        protocols               port
-------+---------------+--------+---------------------+--------------------
400    0060.481b.0115   dynamic ip                    GigabitEthernet2/40

It depends on how your configuration is but it is generally a good idea to disable one of the two interfaces or fix an issue with a logical interface such as EtherChannel/LACP.

For furhter information and Cisco official documentation, http://www.cisco.com/c/en/us/support/docs/switches/catalyst-4000-series-switches/65591-cat4500-high-cpu.html

Leave a Reply