Symptoms
Our network monitoring software found memory usage on some new production switches keep increasing. Those switches are Cisco 2960X and coming with 15.0(2) EX3 IOS.
As we know there are two types of memory in Cisco IOS: process memory and IO memory.
- When a feature is enable on an IOS device (e.g. PIM, HSRP, and etc), IOS allocates process memory for the process.
- IO memory is used when a software switched traffic hits the CPU. The CPU allocates IO memory to store the frame temporarily.
Our case is process memory increasing. What we need to do is to find out which process.
Switch1# show version
Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.0(2)EX3, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Wed 11-Sep-13 02:04 by prod_rel_team
ROM: Bootstrap program is C2960X boot loader
BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.0(2r)EX, RELEASE SOFTWARE (fc1)
CSWP3024 uptime is 5 days, 23 hours, 48 minutes
System returned to ROM by power-on
System restarted at 05:47:41 EST Sat Mar 11 2017
System image file is "flash:/c2960x-universalk9-mz.150-2.EX3/c2960x-universalk9-mz.150-2.EX3.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
[email protected].
cisco WS-C2960X-48TS-LL (APM86ZZZ) processor (revision B0) with 262144K bytes of memory.
Processor board ID FOC1744W3V5
Last reset from power-on
2 Virtual Ethernet interfaces
1 FastEthernet interface
50 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 5C:A4:8A:5C:6F:80
Motherboard assembly number : 73-14196-03
Power supply part number : 341-0537-02
Motherboard serial number : FOC17443CWD
Power supply serial number : LIT17320EU6
Model revision number : B0
Motherboard revision number : B0
Model number : WS-C2960X-48TS-LL
Daughterboard assembly number : 73-14200-03
Daughterboard serial number : FOC17440VRF
System serial number : FOC1744W3V5
Top Assembly Part Number : 800-38011-02
Top Assembly Revision Number : B0
Version ID : V02
CLEI Code Number : CMMNG00ARA
Daughterboard revision number : A0
Hardware Board Revision Number : 0x05
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 50 WS-C2960X-48TS-LL 15.0(2)EX3 C2960X-UNIVERSALK9-M
Configuration register is 0xF
Switch1#show memory statistics history ------------------ History of Processor Mempool ------------------ 777777777777777777777777777777777777777777777777777777777777 222222222222222222222222222222222222222222222222222222222222 100 90 80 70 ********************************************************** 60 ********************************************************** 50 ********************************************************** 40 ********************************************************** 30 ********************************************************** 20 ********************************************************** 10 ********************************************************** 0....5....1....1....2....2....3....3....4....4....5....5....6 0 5 0 5 0 5 0 5 0 5 0 Free memory per second (last 60 seconds) 777777777777777777777777777777777777777777777777777777777777 222222222222222222222222222222222222222222222222222222222222 100 90 80 70 ########################################################## 60 ########################################################## 50 ########################################################## 40 ########################################################## 30 ########################################################## 20 ########################################################## 10 ########################################################## 0....5....1....1....2....2....3....3....4....4....5....5....6 0 5 0 5 0 5 0 5 0 5 0 Free memory per minute (last 60 minutes) * = maximum # = average 777777777777777777777777777777777777777777777777777777777777777777777777 222222222233333333333333444444444444444555555555555555666666666666667777 100 90 80 **############################# 70 ###################################################################### 60 ###################################################################### 50 ###################################################################### 40 ###################################################################### 30 ###################################################################### 20 ###################################################################### 10 ###################################################################### 0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.. 0 5 0 5 0 5 0 5 0 5 0 5 0 Free memory per hour (last 72 hours) * = maximum # = average ------------------ History of I/O Mempool ------------------ 555555555555555555555555555555555555555555555555555555555555 888888888888888888888888888888888888888888888888888888888888 100 90 80 70 60 ********************************************************** 50 ********************************************************** 40 ********************************************************** 30 ********************************************************** 20 ********************************************************** 10 ********************************************************** 0....5....1....1....2....2....3....3....4....4....5....5....6 0 5 0 5 0 5 0 5 0 5 0 Free memory per second (last 60 seconds) 555555555555555555555555555555555555555555555555555555555555 888888888888888888888888888888888888888888888888888888888888 100 90 80 70 60 ########################################################## 50 ########################################################## 40 ########################################################## 30 ########################################################## 20 ########################################################## 10 ########################################################## 0....5....1....1....2....2....3....3....4....4....5....5....6 0 5 0 5 0 5 0 5 0 5 0 Free memory per minute (last 60 minutes) * = maximum # = average 555555555555555555555555555555555555555555555555555555555555555555555555 888888888888888888888888888888888888888888888888888888888888888888888888 100 90 80 70 60 ###################################################################### 50 ###################################################################### 40 ###################################################################### 30 ###################################################################### 20 ###################################################################### 10 ###################################################################### 0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.. 0 5 0 5 0 5 0 5 0 5 0 5 0 Free memory per hour (last 72 hours) * = maximum # = average
From our management software , the free memory diagram shows it is continuous reducing until whole system lost responding .
 |
| Free Switch Memory Going Down in Monitoring Tools |
Troubleshooting
There are two main reasons why a switch runs out of memory.
1. A process does not free up the memory after the memory is no longer needed. This is generally known as memory leak.
2. A process does not limit the amount of memory it allocates. Eventually, the process uses up all process memory on the switch.
The goal is to identify the misbehaving process. There are two approaches.
- Identify any process leaking memory
Show process memory debug leak
Notes: This command only to be run during maintenance window or when the device is not in production .This command is handy to identify any process leaking memory. However, it does not help if the memory problem is caused by a process grabs all memory
- Identify a process grabs all memory
Potential fix:
After googled and I did find many related Cisco bug and suggestions.
1. Cisco Bug: CSCut06609 – Memory leak seen in Auth Manager process- leaks from ASP component (Not our case)
https://quickview.cloudapps.cisco.com/quickview/bug/CSCut06609
Product
Known Affected Releases
Description (partial)
Symptom: ++ Very slow memory leak in "Auth Manager" process seen on 2960s/2960x switches running 15.0(2)SE5. ++ Everyday we see about 64KB leak in Auth Manager process. ++ Leak is seen on a setup having no "Auth session" present (Only auth hidden sessions) and no ports running MAB or dot1X. It is not same as CSCuq53377. ++ The leak is seen in "Auth Manager" process but it is actually being caused by ASP component. Conditions: "macro auto monitor" need to be enabled globally, it could be seen even when no switch port is running MAB/dot1x.
The Fix below fixed all out problems so far:
Apparently, in the IOS there’s an “Auth manager” that can monitor all
sessions in a switch. Starting from the 15.0 IOS stream this feature is
enabled by default. As the bug describes:
“Auth Manager continues to hold more memory in Processor Pool locking
out access to the 3750 stack unless switch is rebooted”
To prevent this ‘leakage’ of memory to the processor, you have to
disable this session monitoring by issuing the following command:
no macro auto monitor
Tried this, but did not help in this case.
2. Cisco Bug: CSCut92215 – Memory Leak in hrpc_incoming_request_handler_process – 15.0(2)EX5 (Not our case)
https://quickview.cloudapps.cisco.com/quickview/bug/CSCut92215
Last Modified
Product
Known Affected Releases
Description (partial)
Symptom:
Show process memory sorted will display MallocLite holding memory but no freeing it
SW#show process memory sorted
Processor Pool Total: ZZZZZZXX Used: ZZZZZZXX Free: ZZZZZZX
I/O Pool Total: ZZZZZZXX Used: ZZZZZZXX Free: ZZZZZZX
Driver te Pool Total: ZZZZZZX Used: XX Free: ZZZZZZX
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 ZZZZZZXX ZZZZZZXX ZZZZZZXX 0 0 *Init*
0 0 X 0 ZZZZZZXX 0 0 *MallocLite*
//////////////////
show mem lite-chunk statistics will show a large disparity between Inuse and Free
SW#show mem lite-chunk statistics
Pool Inuse Free
8-Bytes ZZZ ZZZX
20-Bytes ZZZZZZ ZZZX <<<<<<
44-Bytes ZZZX ZZZ
68-Bytes ZZZ ZZZ
96-Bytes ZZZ ZZZ
128-Bytes XX ZZZ
Conditions:
SW running 15.0(2)EX5
3. SNMP disable (Not our case)
– no macro auto monitor
– no cdp running
– snmp completely disabled
4. Memory leak with hulc_flash init process (Our case)
Switch1#show processes memory sorted | i Total
Processor Pool Total: 174442388 Used: 50534880 Free: 123907508
I/O Pool Total: 16777216 Used: 6924400 Free: 9852816
Driver te Pool Total: 1048576 Used: 40 Free: 1048536
57447240 Total
Switch1#show processes memory sorted | i Total
Processor Pool Total: 174442388 Used: 50661608 Free: 123780780
I/O Pool Total: 16777216 Used: 6924400 Free: 9852816
Driver te Pool Total: 1048576 Used: 40 Free: 1048536
57573808 Total
You can see Total processor Pool Free memory is decreasing, Free I/O and Driver te Pool memory keep same.
Check more details from sorted processes memory command, *Dead* process memory is increasing by comparing the holding memory data with the data collected 5 minutes ago.
Switch1#show processes memory sorted
Processor Pool Total: 174442388 Used: 50519544 Free: 123922844
I/O Pool Total: 16777216 Used: 6924400 Free: 9852816
Driver te Pool Total: 1048576 Used: 40 Free: 1048536
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 59344172 24818044 31046036 0 0 *Init*
0 0 0 0 19543880 0 0 *MallocLite*
0 0 40461580 39067064 1530300 6881007 949166 *Dead*
87 0 715684 146228 451160 0 0 Stack Mgr Notifi
289 0 267228 0 277388 100548 0 EEM ED Syslog
1 0 234876 36256 205780 0 0 Chunk Manager
68 0 206732 232 188500 0 0 USB Startup
205 0 1520792 1070688 154044 0 0 802.1x switch
65 0 108532 400 118292 0 0 EEM ED Identity
180 0 153020 8916 109604 0 0 DHCPD Receive
102 0 93356 212 100304 53760 0 HRPC emac reques
153 1 3418792 3336728 94288 0 0 SSH Process
300 0 70460 0 77620 0 0 EEM Server
178 0 13629416 300 75732 802008 0 CDP Protocol
138 0 65812 232 72740 0 0 REP Helper Proc
53 0 108644 84688 71160 0 0 IF-MGR control p
175 0 530220 373064 62440 0 0 Auth Manager
128 0 232 2204 61160 0 0 HACL Acl Manager
159 0 499433728 499416280 61160 10152 0 HRPC qos request
173 0 49364 0 56524 0 0 ACCT Periodic Pr
280 0 237076 202296 48632 0 0 Crypto CA
21 0 48776 64728 45920 0 9620 Entity MIB API
17 0 211669728 211633028 43860 0 0 ARP Input
189 0 33020 0 40180 0 0 IP ARP Retry Age
297 0 12408 0 37440 0 0 LICENSE AGENT
230 0 50872 9248 34396 0 0 CEF: IPv4 proces
8 0 1547520 3356300 32952 1401600 2400103 Pool Manager
Switch1#show processes memory sorted
Processor Pool Total: 174442388 Used: 50659912 Free: 123782476
I/O Pool Total: 16777216 Used: 6924400 Free: 9852816
Driver te Pool Total: 1048576 Used: 40 Free: 1048536
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 59344172 24818044 31046036 0 0 *Init*
0 0 0 0 19675040 0 0 *MallocLite*
0 0 44037920 42642260 1531252 6881007 949166 *Dead*
87 0 715684 146228 451160 0 0 Stack Mgr Notifi
289 0 267228 0 277388 100548 0 EEM ED Syslog
1 0 234876 36256 205780 0 0 Chunk Manager
68 0 206732 232 188500 0 0 USB Startup
205 0 1520792 1070688 154044 0 0 802.1x switch
65 0 108532 400 118292 0 0 EEM ED Identity
180 0 153020 8916 109604 0 0 DHCPD Receive
102 0 93356 212 100304 53760 0 HRPC emac reques
153 1 193080 111648 93592 0 0 SSH Process
300 0 70460 0 77620 0 0 EEM Server
178 0 13658484 300 75732 802008 0 CDP Protocol
138 0 65812 232 72740 0 0 REP Helper Proc
53 0 108644 84688 71160 0 0 IF-MGR control p
175 0 530220 373064 62440 0 0 Auth Manager
159 0 510696628 510679180 61160 10152 0 HRPC qos request
128 0 232 2204 61160 0 0 HACL Acl Manager
173 0 49364 0 56524 0 0 ACCT Periodic Pr
280 0 237076 202296 48632 0 0 Crypto CA
21 0 48776 64728 45920 0 9620 Entity MIB API
17 0 212097396 212060696 43860 0 0 ARP Input
189 0 33020 0 40180 0 0 IP ARP Retry Age
297 0 12408 0 37440 0 0 LICENSE AGENT
230 0 50872 9248 34396 0 0 CEF: IPv4 proces
112 0 4349612 232 33052 0 0 HLFM address lea
8 0 1547520 3356300 32952 1401600 2400103 Pool Manager
321 0 288460 157320 31420 0 0 SNMP Traps
199 0 5724 0 30884 0 0 SMI Backup Proce
197 0 3160 592 28360 0 0 SMI Director DB
60 0 74608 52800 27056 0 0 Per-Second Jobs
306 0 232 0 25392 0 0 tHUB
From the command ‘show memory dead totals’ output showed there were many instances of hulc_flash init.
Switch1#show memory dead totals
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 49B2FA0 174442388 50508688 123933700 123791240 103785944
I/O E000000 16777216 6924444 9852772 9736080 9805164
Driver te 2B00000 1048576 44 1048532 1048532 1048532
Dead Proc Summary for: Processor
PC Total Count Name
0x020CC35C 942068 47 mifs NOR Sector
0x0036E178 201252 20 SSH Process
0x020D35AC 100220 5 mifs sector buffers
0x01B31C10 65580 1 AAA Large Chunk
0x0017E2D4 64696 517 hulc_flash init
0x020D6A08 25220 5 mifs cache_entry buffers
0x020CC91C 20044 1 mifs NOR Sector Info Pointers
0x0204DE2C 10576 1 master cfg mgr init process
0x02050388 10548 1 HTTP SSL PKI CERTIFICATE CREATE PROC
0x0099C134 9756 1 EEM Shell Director
0x020CC974 8656 10 hulc_flash init
0x0021CBE8 7316 54 Smartport macro autoexec subblock
0x02304ED0 6888 51 LLDP-MED local annex
0x020B0788 6188 1 hulc_flash init
0x01B5F2DC 4652 1 Watcher Info
0x022E3640 4100 52 LIM Index
0x014829EC 3716 3 OBFL UPTIME obfl0
0x020D3DE0 2780 5 hulc_flash init
0x020D3684 2720 5 mifs file_map buffers
0x02883318 2352 7 platform OBFL
0x00477224 2196 1 HTTP SSL PKI CERTIFICATE CREATE PROC
0x004ADE94 2124 1 HTTP SSL PKI CERTIFICATE CREATE PROC
0x020D3618 2040 5 mifs file_info buffers
0x014618C4 1796 5 OBFL UPTIME obfl0
0x020C7A68 1708 24 hulc_flash init
0x020C79F8 1668 24 hulc_flash init
0x020C79BC 1632 24 hulc_flash init
0x01B5E85C 1544 1 Watched Semapho
0x005C40A8 1460 4 HTTP SSL PKI CERTIFICATE CREATE PROC
0x020D6928 1020 15 hulc_flash init
0x020D3DA4 980 5 hulc_flash init
0x020CCA94 460 5 hulc_flash init
0x004ADD80 444 1 HTTP SSL PKI CERTIFICATE CREATE PROC
0x020CCB3C 400 5 hulc_flash init
0x020D08E0 340 5 hulc_flash init
0x020D6C40 340 5 hulc_flash init
0x000730D4 308 1 platform OBFL
0x000739B0 308 1 platform OBFL
0x00073814 308 1 platform OBFL
0x00073B14 308 1 platform OBFL
0x000732A4 308 1 platform OBFL
0x00072E5C 308 1 platform OBFL
0x00343B40 276 3 TTY timer block
0x0045A4EC 232 1 HTTP SSL PKI CERTIFICATE CREATE PROC
0x01489A94 216 1 OBFL Infra Device info
0x01B31A48 208 1 AAA Large Chunk
0x00C1869C 144 1 HTTP
0x003877F0 136 1 Parser Mode
0x020B0734 112 1 hulc_flash init
0x00BF1E80 104 1 HTTP
0x00BFE2FC 100 1 HTTP
0x0059FE88 92 1 crypto_handle_table
0x00BF1B6C 84 1 HTTP
0x020B0808 80 1 hulc_flash init
0x00363AB4 76 1 Parser Linkage
0x002FD83C 68 1 AAA MI SG NAME
0x0038785C 68 1 Parser Mode Q1
0x00387880 68 1 Parser Mode Q2
Dead Proc Summary for: I/O
PC Total Count Name
Dead Proc Summary for: Driver text
PC Total Count Name
Switch1#show memory debug leak
Adding blocks for GD...
Driver text memory
Address Size Alloc_pc PID Alloc-Proc Name
I/O memory
Address Size Alloc_pc PID Alloc-Proc Name
Processor memory
Address Size Alloc_pc PID Alloc-Proc Name
Switch1#
Unfortunately, ‘show memory debug leaks’ did not find any process having leak issue.
I were able to find a post from High Memory Utilization due to hulc_flash to mention CSCth60511 and CSCua52463 are relating to this leak. It seems upgrade will fix this issue, but he did not post if issue resolved after upgraded.
Here is what I searched from Cisco Bug Search Website and there are three related bugs:
 |
| Cisco Bug Search Tool |
 |
| Results from Cisco Bug Search Tool |
4.1 CSCth60511 – Memory leak seen at hulc_flash init process
Upon bootup a switch might leak a small amount of memory for hulc_flash init
Process
Further Info :
Since the leak is small it does not have any functionality impact.
| Product: |
(1)
|
| Known Affected Releases: |
(4)
|
| Known Fixed Releases: |
(23)
|
Unfortunately our devices is 2960x and running on 15.0(2)EX3 , which is different from what this bug says.
4.2 CSCua52463 – C2960 – Memory leak in dead memory – not able to telnet/ssh to switch
Problem observed on multiple switches.
arnsw0034#sh mem dead
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 25D8FA0 23727744 19629156 4098588 3362220 4034820
I/O 2C00000 4194304 2473496 1720808 1646820 1609296
Driver te 1800000 1048576 44 1048532 1048532 1048532
———–
arnsw0034#sh proc mem sor
Processor Pool Total: 23727744 Used: 19628532 Free: 4099212
I/O Pool Total: 4194304 Used: 2473356 Free: 1720948
Driver te Pool Total: 1048576 Used: 40 Free: 1048536
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 22527504 4649688 14708156 0 0 *Init*
0 0 3014844 1663092 1290864 3804375 1341026 *Dead*
150 0 14687348 8409544 659316 2268 0 Auth Manager
71 0 736300 143256 472556 0 0 Stack Mgr Notifi
0 0 0 0 394484 0 0 *MallocLite*
157 0 25864596 43517164 335060 0 0 EAP Framework
179 0 2289784 23580 261832 0 0 CDP Protocol
57 0 198956 1686364 208516 0 0 EEM ED ND
———–
From “show mem dead” we can see one process continuously increasing:
…
026A5598 0000000056 026A5538 026A55F8 001 ——– ——– 000196E0 hulc_flash init
026A55F8 0000000056 026A5598 026A5658 001 ——– ——– 000196E0 hulc_flash init
026A5658 0000000040 026A55F8 026A56A8 001 ——– ——– 005360BC hulc_flash init
026A56A8 0000000028 026A5658 026A56EC 001 ——– ——– 00539ED0 hulc_flash init
026A56EC 0000000044 026A56A8 026A5740 001 ——– ——– 00E4C48C hulc_flash init
…
| Product: |
(104)
|
4.3 CSCua52463 – C2960 – Memory leak in dead memory – not able to telnet/ssh to switch
hulc_flash init and init memory leaks seen during dot1x testing on 3750G
Conditions:
This issue can be seen with or without any dot1x configured on the 3750G switch. Observed this even with the default config.
Workaround:
No workaround exists for this.
Further Problem Description:
| Product: |
(143)
|
Solution
Sadly although we found on bug CSCua52463 seems perfectly matching our case, Cisco does not post any solution and fix for it.
Downgrading to 12.2 or upgrading to 15.2(2)E3 will be our last step to try at this moment. Actually upgrading to 15.2(2)E3 resolved this issue.
Here is the monitoring information from NetMRI. After upgraded IOS to 15.2(2)E3, the free memory is getting stable. It was reducing slowly every minute until upgraded to 15.2(2)E3.
| Type: | Switch (99%) | Vendor: | Cisco |
| O/S Version: | 15.2(2)E3 | Model: | catWsC2960x48tsLL |
| Up Time: | 1d 00h 17m 42s | SNMP Status: | Enabled |
| Last Communication: | 2017-03-27 10:13:30 | MAC Address: | 5C:A4:8A:5C:70:C1 |
| Discovery Blackout: | N/A | Change Blackout: | N/A |
 |
| Free Switch Memory Status in Monitoring Tools |
Reference: