Symptoms:
One of Cisco7606 Routers got a software crash during booting process. Here is output from console.
System Bootstrap, Version 12.2(33r)SRD5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2009 by cisco Systems, Inc.
C7600-RSP720/SP platform with 1048576 Kbytes of main memory
Autoboot executing command: “boot bootdisk:c7600rsp72043-advipservicesk9-mz.122-33.SRD3.bin”
Initializing ATA monitor library…
Self extracting the image… [OK]
Self decompressing the image : ################################################################################################################################################################################################################################# [OK]
Restricted Rights Legend
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software – Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706
Cisco IOS Software, c7600rsp72043_sp Software (c7600rsp72043_sp-ADVIPSERVICESK9-M), Version 12.2(33)SRD3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 10-Sep-09 12:50 by prod_rel_team
Image text-base: 0x0800015C, data-base: 0x0C000000
*Aug 27 13:13:43.659: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.
*Aug 27 13:13:41.851: %PFREDUN-6-ACTIVE: Initializing as ACTIVE processor
*Aug 27 13:13:43.659: %OIR-SP-6-CONSOLE: Changing console ownership to route processor
System Bootstrap, Version 12.2(33r)SRD5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2009 by cisco Systems, Inc.
Warning: monitor nvram area is corrupt … using default values
C7600-RSP720/RP platform with 2097152 Kbytes of main memory
Download Start
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Download Completed! Booting the image.
Self decompressing the image : ##################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################### [OK]
Restricted Rights Legend
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software – Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706
Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRD3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 10-Sep-09 12:19 by prod_rel_team
Image text-base: 0x08000224, data-base: 0x10000000
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
[email protected].
Cisco CISCO7606-S (M8500) processor (revision 1.1) with 1835008K/131072K bytes of memory.
Processor board ID FOX1346GM8T
BASEBOARD: RSP720
CPU: MPC8548_E, Version: 2.0, (0x80390020)
CORE: E500, Version: 2.0, (0x80210020)
CPU:1200MHz, CCB:400MHz, DDR:200MHz,
L1: D-cache 32 kB enabled
I-cache 32 kB enabled
Last reset from power-on
1 SIP-200 controller .
1 Virtual Ethernet interface
50 Gigabit Ethernet interfaces
3964K bytes of non-volatile configuration memory.
507024K bytes of Internal ATA PCMCIA card (Sector size 512 bytes).
Press RETURN to get started!
*Aug 27 13:14:26.503: RP: Currently running ROMMON from S (Gold) region
*Aug 27 13:14:26.847: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Aug 27 08:14:30: %SYS-6-CLOCKUPDATE: System clock has been updated from 13:14:30 UTC Thu Aug 27 2015 to 08:14:30 EDT Thu Aug 27 2015, configured from console by console.
*Aug 27 09:14:30: %SYS-6-CLOCKUPDATE: System clock has been updated from 08:14:30 EDT Thu Aug 27 2015 to 09:14:30 EDT Thu Aug 27 2015, configured from console by console.
*Aug 27 09:14:32: %RTT-4-OPER_TIMEOUT: condition occurred, entry number = 10
*Aug 27 09:14:32: %SYS-5-CONFIG_I: Configured from memory by console
*Aug 27 09:14:34: %SYS-5-RESTART: System restarted —
Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVIPSERVICESK9-M), Version 12.2(33)SRD3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 10-Sep-09 12:19 by prod_rel_team
*Aug 27 09:14:34: %SSH-5-ENABLED: SSH 2.0 has been enabled
*Aug 27 09:14:34: %NTP-6-RESTART: NTP process starts
*Aug 27 13:13:43.975: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.
Firmware compiled 23-Jul-09 11:21 by integ Build [100]
*Aug 27 13:14:24.003: %SPANTREE-SP-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Aug 27 13:14:24.207: SP: SP: Currently running ROMMON from S (Gold) region
*Aug 27 13:14:32.119: %SW_VLAN-SP-6-VTP_DOMAIN_NAME_CHG: VTP domain name changed to CDMA_PMI.
*Aug 27 09:14:34: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 172.16.100.249 port 514 started – CLI initiated
*Aug 27 13:14:34.119: %SYS-SP-5-RESTART: System restarted —
Cisco IOS Software, c7600rsp72043_sp Software (c7600rsp72043_sp-ADVIPSERVICESK9-M), Version 12.2(33)SRD3, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Thu 10-Sep-09 12:50 by prod_rel_team
*Aug 27 13:14:36: %OIR-SP-6-INSPS: Power supply inserted in slot 1
*Aug 27 13:14:36: %C7600_PWR-SP-2-PSFAIL: power supply 1 output failed.
*Aug 27 13:14:36: %C7600_POWER-SP-4-GLITCH: Recovered from glitch in system power supply.
*Aug 27 13:14:36: %OIR-SP-6-INSPS: Power supply inserted in slot 2
*Aug 27 09:14:36: %SNMP-5-CHASSISALARM: Chassis Alarm Trap: tmpAlarm(OFF), minorAlarm(ON), and/or majorAlarm(OFF)
*Aug 27 13:14:36: %C7600_PWR-SP-4-PSOK: power supply 2 turned on.
*Aug 27 09:14:39: %NTP-6-PEERREACH: Peer 127.127.7.1 is reachable
Aug 27 09:14:39: %NTP-5-PEERSYNC: NTP synced to peer 127.127.7.1
Aug 27 09:14:40: %SNMP-5-COLDSTART: SNMP agent on host Horner-C7606A is undergoing a cold start
*Aug 27 09:14:39: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 5.
*Aug 27 09:14:40: %FABRIC-SP-5-FABRIC_MODULE_ACTIVE: The Switch Fabric Module in slot 5 became active.
%Software-forced reload
09:14:44 EDT Thu Aug 27 2015: Unexpected exception to CPU: vector 1500, PC = 0xA7E60E4 , LR = 0xA7E6084
-Traceback= A7E60E4 A7E6084 A4EE088 A380674 A3C18D8 A3C1944 892CC54 892DE20 892EC00 A278880 A279000 A82F640 A82F9B8 A831120 A7DBEAC A883B54
CPU Register Context:
MSR = 0x00029200 CR = 0x24444024 CTR = 0x0A3CCB14 XER = 0x00000000
R0 = 0x0A7E6084 R1 = 0x17ECF058 R2 = 0xFFFCFFFC R3 = 0x18710E64
R4 = 0xFFFFFFFE R5 = 0x00000000 R6 = 0x17ECF030 R7 = 0x100A0000
R8 = 0x00029200 R9 = 0x00000000 R10 = 0x111DA034 R11 = 0x18710E60
R12 = 0xA0000000 R13 = 0x04044000 R14 = 0x00000000 R15 = 0x12DB0000
R16 = 0x12DB49A4 R17 = 0x12DB48B8 R18 = 0x00000000 R19 = 0x79EB1D50
R20 = 0x00000000 R21 = 0x184FF95C R22 = 0x00000000 R23 = 0x000000FF
R24 = 0x1872D21C R25 = 0x00000040 R26 = 0x13DF6EEC R27 = 0x0000001F
R28 = 0x00000001 R29 = 0x103F0000 R30 = 0x13DF6EEC R31 = 0x00000000
Writing crashinfo to bootdisk:crashinfo_20150827-091444-EDT
1073 Unused bytes of context save space
*** System received a Software forced crash ***
signal= 0x17, code= 0x150e
*Aug 27 09:14:45: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.
*Aug 27 09:14:43: %EARL-SP-2-PATCH_INVOCATION_LIMIT: 10 Recovery patch invocations in the last 30 secs have been attempted. Max limit reached
*Aug 27 09:14:45: %OIR-SP-6-CONSOLE: Changing console ownership to switch processor
*** System received a Software forced crash ***
signal= 0x17, code= 0x1500, context= 0xcf6d9ec
PC = 0x82b1ec0, Vector = 0x1500, SP = 0x14c86ff0
Soulution:
It seems RSP module got an issue. Fortunately, there is a standby RSP720 in warehouse. After replaced RSP720, it booted normally.
Route switch processor (RSP) is a module that is installed in one of the card slots in the router. The RSP provides switching and local and remote management for the router and also contains the uplink ports for the router. Both types of modules (supervisor engine and RSP) perform the same functions in the router.
 |
|
RSP720-3CXL-GE (RSP720 with Integrated Switch Fabric/3CXL-GE)
|
 |
| Front Panel of RSP720-3CXL-GE Explanation |
 |
| Cisco Router 7206 with RSP720-3CXL-GE |
Two Gigabit Ethernet uplink ports: port 1 supports a 1-Gbps SFP module; port 2 is configurable with either a 1-Gbps SFP module or a 10/100/1000-Mbps RJ-45 connector
Route processor (RP): 1- to 4-GB DRAM (default 2 GB)Reference:
Cisco 7600 Series Route Switch Processor 720 Data Sheet