Partial Network Outage on Catalyst WS-C6509-E using dual Sup

From: Mister T (romantic24hrs@gmail.com)
Date: Sat Sep 27 2008 - 04:38:12 ART


Dear expert,

I have problem experiencing partial network outages in my one Catalyst
6509-E. It happens many times in almost 2 months, and when it happens
partial network outage takes about 10 minutes (LAN and WAN traffic down).

I have a suspect that something is not right for the dual supervisor
redundancy (when doing switchover). I also noticed that the standby Sup in
slot 6 shows 'Minor Error'. I went to the 'Error Message Decoder' at
cisco.com to interpret the error messages from the log, but it says: no
action is required. For details i enclose the log from the switch.

------------------ show module ------------------

Mod Ports Card Type Model Serial
No.
--- ----- -------------------------------------- ------------------
-----------
  2 16 SFM-capable 16 port 1000mb GBIC WS-X6516-GBIC -
  3 16 SFM-capable 16 port 1000mb GBIC WS-X6516-GBIC -
  4 16 SFM-capable 16 port 1000mb GBIC WS-X6516A-GBIC -
  5 2 Supervisor Engine 720 (Active) WS-SUP720-3B -
  6 2 Supervisor Engine 720 (Hot) WS-SUP720-3B -
  7 16 SFM-capable 16 port 10/100/1000mb RJ45 WS-X6516-GE-TX -
  8 48 SFM-capable 48 port 10/100/1000mb RJ45 WS-X6548-GE-TX -

Mod MAC addresses Hw Fw Sw
Status
--- ---------------------------------- ------ ------------ ------------
-------
  2 0001.6463.0df8 to 0001.6463.0e07 4.4 7.2(1) 8.5(0.46)RFW Ok
  3 0009.11e3.05a4 to 0009.11e3.05b3 5.0 6.3(1) 8.5(0.46)RFW Ok
  4 0019.aa98.6a20 to 0019.aa98.6a2f 4.5 7.2(1) 8.5(0.46)RFW Ok
  5 0014.a982.4ba0 to 0014.a982.4ba3 5.2 8.4(2) 12.2(18)SXF9 Ok
  6 0017.9441.c344 to 0017.9441.c347 5.2 8.4(2) 12.2(18)SXF9 Ok
  7 001a.6d87.0cf0 to 001a.6d87.0cff 2.8 6.3(1) 8.5(0.46)RFW Ok
  8 0009.7c46.ee4a to 0009.7c46.ee79 10.1 7.2(1) 8.5(0.46)RFW Ok

Mod Sub-Module Model Serial Hw
Status
---- --------------------------- ------------------ ----------- -------
-------
  5 Policy Feature Card 3 WS-F6K-PFC3B - 2.3 Ok
  5 MSFC3 Daughterboard WS-SUP720 - 2.5 Ok
  6 Policy Feature Card 3 WS-F6K-PFC3B - 2.3 Ok
  6 MSFC3 Daughterboard WS-SUP720 - 2.5 Ok

Mod Online Diag Status
---- -------------------
  2 Pass
  3 Pass
  4 Pass
  5 Pass
  6 Minor Error
  7 Pass
  8 Pass

***************************************************
****** Information of Last System Crash - SP ******
***************************************************

Using sup-bootflash:crashinfo_20080915-101604.

Writing crashinfo to bootflash:crashinfo_20080915-101604ot 6, interfaces are
now online

Sep 15 17:02:34.005 WIB: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO
mode
Sep 15 17:02:34.005 WIB: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure
console debugging output.

Sep 15 17:02:34.005 WIB: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO
mode
Sep 15 17:02:34.201 WIB: %SYS-SP-3-LOGGER_FLUSHED: System was paused for
00:00:00 to ensure console debugging output.

Sep 15 17:02:35.525 WIB: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
configuration to the standby Router.
Sep 15 17:03:15.510 WIB: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off
for the fabric in slot 6.
Sep 15 17:03:15.606 WIB: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch
Fabric Module in slot 6 became standby
Sep 15 17:03:16.434 WIB: %DIAG-SP-6-RUN_MINIMUM: Module 6: Running Minimal
Diagnostics...
Sep 15 17:03:17.874 WIB: %DIAG-SP-6-DIAG_OK: Module 6: Passed Online
Diagnostics
Sep 15 17:03:18.114 WIB: %OIR-SP-6-INSCARD: Card inserted in slot 6,
interfaces are now online
Sep 15 17:07:55.371 WIB: %PFREDUN-SP-6-ACTIVE: Standby processor removed or
reloaded, changing to Simplex mode
Sep 15 17:10:07.419 WIB: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO
mode
Sep 15 17:10:07.419 WIB: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure
console debugging output.

Sep 15 17:10:07.419 WIB: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO
mode
Sep 15 17:10:07.611 WIB: %SYS-SP-3-LOGGER_FLUSHED: System was paused for
00:00:00 to ensure console debugging output.

Sep 15 17:10:09.644 WIB: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
configuration to the standby Router.
Sep 15 17:10:50.140 WIB: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off
for the fabric in slot 6.
Sep 15 17:10:50.232 WIB: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch
Fabric Module in slot 6 became standby
Sep 15 17:10:51.560 WIB: %DIAG-SP-6-RUN_MINIMUM: Module 6: Running Minimal
Diagnostics...
Sep 15 17:10:53.020 WIB: %DIAG-SP-6-DIAG_OK: Module 6: Passed Online
Diagnostics
Sep 15 17:10:53.216 WIB: %OIR-SP-6-INSCARD: Card inserted in slot 6,
interfaces are now online
Sep 15 17:15:30.617 WIB: %PFREDUN-SP-6-ACTIVE: Standby processor removed or
reloaded, changing to Simplex mode

Any help to resolve this problem would be appreciated.

Regards

mrt

Blogs and organic groups at http://www.ccie.net



This archive was generated by hypermail 2.1.4 : Sat Oct 04 2008 - 09:26:20 ART