CPU hitting 100% very frequently now.
show proc cpu sorted shows "HLFM address lea"
Any idea?
-Sam
show ver
Cisco IOS Software, C3560 Software (C3560-IPSERVICESK9-M), Version
12.2(58)SE2, RELEASE SOFTWARE (fc1)
System image file is
"flash:c3560-ipservicesk9-mz.122-58.SE2/c3560-ipservicesk9-mz.122-58.SE2.bin"
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 52 WS-C3560G-48TS 12.2(58)SE2 C3560-IPSERVICESK9-M
show proc cpu sort 5min
CPU utilization for five seconds: 58%/34%; one minute: 60%; five minutes:
77%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
84 12001788 2278362 5267 6.40% 6.08% 6.69% 0 HLFM
address lea
141 11412738 2821081 4045 2.24% 2.48% 4.59% 0 Hulc LED
Process
142 6411109 74847 85657 2.71% 3.87% 3.70% 0 HL3U bkgrd
proce
71 2043009 4408879 463 1.91% 1.74% 1.23% 0 Fifo Error
Detec
104 1349035 101712 13263 0.95% 0.77% 0.78% 0 hpm
counter proc
88 824936 10065 81960 0.00% 0.41% 0.46% 0 HULC Tcam
Memory
153 531519 20145 26384 0.47% 0.34% 0.32% 0 HQM Stack
Proces
212 381549 459073 831 0.31% 0.18% 0.29% 0 ADJ
resolve proc
4 438141 25503 17179 2.55% 0.35% 0.24% 0 Check
heaps
353 482354 154998 3112 0.00% 0.01% 0.17% 0 SNMP
ENGINE
117 1363 268 5085 0.15% 0.18% 0.17% 1 SSH
Process
101 242757 829674 292 0.15% 0.14% 0.15% 0 hpm main
process
65 204570 1054533 193 0.15% 0.11% 0.11% 0 RedEarth
Tx Mana
205 172987 2278757 75 0.15% 0.03% 0.11% 0 IP ARP
Retry Age
86 160739 2279030 70 0.00% 0.02% 0.09% 0 HLFM
address ret
66 210809 3388709 62 0.47% 0.13% 0.08% 0 RedEarth
Rx Mana
206 423299 339431 1247 0.31% 0.08% 0.07% 0 IP Input
293 127882 7827 16338 0.00% 0.00% 0.05% 0 crypto sw
pk pro
13 115686 106585 1085 0.00% 0.05% 0.05% 0 ARP
Background
271 68733 258357 266 0.15% 0.06% 0.04% 0 DHCPD
Receive
47 86063 1819469 47 0.00% 0.07% 0.04% 0
DownWhenLooped
63 80343 102231 785 0.15% 0.09% 0.04% 0 Per-Second
Jobs
351 146788 310676 472 0.00% 0.00% 0.04% 0 IP SNMP
276 54451 910495 59 0.00% 0.01% 0.01% 0 MDFS RP
process
154 70259 40270 1744 0.15% 0.05% 0.01% 0 HRPC qos
request
287 85385 3024419 28 0.00% 0.00% 0.01% 0 MMON MENG
192 45880 934785 49 0.00% 0.03% 0.00% 0 MDFS MFIB
Proces
62 76605 1761 43500 0.00% 0.03% 0.00% 0 Per-minute
Jobs
220 32249 99435 324 0.00% 0.01% 0.00% 0 Spanning
Tree
199 14121 21086 669 0.00% 0.01% 0.00% 0 CDP
Protocol
37 27579 12669 2176 0.00% 0.02% 0.00% 0 Net
Background
32 83 5 16600 0.00% 0.00% 0.00% 0 PrstVbl
30 337 10187 33 0.00% 0.00% 0.00% 0 IPC Keep
Alive M
31 801 20257 39 0.00% 0.00% 0.00% 0 IPC
Loadometer
35 4053 101714 39 0.00% 0.00% 0.00% 0 GraphIt
36 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy
Client
33 4818 99107 48 0.00% 0.00% 0.00% 0 Dynamic
ARP Insp
34 0 1 0 0.00% 0.00% 0.00% 0 ARP Snoop
39 25141 133455 188 0.00% 0.00% 0.00% 0 Logger
38 0 3 0 0.00% 0.00% 0.00% 0 IDB Work
41 17 8 2125 0.00% 0.00% 0.00% 0 IF-MGR
control p
42 0 67 0 0.00% 0.00% 0.00% 0 IF-MGR
event pro
43 0 1 0 0.00% 0.00% 0.00% 0 AggMgr
Process
show proc cpu his
555555555555556666677777555555555555555555555555555555555555
666677777777777777755555555555555577777888888888888888777775
100
90
80 *****
70 **********
60 **********************************************************
50 **********************************************************
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)
669999999999999999999988766667566696566766567665676666766667
140210101109001011111299702326911100910220907229053041800305
100 *
90 #####################* *
80 #####################** * * * * *
70 ######################* * * * * * *
60 ##########################################################
50 ##########################################################
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
1
9999999999999999989909999999
8648198885774573267909547892
100 ** * ******* ** ***** ***
90 ********##*####********#####
80 ********#######*****#**#####
70 *#*****########****#########
60 ############################
50 ############################
40 ############################
30 ############################
20 ############################
10 ############################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5
0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
show ip traf
IP statistics:
Rcvd: 1126578 total, 326608 local destination
0 format errors, 0 checksum errors, 27025 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 1 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
1 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 couldn't fragment
Bcast: 33115 received, 0 sent
Mcast: 0 received, 0 sent
Sent: 429063 generated, 1536985507 forwarded
Drop: 1731 encapsulation failed, 0 unresolved, 0 no adjacency
0 no route, 0 unicast RPF, 0 forced drop
0 options denied, 0 source IP address zero
ICMP statistics:
Rcvd: 5 format errors, 2 checksum errors, 2 redirects, 172 unreachable
3492 echo, 5 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 info request, 2 other
0 irdp solicitations, 0 irdp advertisements
Sent: 0 redirects, 104560 unreachable, 10 echo, 3492 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp
0 info reply, 27025 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements
TCP statistics:
Rcvd: 48268 total, 1 checksum errors, 52 no port
Sent: 51034 total
UDP statistics:
Rcvd: 274634 total, 0 checksum errors, 349 no port
Sent: 242948 total, 0 forwarded broadcasts
BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh
EIGRP-IPv4 statistics:
Rcvd: 0 total
Sent: 0 total
show controllers cpu-interface
ASIC Rxbiterr Rxunder Fwdctfix Txbuflos Rxbufloc Rxbufdrain
-------------------------------------------------------------------------
ASIC0 0 0 0 0 0 0
ASIC1 0 0 0 0 0 0
ASIC2 0 0 0 0 0 0
ASIC3 0 0 0 0 0 0
ASIC4 0 0 0 0 0 0
ASIC5 0 0 0 0 0 0
ASIC6 0 0 0 0 0 0
ASIC7 0 0 0 0 0 0
ASIC8 0 0 0 0 0 0
ASIC9 0 0 0 0 0 0
ASICA 0 0 0 0 0 0
ASICB 0 0 0 0 0 0
ASICC 0 0 0 0 0 0
HOL Fix Counts
--------------
No Fixes: 0 Added: 1 In Use: 1063 Both: 0
CPU Heartbeat Statistics
Tx Success Tx Fail 1st Thr 2nd Thr Unthr RetryCtMax
---------- ---------- ---------- ---------- ---------- ----------
1993464 0 8 25 8 86
Rx Delay
0 1 2 3 4
---------- ---------- ---------- ---------- ----------
1993062 225 0 0 0
AddlDelay AdvanceCnt
---------- ----------
0 0
Rx Retries by RetryCount
0 1 2 3 4 5
6
---------- ---------- ---------- ---------- ---------- ----------
----------
1992946 0 0 0 25 25
24
7 8 9
---------- ---------- ----------
22 0 0
AddlRetry
----------
245
cpu-queue-frames retrieved dropped invalid hol-block stray
----------------- ---------- ---------- ---------- ---------- ----------
rpc 1 0 0 0 0
stp 50600 0 0 0 0
ipc 0 0 0 0 0
routing protocol 99762 0 0 0 0
L2 protocol 23895 0 0 0 0
remote console 0 0 0 0 0
sw forwarding 199602629 0 0 0 0
host 296387 0 0 0 0
broadcast 130511 0 0 0 0
cbt-to-spt 0 0 0 0 0
igmp snooping 0 0 0 0 0
icmp 96 0 0 0 0
logging 0 0 0 0 0
rpf-fail 0 0 0 0 0
dstats 0 0 0 0 0
cpu heartbeat 1993287 0 0 0 0
cpu-queue static inuse static added
----------------- ------------ ------------
rpc 0 0
stp 0 0
ipc 0 0
routing protocol 0 0
L2 protocol 0 0
remote console 1063 1
sw forwarding 0 0
host 0 0
broadcast 0 0
cbt-to-spt 0 0
igmp snooping 0 0
icmp 0 0
logging 0 0
rpf-fail 0 0
dstats 0 0
cpu heartbeat 0 0
show platform port-asic stats drop
Port-asic Port Drop Statistics - Summary
========================================
RxQueue 0 Drop Stats: 0
RxQueue 1 Drop Stats: 0
RxQueue 2 Drop Stats: 0
RxQueue 3 Drop Stats: 0
Port 0 TxQueue Drop Stats: 0
Port 1 TxQueue Drop Stats: 0
Port 2 TxQueue Drop Stats: 0
Port 3 TxQueue Drop Stats: 0
Supervisor TxQueue Drop Statistics
Queue 0: 0
Queue 1: 0
Queue 2: 0
Queue 3: 0
Queue 4: 0
Queue 5: 0
Queue 6: 48144879
Queue 7: 138
Queue 8: 1001
Queue 9: 0
Queue 10: 0
Queue 11: 0
Queue 12: 0
Queue 13: 0
Queue 14: 0
Queue 15: 177
Port-asic Port Drop Statistics - Details
========================================
RxQueue Drop Statistics
Queue 0
Weight 0 Frames: 0
Weight 1 Frames: 0
Weight 2 Frames: 0
Queue 1
Weight 0 Frames: 0
Weight 1 Frames: 0
Weight 2 Frames: 0
Queue 2
Weight 0 Frames: 0
Weight 1 Frames: 0
Weight 2 Frames: 0
Queue 3
Weight 0 Frames: 0
Weight 1 Frames: 0
Weight 2 Frames: 0
Port 0 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Port 1 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Port 2 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Port 3 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Supervisor TxQueue Drop Statistics
Queue 0: 0
Queue 1: 0
Queue 2: 0
Queue 3: 0
Queue 4: 0
Queue 5: 0
Queue 6: 48144879
Queue 7: 138
Queue 8: 1001
Queue 9: 0
Queue 10: 0
Queue 11: 0
Queue 12: 0
Queue 13: 0
Queue 14: 0
Queue 15: 177
show ip traff
IP statistics:
Rcvd: 1126901 total, 326777 local destination
0 format errors, 0 checksum errors, 27179 bad hop count
0 unknown protocol, 0 not a gateway
0 security failures, 0 bad options, 1 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
1 stream ID, 0 strict source route, 0 alert, 0 cipso, 0 ump
0 other
Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble
0 fragmented, 0 couldn't fragment
Bcast: 33173 received, 0 sent
Mcast: 0 received, 0 sent
Sent: 429529 generated, 1538467221 forwarded
Drop: 1731 encapsulation failed, 0 unresolved, 0 no adjacency
0 no route, 0 unicast RPF, 0 forced drop
0 options denied, 0 source IP address zero
ICMP statistics:
Rcvd: 5 format errors, 2 checksum errors, 2 redirects, 172 unreachable
3508 echo, 5 echo reply, 0 mask requests, 0 mask replies, 0 quench
0 parameter, 0 timestamp, 0 info request, 2 other
0 irdp solicitations, 0 irdp advertisements
Sent: 0 redirects, 104747 unreachable, 10 echo, 3508 echo reply
0 mask requests, 0 mask replies, 0 quench, 0 timestamp
0 info reply, 27179 time exceeded, 0 parameter problem
0 irdp solicitations, 0 irdp advertisements
TCP statistics:
Rcvd: 48365 total, 1 checksum errors, 52 no port
Sent: 51143 total
UDP statistics:
Rcvd: 274690 total, 0 checksum errors, 349 no port
Sent: 242948 total, 0 forwarded broadcasts
BGP statistics:
Rcvd: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh, 0 unrecognized
Sent: 0 total, 0 opens, 0 notifications, 0 updates
0 keepalives, 0 route-refresh
EIGRP-IPv4 statistics:
Rcvd: 0 total
Sent: 0 total
PIMv2 statistics: Sent/Received
Total: 0/0, 0 checksum errors, 0 format errors
Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0, Hellos:
0/0
Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
Queue drops: 0
State-Refresh: 0/0
IGMP statistics: Sent/Received
Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
DVMRP: 0/0, PIM: 0/0
Queue drops: 0
OSPF statistics:
Last clearing of OSPF traffic counters never
Rcvd: 0 total, 0 checksum errors
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
Sent: 0 total
0 hello, 0 database desc, 0 link state req
0 link state updates, 0 link state acks
ARP statistics:
Rcvd: 99220 requests, 650 replies, 0 reverse, 0 other
Sent: 478050 requests, 105 replies (0 proxy), 0 reverse
Drop due to input queue full: 0
show platform tcam utilization
CAM Utilization for ASIC# 0 Max Used
Masks/Values Masks/values
Unicast mac addresses: 1184/9472 32/141
IPv4 IGMP groups + multicast routes: 152/1216 6/26
IPv4 unicast routes: 0/0 0/0
IPv6 Multicast groups: 1184/9472 32/141
IPv6 unicast directly-connected routes: 1184/9472 32/141
IPv6 unicast indirectly-connected routes: 24/192 12/43
IPv4 policy based routing aces: 0/0 0/0
IPv4 qos aces: 768/768 260/260
IPv4 security aces: 1024/1024 40/40
IPv6 policy based routing aces: 0/0 0/0
IPv6 qos aces: 0/0 0/0
IPv6 security aces: 204/510 5/5
Note: Allocation of TCAM entries per feature uses
a complex algorithm. The above information is meant
to provide an abstract view of the current TCAM utilization
show ip rou summ
IP routing table name is default (0x0)
IP routing table maximum-paths is 32
Route Source Networks Subnets Replicates Overhead Memory
(bytes)
connected 0 4 0 240 688
static 1 0 0 60 172
internal 2 932
Total 3 4 0 300 1792
On Mon, Oct 28, 2013 at 9:21 AM, Jay McMickle <jay.mcmickle_at_yahoo.com>wrote:
> Try to look at what is taking up the processes:
> show proc cpu sorted
>
>
> Regards,
> Jay McMickle- 2x CCIE #35355 (R/S,Sec)
> Sent from my iPhone 5
>
> > On Oct 28, 2013, at 7:46 AM, Sampath Weerasinghe <swe20144_at_gmail.com>
> wrote:
> >
> > Hi friends
> >
> > What could be the reason for high cpu utilization on my company
> > switch?
> >
> > show proc cpu sorted 5min
> >
> > show proc cpu sorted 5min
> > CPU utilization for five seconds: 59%/37%; one minute: 60%; five minutes:
> > 72%
> > PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
> > 84 5285196 996140 5305 6.39% 5.91% 6.40% 0 HLFM
> > address lea
> > 141 4953047 1238685 3998 2.39% 2.67% 4.10% 0 Hulc LED
> > Process
> > 142 2836984 33179 85505 3.67% 3.81% 3.74% 0 HL3U
> bkgrd
> > proce
> > 71 922098 1926258 478 1.75% 1.69% 1.35% 0 Fifo
> Error
> > Detec
> > 104 600638 45169 13297 0.79% 0.79% 0.77% 0 hpm
> > counter proc
> > 88 366294 4468 81981 0.00% 0.51% 0.46% 0 HULC Tcam
> > Memory
> > 353 217372 68982 3151 0.00% 0.21% 0.35% 0 SNMP
> > ENGINE
> >
> > show ver
> > System image file is "flash:c3560-ipservicesk9-mz.122-58.SE2
> >
> > show proc cpu his
> >
> >
> >
> > 555555555666665555555555555555555566666666665555555555555555
> > 666666666222225555599999666666666622222222228888866666555555
> > 100
> > 90
> > 80
> > 70
> > 60 **********************************************************
> > 50 **********************************************************
> > 40 **********************************************************
> > 30 **********************************************************
> > 20 **********************************************************
> > 10 **********************************************************
> > 0....5....1....1....2....2....3....3....4....4....5....5....6
> > 0 5 0 5 0 5 0 5 0 5 0
> > CPU% per second (last 60 seconds)
> >
> >
> >
> >
> > 666999989996676666766567656676659765667656676656756667565677
> > 201010090110371001530917090265493019115093251090390316909311
> > 100
> > 90 *######* *
> > 80 *######* * * * * * * * *
> > 70 #######* * * * ** ** * * * *
> > 60 ##########################################################
> > 50 ##########################################################
> > 40 ##########################################################
> > 30 ##########################################################
> > 20 ##########################################################
> > 10 ##########################################################
> > 0....5....1....1....2....2....3....3....4....4....5....5....6
> > 0 5 0 5 0 5 0 5 0 5 0
> > CPU% per minute (last 60 minutes)
> > * = maximum CPU% # = average CPU%
> >
> >
> >
> >
> >
> > 1
> >
> > 989909999999
> >
> > 267909547892
> >
> > 100 ***** ***
> > 90 *******#####
> > 80 ****#**#####
> > 70 ***#########
> > 60 ############
> > 50 ############
> > 40 ############
> > 30 ############
> > 20 ############
> > 10 ############
> >
> > 0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
> > 0 5 0 5 0 5 0 5 0 5 0 5
> > 0
> > CPU% per hour (last 72 hours)
> > * = maximum CPU% # = average CPU%
> >
> > show ip protocols
> > *** IP Routing is NSF aware ***
> >
> >
> >
> >
> > -Sam
> >
> >
> > Blogs and organic groups at http://www.ccie.net
> >
> > _______________________________________________________________________
> > Subscription information may be found at:
> > http://www.groupstudy.com/list/CCIELab.html
Blogs and organic groups at http://www.ccie.net
Received on Mon Oct 28 2013 - 21:42:51 ART
This archive was generated by hypermail 2.2.0 : Fri Nov 01 2013 - 07:35:39 ART