RE: Flexible Netflow causes Cisco 2821 CPU Hogging

     Your interrupt level is extremely high. Is the CPU level that high 24/7
from the CPU History output? When you do open a TAC Case, they should also
ask you for a CPU Profile. Give it to them even if they don't ask for it.
The procedure is at the bottom of this link.

http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a
00801c2af0.shtml

Arick Harris CCIE #23666 R & S

> From: tscott_at_ipexpert.com
> To: joshualixin_at_gmail.com; ccielab_at_groupstudy.com
> Subject: RE: Flexible Netflow causes Cisco 2821 CPU Hogging
> Date: Fri, 15 Jan 2010 09:13:41 -0500
>
> Isn't it time to open a TAC case when you have nothing to account for the
> problems you are seeing. Clearly if you take it off and everything returns
> to normal then you have already narrowed it down to the source of the
> problem.
>
> Regards,
>
> Tyson Scott - CCIE #13513 R&S, Security, and SP
> Technical Instructor - IPexpert, Inc.
> Mailto: tscott_at_ipexpert.com
> Telephone: +1.810.326.1444, ext. 208
> Live Assistance, Please visit: www.ipexpert.com/chat
> eFax: +1.810.454.0130
>
>
>
> -----Original Message-----
> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On Behalf Of
> Joshua
> Sent: Friday, January 15, 2010 2:08 AM
> To: ccielab_at_groupstudy.com
> Subject: Flexible Netflow causes Cisco 2821 CPU Hogging
>
> Hi Guys,
>
> One of our remote site Cisco 2821 is running IPSec VPN in a hub-and-spokes
> topology environment. SolarWinds netflow analyzer is running on a server at
> hub location. For some reasons, as long as flexible netflow applied on
Cisco
> 2821 interfaces, CPU utilization reached closed 100%. But "sh proc cpu"
> shows nothing.
>
> Below is information from "show xxx". Please help!
>
> Thanks,
>
> Joshua
>
> -sh ver
> -sh run
> -sh proc cpu | e 0.00
> - sh int stat
> -sh inter switching
> -sh int | in proto|queue|rate|err
> ==================Remote Router sh run============
> System image file is "flash:c2800nm-advsecurityk9-mz.124-24.T2.bin"
>
> Remote_Office#sh run
> ...
> !
> flow exporter 701-0174
> destination 10.10.50.206
> source GigabitEthernet0/1
> output-features
> transport udp 2055
> export-protocol netflow-v5
> !
> !
> flow monitor flow-monitor
> record netflow-original
> exporter 701-0174
> cache timeout active 1
> !
> ip source-route
> !
> !
> ip cef
> no ip dhcp use vrf connected
> ip dhcp excluded-address 10.9.2.69
> ip dhcp excluded-address 10.9.2.192 10.9.2.254
> ip dhcp excluded-address 10.9.2.1 10.9.2.31
> !
> ip dhcp pool PROD
> network 10.9.2.0 255.255.255.0
> default-router 10.9.2.2
> dns-server 10.9.2.69 10.10.5.155
> domain-name abc.net
> !
> !
> no ip domain lookup
> ip domain name yourdomain.com
> !
> multilink bundle-name authenticated
> !
> !
> !
> !
> username cisco privilege 15 secret 5 $1$U95M$il6Xa8ObGGTerhddWe27y1
> !
> !
> crypto isakmp policy 10
> encr 3des
> hash md5
> authentication pre-share
> crypto isakmp key EK2CLRS2 address 120.239.178.3
> crypto isakmp key EK2CLRS2 address 172.164.230.218
> crypto isakmp key EK2CLRS2 address 167.133.22.142
> crypto isakmp key EK2CLRS2 address 171.6.24.75
> crypto isakmp key EK2CLRS2 address 163.239.217.98
> crypto isakmp key EK2CLRS2 address 165.115.64.18
> !
> !
> crypto ipsec transform-set myset esp-3des esp-md5-hmac
> !
> crypto map mymap 10 ipsec-isakmp
> set peer 124.239.178.3
> set transform-set myset
> match address 150
> crypto map mymap 20 ipsec-isakmp
> set peer 172.164.230.218
> set transform-set myset
> match address 155
> crypto map mymap 30 ipsec-isakmp
> set peer 167.133.22.142
> set transform-set myset
> match address 156
> crypto map mymap 40 ipsec-isakmp
> set peer 171.6.24.75
> set transform-set myset
> match address 165
> crypto map mymap 50 ipsec-isakmp
> set peer 163.239.217.98
> set transform-set myset
> match address 175
> crypto map mymap 60 ipsec-isakmp
> set peer 165.115.64.18
> set transform-set myset
> match address 185
> !
> archive
> log config
> hidekeys
> !
> !
> interface GigabitEthernet0/0
> description Conntect to Internet via T1
> ip address 165.126.217.2 255.255.255.224
> ip flow monitor flow-monitor input
> ip nat outside
> no ip virtual-reassembly
> duplex auto
> speed auto
> no cdp enable
> crypto map mymap
> !
> interface GigabitEthernet0/1
> description Inside
> ip address 10.9.2.2 255.255.255.0
> ip access-group 120 in
> ip accounting output-packets
> ip flow monitor flow-monitor input
> ip nat inside
> ip virtual-reassembly
> duplex auto
> speed auto
> !
> interface Serial0/3/0
> no ip address
> shutdown
> !
> ip forward-protocol nd
> ip route 0.0.0.0 0.0.0.0 165.126.217.1
> no ip http server
> ip http access-class 23
> ip http authentication local
> no ip http secure-server
> ip http timeout-policy idle 60 life 86400 requests 10000
> !
> !
> ip nat inside source route-map internet interface GigabitEthernet0/0
> overload
> !
> logging source-interface GigabitEthernet0/1
> logging 10.10.50.206
> access-list 120 permit tcp any host 10.10.50.132 eq 2967
> access-list 120 permit udp any host 10.10.50.132 eq 2967
> access-list 120 deny tcp any any eq 2967
> access-list 120 deny udp any any eq 2967
> access-list 120 permit ip any any
> access-list 150 permit ip 10.9.2.0 0.0.0.255 10.10.0.0 0.0.127.255
> access-list 155 permit ip 10.9.2.0 0.0.0.255 10.9.18.0 0.0.0.255
> access-list 156 permit ip 10.9.2.0 0.0.0.255 10.9.24.0 0.0.0.255
> access-list 160 deny ip 10.9.2.0 0.0.0.255 10.10.0.0 0.0.255.255
> access-list 160 deny ip 10.9.2.0 0.0.0.255 10.9.18.0 0.0.0.255
> access-list 160 deny ip 10.9.2.0 0.0.0.255 10.9.24.0 0.0.0.255
> access-list 160 deny ip 10.9.2.0 0.0.0.255 10.10.132.0 0.0.0.255
> access-list 160 deny ip 10.9.2.0 0.0.0.255 10.10.136.0 0.0.0.255
> access-list 160 deny ip 10.9.2.0 0.0.0.255 10.9.30.0 0.0.0.255
> access-list 160 permit ip 10.9.2.0 0.0.0.255 any
> access-list 165 permit ip 10.9.2.0 0.0.0.255 10.10.132.0 0.0.0.255
> access-list 175 permit ip 10.9.2.0 0.0.0.255 10.9.30.0 0.0.0.255
> access-list 185 permit ip 10.9.2.0 0.0.0.255 10.10.136.0 0.0.0.255
> !
> !
> !
> route-map internet permit 10
> match ip address 160
> !
> !
> snmp-server community ledcorsnmp RO
> snmp-server enable traps tty
> snmp-server enable traps frame-relay multilink bundle-mismatch
> ...
>
> ==============================================
> Remote_Office#sho proc cpu | e 0.00
> CPU utilization for five seconds: 98%/95%; one minute: 92%; five minutes:
> 86%
> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
> 2 659264 181712 3628 2.16% 1.06% 1.24% 0 Load Meter
> 41 1094688 916216 1194 0.08% 0.04% 0.05% 0 Per-Second
> Jobs
> 82 534812 906685 589 0.08% 0.06% 0.06% 0 Kontrol
> Common
> H
> 111 1974804 1380744 1430 0.17% 0.10% 0.08% 0 IP Input
> 171 1292788 221834927 5 0.34% 0.35% 0.36% 0 HQF Shaper
> Backg
>
> ===================================================
> Remote_Office#sho int stat
> GigabitEthernet0/0
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 387367 58716537 241702 26394423
> Route cache 34135062 2871254989 29089399 3775098211
> Total 34522423 2929963330 29331096 3801492634
> GigabitEthernet0/1
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 355571 41417261 233322 46737972
> Route cache 29076432 2396714220 34048500 1240926749
> Total 29432003 2438131481 34281822 1287664721
> Interface Serial0/3/0 is disabled
> NVI0
> Switching path Pkts In Chars In Pkts Out Chars Out
> Processor 0 0 0 0
> Route cache 0 0 0 0
> Total 0 0 0 0
>
> ===========================================
> Remote_Office#sho interface switching
> GigabitEthernet0/0 Conntect to Internet via T1
> Throttle count 2
> Drops RP 59 SP 0
> SPD Flushes Fast 0 SSE 0
> SPD Aggress Fast 0
> SPD Priority Inputs 585256 Drops 0
> Protocol IP
> Switching path Pkts In Chars In Pkts Out Chars Out
> Process 775766 107911263 619547 78146041
> Cache misses 0 - - -
> Fast 79945420 3102930096 68075262 3761958554
> Auton/SSE 0 0 0 0
> Protocol DEC MOP
> Switching path Pkts In Chars In Pkts Out Chars Out
> Process 0 0 1518 116886
> Cache misses 0 - - -
> Fast 0 0 0 0
> Auton/SSE 0 0 0 0
> Protocol ARP
> Switching path Pkts In Chars In Pkts Out Chars Out
> Process 490959 29457540 1524 91440
> Cache misses 0 - - -
> Fast 0 0 0 0
> Auton/SSE 0 0 0 0
> Protocol Other
> Switching path Pkts In Chars In Pkts Out Chars Out
> Process 0 0 90790 5447400
> Cache misses 0 - - -
> Fast 0 0 0 0
> Auton/SSE 0 0 0 0
> NOTE: all counts are cumulative and reset only after a reload.
> GigabitEthernet0/1 Inside
> Throttle count 15
> Drops RP 2384 SP 0
> SPD Flushes Fast 0 SSE 0
> SPD Aggress Fast 0
> SPD Priority Inputs 142713 Drops 0
> Protocol IP
> Switching path Pkts In Chars In Pkts Out Chars Out
> Process 685757 84939238 358465 68724586
> Cache misses 0 - - -
> Fast 67552473 339769035 79588748 3624304061
> Auton/SSE 0 0 0 0
> Protocol DEC MOP
> Switching path Pkts In Chars In Pkts Out Chars Out
> Process 0 0 1518 116886
> Cache misses 0 - - -
> Fast 0 0 0 0
> Auton/SSE 0 0 0 0
> Protocol ARP
> Switching path Pkts In Chars In Pkts Out Chars Out
>
> ===============================================
> SanDiegoOffice#sh flow mo
> Flow Monitor flow-monitor:
> Description: User defined
> Flow Record: netflow-original
> Flow Exporter: 701-0174
> Cache:
> Type: normal
> Status: allocated
> Size: 4096 entries / 327700 bytes
> Inactive Timeout: 15 secs
> Active Timeout: 1 secs
> Update Timeout: 1800 secs
> ================================================
> Remote_Office#sho int | i proto|queue|rate|err
> GigabitEthernet0/0 is up, line protocol is up
> Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
> Queueing strategy: fifo
> Output queue: 0/40 (size/max)
> 5 minute input rate 1617000 bits/sec, 215 packets/sec
> 5 minute output rate 1007000 bits/sec, 175 packets/sec
> 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
> 29334368 packets output, 3802888567 bytes, 0 underruns
> 0 output errors, 0 collisions, 0 interface resets
> 0 unknown protocol drops
> 0 babbles, 0 late collision, 0 deferred
> GigabitEthernet0/1 is up, line protocol is up
> Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
> Queueing strategy: fifo
> Output queue: 0/40 (size/max)
> 5 minute input rate 931000 bits/sec, 173 packets/sec
> 5 minute output rate 1527000 bits/sec, 212 packets/sec
> 161 input errors, 0 CRC, 0 frame, 0 overrun, 161 ignored
> 34286645 packets output, 1293090925 bytes, 0 underruns
> 0 output errors, 0 collisions, 0 interface resets
> 57181 unknown protocol drops
> 0 babbles, 0 late collision, 0 deferred
> Serial0/3/0 is administratively down, line protocol is down
> Hardware is GT96K with integrated T1 CSU/DSU
> Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
> Queueing strategy: weighted fair
> Output queue: 0/1000/64/0 (size/max total/threshold/drops)
> 5 minute input rate 0 bits/sec, 0 packets/sec
> 5 minute output rate 0 bits/sec, 0 packets/sec
> 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
> 0 packets output, 0 bytes, 0 underruns
> 0 output errors, 0 collisions, 0 interface resets
> 0 unknown protocol drops
> NVI0 is up, line protocol is up
> Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
> 5 minute input rate 0 bits/sec, 0 packets/sec
> 5 minute output rate 0 bits/sec, 0 packets/sec
> 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
> 0 packets output, 0 bytes, 0 underruns
> 0 output errors, 0 collisions, 0 interface resets
> 0 unknown protocol drops
>
>
> Blogs and organic groups at http://www.ccie.net
>
> _______________________________________________________________________
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html
>
>
> Blogs and organic groups at http://www.ccie.net
>
> _______________________________________________________________________
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html
>
>
>
>
>
>
>
Received on Fri Jan 15 2010 - 11:20:05 ART