Re: High CPU load on 7609

Sajjad, the 7600/6500 can handle huge amounts of traffic in hardware, but
very little traffic in software. Therefore it is imperative that you push
as much as possible to the hardware, because CPU is very weak. But even if
it was super strong, it would still be weak compared to the hardware
capabilities.

Think of it this way: Cisco IOS could have been just shaved off all the
commands that would load the CPU, and then you wouldn't have the
functionality at all.

sorry that I've missed that it wasn't caused by the PBR - but I saw that
after the changes the CPU loads have dropped to 30/40%
On Mon, Oct 31, 2011 at 10:41 AM, Sajjad Najafizadeh
<najafizadeh_at_gmail.com>wrote:

> Hi
>
> thanks for your reply ,but as i explainde I removed PBR to check if this
> is PBR issue or not , it wasn't.
> Then I used VRF and i had same issue , i am disappointed that using
> standard PBR or simple VRF light could cause huge CPU load on big router
> such as 7600.
>
> Regards
>
> On Mon, Oct 31, 2011 at 12:51 PM, Pavel Bykov <slidersv_at_gmail.com> wrote:
>
>> Hi.
>> PBR is supported in the hardware on 7600/6500, but only with a very
>> specific configuration. All other config, that is not stated in the
>> documentation, will result in CPU punting, as was mentioned.
>> It is really important to make sure that you use as much hardware as
>> possible on those boxes, because it is easy to get carried away and think
>> of the platform as too versatile, and overload your weak 600MHz SR71000 CPU.
>>
>> In this case it was pretty straigh forward - you knew what was the cause
>> of the problem.
>>
>> In cases where its not that straightforward, you can realize that it's
>> not the process level load that is the problem, but the interrupt level
>> (73% of CPU load in your "sh proc cpu" output came from interrupts).
>> Interrupt level CPU load is either cause by security breaches, malfunctions
>> (e.g. bad IP checksum from other device), or configuration issues.
>> To protect the CPU from extensive load, you can use control plane
>> protection in a form of 10 available hardware policers, that prtect 1G
>> pipes leading to the RP and SP cpus. Software COPP does not really do the
>> trick in this case, as it is more for Process-Level intensive operations
>> (e.g. good for SNMP, BGP, TELNET, FTP etc). But for floods, COPP will put
>> as much load on CPU (as it is a software policer) as dispatching the packet
>> - which defeats the purpose of COPP.
>>
>> In any case, if you really want to know what is your box using the CPU
>> for, you can easily SPAN the pipe to RP and SP CPU in-band, so you'll know
>> what exactly your CPU is working on, and based on that information decide
>> how you can protect it.
>>
>> What you did on the end to reduce the CPU workload doesn't seem bad, so
>> I'm not sure why you're dissapointed. 6500/7600 is not a software platform,
>> so the functionality is fairly limited to PFC hardware capabilities. As I
>> said, PBR is possible in hardware on these platforms, but only with a very
>> specific command set.
>>
>> P.S.: CPU on 6500/7600 will never be able to handle more than 1G of
>> packets, regardless of optimizations, as that is the speed of CPU interface
>> on PORT-ASIC.
>> On Thu, Oct 27, 2011 at 7:36 PM, Sajjad Najafizadeh <
>> najafizadeh_at_gmail.com> wrote:
>>
>>> Friends,
>>>
>>> I have traffic from other network that need to be sent to some packet
>>> analyzer ( About 2 gbps ) then the packet analyser send it back to same
>>> router ( 7609) , first I've used PBR but the CPU load goes very high , I
>>> though VRF might help , I've changed PBR to VRF , same issue , 80% + cpu
>>> load , drop down the traffic to 1gbps , downgrade the IOS from 12.2(33)
>>> to 12.2(18) , no luck , same thing ...
>>> The only idea that worked for me was I removed the VRF and put IP of CRF
>>> to
>>> packet analyser , and make L2 link from other network to packet analyzer
>>> L2
>>> through 7600 , it reduced CPU load from 80%+ to 30-40%.
>>>
>>> really disappointed ...
>>>
>>> Regards
>>>
>>> On Thu, Oct 27, 2011 at 8:52 PM, Yuri Bank <yuribank_at_gmail.com> wrote:
>>>
>>> > So is this a result of using PBR in VRFs? Or will using PBR period,
>>> cause
>>> > all packets ( on that interface ) to hit the CPU?
>>> >
>>> > On Thu, Oct 27, 2011 at 5:16 AM, Radioactive Frog <pbhatkoti_at_gmail.com
>>> >wrote:
>>> >
>>> >> >>>I cant believe given the state of the world that cisco is still
>>> selling
>>> >> products where a feature that can be configured can take slow down the
>>> >> system that much.
>>> >>
>>> >> Totally agreed Joseph.
>>> >> I'd expect at least it captured somewhere in the 'show proc cpu'
>>> command
>>> >> so
>>> >> that u can see what is causing it. once it's PBR'd (L3) , all packets
>>> are
>>> >> gonna punt onto the CPU.
>>> >> May be it's documented somewhere and we don't know that secret
>>> location on
>>> >> CCO yet :)
>>> >>
>>> >> Sajjad -
>>> >> >>>The only workaround that i've think of was to make it L2 toward
>>> >> next-hop
>>> >> and removing VRF in configuration.
>>> >>
>>> >> Do u mean u turned it into L2 switch? If so what a waste of 760X! :(
>>> >>
>>> >>
>>> >> On Thu, Oct 27, 2011 at 1:39 AM, Joseph L. Brunner
>>> >> <joe_at_affirmedsystems.com>wrote:
>>> >>
>>> >> > Very good to know that Sajjad.
>>> >> >
>>> >> > Thanks for posting back what did it.
>>> >> >
>>> >> > I cant believe given the state of the world that cisco is still
>>> selling
>>> >> > products where a feature that can be configured can take slow down
>>> the
>>> >> > system that much.
>>> >> >
>>> >> >
>>> >> >
>>> >> > -----Original Message-----
>>> >> > From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On
>>> Behalf Of
>>> >> > Sajjad Najafizadeh
>>> >> > Sent: Wednesday, October 26, 2011 10:08 AM
>>> >> > To: Radioactive Frog
>>> >> > Cc: Cisco certification
>>> >> > Subject: Re: High CPU load on 7609
>>> >> >
>>> >> > Hi all
>>> >> >
>>> >> > First of all I've change IOS to 12.2(18) , but the issue exist.
>>> >> > I used VRF light to send traffic to next hop as PBR killed the CPU
>>> >> before .
>>> >> > The only workaround that i've think of was to make it L2 toward
>>> next-hop
>>> >> > and
>>> >> > removing VRF in configuration.
>>> >> > The issue solved with this .
>>> >> > I do not believe 7600 router can not handle VRF and BGP with some
>>> PBR in
>>> >> > same time with max traffic of 4gbps.
>>> >> >
>>> >> > Thanks again to all for support.
>>> >> >
>>> >> > REgards
>>> >> >
>>> >> > On Wed, Oct 26, 2011 at 1:21 PM, Radioactive Frog <
>>> pbhatkoti_at_gmail.com
>>> >> > >wrote:
>>> >> >
>>> >> > > last week I had same issue on 6509. Weird thing as nothing will be
>>> >> shown
>>> >> > in
>>> >> > > 'show proc cpu sorted' output.
>>> >> > >
>>> >> > > The root cause of my issue was someone added route-map (matching
>>> ACL
>>> >> and
>>> >> > > set next hop). There were about 8000+ users! University
>>> environment.
>>> >> > > The core 6509 was running like a dog!
>>> >> > >
>>> >> > > The fix: implement VRF's . After removing route-maps CPU was back
>>> to
>>> >> > normal
>>> >> > > 40-55% (was 95-100% constantly for 10 days).
>>> >> > >
>>> >> > >
>>> >> > > HTH
>>> >> > >
>>> >> > > Frog
>>> >> > >
>>> >> > >
>>> >> > > On Wed, Oct 26, 2011 at 6:39 PM, Sajjad Najafizadeh <
>>> >> > najafizadeh_at_gmail.com
>>> >> > > > wrote:
>>> >> > >
>>> >> > >> Hi all
>>> >> > >>
>>> >> > >> we have high CPU load on 7609 router .
>>> >> > >> there is no ip policy and no NAT on this router but here is the
>>> CPU
>>> >> > load.
>>> >> > >> Could any one suggest what to do ??
>>> >> > >>
>>> >> > >> *Output of sho ip proc cpu sorted :*
>>> >> > >>
>>> >> > >>
>>> >> > >> CPU utilization for five seconds: 75%/73%; one minute: 77%; five
>>> >> > minutes:
>>> >> > >> 77%
>>> >> > >> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY
>>> >> Process
>>> >> > >> 220 949800 3116922 304 1.59% 1.58% 1.43% 0
>>> IP
>>> >> > Input
>>> >> > >>
>>> >> > >> 256 2544 3638244 0 0.15% 0.16% 0.15% 0
>>> >> > Ethernet
>>> >> > >> Msec Ti
>>> >> > >> 83 80164 61809 1296 0.15% 0.02% 0.05% 2
>>> >> Virtual
>>> >> > >> Exec
>>> >> > >> 2 30380 5976 5083 0.07% 0.05% 0.06% 0
>>> Load
>>> >> > Meter
>>> >> > >>
>>> >> > >> 219 1132 916608 1 0.07% 0.03% 0.02% 0
>>> IP
>>> >> ARP
>>> >> > >> Retry Age
>>> >> > >> 372 576 43514 13 0.07% 0.00% 0.00% 0
>>> FM
>>> >> core
>>> >> > >>
>>> >> > >> 162 632 29926 21 0.07% 0.02% 0.02% 0
>>> >> > >> Per-Second
>>> >> > >> Jobs
>>> >> > >> 191 52 29866 1 0.07% 0.00% 0.00% 0
>>> CWAN
>>> >> > >> CHOCX
>>> >> > >> PROCE
>>> >> > >> 260 1164 916616 1 0.07% 0.04% 0.05% 0
>>> IPAM
>>> >> > >> Manager
>>> >> > >> 27 1120 29181 38 0.07% 0.02% 0.02% 0
>>> IPC
>>> >> > >> Periodic Tim
>>> >> > >> 326 1292 126588 10 0.07% 0.03% 0.02% 0
>>> TCP
>>> >> > Timer
>>> >> > >>
>>> >> > >> 555 89680 494908 181 0.07% 0.11% 0.11% 0
>>> SNMP
>>> >> > >> ENGINE
>>> >> > >>
>>> >> > >> 379 72 29805 2 0.07% 0.00% 0.00% 0
>>> PfR
>>> >> BR
>>> >> > >> Learn
>>> >> > >> 15 0 2 0 0.00% 0.00% 0.00% 0
>>> ATM
>>> >> Idle
>>> >> > >> Timer
>>> >> > >> 14 316 31552 10 0.00% 0.00% 0.00% 0
>>> ARP
>>> >> > >> Background
>>> >> > >> 17 0 1 0 0.00% 0.00% 0.00% 0
>>> >> > >> AAA_SERVER_DEADT
>>> >> > >> 13 25684 36715 699 0.00% 0.04% 0.05% 0
>>> ARP
>>> >> > Input
>>> >> > >>
>>> >> > >> 12 3748 30473 122 0.00% 0.00% 0.00% 0
>>> >> > WATCH_AFS
>>> >> > >>
>>> >> > >> 16 0 1 0 0.00% 0.00% 0.00% 0
>>> ATM
>>> >> > ASYNC
>>> >> > >> PROC
>>> >> > >> 18 0 1 0 0.00% 0.00% 0.00% 0
>>> >> Policy
>>> >> > >> Manager
>>> >> > >> 22 12 6010 1 0.00% 0.00% 0.00% 0
>>> IPC
>>> >> > Event
>>> >> > >> Notifi
>>> >> > >> 23 64 29182 2 0.00% 0.00% 0.00% 0
>>> IPC
>>> >> > Mcast
>>> >> > >> Pendin
>>> >> > >> 24 0 500 0 0.00% 0.00% 0.00% 0
>>> IPC
>>> >> > >> Dynamic
>>> >> > >> Cach
>>> >> > >> 11 0 2 0 0.00% 0.00% 0.00% 0
>>> >> Timers
>>> >> > >>
>>> >> > >> 26 16 107 149 0.00% 0.00% 0.00% 0
>>> >> PF_Split
>>> >> > >> Sync Pr
>>> >> > >>
>>> >> > >> Regards
>>> >> > >>
>>> >> > >>
>>> >> > >> Blogs and organic groups at http://www.ccie.net
>>> >> > >>
>>> >> > >>
>>> >>
>>> _______________________________________________________________________
>>> >> > >> Subscription information may be found at:
>>> >> > >> http://www.groupstudy.com/list/CCIELab.html
>>> >> >
>>> >> >
>>> >> > Blogs and organic groups at http://www.ccie.net
>>> >> >
>>> >> >
>>> _______________________________________________________________________
>>> >> > Subscription information may be found at:
>>> >> > http://www.groupstudy.com/list/CCIELab.html
>>> >>
>>> >>
>>> >> Blogs and organic groups at http://www.ccie.net
>>> >>
>>> >>
>>> _______________________________________________________________________
>>> >> Subscription information may be found at:
>>> >> http://www.groupstudy.com/list/CCIELab.html
>>>
>>>
>>> Blogs and organic groups at http://www.ccie.net
>>>
>>> _______________________________________________________________________
>>> Subscription information may be found at:
>>> http://www.groupstudy.com/list/CCIELab.html
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> Pavel Bykov
>>
>>
>>
>

-- 
Pavel Bykov
Blogs and organic groups at http://www.ccie.net