Re: Puzzling QoS Issue

Perhaps when I ran the command at that particular time the CPU was fine.
When I looked at sho proc cpu sorted the total CPU was only about 7%.
Also, sh proc cpu history did not show any history of CPU utilization at
dangerously high levels.

Can't say for sure...

On Mon, Apr 22, 2013 at 10:54 AM, Marko Milivojevic <markom_at_ipexpert.com>wrote:

>
> How comes you didn't see "IP Input" unusually hight when you looked at
> CPU? That's a the usual tell-tale sign.
>
> --
> Marko Milivojevic - CCIE #18427 (SP R&S)
> Senior CCIE Instructor / Managing Partner - IPexpert
>
>
> On Mon, Apr 22, 2013 at 9:19 AM, Joe Astorino <joeastorino1982_at_gmail.com>wrote:
>
>> And the root cause was (drumroll please) CEF was disabled somehow.
>>
>>
>> On Sat, Apr 20, 2013 at 12:35 PM, Joe Astorino <joeastorino1982_at_gmail.com
>> >wrote:
>>
>> > Early reports today suggest downgrade to 15.1 fixed the issue!
>> >
>> > Sent from my iPhone
>> >
>> > On Apr 19, 2013, at 10:50 PM, Joe Astorino <joeastorino1982_at_gmail.com>
>> > wrote:
>> >
>> > > The carrier is AT&T. Enough said lol
>> > >
>> > > Sent from my iPad
>> > >
>> > > On Apr 19, 2013, at 10:37 PM, "Joseph L. Brunner"
>> > > <joe_at_affirmedsystems.com> wrote:
>> > >
>> > >> We can only speculate until you get the right person on the phone;
>> > >>
>> > >> But if you remove the shaper and run the line hard - when do the
>> pings
>> > get to 200-400ms?
>> > >>
>> > >> Here's a better question - why does the carrier REQUIRE 802.1q tags?
>> > >>
>> > >> Sounds again like youre not talking to the right person...
>> > >>
>> > >> That's a highly irregular config...
>> > >>
>> > >> What if you didn't have a router capable of 802.1q tagging?
>> > >>
>> > >> Would they let you out of your contract?
>> > >>
>> > >> LOL
>> > >>
>> > >> My policy has all routers at the edge with routed interfaces -
>> wouldn't
>> > I just ignore the tag if they put the ip on a vlan int. without tagging?
>> > >>
>> > >> There is more to this we have to dig into...
>> > >>
>> > >> I think the tags are effecting something - just got to find out
>> where...
>> > >>
>> > >> -----Original Message-----
>> > >> From: Joe Astorino [mailto:joeastorino1982_at_gmail.com]
>> > >> Sent: Friday, April 19, 2013 10:33 PM
>> > >> To: Joseph L. Brunner
>> > >> Cc: Tony Singh; Matt Bentley; Groupstudy
>> > >> Subject: Re: Puzzling QoS Issue
>> > >>
>> > >> Joseph,
>> > >>
>> > >> If it was a carrier shaping/policing issue though, why does removing
>> > the shaper on my end resolve the issue
>> > >>
>> > >> Sent from my iPad
>> > >>
>> > >> On Apr 19, 2013, at 9:20 PM, "Joseph L. Brunner"
>> > >> <joe_at_affirmedsystems.com> wrote:
>> > >>
>> > >>> Your carrier has a aggregate shaper applied or some other type of
>> qos
>> > mechanism applied.
>> > >>>
>> > >>> You're maxing it pal.
>> > >>>
>> > >>> They probably did the wrong "BE" setting, etc.
>> > >>>
>> > >>> :)
>> > >>>
>> > >>> -----Original Message-----
>> > >>> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On
>> Behalf
>> > >>> Of Tony Singh
>> > >>> Sent: Friday, April 19, 2013 8:32 PM
>> > >>> To: Joe Astorino
>> > >>> Cc: Matt Bentley; Groupstudy
>> > >>> Subject: Re: Puzzling QoS Issue
>> > >>>
>> > >>> yeh without further tests it's hard to say what the issue is, what's
>> > >>> the uptime? is the code/license identical to your other working
>> router
>> > >>> ;)
>> > >>>
>> > >>> --
>> > >>> BR
>> > >>>
>> > >>> Tony
>> > >>>
>> > >>> Sent from my iPhone on 3
>> > >>>
>> > >>> On 20 Apr 2013, at 00:50, Joe Astorino <joeastorino1982_at_gmail.com>
>> > wrote:
>> > >>>
>> > >>>> CPU and memory seem to be OK. sh proc cpu shows it sitting about
>> > >>>> 7-10% when
>> > >>> the problem occurs. sh proc cpu history does not show any really
>> bad
>> > spikes.
>> > >>> sho proc mem seems to show I have plenty of free RAM as well.
>> > >>>>
>> > >>>>
>> > >>>> On Fri, Apr 19, 2013 at 7:43 PM, Joe Astorino
>> > >>>> <joeastorino1982_at_gmail.com>
>> > >>> wrote:
>> > >>>> Sorry, 2GB of RAM is what it has
>> > >>>>
>> > >>>>
>> > >>>> On Fri, Apr 19, 2013 at 7:41 PM, Matt Bentley
>> > >>>> <mattdbentley_at_gmail.com>
>> > >>> wrote:
>> > >>>> Interesting. I've done testing with Spirent and other traffic-gen
>> > >>> platforms. I never saw shaping add to the latency - until I
>> actually
>> > got up close to the CIR - maybe slightly with a lower-end platform.
>> But I
>> > don't think shaping even activates until it senses backpressure and
>> > activates "itself".
>> > >>>>
>> > >>>>
>> > >>>> On Fri, Apr 19, 2013 at 4:17 PM, Tony Singh <mothafungla_at_gmail.com
>> >
>> > wrote:
>> > >>>> The reason why you get the extra delay when the policy is applied
>> is
>> > >>>> down to
>> > >>> the extra work the CPU has to do on top of inbound congestion,
>> what's
>> > your memory/free?
>> > >>>>
>> > >>>> I would also raise a TAC to get the definitive answer "show
>> > >>>> tech-support" &
>> > >>> pass to them
>> > >>>>
>> > >>>> --
>> > >>>> BR
>> > >>>>
>> > >>>> Sent from my iPhone on 3
>> > >>>>
>> > >>>> On 19 Apr 2013, at 23:13, Joe Astorino <joeastorino1982_at_gmail.com>
>> > wrote:
>> > >>>>
>> > >>>>> Another interesting fact - When the policy is applied as shown
>> > >>>>> above, the problem only occurs when the inbound RX is congested.
>> > >>>>>
>> > >>>>> So basically, if the service-policy is applied outbound towards
>> the
>> > >>>>> WAN
>> > >>> as
>> > >>>>> shown, and the output utilization of the WAN link (TX) is low, and
>> > >>>>> the inbound (RX) utilization is low, everything is fine.
>> > >>>>>
>> > >>>>> In the event that the inbound utilization of the interface (RX) is
>> > >>>>> significant, and the policy is applied outbound as usual, the
>> policy
>> > >>>>> is adding the 200-400ms extra delay. If I remove the policy, the
>> > >>>>> extra delay goes away.
>> > >>>>>
>> > >>>>> I guess I don't understand why inbound congestion would change how
>> > >>>>> the outbound queue / shaping works.
>> > >>>>>
>> > >>>>> input is low, output is low, policy applied: fine input is high,
>> > >>>>> output is low, policy not applied: expected poor response times
>> > >>>>> input is high, output is low, policy is applied: expected poor
>> > >>>>> response time PLUS an additional 200-400ms of delay
>> > >>>>>
>> > >>>>>
>> > >>>>> On Fri, Apr 19, 2013 at 5:49 PM, Joe Astorino
>> > >>> <joeastorino1982_at_gmail.com>wrote:
>> > >>>>>
>> > >>>>>> I have a 3945 ISR router connected via a GigabitEthernet link to
>> a
>> > >>> service
>> > >>>>>> provider that is providing 10Mbps WAN access. I wish to use
>> CBWFQ
>> > >>>>>> and
>> > >>> the
>> > >>>>>> service provider requires dot1q tagging. As such, I must shape
>> my
>> > >>>>>> sub-interface and use a hierarchical type QoS policy...no big
>> deal
>> > >>>>>>
>> > >>>>>> policy-map WAN-EDGE
>> > >>>>>> class VOICE
>> > >>>>>> priority percent 20
>> > >>>>>> !
>> > >>>>>> class VIDEO
>> > >>>>>> bandwidth remaining percent 60
>> > >>>>>> queue-limit 128 packets
>> > >>>>>> !
>> > >>>>>> class APPS_SIGNALING
>> > >>>>>> set dscp af21
>> > >>>>>> bandwidth remaining percent 30
>> > >>>>>> !
>> > >>>>>> class class-default
>> > >>>>>> bandwidth remaining percent 10
>> > >>>>>> !
>> > >>>>>> !
>> > >>>>>> policy-map SHAPE-OUT
>> > >>>>>> class class-default
>> > >>>>>> shape average 10000000
>> > >>>>>> service-policy WAN-EDGE
>> > >>>>>> !
>> > >>>>>> !
>> > >>>>>> int gi0/1.2
>> > >>>>>> encapsulation dot1q 2
>> > >>>>>> ip address ...
>> > >>>>>> service-policy output SHAPE-OUT
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Here is the very strange thing. Even when the TX of this WAN
>> > >>>>>> interface
>> > >>> is
>> > >>>>>> barely being used, when and only when the service-policy is
>> > >>>>>> applied, the response in pings to anything behind this router
>> > >>>>>> increase immediately by 200-400ms. Immediately after removing
>> the
>> > >>>>>> service-policy things return
>> > >>> to
>> > >>>>>> normal.
>> > >>>>>>
>> > >>>>>> Investigating the output of "show policy-map interface" reveals a
>> > >>>>>> large number of output drops in the shaper class-default. So far
>> I
>> > >>>>>> have tried
>> > >>>>>>
>> > >>>>>> - increasing the queue-limit to many different combinations
>> > >>>>>> - increasing the burst size
>> > >>>>>>
>> > >>>>>> I also have the exact same configuration on another remote site
>> > >>>>>> router running the same version of IOS with the same setup
>> (10Mbps
>> > >>>>>> Ethernet WAN
>> > >>>>>> link) on the same platform.
>> > >>>>>>
>> > >>>>>> I'm really baffled as to what would cause this. Any insight
>> > >>> appreciated.
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> --
>> > >>>>>> Regards,
>> > >>>>>>
>> > >>>>>> Joe Astorino
>> > >>>>>> CCIE #24347
>> > >>>>>> http://astorinonetworks.com
>> > >>>>>>
>> > >>>>>> "He not busy being born is busy dying" - Dylan
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> --
>> > >>>>> Regards,
>> > >>>>>
>> > >>>>> Joe Astorino
>> > >>>>> CCIE #24347
>> > >>>>> http://astorinonetworks.com
>> > >>>>>
>> > >>>>> "He not busy being born is busy dying" - Dylan
>> > >>>>>
>> > >>>>>
>> > >>>>> Blogs and organic groups at http://www.ccie.net
>> > >>>>>
>> > >>>>>
>> ____________________________________________________________________
>> > >>>>> ___ Subscription information may be found at:
>> > >>>>> http://www.groupstudy.com/list/CCIELab.html
>> > >>>>
>> > >>>>
>> > >>>> Blogs and organic groups at http://www.ccie.net
>> > >>>>
>> > >>>>
>> _____________________________________________________________________
>> > >>>> _ _ Subscription information may be found at:
>> > >>>> http://www.groupstudy.com/list/CCIELab.html
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> --
>> > >>>> Regards,
>> > >>>>
>> > >>>> Joe Astorino
>> > >>>> CCIE #24347
>> > >>>> http://astorinonetworks.com
>> > >>>>
>> > >>>> "He not busy being born is busy dying" - Dylan
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> --
>> > >>>> Regards,
>> > >>>>
>> > >>>> Joe Astorino
>> > >>>> CCIE #24347
>> > >>>> http://astorinonetworks.com
>> > >>>>
>> > >>>> "He not busy being born is busy dying" - Dylan
>> > >>>
>> > >>>
>> > >>> Blogs and organic groups at http://www.ccie.net
>> > >>>
>> > >>>
>> ______________________________________________________________________
>> > >>> _ Subscription information may be found at:
>> > >>> http://www.groupstudy.com/list/CCIELab.html
>> > >>
>> > >>
>> >
>>
>>
>>
>> --
>> Regards,
>>
>> Joe Astorino
>> CCIE #24347
>> http://astorinonetworks.com
>>
>> "He not busy being born is busy dying" - Dylan
>>
>>
>> Blogs and organic groups at http://www.ccie.net
>>
>> _______________________________________________________________________
>> Subscription information may be found at:
>> http://www.groupstudy.com/list/CCIELab.html
>>
>>
>>
>>
>>
>>
>>
>>
>

-- 
Regards,
Joe Astorino
CCIE #24347
http://astorinonetworks.com
"He not busy being born is busy dying" - Dylan
Blogs and organic groups at http://www.ccie.net