Re: Puzzling QoS Issue

How comes you didn't see "IP Input" unusually hight when you looked at CPU?
That's a the usual tell-tale sign.

--
Marko Milivojevic - CCIE #18427 (SP R&S)
Senior CCIE Instructor / Managing Partner - IPexpert
On Mon, Apr 22, 2013 at 9:19 AM, Joe Astorino <joeastorino1982_at_gmail.com>wrote:
> And the root cause was (drumroll please) CEF was disabled somehow.
>
>
> On Sat, Apr 20, 2013 at 12:35 PM, Joe Astorino <joeastorino1982_at_gmail.com
> >wrote:
>
> > Early reports today suggest downgrade to 15.1 fixed the issue!
> >
> > Sent from my iPhone
> >
> > On Apr 19, 2013, at 10:50 PM, Joe Astorino <joeastorino1982_at_gmail.com>
> > wrote:
> >
> > > The carrier is AT&T. Enough said lol
> > >
> > > Sent from my iPad
> > >
> > > On Apr 19, 2013, at 10:37 PM, "Joseph L. Brunner"
> > > <joe_at_affirmedsystems.com> wrote:
> > >
> > >> We can only speculate until you get the right person on the phone;
> > >>
> > >> But if you remove the shaper and run the line hard - when do the pings
> > get to 200-400ms?
> > >>
> > >> Here's a better question - why does the carrier REQUIRE 802.1q tags?
> > >>
> > >> Sounds again like youre not talking to the right person...
> > >>
> > >> That's a highly irregular config...
> > >>
> > >> What if you didn't have a router capable of 802.1q tagging?
> > >>
> > >> Would they let you out of your contract?
> > >>
> > >> LOL
> > >>
> > >> My policy has all routers at the edge with routed interfaces -
> wouldn't
> > I just ignore the tag if they put the ip on a vlan int. without tagging?
> > >>
> > >> There is more to this we have to dig into...
> > >>
> > >> I think the tags are effecting something - just got to find out
> where...
> > >>
> > >> -----Original Message-----
> > >> From: Joe Astorino [mailto:joeastorino1982_at_gmail.com]
> > >> Sent: Friday, April 19, 2013 10:33 PM
> > >> To: Joseph L. Brunner
> > >> Cc: Tony Singh; Matt Bentley; Groupstudy
> > >> Subject: Re: Puzzling QoS Issue
> > >>
> > >> Joseph,
> > >>
> > >> If it was a carrier shaping/policing issue though, why does removing
> > the shaper on my end resolve the issue
> > >>
> > >> Sent from my iPad
> > >>
> > >> On Apr 19, 2013, at 9:20 PM, "Joseph L. Brunner"
> > >> <joe_at_affirmedsystems.com> wrote:
> > >>
> > >>> Your carrier has a aggregate shaper applied or some other type of qos
> > mechanism applied.
> > >>>
> > >>> You're maxing it pal.
> > >>>
> > >>> They probably did the wrong "BE" setting, etc.
> > >>>
> > >>> :)
> > >>>
> > >>> -----Original Message-----
> > >>> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On Behalf
> > >>> Of Tony Singh
> > >>> Sent: Friday, April 19, 2013 8:32 PM
> > >>> To: Joe Astorino
> > >>> Cc: Matt Bentley; Groupstudy
> > >>> Subject: Re: Puzzling QoS Issue
> > >>>
> > >>> yeh without further tests it's hard to say what the issue is, what's
> > >>> the uptime? is the code/license identical to your other working
> router
> > >>> ;)
> > >>>
> > >>> --
> > >>> BR
> > >>>
> > >>> Tony
> > >>>
> > >>> Sent from my iPhone on 3
> > >>>
> > >>> On 20 Apr 2013, at 00:50, Joe Astorino <joeastorino1982_at_gmail.com>
> > wrote:
> > >>>
> > >>>> CPU and memory seem to be OK.  sh proc cpu shows it sitting about
> > >>>> 7-10% when
> > >>> the problem occurs.  sh proc cpu history does not show any really bad
> > spikes.
> > >>> sho proc mem seems to show I have plenty of free RAM as well.
> > >>>>
> > >>>>
> > >>>> On Fri, Apr 19, 2013 at 7:43 PM, Joe Astorino
> > >>>> <joeastorino1982_at_gmail.com>
> > >>> wrote:
> > >>>> Sorry, 2GB of RAM is what it has
> > >>>>
> > >>>>
> > >>>> On Fri, Apr 19, 2013 at 7:41 PM, Matt Bentley
> > >>>> <mattdbentley_at_gmail.com>
> > >>> wrote:
> > >>>> Interesting.  I've done testing with Spirent and other traffic-gen
> > >>> platforms.  I never saw shaping add to the latency - until I actually
> > got up close to the CIR - maybe slightly with a lower-end platform.  But
> I
> > don't think shaping even activates until it senses backpressure and
> > activates "itself".
> > >>>>
> > >>>>
> > >>>> On Fri, Apr 19, 2013 at 4:17 PM, Tony Singh <mothafungla_at_gmail.com>
> > wrote:
> > >>>> The reason why you get the extra delay when the policy is applied is
> > >>>> down to
> > >>> the extra work the CPU has to do on top of inbound congestion, what's
> > your memory/free?
> > >>>>
> > >>>> I would also raise a TAC to get the definitive answer "show
> > >>>> tech-support" &
> > >>> pass to them
> > >>>>
> > >>>> --
> > >>>> BR
> > >>>>
> > >>>> Sent from my iPhone on 3
> > >>>>
> > >>>> On 19 Apr 2013, at 23:13, Joe Astorino <joeastorino1982_at_gmail.com>
> > wrote:
> > >>>>
> > >>>>> Another interesting fact - When the policy is applied as shown
> > >>>>> above, the problem only occurs when the inbound RX is congested.
> > >>>>>
> > >>>>> So basically, if the service-policy is applied outbound towards the
> > >>>>> WAN
> > >>> as
> > >>>>> shown, and the output utilization of the WAN link (TX) is low, and
> > >>>>> the inbound (RX) utilization is low,  everything is fine.
> > >>>>>
> > >>>>> In the event that the inbound utilization of the interface (RX) is
> > >>>>> significant, and the policy is applied outbound as usual, the
> policy
> > >>>>> is adding the 200-400ms extra delay. If I remove the policy, the
> > >>>>> extra delay goes away.
> > >>>>>
> > >>>>> I guess I don't understand why inbound congestion would change how
> > >>>>> the outbound queue / shaping works.
> > >>>>>
> > >>>>> input is low, output is low, policy applied: fine input is high,
> > >>>>> output is low, policy not applied: expected poor response times
> > >>>>> input is high, output is low, policy is applied: expected poor
> > >>>>> response time PLUS an additional 200-400ms of delay
> > >>>>>
> > >>>>>
> > >>>>> On Fri, Apr 19, 2013 at 5:49 PM, Joe Astorino
> > >>> <joeastorino1982_at_gmail.com>wrote:
> > >>>>>
> > >>>>>> I have a 3945 ISR router connected via a GigabitEthernet link to a
> > >>> service
> > >>>>>> provider that is providing 10Mbps WAN access.  I wish to use CBWFQ
> > >>>>>> and
> > >>> the
> > >>>>>> service provider requires dot1q tagging.  As such, I must shape my
> > >>>>>> sub-interface and use a hierarchical type QoS policy...no big deal
> > >>>>>>
> > >>>>>> policy-map WAN-EDGE
> > >>>>>> class VOICE
> > >>>>>> priority percent 20
> > >>>>>> !
> > >>>>>> class VIDEO
> > >>>>>> bandwidth remaining percent 60
> > >>>>>> queue-limit 128 packets
> > >>>>>> !
> > >>>>>> class APPS_SIGNALING
> > >>>>>> set dscp af21
> > >>>>>> bandwidth remaining percent 30
> > >>>>>> !
> > >>>>>> class class-default
> > >>>>>> bandwidth remaining percent 10
> > >>>>>> !
> > >>>>>> !
> > >>>>>> policy-map SHAPE-OUT
> > >>>>>> class class-default
> > >>>>>> shape average 10000000
> > >>>>>> service-policy WAN-EDGE
> > >>>>>> !
> > >>>>>> !
> > >>>>>> int gi0/1.2
> > >>>>>> encapsulation dot1q 2
> > >>>>>> ip address ...
> > >>>>>> service-policy output SHAPE-OUT
> > >>>>>>
> > >>>>>>
> > >>>>>> Here is the very strange thing.  Even when the TX of this WAN
> > >>>>>> interface
> > >>> is
> > >>>>>> barely being used, when and only when the service-policy is
> > >>>>>> applied, the response in pings to anything behind this router
> > >>>>>> increase immediately by 200-400ms.  Immediately after removing the
> > >>>>>> service-policy things return
> > >>> to
> > >>>>>> normal.
> > >>>>>>
> > >>>>>> Investigating the output of "show policy-map interface" reveals a
> > >>>>>> large number of output drops in the shaper class-default. So far I
> > >>>>>> have tried
> > >>>>>>
> > >>>>>> - increasing the queue-limit to many different combinations
> > >>>>>> - increasing the burst size
> > >>>>>>
> > >>>>>> I also have the exact same configuration on another remote site
> > >>>>>> router running the same version of IOS with the same setup (10Mbps
> > >>>>>> Ethernet WAN
> > >>>>>> link) on the same platform.
> > >>>>>>
> > >>>>>> I'm really baffled as to what would cause this.  Any insight
> > >>> appreciated.
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Regards,
> > >>>>>>
> > >>>>>> Joe Astorino
> > >>>>>> CCIE #24347
> > >>>>>> http://astorinonetworks.com
> > >>>>>>
> > >>>>>> "He not busy being born is busy dying" - Dylan
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> Regards,
> > >>>>>
> > >>>>> Joe Astorino
> > >>>>> CCIE #24347
> > >>>>> http://astorinonetworks.com
> > >>>>>
> > >>>>> "He not busy being born is busy dying" - Dylan
> > >>>>>
> > >>>>>
> > >>>>> Blogs and organic groups at http://www.ccie.net
> > >>>>>
> > >>>>>
> ____________________________________________________________________
> > >>>>> ___ Subscription information may be found at:
> > >>>>> http://www.groupstudy.com/list/CCIELab.html
> > >>>>
> > >>>>
> > >>>> Blogs and organic groups at http://www.ccie.net
> > >>>>
> > >>>>
> _____________________________________________________________________
> > >>>> _ _ Subscription information may be found at:
> > >>>> http://www.groupstudy.com/list/CCIELab.html
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Regards,
> > >>>>
> > >>>> Joe Astorino
> > >>>> CCIE #24347
> > >>>> http://astorinonetworks.com
> > >>>>
> > >>>> "He not busy being born is busy dying" - Dylan
> > >>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Regards,
> > >>>>
> > >>>> Joe Astorino
> > >>>> CCIE #24347
> > >>>> http://astorinonetworks.com
> > >>>>
> > >>>> "He not busy being born is busy dying" - Dylan
> > >>>
> > >>>
> > >>> Blogs and organic groups at http://www.ccie.net
> > >>>
> > >>>
> ______________________________________________________________________
> > >>> _ Subscription information may be found at:
> > >>> http://www.groupstudy.com/list/CCIELab.html
> > >>
> > >>
> >
>
>
>
> --
> Regards,
>
> Joe Astorino
> CCIE #24347
> http://astorinonetworks.com
>
> "He not busy being born is busy dying" - Dylan
>
>
> Blogs and organic groups at http://www.ccie.net
>
> _______________________________________________________________________
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html
Blogs and organic groups at http://www.ccie.net