Re: Puzzling QoS Issue

And the root cause was (drumroll please) CEF was disabled somehow.

On Sat, Apr 20, 2013 at 12:35 PM, Joe Astorino <joeastorino1982_at_gmail.com>wrote:

> Early reports today suggest downgrade to 15.1 fixed the issue!
>
> Sent from my iPhone
>
> On Apr 19, 2013, at 10:50 PM, Joe Astorino <joeastorino1982_at_gmail.com>
> wrote:
>
> > The carrier is AT&T. Enough said lol
> >
> > Sent from my iPad
> >
> > On Apr 19, 2013, at 10:37 PM, "Joseph L. Brunner"
> > <joe_at_affirmedsystems.com> wrote:
> >
> >> We can only speculate until you get the right person on the phone;
> >>
> >> But if you remove the shaper and run the line hard - when do the pings
> get to 200-400ms?
> >>
> >> Here's a better question - why does the carrier REQUIRE 802.1q tags?
> >>
> >> Sounds again like youre not talking to the right person...
> >>
> >> That's a highly irregular config...
> >>
> >> What if you didn't have a router capable of 802.1q tagging?
> >>
> >> Would they let you out of your contract?
> >>
> >> LOL
> >>
> >> My policy has all routers at the edge with routed interfaces - wouldn't
> I just ignore the tag if they put the ip on a vlan int. without tagging?
> >>
> >> There is more to this we have to dig into...
> >>
> >> I think the tags are effecting something - just got to find out where...
> >>
> >> -----Original Message-----
> >> From: Joe Astorino [mailto:joeastorino1982_at_gmail.com]
> >> Sent: Friday, April 19, 2013 10:33 PM
> >> To: Joseph L. Brunner
> >> Cc: Tony Singh; Matt Bentley; Groupstudy
> >> Subject: Re: Puzzling QoS Issue
> >>
> >> Joseph,
> >>
> >> If it was a carrier shaping/policing issue though, why does removing
> the shaper on my end resolve the issue
> >>
> >> Sent from my iPad
> >>
> >> On Apr 19, 2013, at 9:20 PM, "Joseph L. Brunner"
> >> <joe_at_affirmedsystems.com> wrote:
> >>
> >>> Your carrier has a aggregate shaper applied or some other type of qos
> mechanism applied.
> >>>
> >>> You're maxing it pal.
> >>>
> >>> They probably did the wrong "BE" setting, etc.
> >>>
> >>> :)
> >>>
> >>> -----Original Message-----
> >>> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On Behalf
> >>> Of Tony Singh
> >>> Sent: Friday, April 19, 2013 8:32 PM
> >>> To: Joe Astorino
> >>> Cc: Matt Bentley; Groupstudy
> >>> Subject: Re: Puzzling QoS Issue
> >>>
> >>> yeh without further tests it's hard to say what the issue is, what's
> >>> the uptime? is the code/license identical to your other working router
> >>> ;)
> >>>
> >>> --
> >>> BR
> >>>
> >>> Tony
> >>>
> >>> Sent from my iPhone on 3
> >>>
> >>> On 20 Apr 2013, at 00:50, Joe Astorino <joeastorino1982_at_gmail.com>
> wrote:
> >>>
> >>>> CPU and memory seem to be OK. sh proc cpu shows it sitting about
> >>>> 7-10% when
> >>> the problem occurs. sh proc cpu history does not show any really bad
> spikes.
> >>> sho proc mem seems to show I have plenty of free RAM as well.
> >>>>
> >>>>
> >>>> On Fri, Apr 19, 2013 at 7:43 PM, Joe Astorino
> >>>> <joeastorino1982_at_gmail.com>
> >>> wrote:
> >>>> Sorry, 2GB of RAM is what it has
> >>>>
> >>>>
> >>>> On Fri, Apr 19, 2013 at 7:41 PM, Matt Bentley
> >>>> <mattdbentley_at_gmail.com>
> >>> wrote:
> >>>> Interesting. I've done testing with Spirent and other traffic-gen
> >>> platforms. I never saw shaping add to the latency - until I actually
> got up close to the CIR - maybe slightly with a lower-end platform. But I
> don't think shaping even activates until it senses backpressure and
> activates "itself".
> >>>>
> >>>>
> >>>> On Fri, Apr 19, 2013 at 4:17 PM, Tony Singh <mothafungla_at_gmail.com>
> wrote:
> >>>> The reason why you get the extra delay when the policy is applied is
> >>>> down to
> >>> the extra work the CPU has to do on top of inbound congestion, what's
> your memory/free?
> >>>>
> >>>> I would also raise a TAC to get the definitive answer "show
> >>>> tech-support" &
> >>> pass to them
> >>>>
> >>>> --
> >>>> BR
> >>>>
> >>>> Sent from my iPhone on 3
> >>>>
> >>>> On 19 Apr 2013, at 23:13, Joe Astorino <joeastorino1982_at_gmail.com>
> wrote:
> >>>>
> >>>>> Another interesting fact - When the policy is applied as shown
> >>>>> above, the problem only occurs when the inbound RX is congested.
> >>>>>
> >>>>> So basically, if the service-policy is applied outbound towards the
> >>>>> WAN
> >>> as
> >>>>> shown, and the output utilization of the WAN link (TX) is low, and
> >>>>> the inbound (RX) utilization is low, everything is fine.
> >>>>>
> >>>>> In the event that the inbound utilization of the interface (RX) is
> >>>>> significant, and the policy is applied outbound as usual, the policy
> >>>>> is adding the 200-400ms extra delay. If I remove the policy, the
> >>>>> extra delay goes away.
> >>>>>
> >>>>> I guess I don't understand why inbound congestion would change how
> >>>>> the outbound queue / shaping works.
> >>>>>
> >>>>> input is low, output is low, policy applied: fine input is high,
> >>>>> output is low, policy not applied: expected poor response times
> >>>>> input is high, output is low, policy is applied: expected poor
> >>>>> response time PLUS an additional 200-400ms of delay
> >>>>>
> >>>>>
> >>>>> On Fri, Apr 19, 2013 at 5:49 PM, Joe Astorino
> >>> <joeastorino1982_at_gmail.com>wrote:
> >>>>>
> >>>>>> I have a 3945 ISR router connected via a GigabitEthernet link to a
> >>> service
> >>>>>> provider that is providing 10Mbps WAN access. I wish to use CBWFQ
> >>>>>> and
> >>> the
> >>>>>> service provider requires dot1q tagging. As such, I must shape my
> >>>>>> sub-interface and use a hierarchical type QoS policy...no big deal
> >>>>>>
> >>>>>> policy-map WAN-EDGE
> >>>>>> class VOICE
> >>>>>> priority percent 20
> >>>>>> !
> >>>>>> class VIDEO
> >>>>>> bandwidth remaining percent 60
> >>>>>> queue-limit 128 packets
> >>>>>> !
> >>>>>> class APPS_SIGNALING
> >>>>>> set dscp af21
> >>>>>> bandwidth remaining percent 30
> >>>>>> !
> >>>>>> class class-default
> >>>>>> bandwidth remaining percent 10
> >>>>>> !
> >>>>>> !
> >>>>>> policy-map SHAPE-OUT
> >>>>>> class class-default
> >>>>>> shape average 10000000
> >>>>>> service-policy WAN-EDGE
> >>>>>> !
> >>>>>> !
> >>>>>> int gi0/1.2
> >>>>>> encapsulation dot1q 2
> >>>>>> ip address ...
> >>>>>> service-policy output SHAPE-OUT
> >>>>>>
> >>>>>>
> >>>>>> Here is the very strange thing. Even when the TX of this WAN
> >>>>>> interface
> >>> is
> >>>>>> barely being used, when and only when the service-policy is
> >>>>>> applied, the response in pings to anything behind this router
> >>>>>> increase immediately by 200-400ms. Immediately after removing the
> >>>>>> service-policy things return
> >>> to
> >>>>>> normal.
> >>>>>>
> >>>>>> Investigating the output of "show policy-map interface" reveals a
> >>>>>> large number of output drops in the shaper class-default. So far I
> >>>>>> have tried
> >>>>>>
> >>>>>> - increasing the queue-limit to many different combinations
> >>>>>> - increasing the burst size
> >>>>>>
> >>>>>> I also have the exact same configuration on another remote site
> >>>>>> router running the same version of IOS with the same setup (10Mbps
> >>>>>> Ethernet WAN
> >>>>>> link) on the same platform.
> >>>>>>
> >>>>>> I'm really baffled as to what would cause this. Any insight
> >>> appreciated.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Regards,
> >>>>>>
> >>>>>> Joe Astorino
> >>>>>> CCIE #24347
> >>>>>> http://astorinonetworks.com
> >>>>>>
> >>>>>> "He not busy being born is busy dying" - Dylan
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Regards,
> >>>>>
> >>>>> Joe Astorino
> >>>>> CCIE #24347
> >>>>> http://astorinonetworks.com
> >>>>>
> >>>>> "He not busy being born is busy dying" - Dylan
> >>>>>
> >>>>>
> >>>>> Blogs and organic groups at http://www.ccie.net
> >>>>>
> >>>>> ____________________________________________________________________
> >>>>> ___ Subscription information may be found at:
> >>>>> http://www.groupstudy.com/list/CCIELab.html
> >>>>
> >>>>
> >>>> Blogs and organic groups at http://www.ccie.net
> >>>>
> >>>> _____________________________________________________________________
> >>>> _ _ Subscription information may be found at:
> >>>> http://www.groupstudy.com/list/CCIELab.html
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Regards,
> >>>>
> >>>> Joe Astorino
> >>>> CCIE #24347
> >>>> http://astorinonetworks.com
> >>>>
> >>>> "He not busy being born is busy dying" - Dylan
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Regards,
> >>>>
> >>>> Joe Astorino
> >>>> CCIE #24347
> >>>> http://astorinonetworks.com
> >>>>
> >>>> "He not busy being born is busy dying" - Dylan
> >>>
> >>>
> >>> Blogs and organic groups at http://www.ccie.net
> >>>
> >>> ______________________________________________________________________
> >>> _ Subscription information may be found at:
> >>> http://www.groupstudy.com/list/CCIELab.html
> >>
> >>
>

-- 
Regards,
Joe Astorino
CCIE #24347
http://astorinonetworks.com
"He not busy being born is busy dying" - Dylan
Blogs and organic groups at http://www.ccie.net