Re: Puzzling QoS Issue

Ha ha...

Amazing

From: Joe Astorino [mailto:joeastorino1982_at_gmail.com]
Sent: Monday, April 22, 2013 09:19 AM
To: Joseph L. Brunner
Cc: Tony Singh <mothafungla_at_gmail.com>; Matt Bentley <mattdbentley_at_gmail.com>; Groupstudy <ccielab_at_groupstudy.com>
Subject: Re: Puzzling QoS Issue

And the root cause was (drumroll please) CEF was disabled somehow.

On Sat, Apr 20, 2013 at 12:35 PM, Joe Astorino <joeastorino1982_at_gmail.com<mailto:joeastorino1982_at_gmail.com>> wrote:
Early reports today suggest downgrade to 15.1 fixed the issue!

Sent from my iPhone

On Apr 19, 2013, at 10:50 PM, Joe Astorino <joeastorino1982_at_gmail.com<mailto:joeastorino1982_at_gmail.com>> wrote:

> The carrier is AT&T. Enough said lol
>
> Sent from my iPad
>
> On Apr 19, 2013, at 10:37 PM, "Joseph L. Brunner"
> <joe_at_affirmedsystems.com<mailto:joe_at_affirmedsystems.com>> wrote:
>
>> We can only speculate until you get the right person on the phone;
>>
>> But if you remove the shaper and run the line hard - when do the pings get to 200-400ms?
>>
>> Here's a better question - why does the carrier REQUIRE 802.1q tags?
>>
>> Sounds again like youre not talking to the right person...
>>
>> That's a highly irregular config...
>>
>> What if you didn't have a router capable of 802.1q tagging?
>>
>> Would they let you out of your contract?
>>
>> LOL
>>
>> My policy has all routers at the edge with routed interfaces - wouldn't I just ignore the tag if they put the ip on a vlan int. without tagging?
>>
>> There is more to this we have to dig into...
>>
>> I think the tags are effecting something - just got to find out where...
>>
>> -----Original Message-----
>> From: Joe Astorino [mailto:joeastorino1982_at_gmail.com<mailto:joeastorino1982_at_gmail.com>]
>> Sent: Friday, April 19, 2013 10:33 PM
>> To: Joseph L. Brunner
>> Cc: Tony Singh; Matt Bentley; Groupstudy
>> Subject: Re: Puzzling QoS Issue
>>
>> Joseph,
>>
>> If it was a carrier shaping/policing issue though, why does removing the shaper on my end resolve the issue
>>
>> Sent from my iPad
>>
>> On Apr 19, 2013, at 9:20 PM, "Joseph L. Brunner"
>> <joe_at_affirmedsystems.com<mailto:joe_at_affirmedsystems.com>> wrote:
>>
>>> Your carrier has a aggregate shaper applied or some other type of qos mechanism applied.
>>>
>>> You're maxing it pal.
>>>
>>> They probably did the wrong "BE" setting, etc.
>>>
>>> :)
>>>
>>> -----Original Message-----
>>> From: nobody_at_groupstudy.com<mailto:nobody_at_groupstudy.com> [mailto:nobody_at_groupstudy.com<mailto:nobody_at_groupstudy.com>] On Behalf
>>> Of Tony Singh
>>> Sent: Friday, April 19, 2013 8:32 PM
>>> To: Joe Astorino
>>> Cc: Matt Bentley; Groupstudy
>>> Subject: Re: Puzzling QoS Issue
>>>
>>> yeh without further tests it's hard to say what the issue is, what's
>>> the uptime? is the code/license identical to your other working router
>>> ;)
>>>
>>> --
>>> BR
>>>
>>> Tony
>>>
>>> Sent from my iPhone on 3
>>>
>>> On 20 Apr 2013, at 00:50, Joe Astorino <joeastorino1982_at_gmail.com<mailto:joeastorino1982_at_gmail.com>> wrote:
>>>
>>>> CPU and memory seem to be OK. sh proc cpu shows it sitting about
>>>> 7-10% when
>>> the problem occurs. sh proc cpu history does not show any really bad spikes.
>>> sho proc mem seems to show I have plenty of free RAM as well.
>>>>
>>>>
>>>> On Fri, Apr 19, 2013 at 7:43 PM, Joe Astorino
>>>> <joeastorino1982_at_gmail.com<mailto:joeastorino1982_at_gmail.com>>
>>> wrote:
>>>> Sorry, 2GB of RAM is what it has
>>>>
>>>>
>>>> On Fri, Apr 19, 2013 at 7:41 PM, Matt Bentley
>>>> <mattdbentley_at_gmail.com<mailto:mattdbentley_at_gmail.com>>
>>> wrote:
>>>> Interesting. I've done testing with Spirent and other traffic-gen
>>> platforms. I never saw shaping add to the latency - until I actually got up close to the CIR - maybe slightly with a lower-end platform. But I don't think shaping even activates until it senses backpressure and activates "itself".
>>>>
>>>>
>>>> On Fri, Apr 19, 2013 at 4:17 PM, Tony Singh <mothafungla_at_gmail.com<mailto:mothafungla_at_gmail.com>> wrote:
>>>> The reason why you get the extra delay when the policy is applied is
>>>> down to
>>> the extra work the CPU has to do on top of inbound congestion, what's your memory/free?
>>>>
>>>> I would also raise a TAC to get the definitive answer "show
>>>> tech-support" &
>>> pass to them
>>>>
>>>> --
>>>> BR
>>>>
>>>> Sent from my iPhone on 3
>>>>
>>>> On 19 Apr 2013, at 23:13, Joe Astorino <joeastorino1982_at_gmail.com<mailto:joeastorino1982_at_gmail.com>> wrote:
>>>>
>>>>> Another interesting fact - When the policy is applied as shown
>>>>> above, the problem only occurs when the inbound RX is congested.
>>>>>
>>>>> So basically, if the service-policy is applied outbound towards the
>>>>> WAN
>>> as
>>>>> shown, and the output utilization of the WAN link (TX) is low, and
>>>>> the inbound (RX) utilization is low, everything is fine.
>>>>>
>>>>> In the event that the inbound utilization of the interface (RX) is
>>>>> significant, and the policy is applied outbound as usual, the policy
>>>>> is adding the 200-400ms extra delay. If I remove the policy, the
>>>>> extra delay goes away.
>>>>>
>>>>> I guess I don't understand why inbound congestion would change how
>>>>> the outbound queue / shaping works.
>>>>>
>>>>> input is low, output is low, policy applied: fine input is high,
>>>>> output is low, policy not applied: expected poor response times
>>>>> input is high, output is low, policy is applied: expected poor
>>>>> response time PLUS an additional 200-400ms of delay
>>>>>
>>>>>
>>>>> On Fri, Apr 19, 2013 at 5:49 PM, Joe Astorino
>>> <joeastorino1982_at_gmail.com<mailto:joeastorino1982_at_gmail.com>>wrote:
>>>>>
>>>>>> I have a 3945 ISR router connected via a GigabitEthernet link to a
>>> service
>>>>>> provider that is providing 10Mbps WAN access. I wish to use CBWFQ
>>>>>> and
>>> the
>>>>>> service provider requires dot1q tagging. As such, I must shape my
>>>>>> sub-interface and use a hierarchical type QoS policy...no big deal
>>>>>>
>>>>>> policy-map WAN-EDGE
>>>>>> class VOICE
>>>>>> priority percent 20
>>>>>> !
>>>>>> class VIDEO
>>>>>> bandwidth remaining percent 60
>>>>>> queue-limit 128 packets
>>>>>> !
>>>>>> class APPS_SIGNALING
>>>>>> set dscp af21
>>>>>> bandwidth remaining percent 30
>>>>>> !
>>>>>> class class-default
>>>>>> bandwidth remaining percent 10
>>>>>> !
>>>>>> !
>>>>>> policy-map SHAPE-OUT
>>>>>> class class-default
>>>>>> shape average 10000000
>>>>>> service-policy WAN-EDGE
>>>>>> !
>>>>>> !
>>>>>> int gi0/1.2
>>>>>> encapsulation dot1q 2
>>>>>> ip address ...
>>>>>> service-policy output SHAPE-OUT
>>>>>>
>>>>>>
>>>>>> Here is the very strange thing. Even when the TX of this WAN
>>>>>> interface
>>> is
>>>>>> barely being used, when and only when the service-policy is
>>>>>> applied, the response in pings to anything behind this router
>>>>>> increase immediately by 200-400ms. Immediately after removing the
>>>>>> service-policy things return
>>> to
>>>>>> normal.
>>>>>>
>>>>>> Investigating the output of "show policy-map interface" reveals a
>>>>>> large number of output drops in the shaper class-default. So far I
>>>>>> have tried
>>>>>>
>>>>>> - increasing the queue-limit to many different combinations
>>>>>> - increasing the burst size
>>>>>>
>>>>>> I also have the exact same configuration on another remote site
>>>>>> router running the same version of IOS with the same setup (10Mbps
>>>>>> Ethernet WAN
>>>>>> link) on the same platform.
>>>>>>
>>>>>> I'm really baffled as to what would cause this. Any insight
>>> appreciated.
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>> Joe Astorino
>>>>>> CCIE #24347
>>>>>> http://astorinonetworks.com
>>>>>>
>>>>>> "He not busy being born is busy dying" - Dylan
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Joe Astorino
>>>>> CCIE #24347
>>>>> http://astorinonetworks.com
>>>>>
>>>>> "He not busy being born is busy dying" - Dylan
>>>>>
>>>>>
>>>>> Blogs and organic groups at http://www.ccie.net
>>>>>
>>>>> ____________________________________________________________________
>>>>> ___ Subscription information may be found at:
>>>>> http://www.groupstudy.com/list/CCIELab.html
>>>>
>>>>
>>>> Blogs and organic groups at http://www.ccie.net
>>>>
>>>> _____________________________________________________________________
>>>> _ _ Subscription information may be found at:
>>>> http://www.groupstudy.com/list/CCIELab.html
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Joe Astorino
>>>> CCIE #24347
>>>> http://astorinonetworks.com
>>>>
>>>> "He not busy being born is busy dying" - Dylan
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Joe Astorino
>>>> CCIE #24347
>>>> http://astorinonetworks.com
>>>>
>>>> "He not busy being born is busy dying" - Dylan
>>>
>>>
>>> Blogs and organic groups at http://www.ccie.net
>>>
>>> ______________________________________________________________________
>>> _ Subscription information may be found at:
>>> http://www.groupstudy.com/list/CCIELab.html
>>
>>

--
Regards,
Joe Astorino
CCIE #24347
http://astorinonetworks.com
"He not busy being born is busy dying" - Dylan
Blogs and organic groups at http://www.ccie.net