Re: Shapers, CBWFQ, and tx_ring, oh my!

Found another graphic in the CCIE R&S Cert Guide that again redefined what
I think is happening! I think there is a shaping active check early in the
process that doesn't actually use a shaping queue -- it just checks to see
if shaping is currently active or not.

This is with a parent policy-map with a shaper, and a child policy-map with
many classes and reserved bandwidth.

- Packet needs to be sent out an interface
- If shaping is not active, go directly to the interface software queue
(FIFO, WFQ, etc) and then the tx_ring
- if shaping is active, pass the packet into the child policy-map for
prioritization, marking, policing, etc
- the child policy-map immediately processes the traffic and puts it into
the shaping queue in the correct prioritized order
- the shaper dequeues the shaping queue into the interface software queue
(FIFO, WFQ) at the shaping rate as normal
- the interface software queue feeds into the tx_ring, where the packet is
finally transmitted

The thing that might disprove this is the 'show policy-map interface'
command, which shows that both the shaping queue and the child policy-map
queues fill up. If the above were true, I would expect the child
policy-map queues to process data, but not actually increase in size. This
may be a show command anomaly?

Okay, hit me -- where have I got it wrong?

Keller Giacomarro
keller.g_at_gmail.com

On Sat, Jul 14, 2012 at 7:32 PM, Keller Giacomarro <keller.g_at_gmail.com>wrote:

> Thanks everyone for the good information -- I think that this is a more
> complicated topic than I realized.
>
>
> I think Carlos hit it on the head; what's going on the router doesn't
> exactly line up with the metaphors of queues that we are taught. In this
> case, although we configure a CBWFQ policy as a child policy of a shaper
> policy-map, the router doesn't really handle it that way.
>
> Traffic shaping is its own thing.
> CBWFQ using 'bandwidth' and 'priority' is its own thing.
> Combining the two doesn't chain them together -- it is its own separate
> process.
>
> The way I'm interpreting what everyone is saying is this:
> tx_ring is minimal
> When a shaper policy-map is configured with a bandwidth/priority policy as
> its child, the shaping is being done by directly interacting with the child
> policy-map's queues. There is no shaper queue that backs up into the CBWFQ
> queues. Instead, the CBWFQ queues are being shaped themselves.
>
> Am I correct in saying that we only have two layers of queues in play?
>
> What I thought it was before:
> CBWFQ queues (multiple) -> Traffic Shaper (single) -> tx_ring -> interface
> This would imply that as the shaper queues packets because the CIR is
> exceeded, they are fed back into the CBWFQ queues. The traffic shaper then
> de-queues from the CBWFQ queues in accordance with the bandwidth/priority
> settings in the CBWFQ queues.
>
> However, it is sounding like the process is not nearly that
> straightforward. It's almost like the router is using the CBWFQ queues IN
> PLACE OF the shaper queue, that they are one and the same.
>
> Regardless, I think I have a handle enough on the functionality for the
> exam -- the nitty gritty is just a matter of curiosity now. Interested in
> seeing the internal workings of it.
>
> Responses:
>
> I don't fully understand what you are saying here. What do you mean by
> "without full flow information" ?
>
>
> I think I was confused here, my understanding is a little different now.
> I still don't know what the Cisco poster meant by the shaper using WFQ/HQF
> to dequeue the CBWFQ queues, as I would expect dequeueing to be determined
> by the settings in the CBWFQ queues.
>
> Please attach the policy config and what is that drives your attention
> from this output!
>
>
> It is based on the simplified config I posted earlier in the thread:
>
> policy-map pm-shaper
> class class-default
> shaper average percent 100
> service-policy pm-cbwfq
> !
> policy-map pm-cbwfq
> class cm-ssh
> bandwidth percent 5
> class cm-http
> bandwidth percent 20
> class class-default
> fair-queue
>
> And in the original output I highlighted in bold, which apparently
> doesn't stick when sent via the mailing list.
>
> gateway#show policy-map int f0/1
> FastEthernet0/1
>
> Service-policy output: pm-wan-out-shaper
>
> Class-map: class-default (match-any)
> 600317 packets, 143960388 bytes
> 30 second offered rate 631000 bps, drop rate 14000 bps
> Match: any
> Traffic Shaping
> Target/Average Byte Sustain Excess Interval Increment
> Rate Limit bits/int bits/int (ms) (bytes)
> 600000/600000 937 3750 3750 6 468
>
> Adapt Queue Packets Bytes Packets Bytes Shaping
> Active Depth Delayed Delayed Active
> ---> - 3 598394 141284492 90698 94225802 yes
>
>
> Class-map: cm-ssh (match-all)
> 70238 packets, 105192011 bytes
> 30 second offered rate 621000 bps, drop rate 14000 bps
> Match: protocol ssh
> Queueing
> Output Queue: Conversation 74
> Bandwidth 5 (%)
> Bandwidth 30 (kbps)
> (pkts matched/bytes matched) 62464/93671468
> ----> (depth/total drops/no-buffer drops) 2/1816/3
> exponential weight: 9
> mean queue depth: 1
>
>
> The traffic shaper depth is 3, while the only class with any sort of
> traffic is 2. I am now thinking that this is a show command anomaly, they
> should be the same.
>
> Thanks again, all -- your comments have really helped!
>
> Keller Giacomarro
> keller.g_at_gmail.com
>
>
>
> On Sat, Jul 14, 2012 at 7:10 PM, Carlos G Mendioroz <tron_at_huapi.ba.ar>wrote:
>
>> Keller Giacomarro @ 14/07/2012 07:07 -0300 dixit:
>>
>> I am trying to wrap my head around how QoS queueing actually functions.
>>> If
>>> you're able, please confirm or debunk my understanding!
>>>
>>
>> Let's see if I can help. In any case, the guys that usually chime in with
>> details are Peter Lapukhov and, in other forums, Ivan Pepelnjak.
>> One issue with queueing is that usually a metaphor is used to
>> understand/explain it, but it does not always play nice down to
>> implementation details.
>>
>>
>> Your 'normal' QoS setup involves a CBWFQ policy-map applied outbound on a
>>> WAN interface. As the tx_ring fills up, packets are queued into the
>>> CBWFQ
>>> policy-map queues (one per class-map), and are dequeued as normal. The
>>> tx_ring filling up is the trigger for filling queues. Okay, simple so
>>> far.
>>>
>>
>> Sort of. The tx_ring filling up is the trigger to start "software
>> queues". But some of the CBWFQ configuration may be active even if the
>> tx_ring does not fill: police and shape commands (and remarking).
>>
>>
>> The complication comes when you have an interface with a rate limit
>>> higher
>>> than your CIR (like a home Internet connection via 100Mbps ethernet with
>>> a
>>> CIR of 512Kbps). If CBWFQ is applied directly to the interface, even if
>>> the bandwidth is set, the tx_ring clears faster than the WAN circuit will
>>> take the data, and the software queues are bypassed entirely. In this
>>> situation, applying a CBWFQ policy-map directly to the interface, even
>>> setting the bandwidth command, does absolutely nothing.
>>>
>>> Here's where I get fuzzier. The solution to this is to put something
>>> else
>>> between the CBWFQ policy-map and the tx_ring: a shaper via nested policy
>>> maps. The shaper is configured to the correct CIR. As the shaper sees
>>> that the interface is transmitting too fast, it begins to fill up the
>>> CBWFQ
>>> policy-map queues instead of transmitting. In this way, the physical
>>> interface is faster than the CIR but we still create the necessary
>>> 'backpressure' to fill up the software queues.
>>>
>>
>> I would not call this "backpressure". And the shaper is not "seeing" the
>> interface txing too fast, because it will never do so.
>>
>> Let's recap, it's all about when something happens, and that something is
>> packet transmission.
>>
>> When tx_ring is empty, a packet is transmitted when a packet is received.
>> So the trigger is packet reception. (And the code that does the tx is in
>> the rx interrupt service). (Well, not really transmitted, but enqueued at a
>> hardware level, so forget about it, its gone :)
>>
>> Now, if the tx_ring is full, the rx interrupt service just lets the
>> packet in "a queue". It has to check first if the "queue" has space,
>> and this is quite involved too.
>> Then, when the tx_ring gets a place, it looks for which packet from the
>> queue has to be transmitted next. The tx is now controlled by the AR, the
>> rate of the tx interface. And CBWFQ is just a fancy way of defining which
>> should be the next packet to be xmitted.
>>
>> When a shaper enters the mix, basically there's another trigger. If the
>> tx_ring gets space, it now has to check if actually getting a packet from
>> the queue would exceed the shaper rate, and if it would... it has to do
>> nothing. Well, nothing for a calculated amount of time. Call it sleep,
>> delay, whatever.
>>
>>
>> Two things muck with my understanding:
>>> In https://supportforums.cisco.**com/thread/2132501<https://supportforums.cisco.com/thread/2132501>, a Cisco employee says
>>> that the shaper uses WFQ (or HQF in the newest releases) to de-queue the
>>> CBWFQ queues. Why is the shaper implementing any dequeueing strategy at
>>> all? Shouldn't the CBWFQ policy-map be handling that (such as policy
>>> queues going first, etc)? And how can it possibly do that without full
>>> flow information?
>>>
>>
>> I don't fully understand what you are saying here. What do you mean by
>> "without full flow information" ?
>> But the "shaper" has to choose the next packet to tx, and that usually
>> comes from a child policy. Or may be fair-queues, parallel flows
>> (conversations) with a neat selection algorithm.
>>
>> The other issue is that the show commands on my router support my
>>> understanding...almost. If I'm moving a lot of ssh data upstream (via
>>> scp), I can see the shaper queue fill and the CBWFQ queue fill, makes
>>> sense. Most of the time their values are the same. However, they do
>>> on occasion differ by a number or two. Show command artifact, or an
>>> indication that I have no idea what I'm talking about?
>>>
>>> gateway#show policy-map int f0/1
>>> FastEthernet0/1
>>>
>>> Service-policy output: pm-wan-out-shaper
>>>
>>> Class-map: class-default (match-any)
>>> 600317 packets, 143960388 bytes
>>> 30 second offered rate 631000 bps, drop rate 14000 bps
>>> Match: any
>>> Traffic Shaping
>>> Target/Average Byte Sustain Excess Interval
>>> Increment
>>> Rate Limit bits/int bits/int (ms) (bytes)
>>> 600000/600000 937 3750 3750 6 468
>>>
>>> Adapt *Queue* Packets Bytes Packets Bytes
>>> Shaping
>>> Active *Depth* Delayed Delayed
>>> Active
>>> - *3* 598394 141284492 90698 94225802 yes
>>>
>>>
>>> Class-map: cm-ssh (match-all)
>>> 70238 packets, 105192011 bytes
>>> 30 second offered rate 621000 bps, drop rate 14000 bps
>>> Match: protocol ssh
>>> Queueing
>>> Output Queue: Conversation 74
>>> Bandwidth 5 (%)
>>> Bandwidth 30 (kbps)
>>> (pkts matched/bytes matched) 62464/93671468
>>> *(depth/total drops/no-buffer drops) 2*/1816/3
>>>
>>> exponential weight: 9
>>> mean queue depth: 1
>>>
>>
>> Please attach the policy config and what is that drives your attention
>> from this output!
>>
>>
>>
>>> Appreciate your input -- hopefully this helps someone else too, as none
>>> of
>>> the standard study resources I've read have adequately explained how this
>>> works!
>>>
>>> Keller Giacomarro
>>> keller.g_at_gmail.com
>>>
>>>
>> HTH,
>> -Carlos
>>
>> --
>> Carlos G Mendioroz <tron_at_huapi.ba.ar> LW7 EQI Argentina

Blogs and organic groups at http://www.ccie.net
Received on Sat Jul 14 2012 - 23:57:31 ART