Re: Shapers, CBWFQ, and tx_ring, oh my!

Keller Giacomarro @ 15/07/2012 01:57 -0300 dixit:
> Found another graphic in the CCIE R&S Cert Guide that again redefined what
> I think is happening! I think there is a shaping active check early in the
> process that doesn't actually use a shaping queue -- it just checks to see
> if shaping is currently active or not.

Sure. As I said, explicit policing & shaping work even if software
queues are not engaged. This is just what's needed: have we exceeded our
rate ? (i.e. ar/should we queue ?) More or less same thing with
policing, but decision there is to tx or drop.

> This is with a parent policy-map with a shaper, and a child policy-map with
> many classes and reserved bandwidth.
>
> - Packet needs to be sent out an interface
> - If shaping is not active, go directly to the interface software queue
> (FIFO, WFQ, etc) and then the tx_ring
> - if shaping is active, pass the packet into the child policy-map for
> prioritization, marking, policing, etc

This is a two part process. I would guess that the first part is just
queue (rx interrupt time) after checking for queue limits.
(i.e. the queue actually exists in the sense of # of buffers and
scheduling policy: # of buffers matters when accepting a new packet)
The child map actually defines the way the scheduler works, i.e., order
of tx.

> - the child policy-map immediately processes the traffic and puts it into
> the shaping queue in the correct prioritized order

Not the way I see it.

> - the shaper dequeues the shaping queue into the interface software queue
> (FIFO, WFQ) at the shaping rate as normal

Well, this is something I believe does not happen.
The actual "queue" depends on where the shaping is "activated". It may
be an interface queue (1000 packets AFAIK), or a tunnel queue, or a
subinterface queue that has to be created adhoc because of shaping.
What matters, I guess, is that you have to be able to understand how it
behaves even though some of the details of how it is implemented are not
readily available.
Look at
http://www.cisco.com/en/US/products/hw/routers/ps341/products_tech_note09186a0080af893d.shtml

> - the interface software queue feeds into the tx_ring, where the packet is
> finally transmitted

Yes, but I would says tx_ring feeds from the queue, to make it evident
that "the scheduler" is driven by the tx_ring needing data, may be
slowed by a shaper :)

> The thing that might disprove this is the 'show policy-map interface'
> command, which shows that both the shaping queue and the child policy-map
> queues fill up. If the above were true, I would expect the child
> policy-map queues to process data, but not actually increase in size. This
> may be a show command anomaly?

I still think that it is "the same queue" as in "the same buffer space",
but controlled by chained logic. The class queue is "inside" the
interface queue, so to say. But again, if you get into a level of
understanding that gives you no surprises, your model of the working has
not to be an exact replica of mine or anybody elses. But I digress.

-Carlos

>
> Okay, hit me -- where have I got it wrong?
>
> Keller Giacomarro
> keller.g_at_gmail.com
>
>
> On Sat, Jul 14, 2012 at 7:32 PM, Keller Giacomarro <keller.g_at_gmail.com>wrote:
>
>> Thanks everyone for the good information -- I think that this is a more
>> complicated topic than I realized.
>>
>>
>> I think Carlos hit it on the head; what's going on the router doesn't
>> exactly line up with the metaphors of queues that we are taught. In this
>> case, although we configure a CBWFQ policy as a child policy of a shaper
>> policy-map, the router doesn't really handle it that way.
>>
>> Traffic shaping is its own thing.
>> CBWFQ using 'bandwidth' and 'priority' is its own thing.
>> Combining the two doesn't chain them together -- it is its own separate
>> process.
>>
>> The way I'm interpreting what everyone is saying is this:
>> tx_ring is minimal
>> When a shaper policy-map is configured with a bandwidth/priority policy as
>> its child, the shaping is being done by directly interacting with the child
>> policy-map's queues. There is no shaper queue that backs up into the CBWFQ
>> queues. Instead, the CBWFQ queues are being shaped themselves.
>>
>> Am I correct in saying that we only have two layers of queues in play?
>>
>> What I thought it was before:
>> CBWFQ queues (multiple) -> Traffic Shaper (single) -> tx_ring -> interface
>> This would imply that as the shaper queues packets because the CIR is
>> exceeded, they are fed back into the CBWFQ queues. The traffic shaper then
>> de-queues from the CBWFQ queues in accordance with the bandwidth/priority
>> settings in the CBWFQ queues.
>>
>> However, it is sounding like the process is not nearly that
>> straightforward. It's almost like the router is using the CBWFQ queues IN
>> PLACE OF the shaper queue, that they are one and the same.
>>
>> Regardless, I think I have a handle enough on the functionality for the
>> exam -- the nitty gritty is just a matter of curiosity now. Interested in
>> seeing the internal workings of it.
>>
>> Responses:
>>
>> I don't fully understand what you are saying here. What do you mean by
>> "without full flow information" ?
>>
>>
>> I think I was confused here, my understanding is a little different now.
>> I still don't know what the Cisco poster meant by the shaper using WFQ/HQF
>> to dequeue the CBWFQ queues, as I would expect dequeueing to be determined
>> by the settings in the CBWFQ queues.
>>
>> Please attach the policy config and what is that drives your attention
>> from this output!
>>
>>
>> It is based on the simplified config I posted earlier in the thread:
>>
>> policy-map pm-shaper
>> class class-default
>> shaper average percent 100
>> service-policy pm-cbwfq
>> !
>> policy-map pm-cbwfq
>> class cm-ssh
>> bandwidth percent 5
>> class cm-http
>> bandwidth percent 20
>> class class-default
>> fair-queue
>>
>> And in the original output I highlighted in bold, which apparently
>> doesn't stick when sent via the mailing list.
>>
>> gateway#show policy-map int f0/1
>> FastEthernet0/1
>>
>> Service-policy output: pm-wan-out-shaper
>>
>> Class-map: class-default (match-any)
>> 600317 packets, 143960388 bytes
>> 30 second offered rate 631000 bps, drop rate 14000 bps
>> Match: any
>> Traffic Shaping
>> Target/Average Byte Sustain Excess Interval Increment
>> Rate Limit bits/int bits/int (ms) (bytes)
>> 600000/600000 937 3750 3750 6 468
>>
>> Adapt Queue Packets Bytes Packets Bytes Shaping
>> Active Depth Delayed Delayed Active
>> ---> - 3 598394 141284492 90698 94225802 yes
>>
>>
>> Class-map: cm-ssh (match-all)
>> 70238 packets, 105192011 bytes
>> 30 second offered rate 621000 bps, drop rate 14000 bps
>> Match: protocol ssh
>> Queueing
>> Output Queue: Conversation 74
>> Bandwidth 5 (%)
>> Bandwidth 30 (kbps)
>> (pkts matched/bytes matched) 62464/93671468
>> ----> (depth/total drops/no-buffer drops) 2/1816/3
>> exponential weight: 9
>> mean queue depth: 1
>>
>>
>> The traffic shaper depth is 3, while the only class with any sort of
>> traffic is 2. I am now thinking that this is a show command anomaly, they
>> should be the same.
>>
>> Thanks again, all -- your comments have really helped!
>>
>> Keller Giacomarro
>> keller.g_at_gmail.com
>>
>>
>>
>> On Sat, Jul 14, 2012 at 7:10 PM, Carlos G Mendioroz <tron_at_huapi.ba.ar>wrote:
>>
>>> Keller Giacomarro @ 14/07/2012 07:07 -0300 dixit:
>>>
>>> I am trying to wrap my head around how QoS queueing actually functions.
>>>> If
>>>> you're able, please confirm or debunk my understanding!
>>>>
>>>
>>> Let's see if I can help. In any case, the guys that usually chime in with
>>> details are Peter Lapukhov and, in other forums, Ivan Pepelnjak.
>>> One issue with queueing is that usually a metaphor is used to
>>> understand/explain it, but it does not always play nice down to
>>> implementation details.
>>>
>>>
>>> Your 'normal' QoS setup involves a CBWFQ policy-map applied outbound on a
>>>> WAN interface. As the tx_ring fills up, packets are queued into the
>>>> CBWFQ
>>>> policy-map queues (one per class-map), and are dequeued as normal. The
>>>> tx_ring filling up is the trigger for filling queues. Okay, simple so
>>>> far.
>>>>
>>>
>>> Sort of. The tx_ring filling up is the trigger to start "software
>>> queues". But some of the CBWFQ configuration may be active even if the
>>> tx_ring does not fill: police and shape commands (and remarking).
>>>
>>>
>>> The complication comes when you have an interface with a rate limit
>>>> higher
>>>> than your CIR (like a home Internet connection via 100Mbps ethernet with
>>>> a
>>>> CIR of 512Kbps). If CBWFQ is applied directly to the interface, even if
>>>> the bandwidth is set, the tx_ring clears faster than the WAN circuit will
>>>> take the data, and the software queues are bypassed entirely. In this
>>>> situation, applying a CBWFQ policy-map directly to the interface, even
>>>> setting the bandwidth command, does absolutely nothing.
>>>>
>>>> Here's where I get fuzzier. The solution to this is to put something
>>>> else
>>>> between the CBWFQ policy-map and the tx_ring: a shaper via nested policy
>>>> maps. The shaper is configured to the correct CIR. As the shaper sees
>>>> that the interface is transmitting too fast, it begins to fill up the
>>>> CBWFQ
>>>> policy-map queues instead of transmitting. In this way, the physical
>>>> interface is faster than the CIR but we still create the necessary
>>>> 'backpressure' to fill up the software queues.
>>>>
>>>
>>> I would not call this "backpressure". And the shaper is not "seeing" the
>>> interface txing too fast, because it will never do so.
>>>
>>> Let's recap, it's all about when something happens, and that something is
>>> packet transmission.
>>>
>>> When tx_ring is empty, a packet is transmitted when a packet is received.
>>> So the trigger is packet reception. (And the code that does the tx is in
>>> the rx interrupt service). (Well, not really transmitted, but enqueued at a
>>> hardware level, so forget about it, its gone :)
>>>
>>> Now, if the tx_ring is full, the rx interrupt service just lets the
>>> packet in "a queue". It has to check first if the "queue" has space,
>>> and this is quite involved too.
>>> Then, when the tx_ring gets a place, it looks for which packet from the
>>> queue has to be transmitted next. The tx is now controlled by the AR, the
>>> rate of the tx interface. And CBWFQ is just a fancy way of defining which
>>> should be the next packet to be xmitted.
>>>
>>> When a shaper enters the mix, basically there's another trigger. If the
>>> tx_ring gets space, it now has to check if actually getting a packet from
>>> the queue would exceed the shaper rate, and if it would... it has to do
>>> nothing. Well, nothing for a calculated amount of time. Call it sleep,
>>> delay, whatever.
>>>
>>>
>>> Two things muck with my understanding:
>>>> In https://supportforums.cisco.**com/thread/2132501<https://supportforums.cisco.com/thread/2132501>, a Cisco employee says
>>>> that the shaper uses WFQ (or HQF in the newest releases) to de-queue the
>>>> CBWFQ queues. Why is the shaper implementing any dequeueing strategy at
>>>> all? Shouldn't the CBWFQ policy-map be handling that (such as policy
>>>> queues going first, etc)? And how can it possibly do that without full
>>>> flow information?
>>>>
>>>
>>> I don't fully understand what you are saying here. What do you mean by
>>> "without full flow information" ?
>>> But the "shaper" has to choose the next packet to tx, and that usually
>>> comes from a child policy. Or may be fair-queues, parallel flows
>>> (conversations) with a neat selection algorithm.
>>>
>>> The other issue is that the show commands on my router support my
>>>> understanding...almost. If I'm moving a lot of ssh data upstream (via
>>>> scp), I can see the shaper queue fill and the CBWFQ queue fill, makes
>>>> sense. Most of the time their values are the same. However, they do
>>>> on occasion differ by a number or two. Show command artifact, or an
>>>> indication that I have no idea what I'm talking about?
>>>>
>>>> gateway#show policy-map int f0/1
>>>> FastEthernet0/1
>>>>
>>>> Service-policy output: pm-wan-out-shaper
>>>>
>>>> Class-map: class-default (match-any)
>>>> 600317 packets, 143960388 bytes
>>>> 30 second offered rate 631000 bps, drop rate 14000 bps
>>>> Match: any
>>>> Traffic Shaping
>>>> Target/Average Byte Sustain Excess Interval
>>>> Increment
>>>> Rate Limit bits/int bits/int (ms) (bytes)
>>>> 600000/600000 937 3750 3750 6 468
>>>>
>>>> Adapt *Queue* Packets Bytes Packets Bytes
>>>> Shaping
>>>> Active *Depth* Delayed Delayed
>>>> Active
>>>> - *3* 598394 141284492 90698 94225802 yes
>>>>
>>>>
>>>> Class-map: cm-ssh (match-all)
>>>> 70238 packets, 105192011 bytes
>>>> 30 second offered rate 621000 bps, drop rate 14000 bps
>>>> Match: protocol ssh
>>>> Queueing
>>>> Output Queue: Conversation 74
>>>> Bandwidth 5 (%)
>>>> Bandwidth 30 (kbps)
>>>> (pkts matched/bytes matched) 62464/93671468
>>>> *(depth/total drops/no-buffer drops) 2*/1816/3
>>>>
>>>> exponential weight: 9
>>>> mean queue depth: 1
>>>>
>>>
>>> Please attach the policy config and what is that drives your attention
>>> from this output!
>>>
>>>
>>>
>>>> Appreciate your input -- hopefully this helps someone else too, as none
>>>> of
>>>> the standard study resources I've read have adequately explained how this
>>>> works!
>>>>
>>>> Keller Giacomarro
>>>> keller.g_at_gmail.com
>>>>
>>>>
>>> HTH,
>>> -Carlos
>>>
>>> --
>>> Carlos G Mendioroz <tron_at_huapi.ba.ar> LW7 EQI Argentina
>
>
> Blogs and organic groups at http://www.ccie.net
>
> _______________________________________________________________________
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html
>
>
>
>
>
>
>

-- 
Carlos G Mendioroz  <tron_at_huapi.ba.ar>  LW7 EQI  Argentina
Blogs and organic groups at http://www.ccie.net