Re: Shapers, CBWFQ, and tx_ring, oh my!

Thanks everyone for the good information -- I think that this is a more
complicated topic than I realized.

I think Carlos hit it on the head; what's going on the router doesn't
exactly line up with the metaphors of queues that we are taught. In this
case, although we configure a CBWFQ policy as a child policy of a shaper
policy-map, the router doesn't really handle it that way.

Traffic shaping is its own thing.
CBWFQ using 'bandwidth' and 'priority' is its own thing.
Combining the two doesn't chain them together -- it is its own separate
process.

The way I'm interpreting what everyone is saying is this:
tx_ring is minimal
When a shaper policy-map is configured with a bandwidth/priority policy as
its child, the shaping is being done by directly interacting with the child
policy-map's queues. There is no shaper queue that backs up into the CBWFQ
queues. Instead, the CBWFQ queues are being shaped themselves.

Am I correct in saying that we only have two layers of queues in play?

What I thought it was before:
CBWFQ queues (multiple) -> Traffic Shaper (single) -> tx_ring -> interface
This would imply that as the shaper queues packets because the CIR is
exceeded, they are fed back into the CBWFQ queues. The traffic shaper then
de-queues from the CBWFQ queues in accordance with the bandwidth/priority
settings in the CBWFQ queues.

However, it is sounding like the process is not nearly that
straightforward. It's almost like the router is using the CBWFQ queues IN
PLACE OF the shaper queue, that they are one and the same.

Regardless, I think I have a handle enough on the functionality for the
exam -- the nitty gritty is just a matter of curiosity now. Interested in
seeing the internal workings of it.

Responses:

I don't fully understand what you are saying here. What do you mean by
"without full flow information" ?

I think I was confused here, my understanding is a little different now. I
still don't know what the Cisco poster meant by the shaper using WFQ/HQF to
dequeue the CBWFQ queues, as I would expect dequeueing to be determined by
the settings in the CBWFQ queues.

Please attach the policy config and what is that drives your attention from
this output!

It is based on the simplified config I posted earlier in the thread:

policy-map pm-shaper
class class-default
shaper average percent 100
service-policy pm-cbwfq
!
policy-map pm-cbwfq
class cm-ssh
bandwidth percent 5
class cm-http
bandwidth percent 20
class class-default
fair-queue

And in the original output I highlighted in bold, which apparently doesn't
stick when sent via the mailing list.

gateway#show policy-map int f0/1
 FastEthernet0/1

  Service-policy output: pm-wan-out-shaper

    Class-map: class-default (match-any)
      600317 packets, 143960388 bytes
      30 second offered rate 631000 bps, drop rate 14000 bps
      Match: any
      Traffic Shaping
           Target/Average Byte Sustain Excess Interval Increment
             Rate Limit bits/int bits/int (ms) (bytes)
           600000/600000 937 3750 3750 6 468

        Adapt Queue Packets Bytes Packets Bytes Shaping
        Active Depth Delayed Delayed Active
---> - 3 598394 141284492 90698 94225802 yes

        Class-map: cm-ssh (match-all)
          70238 packets, 105192011 bytes
          30 second offered rate 621000 bps, drop rate 14000 bps
          Match: protocol ssh
          Queueing
            Output Queue: Conversation 74
            Bandwidth 5 (%)
            Bandwidth 30 (kbps)
            (pkts matched/bytes matched) 62464/93671468
----> (depth/total drops/no-buffer drops) 2/1816/3
             exponential weight: 9
             mean queue depth: 1

The traffic shaper depth is 3, while the only class with any sort of
traffic is 2. I am now thinking that this is a show command anomaly, they
should be the same.

Thanks again, all -- your comments have really helped!

Keller Giacomarro
keller.g_at_gmail.com

On Sat, Jul 14, 2012 at 7:10 PM, Carlos G Mendioroz <tron_at_huapi.ba.ar>wrote:

> Keller Giacomarro @ 14/07/2012 07:07 -0300 dixit:
>
> I am trying to wrap my head around how QoS queueing actually functions.
>> If
>> you're able, please confirm or debunk my understanding!
>>
>
> Let's see if I can help. In any case, the guys that usually chime in with
> details are Peter Lapukhov and, in other forums, Ivan Pepelnjak.
> One issue with queueing is that usually a metaphor is used to
> understand/explain it, but it does not always play nice down to
> implementation details.
>
>
> Your 'normal' QoS setup involves a CBWFQ policy-map applied outbound on a
>> WAN interface. As the tx_ring fills up, packets are queued into the CBWFQ
>> policy-map queues (one per class-map), and are dequeued as normal. The
>> tx_ring filling up is the trigger for filling queues. Okay, simple so
>> far.
>>
>
> Sort of. The tx_ring filling up is the trigger to start "software queues".
> But some of the CBWFQ configuration may be active even if the tx_ring does
> not fill: police and shape commands (and remarking).
>
>
> The complication comes when you have an interface with a rate limit higher
>> than your CIR (like a home Internet connection via 100Mbps ethernet with a
>> CIR of 512Kbps). If CBWFQ is applied directly to the interface, even if
>> the bandwidth is set, the tx_ring clears faster than the WAN circuit will
>> take the data, and the software queues are bypassed entirely. In this
>> situation, applying a CBWFQ policy-map directly to the interface, even
>> setting the bandwidth command, does absolutely nothing.
>>
>> Here's where I get fuzzier. The solution to this is to put something else
>> between the CBWFQ policy-map and the tx_ring: a shaper via nested policy
>> maps. The shaper is configured to the correct CIR. As the shaper sees
>> that the interface is transmitting too fast, it begins to fill up the
>> CBWFQ
>> policy-map queues instead of transmitting. In this way, the physical
>> interface is faster than the CIR but we still create the necessary
>> 'backpressure' to fill up the software queues.
>>
>
> I would not call this "backpressure". And the shaper is not "seeing" the
> interface txing too fast, because it will never do so.
>
> Let's recap, it's all about when something happens, and that something is
> packet transmission.
>
> When tx_ring is empty, a packet is transmitted when a packet is received.
> So the trigger is packet reception. (And the code that does the tx is in
> the rx interrupt service). (Well, not really transmitted, but enqueued at a
> hardware level, so forget about it, its gone :)
>
> Now, if the tx_ring is full, the rx interrupt service just lets the packet
> in "a queue". It has to check first if the "queue" has space,
> and this is quite involved too.
> Then, when the tx_ring gets a place, it looks for which packet from the
> queue has to be transmitted next. The tx is now controlled by the AR, the
> rate of the tx interface. And CBWFQ is just a fancy way of defining which
> should be the next packet to be xmitted.
>
> When a shaper enters the mix, basically there's another trigger. If the
> tx_ring gets space, it now has to check if actually getting a packet from
> the queue would exceed the shaper rate, and if it would... it has to do
> nothing. Well, nothing for a calculated amount of time. Call it sleep,
> delay, whatever.
>
>
> Two things muck with my understanding:
>> In https://supportforums.cisco.**com/thread/2132501<https://supportforums.cisco.com/thread/2132501>, a Cisco employee says
>> that the shaper uses WFQ (or HQF in the newest releases) to de-queue the
>> CBWFQ queues. Why is the shaper implementing any dequeueing strategy at
>> all? Shouldn't the CBWFQ policy-map be handling that (such as policy
>> queues going first, etc)? And how can it possibly do that without full
>> flow information?
>>
>
> I don't fully understand what you are saying here. What do you mean by
> "without full flow information" ?
> But the "shaper" has to choose the next packet to tx, and that usually
> comes from a child policy. Or may be fair-queues, parallel flows
> (conversations) with a neat selection algorithm.
>
> The other issue is that the show commands on my router support my
>> understanding...almost. If I'm moving a lot of ssh data upstream (via
>> scp), I can see the shaper queue fill and the CBWFQ queue fill, makes
>> sense. Most of the time their values are the same. However, they do
>> on occasion differ by a number or two. Show command artifact, or an
>> indication that I have no idea what I'm talking about?
>>
>> gateway#show policy-map int f0/1
>> FastEthernet0/1
>>
>> Service-policy output: pm-wan-out-shaper
>>
>> Class-map: class-default (match-any)
>> 600317 packets, 143960388 bytes
>> 30 second offered rate 631000 bps, drop rate 14000 bps
>> Match: any
>> Traffic Shaping
>> Target/Average Byte Sustain Excess Interval
>> Increment
>> Rate Limit bits/int bits/int (ms) (bytes)
>> 600000/600000 937 3750 3750 6 468
>>
>> Adapt *Queue* Packets Bytes Packets Bytes
>> Shaping
>> Active *Depth* Delayed Delayed Active
>> - *3* 598394 141284492 90698 94225802 yes
>>
>>
>> Class-map: cm-ssh (match-all)
>> 70238 packets, 105192011 bytes
>> 30 second offered rate 621000 bps, drop rate 14000 bps
>> Match: protocol ssh
>> Queueing
>> Output Queue: Conversation 74
>> Bandwidth 5 (%)
>> Bandwidth 30 (kbps)
>> (pkts matched/bytes matched) 62464/93671468
>> *(depth/total drops/no-buffer drops) 2*/1816/3
>>
>> exponential weight: 9
>> mean queue depth: 1
>>
>
> Please attach the policy config and what is that drives your attention
> from this output!
>
>
>
>> Appreciate your input -- hopefully this helps someone else too, as none of
>> the standard study resources I've read have adequately explained how this
>> works!
>>
>> Keller Giacomarro
>> keller.g_at_gmail.com
>>
>>
> HTH,
> -Carlos
>
> --
> Carlos G Mendioroz <tron_at_huapi.ba.ar> LW7 EQI Argentina

Blogs and organic groups at http://www.ccie.net
Received on Sat Jul 14 2012 - 19:32:29 ART