Re: LLQ

Hey Ed,

Depending on the platform, you can generate several Mbps of traffic
with ICMP Echo. You simply need to specify the timeout as zero so
that there is no wait for a reply. It helps to specify a large echo
request size as well. The only time I ever saw a 6500 CPU spike to
and remain near 100% was when I used this technique to remotely
troubleshoot a throughput problem (the 6500 being the only "host" on
the network over which I had any control from afar).

Having said that, this is one area where Dynamips falls flat on its
face; I wouldn't trust any results involving Dynamips and any QoS
function, as there is no physical interface into which and out of
which to clock traffic. Dynamips is OK for practicing the
configuration of QoS, but not for capturing any meaningful results of
said configuration.

Regards,

Scott

On Dec 8, 2009, at 1:57 , Edison Ortiz wrote:

> You won't be able to generate 512kbps from a ping on a router. Lower
> that
> value - as you noticed I went for the minimum allowed to notice the
> behavior
> right away.
>
>
>
>
>
> Edison Ortiz
>
> Routing and Switching, CCIE # 17943
>
>
>
> -----Original Message-----
> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On Behalf
> Of ron
> wilkerson
> Sent: Tuesday, December 08, 2009 3:48 PM
> To: Tony Varriale
> Cc: Cisco certification
> Subject: Re: LLQ
>
>
>
> don't mean to add to the confusion but after reading edison's test, i
>
> couldn't believe the results, so using dynamips, i conducted my own
> test
>
> between llq and policing. i didn't expect the policer to behave
> differently
>
> than llq for excess traffic (which is what ed's test showed).
>
>
>
> my test: the first ping is with llq at 512k, 93% success. the
> second test
>
> is with the policer at 512k, 94% success. this is what i expected,
> which is
>
> completely different that ed's test which showed that the policer
> dropped
>
> all traffic.
>
>
>
> i don't understand how ed's policer test dropped all traffic
>
>
>
>
>
>
>
> Rack1R1#ping 3.3.3.3 si 1500 re 100
>
>
>
> Type escape sequence to abort.
>
> Sending 100, 1500-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
>
> !!!!!!!!!!!!!.!!!!!!!!!!!!!!!.!!!!!!!!!!!!.!!!!!!!!!!!!.!!!!!!!!!!!!.!
>
> !!!!!!!!!!!!.!!!!!!!!!!!!!!!!.
>
> Success rate is 93 percent (93/100), round-trip min/avg/max =
> 4/10/52 ms
>
> Rack1R1#conf
>
> Configuring from terminal, memory, or network [terminal]?
>
> Enter configuration commands, one per line. End with CNTL/Z.
>
> Rack1R1(config)#poli
>
> Rack1R1(config)#policy-map 2hip
>
> Rack1R1(config-pmap)#class 2hip
>
> Rack1R1(config-pmap-c)#no pri
>
> Rack1R1(config-pmap-c)#no priority 512
>
> Rack1R1(config-pmap-c)#police 512000
>
> Rack1R1(config-pmap-c-police)#^Z
>
> Rack1R1#ping 3.3.3.3 si 1500 re 100
>
> *Mar 1 00:18:29.671: %SYS-5-CONFIG_I: Configured from console by
> console
>
> Rack1R1#ping 3.3.3.3 si 1500 re 100
>
>
>
> Type escape sequence to abort.
>
> Sending 100, 1500-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
>
> !!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!.!!!
>
> !!!!!!!!!!!.!!!!!!!!!!!!!!!.!!
>
> Success rate is 94 percent (94/100), round-trip min/avg/max = 1/9/64
> ms
>
> Rack1R1#
>
>
>
> On Tue, Dec 8, 2009 at 3:36 PM, Tony Varriale
> <tvarriale_at_flamboyaninc.com>wrote:
>
>
>
>> I think based on SOME of the testing that's been posted here that's
>> not
>
>> necessary. Hence, my post for clarification.
>
>>
>
>> Simply restating what the docs say is easy. Anyone tested this on
>> IOS?
>
>> Maybe try 12.4T and 12.4 mainline?
>
>>
>
>> tv
>
>>
>
>>
>
>> -----Original Message-----
>
>> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On
>> Behalf Of
>
>> Edison Ortiz
>
>> Sent: Tuesday, December 08, 2009 2:24 PM
>
>> To: 'Cisco certification'
>
>> Subject: RE: LLQ
>
>>
>
>> "So from what I understand, the above text is saying that this
>
>> rate-limiting
>
>> will only take place under interface congestion; thus if the
>> interface is
>
>> not congested the priority queue is not really a policer, and might
>> take
>
>> more than what is configured with the priority command."
>
>>
>
>>
>
>>
>
>> Correct. If you want to police a class with LLQ under all conditions
>
>> (congestion or not) - as Narbik noted add the police command under
>> that
>
>> class as well.
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> Edison Ortiz
>
>>
>
>> Routing and Switching, CCIE # 17943
>
>>
>
>> _____
>
>>
>
>> From: karim jamali [mailto:karim.jamali_at_gmail.com]
>
>> Sent: Tuesday, December 08, 2009 3:17 PM
>
>> To: Edison Ortiz; Cisco certification
>
>> Subject: Re: LLQ
>
>>
>
>>
>
>>
>
>> Hi Experts,
>
>>
>
>>
>
>>
>
>> Quote from Petr's post Insights on CBWFQ on the following link:
>
>>
>
>> http://blog.internetworkexpert.com/2008/08/17/insights-on-cbwfq/
>
>>
>
>>
>
>>
>
>> If you have priority bandwidth configured in your policy map,
>> subtract
> this
>
>> value from total interface bandwidth to yield the amount of bandwidth
>
>> available to other classes. The priority queue is only rate-limited
>> under
>
>> interface congestion, and in such case, it cannot get more
>> bandwidth than
>
>> configured with priority statement.
>
>>
>
>> So from what I understand, the above text is saying that this
> rate-limiting
>
>> will only take place under interface congestion; thus if the
>> interface is
>
>> not congested the priority queue is not really a policer, and might
>> take
>
>> more than what is configured with the priority command.
>
>>
>
>>
>
>>
>
>> Thank You Edison for the testing!
>
>>
>
>>
>
>>
>
>> Best Regards,
>
>>
>
>>
>
>>
>
>> On Tue, Dec 8, 2009 at 9:59 PM, Edison Ortiz <edisonmortiz_at_gmail.com>
>
>> wrote:
>
>>
>
>> I don't have the original testing at the moment but I quickly pull
>> up a
>
>> scenario.
>
>>
>
>>
>
>>
>
>> R2 <----> R0 <----> R3
>
>>
>
>>
>
>>
>
>> R0 will LLQ traffic from R2 towards R3 - for this example, I
>> lowered the
>
>> priority to a minimum value to observe any drops with a simple ping
>> (size
>
>> 1500 bytes).
>
>>
>
>>
>
>>
>
>> R0 config:
>
>>
>
>> class-map match-all EF
>
>>
>
>> match ip dscp ef
>
>>
>
>> !
>
>>
>
>> !
>
>>
>
>> policy-map WAN_QOS
>
>>
>
>> class EF
>
>>
>
>> priority 9
>
>>
>
>> !
>
>>
>
>> R0#sh policy-map interface
>
>>
>
>> FastEthernet0/1
>
>>
>
>>
>
>>
>
>> Service-policy output: WAN_QOS
>
>>
>
>>
>
>>
>
>> Class-map: EF (match-all)
>
>>
>
>> 0 packets, 0 bytes
>
>>
>
>> 5 minute offered rate 0 bps, drop rate 0 bps
>
>>
>
>> Match: ip dscp ef (46)
>
>>
>
>> Queueing
>
>>
>
>> Strict Priority
>
>>
>
>> Output Queue: Conversation 264
>
>>
>
>> Bandwidth 9 (kbps) Burst 225 (Bytes)
>
>>
>
>> (pkts matched/bytes matched) 0/0
>
>>
>
>> (total drops/bytes drops) 0/0
>
>>
>
>>
>
>>
>
>> Class-map: class-default (match-any)
>
>>
>
>> 1 packets, 74 bytes
>
>>
>
>> 5 minute offered rate 0 bps, drop rate 0 bps
>
>>
>
>> Match: any
>
>>
>
>>
>
>>
>
>> Generate some traffic from R2
>
>>
>
>>
>
>>
>
>> R2#ping
>
>>
>
>> Protocol [ip]:
>
>>
>
>> Target IP address: 10.1.1.2
>
>>
>
>> Repeat count [5]: 10000
>
>>
>
>> Datagram size [100]: 1500
>
>>
>
>> Timeout in seconds [2]:
>
>>
>
>> Extended commands [n]: y
>
>>
>
>> Source address or interface:
>
>>
>
>> Type of service [0]: 0xB8
>
>>
>
>> Set DF bit in IP header? [no]:
>
>>
>
>> Validate reply data? [no]:
>
>>
>
>> Data pattern [0xABCD]:
>
>>
>
>> Loose, Strict, Record, Timestamp, Verbose[none]:
>
>>
>
>> Sweep range of sizes [n]:
>
>>
>
>> Type escape sequence to abort.
>
>>
>
>> Sending 10000, 1500-byte ICMP Echos to 10.1.1.2, timeout is 2
>> seconds:
>
>>
>
>> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>>
>
>>
>
>>
>
>> On R0:
>
>>
>
>>
>
>>
>
>> R0#sh policy-map interface
>
>>
>
>> FastEthernet0/1
>
>>
>
>>
>
>>
>
>> Service-policy output: WAN_QOS
>
>>
>
>>
>
>>
>
>> Class-map: EF (match-all)
>
>>
>
>> 697 packets, 1055258 bytes
>
>>
>
>> 5 minute offered rate 27000 bps, drop rate 1000 bps
>
>>
>
>> Match: ip dscp ef (46)
>
>>
>
>> Queueing
>
>>
>
>> Strict Priority
>
>>
>
>> Output Queue: Conversation 264
>
>>
>
>> Bandwidth 9 (kbps) Burst 225 (Bytes)
>
>>
>
>> (pkts matched/bytes matched) 2/3028
>
>>
>
>> (total drops/bytes drops) 2/3028
>
>>
>
>>
>
>>
>
>> You will notice some drop rate on the output and some drop was
>> noticed on
>
>> R2
>
>>
>
>> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>>
>
>> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>>
>
>> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>>
>
>> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>>
>
>> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>>
>
>> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>>
>
>> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>
>>
>
>> !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!
>
>>
>
>>
>
>>
>
>> but the flow was not completely dropped like a policer.
>
>>
>
>>
>
>>
>
>> Let's test with a policer..
>
>>
>
>>
>
>>
>
>> On R0:
>
>>
>
>>
>
>>
>
>> class-map match-all EF
>
>>
>
>> match ip dscp ef
>
>>
>
>> !
>
>>
>
>> !
>
>>
>
>> policy-map WAN_QOS
>
>>
>
>> class EF
>
>>
>
>> police 9000
>
>>
>
>>
>
>>
>
>> On R2:
>
>>
>
>>
>
>>
>
>> ...
>
>>
>
>> ..........................
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> Feel free to perform your own test as well.
>
>>
>
>>
>
>>
>
>>
>
>> Edison Ortiz
>
>>
>
>> Routing and Switching, CCIE # 17943
>
>>
>
>>
>
>>
>
>> -----Original Message-----
>
>>
>
>> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On
>> Behalf Of
>
>> Tony
>
>> Varriale
>
>> Sent: Tuesday, December 08, 2009 2:23 PM
>
>> To: 'Cisco certification'
>
>> Subject: RE: LLQ
>
>>
>
>>
>
>>
>
>> I'm aware of what the docs say. I thought this was discussed here
>> and
>
>> found
>
>>
>
>> that anything over the priority statement was dropped. I could be
>
>>
>
>> remembering incorrectly.
>
>>
>
>>
>
>>
>
>> Do you have any of your testing that you care to share publically?
>
>>
>
>>
>
>>
>
>> Tony Varriale
>
>>
>
>> Flamboyan, Inc.
>
>>
>
>> C: 630.546.7610
>
>>
>
>> F: 815.717.9436
>
>>
>
>> tvarriale_at_flamboyaninc.com
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> -----Original Message-----
>
>>
>
>> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On
>> Behalf Of
>
>>
>
>> Edison Ortiz
>
>>
>
>> Sent: Tuesday, December 08, 2009 12:32 PM
>
>>
>
>> Cc: 'Cisco certification'
>
>>
>
>> Subject: RE: LLQ
>
>>
>
>>
>
>>
>
>> The documentation and my testing say otherwise:
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> "When the device is not congested, the priority class traffic is
>> allowed
> to
>
>>
>
>> exceed its allocated bandwidth. When the device is congested, the
>> priority
>
>>
>
>> class traffic above the allocated bandwidth is discarded."
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
> http://www.cisco.com/en/US/docs/ios/qos/command/reference/qos_n1.html#wp1048
>
>>
>
>> 842
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> Edison Ortiz
>
>>
>
>>
>
>>
>
>> Routing and Switching, CCIE # 17943
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> -----Original Message-----
>
>>
>
>> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On
>> Behalf Of
>
>>
>
>> Ahmed Elhoussiny
>
>>
>
>> Sent: Tuesday, December 08, 2009 1:19 PM
>
>>
>
>> To: Tony Varriale
>
>>
>
>> Cc: Cisco certification
>
>>
>
>> Subject: Re: LLQ
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> Dear all,
>
>>
>
>>
>
>>
>
>> if LLQ is used for VOIP, it will get dropped/policed in
>> case
>
>>
>
>>
>
>>
>
>> the traffic exceeds the LLQ size.
>
>>
>
>>
>
>>
>
>> And this in case there is congestion and same if there is no
>> congestion.`
>
>>
>
>>
>
>>
>
>> LLQ got nothin to do with congestion, this is based on IOS & IOS XR
>
>>
>
>>
>
>>
>
>> features & also my testing while designing QOS for my Mobile
>> operator.
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> In some references you may find LLQ congestion aware....but this
>> didn't
>
>>
>
>>
>
>>
>
>> successfully being implemented till now...
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> WHY ?
>
>>
>
>>
>
>>
>
>> simply imagine u have an LLQ class with 1 M , and no interface BW
>
>>
>
>>
>
>>
>
>> congestion.
>
>>
>
>>
>
>>
>
>> VOIP traffic increased to reach 2 M, and no drops cuz of no
>> congestion due
>
>>
>
>>
>
>>
>
>> to not used BW on other classes...
>
>>
>
>>
>
>>
>
>> SO what if the traffic in other queues increased, and reached its
>> 100 %,
>
>> now
>
>>
>
>>
>
>>
>
>> the LLQ will decrease to reach 1 M, and all VOIP calls will get some
>
>> packets
>
>>
>
>>
>
>>
>
>> dropped which will affect most of VOIP calls...
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> Hope this might help
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> Thanks & B.regards
>
>>
>
>>
>
>>
>
>> Ahmed Elhoussiny,2x CCIE# 21988 (R&S-SP)
>
>>
>
>>
>
>>
>
>> Network Consultant & Cisco Academy Instructor
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> On Tue, Dec 8, 2009 at 6:54 PM, Tony Varriale
>
>>
>
>> <tvarriale_at_flamboyaninc.com>wrote:
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>> I thought the priority queue won't use the general bucket when
>>> it's over
>
>>
>
>>
>
>>
>
>>> its
>
>>
>
>>
>
>>
>
>>> defined number? Hence, all packets will get dropped.
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>> tv
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>> -----Original Message-----
>
>>
>
>>
>
>>
>
>>> From: nobody_at_groupstudy.com [mailto:nobody_at_groupstudy.com] On
>>> Behalf Of
>
>>
>
>>
>
>>
>
>>> Narbik Kocharians
>
>>
>
>>
>
>>
>
>>> Sent: Tuesday, December 08, 2009 10:20 AM
>
>>
>
>>
>
>>
>
>>> To: Wouter Prins
>
>>
>
>>
>
>>
>
>>> Cc: jack daniels; Cisco certification
>
>>
>
>>
>
>>
>
>>> Subject: Re: LLQ
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>> If you like the voice traffic to get 1M and 1M ONLY, then, provide
>>> LLQ
>
>> and
>
>>
>
>>
>
>>
>
>>> Police that traffic at the same time. Very common.
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>> On Tue, Dec 8, 2009 at 8:10 AM, Wouter Prins <wp_at_null0.nl> wrote:
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>> Hello Jack,
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>> What do you think would happen to the other traffic if the voice
>
>> traffic
>
>>
>
>>
>
>>
>
>>>> was
>
>>
>
>>
>
>>
>
>>>> allowed to burst to 2M in a LLQ?
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>> Depending on whether the interface is congested or not, the traffic
>
>>
>
>> would
>
>>
>
>>
>
>>
>
>>>> be
>
>>
>
>>
>
>>
>
>>>> dropped if it exceeds the bw you specify in the priority command.
>>>> It's
>
>>
>
>>
>
>>
>
>>> sort
>
>>
>
>>
>
>>
>
>>>> of a conditional policer. The traffic will not end up in the
>>>> default
>
>>
>
>>
>
>>
>
>>> class.
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>> 2009/12/8 jack daniels <jckdaniels12_at_gmail.com>
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>>> Hi guys,
>
>>
>
>>
>
>>
>
>>>>>
>
>>
>
>>
>
>>
>
>>>>> Please help me with the understanding of LLQ -
>
>>
>
>>
>
>>
>
>>>>>
>
>>
>
>>
>
>>
>
>>>>> If I have a link of 2 MB
>
>>
>
>>
>
>>
>
>>>>>
>
>>
>
>>
>
>>
>
>>>>> and I reserve 1 MB for VOICE ( LLQ) the if voice exceeds 1 MB will
> it
>
>>
>
>>
>
>>
>
>>> be
>
>>
>
>>
>
>>
>
>>>>> droppped or be sent in default class.
>
>>
>
>>
>
>>
>
>>>>>
>
>>
>
>>
>
>>
>
>>>>>
>
>>
>
>>
>
>>
>
>>>>>
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>> --
>
>>
>
>>
>
>>
>
>>>> Wouter Prins
>
>>
>
>>
>
>>
>
>>>> wp_at_null0.nl
>
>>
>
>>
>
>>
>
>>>> CCIE #25628
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>> Blogs and organic groups at http://www.ccie.net <http://www.ccie.net/
>>>> >
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>>
> _______________________________________________________________________
>
>>
>
>>
>
>>
>
>>>> Subscription information may be found at:
>
>>
>
>>
>
>>
>
>>>> http://www.groupstudy.com/list/CCIELab.html
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>> --
>
>>
>
>>
>
>>
>
>>> Narbik Kocharians
>
>>
>
>>
>
>>
>
>>> CCSI#30832, CCIE# 12410 (R&S, SP, Security)
>
>>
>
>>
>
>>
>
>>> www.MicronicsTraining.com <http://www.micronicstraining.com/>
>
>>
>
>>
>
>>
>
>>> Sr. Technical Instructor
>
>>
>
>>
>
>>
>
>>> YES! We take Cisco Learning Credits!
>
>>
>
>>
>
>>
>
>>> Training And Remote Racks available
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>> Blogs and organic groups at http://www.ccie.net <http://www.ccie.net/
>>> >
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>> _______________________________________________________________________
>
>>
>
>>
>
>>
>
>>> Subscription information may be found at:
>
>>
>
>>
>
>>
>
>>> http://www.groupstudy.com/list/CCIELab.html
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>> Blogs and organic groups at http://www.ccie.net <http://www.ccie.net/
>>> >
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>> _______________________________________________________________________
>
>>
>
>>
>
>>
>
>>> Subscription information may be found at:
>
>>
>
>>
>
>>
>
>>> http://www.groupstudy.com/list/CCIELab.html
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> --
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> Blogs and organic groups at http://www.ccie.net <http://
>> www.ccie.net/>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> _______________________________________________________________________
>
>>
>
>>
>
>>
>
>> Subscription information may be found at:
>
>>
>
>>
>
>>
>
>> http://www.groupstudy.com/list/CCIELab.html
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> Blogs and organic groups at http://www.ccie.net <http://
>> www.ccie.net/>
>
>>
>
>>
>
>>
>
>> _______________________________________________________________________
>
>>
>
>> Subscription information may be found at:
>
>>
>
>> http://www.groupstudy.com/list/CCIELab.html
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> Blogs and organic groups at http://www.ccie.net <http://
>> www.ccie.net/>
>
>>
>
>>
>
>>
>
>> _______________________________________________________________________
>
>>
>
>> Subscription information may be found at:
>
>>
>
>> http://www.groupstudy.com/list/CCIELab.html
>
>>
>
>>
>
>> Blogs and organic groups at http://www.ccie.net <http://
>> www.ccie.net/>
>
>>
>
>> _______________________________________________________________________
>
>> Subscription information may be found at:
>
>> http://www.groupstudy.com/list/CCIELab.html
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>> --
>
>> KJ
>
>>
>
>>
>
>> Blogs and organic groups at http://www.ccie.net
>
>>
>
>> _______________________________________________________________________
>
>> Subscription information may be found at:
>
>> http://www.groupstudy.com/list/CCIELab.html
>
>>
>
>>
>
>> Blogs and organic groups at http://www.ccie.net
>
>>
>
>> _______________________________________________________________________
>
>> Subscription information may be found at:
>
>> http://www.groupstudy.com/list/CCIELab.html
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>>
>
>
>
>
>
> --
>
> stop talking
>
>
>
>
>
> Blogs and organic groups at http://www.ccie.net
>
>
>
> _______________________________________________________________________
>
> Subscription information may be found at:
>
> http://www.groupstudy.com/list/CCIELab.html
>
>
> Blogs and organic groups at http://www.ccie.net
>
> _______________________________________________________________________
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html

Blogs and organic groups at http://www.ccie.net
Received on Tue Dec 08 2009 - 14:19:12 ART