Re: LLQ- help (with elephants)

From: Paul Negron <negron.paul_at_gmail.com>
Date: Wed, 19 Dec 2012 19:02:41 -0500

I was going to lab it up when I get back from Vacation with My 6 Kids and
Beautiful wife Bro!

Get off me!!!

I trust my own notes dude.

Paul

Paul Negron
CCIE# 14856
negron.paul_at_gmail.com
303-725-8162

On Dec 19, 2012, at 6:57 PM, Paul Negron <negron.paul_at_gmail.com> wrote:

> I'm sorry. I lost you in you making love to yourself.
>
> What are you saying?
>
> I am incorrect about the BURST?
>
>
> Paul
> Paul Negron
> CCIE# 14856
> negron.paul_at_gmail.com
> 303-725-8162
>
>
>
> On Dec 19, 2012, at 6:52 PM, Marko Milivojevic <markom_at_ipexpert.com> wrote:
>
>> You are just not giving up, are you? You are using cyclic logic to
>> explain what BURST (to use your spelling) does, still not explaining
>> it.
>>
>> If the policer works as a single-rate two-color policer, then BURST is
>> used to calculate the measurement interval. However, read one of my
>> previous messages where I offered two explanrtions for the burst: one
>> according to what Cisco documentation says, which is *decidedly* in
>> contrast to what YOU claim, but never show. Another one was the
>> explanation that's in-line with your CLAIMS, but in contrast to Cisco
>> documentation. Which one is it then? Cisco or you? If it's you,
>> where's proof? If it's Cisco, then you're wrong all along...
>>
>> So, dear Paul - don't go back to a single statement I made and keep
>> putting it *completely* out of contest only because you cannot make
>> any reasonable argument and have lost all credibility in this
>> discussion. Go, get some gear and lab it up, as both Carlos and I have
>> done... or are you just talk and no walk?
>>
>> --
>> Marko Milivojevic - CCIE #18427 (SP R&S)
>>
>>
>> On Wed, Dec 19, 2012 at 3:43 PM, Paul Negron <negron.paul_at_gmail.com>
wrote:
>>> ALL,
>>>
>>> I spoke with my friend Wendell and he still had notes on my test. No
>>> configssorry:(
>>>
>>> The one thing I forgot was that the link was 128K that I was testing on
at
>>> the time. Very similar to the the 128K test that Carlos accidentally
forced.
>>>
>>> Therefore, when I placed 4 calls that tootled approx 112K of the Pipe.
All
>>> worked well. The 5th call went through fine cause it was not enough to
>>> congest the link. The 6th call had to have had congested the link which
>>> kicked in the BURST parameter set in the Priority Queue and traffic
dropped
>>> traffic out of the VOICE CLASS.
>>>
>>> WHen I used Video, I used enough Video traffic to barely congest the link
>>> but the BURST was much greater to cause more drops.
>>>
>>> I also have in my notes that the POLICE action worked very similar to a
>>> Single Rate 2 Color Policer. '
>>>
>>> The conclusion was that the Priority Class Burst Parameter will drop
traffic
>>> only when the link is congested, even if it's by a little depending on
the
>>> traffic.
>>>
>>> This is how the BURST parameter works.
>>>
>>> I believe Mr Neiberger said this exact same thing in a previous post.
(Good
>>> Job Neiberger).
>>>
>>> Marko was correct about the congestion but I never argued about the CBWFQ
>>> portion of this. Just the Priority Queue behavior with the BURST
parameter.
>>> It's just that the BURST parameter ONLY kicks in when there is congestion
on
>>> the physical interface. I believe Carlos also agrees with this but I will
>>> let him speak for himself. Not sure if Marko agrees since he admitted he
did
>>> not know how the BURST worked. (Not that there is anything wrong with
that)
>>> :-)
>>>
>>>
>>> Paul
>>>
>>>
>>> Paul Negron
>>> CCIE# 14856
>>> negron.paul_at_gmail.com
>>> 303-725-8162
>>>
>>>
>>>
>>> On Dec 19, 2012, at 7:28 AM, Carlos G Mendioroz <tron_at_huapi.ba.ar> wrote:
>>>
>>> Oh my.
>>> I found why the policer was dropping in FR: the serial was clocked at
128K
>>> on the other side :(
>>>
>>> (This was cloaked by SNMP showing TX at 200k for as long as I tested, w/o
>>> the policy... this is indeed a heads up on SNMP metering for me).
>>>
>>> Sorry about that.
>>>
>>> On the other hand, I would like to share an example on one of the reasons
we
>>> need bursting:
>>>
>>> Assume we have someone that has to send some elephants through a path
that
>>> crosses a river with a bridge. The brisge has a guard that is told not to
>>> let more than 3 elephants per minute through, and just kill any violating
>>> elephants that may arrive at bridge entrance.
>>>
>>> Knowing the bridge policy, the herd owner ask you to be carefull not to
send
>>> more than 3 epm. (elephants per minute).
>>>
>>> So you start your clock and send at times:
>>> 0'10
>>> 0'30
>>> 0'50
>>> 1'00
>>> 1'15
>>> 1'30
>>> 2'12
>>> 2'40
>>> 2'55
>>>
>>> Right ? No minute had more than 3. All elephants walk preciselly at speed
of
>>> light :) and arrive w/o jitter at the bridge, but the gatekeeper's watch
is
>>> not synchronized, so let's add 15 seconds and see:
>>>
>>> 0'25
>>> 0'45
>>> 1'05
>>> 1'15
>>> 1'30
>>> *1'45* Kill it, 4th in a minute!
>>> 2'27
>>> 2'55
>>> 3'10
>>>
>>>
>>> What went wrong ? Well, metering. That's one of the reasons we need
>>> bursting.
>>>
>>> Hope nobody gets under one elephant foot :)
>>>
>>> -Carlos
>>> P.S.
>>> No animal was hurt in this test!
>>>
>>>
>>>
>>>
>>> Carlos G Mendioroz @ 19/12/2012 06:50 -0300 dixit:
>>>
>>> Paul,
>>> I would kindly ask you to apply some policing on words like CRAP. Those
>>> raise the emotions w/o adding any technical data...
>>>
>>> Now, it would be great if we agree on what we say. Congestion is a not
>>> very precise word. It might be that your are dropping packets, that you
>>> have some packets backlog (queue), or that your processor is running at
>>> top speed. In cisco's queue management, it's usually used to denote that
>>> one interface's TX ring is full, so the router can not just FIFO packets
>>> at interrupt time and has to "engage" software queueing.
>>>
>>> Marko, I and a lot of people at cisco (e.g. Tim Szigeti) kind of konw
>>> that CBWFQ or LLQ vanish when not congested, for good reason.
>>> Well, not if you have "police" commands. Those work always.
>>>
>>> So one or the arguments (main one ?) is that the "priority xx yy" is not
>>> going to produce drops unless somehoy you force LLQ to engage.
>>>
>>> It seems that there is some other way that cisco implements that also
>>> takes the "priority xx yy" into consideration that I stumbled on, while
>>> trying to prove this with a test.
>>>
>>> Now, real phone stream is not golden test data. Cisco has special images
>>> to generate test data and if you have a programmer side, you can do that
>>> too, or use one of the many tools to do CBR, or some VBR.
>>> As long as you send below your output TX rate, you should not congest,
>>> and testing with low values of priority should be enough to see test
>>> results.
>>>
>>> BTW, the reason for the implicit policer is fairness. The priority queue
>>> is served always first, and there is no scheduler check for that. That's
>>> why you need to prevent much traffic from entering the queue or else it
>>> will starve others. "Queue" here is a control mechanism, it cisco says
>>> it has not packet count limit.
>>>
>>> -Carlos
>>>
>>>
>>> Paul Negron @ 19/12/2012 01:30 -0300 dixit:
>>>
>>> Guys,
>>>
>>> The reason why we are going through this is because of a statement I
>>> said a while back in the thread that Marko is disagreeing with.
>>>
>>> I tested this crap with REAL Voice packets from a real phone from a
>>> multiple real FXS ports.
>>> Not your crappy UDP tests that may not set up real BURST behavior like
>>> real Video.
>>>
>>> I remember trying to prove that the Priority Queue would NOT drop
>>> packets if I sent more Call traffic through it in the event that there
>>> was NO congestion. Very similar to what Marko is saying. To my surprise
>>> I saw the Priority Queue actually drop traffic when I had placed 2 more
>>> calls than the "Priority" command was configured for.
>>>
>>> Marko said I had done an" Invalid test" which I felt was very Cocky on
>>> his part especially when he could not recall how the Burst command
>>> actually worked. It's kinda like saying someone else is wrong before you
>>> know ALL the facets involved. He could be right but I guarantee he is
>>> NOT performing the same test with REAL traffic from REAL devices. He is
>>> NOT listening he is just trying to win an argument. WHATEVER!!
>>>
>>> The reason why the Priority Queue introduced the policer was to
>>> protect the Priority Class from using too much traffic in the event that
>>> other classes were not using their share. Now this was when QOS was
>>> first introduced to the code and I was testing on Frame-Relay links so
>>> it COULD have been flawed side it looked like Carlos's test showed but I
>>> am NOT completely sure since I am getting different results from
>>> different people now (like Carlos and Marko). I remember what I saw
>>> cause I repeated the same test with REAL Video Traffic and got similar
>>> results. The priority Queue would drop traffic prior to congesting the
>>> link due to very high burst in a very small period of time. The behavior
>>> was almost like CAR not Class Based Policing.
>>>
>>> I may be getting old and soft in the brain but I remember that test. It
>>> was the Original QOS lab that SKYLINE would run for QOS back in 2002 I
>>> think. One 2600 with FXS ports connected to another 2600 with FXS ports.
>>> We would set up an HTTP server and create Video Flows from Cisco IPTV to
>>> congest the link.
>>>
>>> I put in a call to Wendell Odem to refresh my memory since we set up
>>> that lab together.
>>>
>>> I almost hope he agrees with Marko cause it would make more sense that
>>> way since it would behave most like a pure Queuing mechanism. John
>>> Neiberger also added very accurate insight that agreed with Marko but I
>>> KNOW he is listening to the whole conversation with an unbiased eye.
>>> (thanks man). This discussion is waking up my brain on this subject and
>>> I actually enjoy the dialogue.
>>>
>>>
>>> Paul Negron
>>> CCIE# 14856
>>> negron.paul_at_gmail.com <mailto:negron.paul_at_gmail.com>
>>>
>>>
>>>
>>> On Dec 18, 2012, at 6:46 PM, Marko Milivojevic <markom_at_ipexpert.com
>>> <mailto:markom_at_ipexpert.com>> wrote:
>>>
>>> That's exactly my point. We don't (you talked about single-rate
>>> policer, not me, btw.). This is what the description of the command
>>> says:
>>>
>>> "(Optional) Burst size in bytes. The burst size configures the network
>>> to accommodate temporary bursts of traffic. The default burst value,
>>> which is computed as 200 milliseconds of traffic at the configured
>>> bandwidth rate, is used when theburst argument is not specified. The
>>> range of the burst is from 32 to 2000000 bytes."
>>>
>>> Further down, there's an example:
>>>
>>> "Examples
>>>
>>> The following example shows how to configure PQ with a guaranteed
>>> bandwidth of 50 kbps and a one-time allowable burst size of 60 bytes
>>> for the policy map named policy1:
>>>
>>> Router(config)# policy-map policy1
>>> Router(config-pmap)# class voice
>>> Router(config-pmap-c)# priority 50 60"
>>>
>>> Now, look me in the eye and say this sounds like Bc to you? To me,
>>> this sounds more like a Be (which implies two rate policer. This is
>>> not the three color policer, as there seems to be no option to specify
>>> three actions, hence my comment that it sounds like be, but not even
>>> that, as it lacks the functionality I'd expect to see...
>>>
>>> Now, there are two possibilites:
>>>
>>> 1) Cisco documentation is wrong and this indeed specifies Bc for a
>>> single rate, two color policer. It would actually make sense, as it
>>> allows the possibility to adjust metering on transmission side to the
>>> metering on the receiving side (other side of the link), should they
>>> be mismatched. This is not an unlikely scenario and it's what you're
>>> saying.
>>>
>>> 2) This is indeed a temporary "excess burst", but with the implied
>>> "transmit" action, meaning that it can be used to create a two rate,
>>> two color policer. According to the documentation, this is what it is.
>>>
>>>
>>> --
>>> Marko Milivojevic - CCIE #18427 (SP R&S)
>>> Senior CCIE Instructor - IPexpert
>>>
>>> On Tue, Dec 18, 2012 at 3:39 PM, Narbik Kocharians <narbikk_at_gmail.com
>>> <mailto:narbikk_at_gmail.com>> wrote:
>>>
>>> Since when do we have a Be in single rate two color policer?
>>>
>>>
>>> On Tue, Dec 18, 2012 at 3:33 PM, Marko Milivojevic
>>> <markom_at_ipexpert.com <mailto:markom_at_ipexpert.com>>
>>> wrote:
>>>
>>>
>>> Well, let's talk about the burst parameter, because I believe we
>>> reached a conclusion it does not work like Bc (i.e. to determine the
>>> Tc). Rather it works more closely to Be, but not even that... :-)
>>>
>>> I'm not confused about it - I just can't seem to find the information
>>> what it does. I can *believe* what it does, but that doesn't explain
>>> it to me ;-)
>>>
>>> --
>>> Marko Milivojevic - CCIE #18427 (SP R&S)
>>> Senior CCIE Instructor - IPexpert
>>>
>>> On Tue, Dec 18, 2012 at 3:30 PM, Narbik Kocharians
>>> <narbikk_at_gmail.com <mailto:narbikk_at_gmail.com>>
>>> wrote:
>>>
>>> I have not, why would you say that? I am not going to participate
>>> in you
>>> said he said stuff. The burst in LLQ works the way I explained it,
>>> and I
>>> still say that, when did I say the burst does not work the way i
>>> mentioned?
>>>
>>> The burst in LLQ works just like the Bc in a single rate 2 color
>>> policer.
>>> NOW...whether it's behavior is that way during congestion or not it
>>> needs to
>>> be tested, I believe it works that way during congestion, but i do
>>> not
>>> have
>>> the testing gear to provide results, the tests shown in some of the
>>> examples
>>> are ridiculous to test the complete package, things like burst and
>>> playing
>>> with the burst value.
>>>
>>> It was you that was confused about the burst parameter and not me. Go
>>> back
>>> and read what you wrote.
>>>
>>>
>>> On Tue, Dec 18, 2012 at 3:15 PM, Marko Milivojevic
>>> <markom_at_ipexpert.com <mailto:markom_at_ipexpert.com>>
>>> wrote:
>>>
>>>
>>> Thank you for agreeing with what I was saying all along (even though
>>> it kind-of contradicts your previous burst statement) ;-)
>>>
>>> Now, testing LLQ is tricky and making calls is not exactly the best
>>> way to test it, should you intend to use it for something other than
>>> voice. Any throughput, queueing, performance testing is best left to
>>> the professional tools designed for that purpose. Human ears are
>>> not a
>>> good tool :-).
>>>
>>> For anyone with (serious) production concerns about the LLQ and a
>>> (serious) testing budget, I can only highly recommend Spirent
>>> SmartBits or TestCenter products. They are *unparalleled* to what
>>> you'll be able to learn about your routers and switches and their
>>> (lack of) performance... :-)
>>>
>>> --
>>> Marko Milivojevic - CCIE #18427 (SP R&S)
>>> Senior CCIE Instructor - IPexpert
>>>
>>> On Tue, Dec 18, 2012 at 3:09 PM, Narbik Kocharians
>>> <narbikk_at_gmail.com <mailto:narbikk_at_gmail.com>>
>>> wrote:
>>>
>>> Totally agree with what was stated in that link.
>>> We have to remember that PQ, CBWFQ, are all congestion management
>>> tool.
>>> but
>>> the best way to test LLQ is to actually make calls and test the
>>> quality.
>>>
>>> On Tue, Dec 18, 2012 at 2:40 PM, Joe Sanchez <marco207p_at_gmail.com
>>> <mailto:marco207p_at_gmail.com>>
>>> wrote:
>>>
>>>
>>> Here is a great view:
>>>
>>>
>>>
>>>
>>>
http://www.cisco.com/en/US/docs/ios/12_0t/12_0t7/feature/guide/pqcbwfq.html#w
p5329
>>>
>>>
>>> JS
>>>
>>> On Tue, Dec 18, 2012 at 3:28 PM, Marko Milivojevic
>>> <markom_at_ipexpert.com>
>>> wrote:
>>>
>>>
>>> For fun, I added another source of the traffic and lowered
>>> bandwidth
>>> on the interface to 768000 (in an attempt to actually create a
>>> congestion). This is by no means an exhaustive test of LLQ and
>>> the
>>> quality, but it shows a conditional policer in action.
>>>
>>> ------------------------------8<------------------------------
>>>
>>> !!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!
>>>
>>>
>>> !!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>
>>>
>>> !.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!
>>>
>>>
>>> !!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!
>>>
>>>
>>> !!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!
>>>
>>>
>>> !!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>
>>>
>>> !!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!
>>>
>>>
>>> !!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!
>>>
>>>
>>> !!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!
>>>
>>>
>>> !!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!
>>>
>>>
>>> !!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>
>>>
>>> !.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!
>>>
>>>
>>> !!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!
>>>
>>>
>>> !!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!
>>>
>>>
>>> !!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!
>>>
>>>
>>> !!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!
>>>
>>>
>>> !!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!
>>>
>>>
>>> !!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!
>>>
>>>
>>> !!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!
>>>
>>> ------------------------------8<------------------------------
>>>
>>> If you look closely at the pattern of drops above, you will
>>> observe
>>> the regular intervals at which they appear. This *can* be an
>>> indicator
>>> of active traffic conditioner. Let's see what our policy says.
>>> Keep
>>> in
>>> mind that the other traffic generator is much more aggressive
>>> than
>>> the
>>> first (timeout 0).
>>>
>>> ------------------------------8<------------------------------
>>> R2#show policy-map interface Serial0/2/0 output class PRIORITY
>>>
>>> Serial0/2/0
>>>
>>> Service-policy output: LLQ
>>>
>>> queue stats for all priority classes:
>>>
>>> queue limit 64 packets
>>> (queue depth/total drops/no-buffer drops) 0/0/0
>>> (pkts output/bytes output) 158281/238054624
>>>
>>> Class-map: PRIORITY (match-all)
>>> 344671 packets, 518385060 bytes
>>> 30 second offered rate 10173000 bps, drop rate 9649000 bps
>>> Match: protocol icmp
>>> Priority: 100 kbps, burst bytes 2500, b/w exceed drops:
>>> 186359
>>> ------------------------------8<------------------------------
>>>
>>> The problem is that this test is flawed when it comes to
>>> reproducing
>>> the real congestion, as here we also have to take into account
>>> that
>>> the traffic is arriving from the same input interface, so we have
>>> 1:1
>>> input:output mapping (again, something I mentioned very early in
>>> this
>>> thread as an important factor).
>>>
>>> I could spend even more time labbing it up for you, but I suppose
>>> I've
>>> done enough train-the-trainer for one day :-).
>>>
>>> --
>>> Marko Milivojevic - CCIE #18427 (SP R&S)
>>> Senior CCIE Instructor - IPexpert
>>>
>>> On Tue, Dec 18, 2012 at 1:16 PM, Marko Milivojevic
>>> <markom_at_ipexpert.com>
>>> wrote:
>>>
>>> On Tue, Dec 18, 2012 at 1:14 PM, Narbik Kocharians
>>> <narbikk_at_gmail.com>
>>> wrote:
>>>
>>> How did you prove that LLQ kicks in during congestion? How did
>>> you
>>> measure
>>> the quality of these packets?
>>>
>>>
>>> Well, that's not at all what I was testing, as the test was
>>> built
>>> specifically to avoid congestion to show how there's no policing
>>> when
>>> congestion does not occur. Apples and Oranges.
>>>
>>> Testing the quality of these packets would be difficult (not
>>> impossible) to measure using IOS-only and Voice SLA probes as
>>> traffic
>>> generators, but as I said... this was not the point here.
>>>
>>> --
>>> Marko Milivojevic - CCIE #18427 (SP R&S)
>>> Senior CCIE Instructor - IPexpert
>>>
>>>
>>>
>>> Blogs and organic groups at http://www.ccie.net
>>>
>>>
>>>
>>> _______________________________________________________________________
>>>
>>> Subscription information may be found at:
>>> http://www.groupstudy.com/list/CCIELab.html
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Narbik Kocharians
>>> CCSI#30832, CCIE# 12410 (R&S, SP, Security)
>>> www.MicronicsTraining.com <http://www.MicronicsTraining.com>
>>> Sr. Technical Instructor
>>> YES! We take Cisco Learning Credits!
>>> A Cisco Learning Partner
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Narbik Kocharians
>>> CCSI#30832, CCIE# 12410 (R&S, SP, Security)
>>> www.MicronicsTraining.com <http://www.MicronicsTraining.com>
>>> Sr. Technical Instructor
>>> YES! We take Cisco Learning Credits!
>>> A Cisco Learning Partner
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Narbik Kocharians
>>> CCSI#30832, CCIE# 12410 (R&S, SP, Security)
>>> www.MicronicsTraining.com <http://www.MicronicsTraining.com>
>>> Sr. Technical Instructor
>>> YES! We take Cisco Learning Credits!
>>> A Cisco Learning Partner
>>>
>>>
>>>
>>>
>>> --
>>> Carlos G Mendioroz <tron_at_huapi.ba.ar> LW7 EQI Argentina

Blogs and organic groups at http://www.ccie.net
Received on Wed Dec 19 2012 - 19:02:41 ART

This archive was generated by hypermail 2.2.0 : Tue Jan 01 2013 - 09:36:53 ART