Re: Multicast Helper-map issue

<Have added Narbik to the CC' line>

I tried this again w/ RIP and cannot get it to work either. I tried a few
combinations of access-lists as well ... and it does not seem to work.

One theory is that the RIP packets have a TTL of 1 and do not get processed
by the last hope router. When these packets are taken out of the mcast
stream, they are expired.

Interesting to note, when you have a sender, it will inform the RP; looks
good so far. No receivers just yet though ...

After you add the helper-map to the last hop router, the RP will show that
it has a receiver and sender. The show ip mroute command appears good.

The last hop router must be informing the RP, and this is how the mroute is
getting populated. All is good thus far.

The last hop router receives the traffic according to the show ip mroute
command on the correct interface / RFP neighbor, but sends back a prune. Is
this what you all are seeing? This kind of goes against the idea that it is
a TTL problem ...

Narbik - Previously you added an example (a few months ago) in which you
showed a lab w/ RIP as the broadcast test. Any thoughts to add to this? In
my searches, it appears you sent this on May 25th. For anyone looking for
this, just search the archives for helper-map

DNS works fine, easy to test ... RIP does not.

Andrew Lee Lissitz

On Sun, Nov 22, 2009 at 12:00 AM, Roy Waterman <roy.waterman_at_gmail.com>wrote:

> Weird...
> Regards the use of ip pim sparse on R2 f0/0 that is needed for the RPF
> check
> for 10.1.12.1, and to enable the forwarding of multicast packets.
> You would see that if you do not enable pim on r2 f0/0 that the Incoming
> Interface for sh ip mroute on R2 would show as Null, whereas with the
> command, you will see FastEthernet0/0.
> Ive tested without using this command, & I was not getting the broadcast to
> multicast conversion.
> As you do not seem to be having any luck with RIP though,but DNS is fine,
> something else is clearly going on here.
>
> Perhaps some others can comment?
>
>
>
> 2009/11/22 Nate Lee <natetlee_at_gmail.com>
>
> > So with the fact that the DNS broadcast works with the multicast helper
> map
> > from the same source host that I am trying to get the RIPv2 broadcasts to
> > work, is it safe to assume that the configs are correct and there is
> > something else going on that is preventing R2 from converting the RIP
> > broadcasts to multicast?
> >
> >
> >
> > On Sat, Nov 21, 2009 at 9:39 PM, Nate Lee <natetlee_at_gmail.com> wrote:
> >
> > >
> > >
> > >
> > > Nice catch Roy, I'm not sure how that got in there, I just checked it
> > again
> > > and it is as you pointed out that it should be, 10.1.23.2 for the RPF
> on
> > > *,224.1.1.1.
> > >
> > > One thing of note, I am not able to see an mroute entry for 10.1.12.1,
> > > 224.1.1.1 unless I kick off a broadcasted DNS lookup on R1. Even
> though
> > the
> > > RIPv2 broadcasts from R1 can be seen on R2, they don't cause the
> creation
> > of
> > > the 10.1.12.1, 224.1.1.1 S,G entries on the transit routers.
> > >
> > > The DNS broadcasts seem to work fine, I can see them on R5 as expected.
> > > For some reason, the RIP broadcasts do not get forwarded from R2
> though.
> > >
> > >
> > >
> > >
> > > On Sat, Nov 21, 2009 at 7:43 PM, Roy Waterman <roy.waterman_at_gmail.com
> > >wrote:
> > >
> > >> Hi Nate
> > >>
> > >> Checking your sh ip mroute, I see a cause for concern...
> > >>
> > >>
> > >> R3(config)#do sh ip mroute 224.1.1.1
> > >>
> > >> (*, 224.1.1.1), 00:36:26/00:02:59, RP 2.2.2.2, flags: SJCL
> > >>
> > >> Incoming interface: FastEthernet0/1, RPF nbr *10.1.34.3* <--------
> > >> Outgoing interface list:
> > >> FastEthernet0/0, Forward/Sparse, 00:36:26/00:02:02
> > >>
> > >> (10.1.12.1, 224.1.1.1), 00:01:33/00:01:26, flags: LJ
> > >>
> > >> Incoming interface: Null, RPF nbr 0.0.0.0
> > >> Outgoing interface list:
> > >> FastEthernet0/1, Forward/Sparse, 00:01:33/00:02:58
> > >> FastEthernet0/0, Forward/Sparse, 00:01:33/00:02:02
> > >>
> > >> RPF nbr should be 10.1.23.2
> > >> Can you investigate why your route to rp 2.2.2.2 has next hop as
> > 10.1.34.3
> > >> instead of 10.1.23.2?
> > >>
> > >> I think your issues are being caused as a result of you using LAN
> > >> interfaces all the way through, and perhaps all your lan interfaces
> are
> > in
> > >> the same VLAN.
> > >> I imagine if you used point to point serial interfaces between R2 &
> R3,
> > >> and between R3 & R4, you would not see this.
> > >> You should be able to get this to work, just make sure your routing is
> > as
> > >> it should be.
> > >>
> > >> Either this or you made a typo when posting the output into gs, and
> the
> > >> mroute output is not from R3 :)
> > >>
> > >> Just so you know, I replicated your setup using point to point serials
> > and
> > >> all is well:
> > >>
> > >> In my setup:
> > >> *Mar 1 03:50:38.387: IP(0): s=10.1.12.1 (FastEthernet0/0) d=224.1.1.1
> > >> (Serial0/0.23) id=0, ttl=4, prot=17, len=52(52), mforward
> > >> R2(config)#
> > >>
> > >> R5(config-if)#do sh ip ro rip
> > >> R 1.0.0.0/8 [120/1] via 10.1.12.1, 00:00:19 --- R1s loopback
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> 2009/11/21 Nate Lee <natetlee_at_gmail.com>
> > >>
> > >> Thanks again for taking the time to have a look at this. Other than
> > >>> Groupstudy, there isn't a very large pool of people you can ask for
> > help on
> > >>> some of the CCIE lab topics without watching the other person's eyes
> > glaze
> > >>> over :)
> > >>>
> > >>> Roy - I was under the impression that I would not need PIM enabled on
> > the
> > >>> fa0/0 of R2. Is that incorrect? I tried enabling it, but was still
> > unable
> > >>> to pass the traffic.
> > >>>
> > >>> Andrew - I gave it a run with DNS and could not get it working
> either,
> > I
> > >>> added udp 53 to the ip forward-protocol as well as to my ACLs but
> > wasn't
> > >>> able to see it on the other side or able to see it leave R2 as a
> > multicast
> > >>> for that matter. One thing I noticed is that even after typing in ip
> > >>> forward-protocol 53 or ip forward-protocol domain that it does not
> show
> > up
> > >>> in the config. Is this a default setting?
> > >>>
> > >>> I took a look at my mroute statements as well, and the entries are
> > there,
> > >>> however R3 and R4 have Null incoming interfaces for the 10.1.12.1,
> > 224.1.1.1
> > >>> entry. I am assuming that this is because they never receive an
> > incoming
> > >>> packet from a source for group 224.1.1.1?
> > >>>
> > >>> Here are the Mroutes:
> > >>>
> > >>> R2(config)#do sh ip mroute 224.1.1.1
> > >>>
> > >>>
> > >>> (*, 224.1.1.1), 00:34:35/00:03:20, RP 2.2.2.2, flags: S
> > >>>
> > >>> Incoming interface: Null, RPF nbr 0.0.0.0
> > >>> Outgoing interface list:
> > >>> FastEthernet0/1, Forward/Sparse, 00:34:35/00:03:20
> > >>>
> > >>> (10.1.12.1, 224.1.1.1), 00:13:16/00:03:24, flags: T
> > >>> Incoming interface: FastEthernet0/0, RPF nbr 0.0.0.0
> > >>> Outgoing interface list:
> > >>> FastEthernet0/1, Forward/Sparse, 00:13:16/00:03:20
> > >>>
> > >>> R3(config)#do sh ip mroute 224.1.1.1
> > >>>
> > >>> (*, 224.1.1.1), 00:36:26/00:02:59, RP 2.2.2.2, flags: SJCL
> > >>>
> > >>> Incoming interface: FastEthernet0/1, RPF nbr 10.1.34.3
> > >>> Outgoing interface list:
> > >>> FastEthernet0/0, Forward/Sparse, 00:36:26/00:02:02
> > >>>
> > >>> (10.1.12.1, 224.1.1.1), 00:01:33/00:01:26, flags: LJ
> > >>>
> > >>> Incoming interface: Null, RPF nbr 0.0.0.0
> > >>> Outgoing interface list:
> > >>> FastEthernet0/1, Forward/Sparse, 00:01:33/00:02:58
> > >>> FastEthernet0/0, Forward/Sparse, 00:01:33/00:02:02
> > >>>
> > >>> R4(config)#do sh ip mroute
> > >>>
> > >>> (*, 224.1.1.1), 00:28:06/00:02:59, RP 2.2.2.2, flags: SJCL
> > >>>
> > >>> Incoming interface: FastEthernet0/1, RPF nbr 10.1.34.3
> > >>> Outgoing interface list:
> > >>> FastEthernet0/0, Forward/Sparse, 00:28:06/00:02:20
> > >>>
> > >>> (10.1.12.1, 224.1.1.1), 00:02:19/00:00:40, flags: LJ
> > >>>
> > >>> Incoming interface: Null, RPF nbr 0.0.0.0
> > >>> Outgoing interface list:
> > >>> FastEthernet0/1, Forward/Sparse, 00:02:19/00:02:26
> > >>> FastEthernet0/0, Forward/Sparse, 00:02:19/00:02:20
> > >>>
> > >>>
> > >>> Here is my configs from R2, R3 and R4
> > >>>
> > >>> R2:
> > >>>
> > >>> interface Loopback0
> > >>> ip address 2.2.2.2 255.0.0.0
> > >>> ip pim sparse-mode
> > >>> !
> > >>>
> > >>> interface FastEthernet0/0
> > >>> ip address 10.1.12.2 255.255.255.0
> > >>> ip pim sparse-mode
> > >>> ip multicast helper-map broadcast 224.1.1.1 100 ttl 5
> > >>> duplex auto
> > >>> speed auto
> > >>>
> > >>> !
> > >>> interface FastEthernet0/1
> > >>> ip address 10.1.23.2 255.255.255.0
> > >>> ip pim sparse-mode
> > >>> duplex auto
> > >>> speed auto
> > >>> !
> > >>> router ospf 100
> > >>> log-adjacency-changes
> > >>>
> > >>> network 2.2.2.2 0.0.0.0 area 0
> > >>> network 10.1.23.2 0.0.0.0 area 0
> > >>> !
> > >>> ip forward-protocol nd
> > >>> ip forward-protocol udp rip
> > >>> !
> > >>> !
> > >>> ip http server
> > >>> no ip http secure-server
> > >>>
> > >>> ip pim bsr-candidate Loopback0 0
> > >>> ip pim rp-candidate Loopback0
> > >>> !
> > >>> access-list 100 permit udp any any eq rip
> > >>> access-list 100 permit udp any any eq domain
> > >>>
> > >>> R3:
> > >>>
> > >>>
> > >>> interface Loopback0
> > >>> ip address 3.3.3.3 255.0.0.0
> > >>>
> > >>> !
> > >>> interface FastEthernet0/0
> > >>> ip address 10.1.23.3 255.255.255.0
> > >>> ip pim sparse-mode
> > >>> duplex auto
> > >>> speed auto
> > >>>
> > >>> !
> > >>> interface FastEthernet0/1
> > >>> ip address 10.1.34.3 255.255.255.0
> > >>> ip pim sparse-mode
> > >>> duplex auto
> > >>> speed auto
> > >>> !
> > >>> router ospf 100
> > >>> log-adjacency-changes
> > >>>
> > >>> network 0.0.0.0 255.255.255.255 area 0
> > >>>
> > >>> R4:
> > >>>
> > >>> interface Loopback0
> > >>> ip address 4.4.4.4 255.0.0.0
> > >>> ip pim sparse-mode
> > >>> !
> > >>>
> > >>> interface FastEthernet0/0
> > >>> ip address 10.1.45.4 255.255.255.0
> > >>> ip directed-broadcast
> > >>> ip pim sparse-mode
> > >>> duplex auto
> > >>> speed auto
> > >>>
> > >>> !
> > >>> interface FastEthernet0/1
> > >>> ip address 10.1.34.4 255.255.255.0
> > >>> ip pim sparse-mode
> > >>> ip multicast helper-map 224.1.1.1 10.1.45.255 100
> > >>> duplex auto
> > >>> speed auto
> > >>> !
> > >>> router ospf 100
> > >>> log-adjacency-changes
> > >>>
> > >>> network 4.4.4.4 0.0.0.0 area 0
> > >>> network 10.1.34.4 0.0.0.0 area 0
> > >>> !
> > >>> ip forward-protocol nd
> > >>> ip forward-protocol udp rip
> > >>> !
> > >>> !
> > >>> ip http server
> > >>> no ip http secure-server
> > >>>
> > >>> !
> > >>> access-list 100 permit udp any any eq rip
> > >>> access-list 100 permit udp any any eq domain
> > >>>
> > >>>
> > >>>
> > >>> On Fri, Nov 20, 2009 at 10:56 AM, ALL From_NJ <all.from.nj_at_gmail.com
> > >wrote:
> > >>>
> > >>>> I have to run out in 2 minutes, but can add more color later.
> > >>>>
> > >>>> Nate - can you try this with DNS and then paste your configs if it
> > does
> > >>>> not work?
> > >>>>
> > >>>> I labbed this and got DNS lookups to work fairly easily, but did not
> > get
> > >>>> RIP to be forwarded. I need a little more time to look at RIP,
> > however dns
> > >>>> bcasts on one side are forwarded across to the other ... I am
> trying
> > to
> > >>>> squeeze this lab test in between a mountain of other end of week
> > tasks. ;-)
> > >>>>
> > >>>>
> > >>>> Not to be silly with this next suggestion Nate, however ... when
> > >>>> forwarding the DNS, make sure that the correct port numbers are
> > specified in
> > >>>> the forward-protocol command and the access-list. I originally
> copied
> > and
> > >>>> pasted the command reference into notepad and forgot to change
> these.
> > Oh
> > >>>> well ... trying to be too quick.
> > >>>>
> > >>>> Also, a pretty cool thing to do across the network before and during
> > the
> > >>>> test is the show ip mroute command. You will see incoming and
> > outgoing
> > >>>> interfaces added accordingly. Keep it nerdy!
> > >>>>
> > >>>> Team ---> TGIF!
> > >>>>
> > >>>> Chat soon,
> > >>>>
> > >>>> Andrew
> > >>>>
> > >>>>
> > >>>> On Fri, Nov 20, 2009 at 4:33 AM, Roy Waterman <
> roy.waterman_at_gmail.com
> > >wrote:
> > >>>>
> > >>>>> Hi Nate
> > >>>>>
> > >>>>> It looks like you are missing:
> > >>>>>
> > >>>>> *ip pim sparse-mode* on R2 f0/0.
> > >>>>>
> > >>>>> This command does show on the config guide.
> > >>>>> With all the commands needed for mcast helper map, it is so easy to
> > >>>>> miss the
> > >>>>> odd command.
> > >>>>> Other than that, your config looks great to me.
> > >>>>>
> > >>>>> Give that a go and let us know if that resolves matters.
> > >>>>>
> > >>>>> Regards
> > >>>>> Roy
> > >>>>>
> > >>>>> 2009/11/14 Nate Lee <natetlee_at_gmail.com>
> > >>>>>
> > >>>>> > I am having an issue getting the ip helper-map to work and after
> > >>>>> going over
> > >>>>> > my configs 3 or 4 times am still not sure what the problem is.
> Any
> > >>>>> hints
> > >>>>> > or
> > >>>>> > pointers in the right direction would be greatly appreciated. I
> am
> > >>>>> trying
> > >>>>> > to get one router to send a RIPv2 broadcast which should be
> picked
> > up
> > >>>>> by
> > >>>>> > another router, translated using the multicast helper-map to
> > >>>>> 224.1.1.1 and
> > >>>>> > then sent across an OSPF domain where it is converted back to a
> > >>>>> broadcast
> > >>>>> > at
> > >>>>> > the other side and received by another RIP router.
> > >>>>> >
> > >>>>> > *The setup:*
> > >>>>> >
> > >>>>> > I have 5 routers setup in a straight line R1 <> R2 <> R3 <> R4 <>
> > R5.
> > >>>>> > Multicast routing is enabled on R2, R3 and R4.
> > >>>>> >
> > >>>>> > R2, R3 and R4 are all running PIM sparse mode on their loopbacks
> > and
> > >>>>> their
> > >>>>> > interfaces facing each other.
> > >>>>> >
> > >>>>> > R1 and R5 are speaking RIPv2 and R2, R3 and R4 are using OSPF. I
> > am
> > >>>>> trying
> > >>>>> > to get R5 to receive RIP routes from R1.
> > >>>>> >
> > >>>>> > R1 has the v2-broadcast turned on on its interface facing R2. R2
> > is
> > >>>>> able
> > >>>>> > to
> > >>>>> > see these broadcasts with a packet debug.
> > >>>>> >
> > >>>>> > R2 is the BSR and RP for the three routers running OSPF,
> multicast
> > >>>>> routing
> > >>>>> > is not enabled on R1 or R5.
> > >>>>> >
> > >>>>> > The helper map is configured on R2's interface fa0/0 which is
> > facing
> > >>>>> R1 and
> > >>>>> > should take RIP broadcasts and convert them to the 224.1.1.1
> group.
> > >>>>> >
> > >>>>> > There is another helper-map on R4's fa0/1 interface which faces
> R3
> > >>>>> and is
> > >>>>> > configured to receive the 224.1.1.1 group and send a broadcast to
> > >>>>> > 10.1.45.255.
> > >>>>> >
> > >>>>> > IP Directed-broadcast is enabled on R4's fa0/0 interface facing
> R5.
> > >>>>> >
> > >>>>> > R5 is configured with no validate-update-source to prevent it
> from
> > >>>>> dropping
> > >>>>> > any packets that appear from R1's 10.1.12.1 address.
> > >>>>> >
> > >>>>> > *The problem:*
> > >>>>> >
> > >>>>> > When I debug IP mpacket and debug IP packet on R2, I am able to
> see
> > >>>>> the RIP
> > >>>>> > broadcasts from R1, but it appears that they are never sent out
> as
> > >>>>> the
> > >>>>> > 224.1.1.1 multicast address
> > >>>>> >
> > >>>>> > R5 never receives any packets from R1, a debug IP mpacket on R2
> R3
> > >>>>> and R4
> > >>>>> > show no packets to 224.1.1.1 ever being sent anywhere.
> > >>>>> >
> > >>>>> > *The configs:
> > >>>>> > *
> > >>>>> > R1:
> > >>>>> >
> > >>>>> > interface FastEthernet0/0
> > >>>>> > ip address 10.1.12.1 255.255.255.0
> > >>>>> > ip rip v2-broadcast
> > >>>>> >
> > >>>>> > router rip
> > >>>>> > version 2
> > >>>>> > network 1.0.0.0
> > >>>>> > network 10.0.0.0
> > >>>>> > no auto-summary
> > >>>>> >
> > >>>>> > R2:
> > >>>>> >
> > >>>>> > interface FastEthernet0/0
> > >>>>> > ip address 10.1.12.2 255.255.255.0
> > >>>>> > ip multicast helper-map broadcast 224.1.1.1 100 ttl 4
> > >>>>> > !
> > >>>>> > interface FastEthernet0/1
> > >>>>> > ip address 10.1.23.2 255.255.255.0
> > >>>>> > ip pim sparse-mode
> > >>>>> > !
> > >>>>> > router ospf 100
> > >>>>> > network 2.2.2.2 0.0.0.0 area 0
> > >>>>> > network 10.1.23.2 0.0.0.0 area 0
> > >>>>> > !
> > >>>>> > ip forward-protocol udp rip
> > >>>>> > !
> > >>>>> > ip pim bsr-candidate Loopback0 0
> > >>>>> > ip pim rp-candidate Loopback0
> > >>>>> > !
> > >>>>> > access-list 100 permit udp any any eq rip
> > >>>>> >
> > >>>>> > R3:
> > >>>>> >
> > >>>>> > interface FastEthernet0/0
> > >>>>> > ip address 10.1.23.3 255.255.255.0
> > >>>>> > ip pim sparse-mode
> > >>>>> > !
> > >>>>> > interface FastEthernet0/1
> > >>>>> > ip address 10.1.34.3 255.255.255.0
> > >>>>> > ip pim sparse-mode
> > >>>>> > !
> > >>>>> > router ospf 100
> > >>>>> > network 0.0.0.0 255.255.255.255 area 0
> > >>>>> >
> > >>>>> > R4:
> > >>>>> >
> > >>>>> > interface FastEthernet0/0
> > >>>>> > ip address 10.1.45.4 255.255.255.0
> > >>>>> > ip directed-broadcast
> > >>>>> > ip igmp join-group 224.1.1.1
> > >>>>> > !
> > >>>>> > interface FastEthernet0/1
> > >>>>> > ip address 10.1.34.4 255.255.255.0
> > >>>>> > ip pim sparse-mode
> > >>>>> > ip multicast helper-map 224.1.1.1 10.1.45.255 100
> > >>>>> > !
> > >>>>> > router ospf 100
> > >>>>> > network 4.4.4.4 0.0.0.0 area 0
> > >>>>> > network 10.1.34.4 0.0.0.0 area 0
> > >>>>> > !
> > >>>>> > ip forward-protocol nd
> > >>>>> > ip forward-protocol udp rip
> > >>>>> >
> > >>>>> > R5:
> > >>>>> >
> > >>>>> > interface FastEthernet0/0
> > >>>>> > ip address 10.1.45.5 255.255.255.0
> > >>>>> > !
> > >>>>> > router rip
> > >>>>> > version 2
> > >>>>> > no validate-update-source
> > >>>>> > network 10.0.0.0
> > >>>>> > no auto-summary
> > >>>>> >
> > >>>>> > *The shows and debugs:
> > >>>>> >
> > >>>>> > *
> > >>>>> > From R2 with debug ip packet and mpacket enabled, notice no
> packets
> > >>>>> to
> > >>>>> > 224.1.1.1 are ever sent:
> > >>>>> >
> > >>>>> > *Mar 1 01:29:53.599: IP: s=10.1.12.1 (FastEthernet0/0),
> > >>>>> d=255.255.255.255,
> > >>>>> > len 52, rcvd 2
> > >>>>> > *Mar 1 01:29:53.603: UDP src=520, dst=520
> > >>>>> >
> > >>>>> > From R4:
> > >>>>> >
> > >>>>> > R4(config-if)#*do sh ip mroute*
> > >>>>> >
> > >>>>> > IP Multicast Routing Table
> > >>>>> > <output omitted>
> > >>>>> >
> > >>>>> > (*, 224.1.1.1), 01:21:43/00:02:57, RP 2.2.2.2, flags: SJCL
> > >>>>> > Incoming interface: FastEthernet0/1, RPF nbr 10.1.34.3
> > >>>>> > Outgoing interface list:
> > >>>>> > FastEthernet0/0, Forward/Sparse, 00:41:58/00:02:54
> > >>>>> >
> > >>>>> > (*, 224.0.1.40), 01:24:27/00:02:52, RP 0.0.0.0, flags: DCL
> > >>>>> > Incoming interface: Null, RPF nbr 0.0.0.0
> > >>>>> > Outgoing interface list:
> > >>>>> > Loopback0, Forward/Sparse, 01:24:27/00:02:52
> > >>>>> >
> > >>>>> >
> > >>>>> > R4(config-if)#*do sh ip pim rp map 224.1.1.1*
> > >>>>> >
> > >>>>> > PIM Group-to-RP Mappings
> > >>>>> >
> > >>>>> > Group(s) 224.0.0.0/4
> > >>>>> > RP 2.2.2.2 (?), v2
> > >>>>> > Info source: 2.2.2.2 (?), via bootstrap, priority 0, holdtime
> > 150
> > >>>>> > Uptime: 01:24:07, expires: 00:01:48
> > >>>>> >
> > >>>>> >
> > >>>>> > Blogs and organic groups at http://www.ccie.net
> > >>>>> >
> > >>>>> >
> > >>>>>
> > _______________________________________________________________________
> > >>>>> > Subscription information may be found at:
> > >>>>> > http://www.groupstudy.com/list/CCIELab.html
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>> >
> > >>>>>
> > >>>>>
> > >>>>> --
> > >>>>> Regards
> > >>>>> Roy
> > >>>>>
> > >>>>>
> > >>>>> Blogs and organic groups at http://www.ccie.net
> > >>>>>
> > >>>>>
> > _______________________________________________________________________
> > >>>>> Subscription information may be found at:
> > >>>>> http://www.groupstudy.com/list/CCIELab.html
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Andrew Lee Lissitz
> > >>>> all.from.nj_at_gmail.com
> > >>>>
> > >>>
> > >>>
> > >>
> > >>
> > >> --
> > >> Regards
> > >> Roy
> >
> >
> > Blogs and organic groups at http://www.ccie.net
> >
> > _______________________________________________________________________
> > Subscription information may be found at:
> > http://www.groupstudy.com/list/CCIELab.html
> >
> >
> >
> >
> >
> >
> >
> >
>
>
> --
> Regards
> Roy
>
>
> Blogs and organic groups at http://www.ccie.net
>
> _______________________________________________________________________
> Subscription information may be found at:
> http://www.groupstudy.com/list/CCIELab.html
>
>
>
>
>
>
>
>

-- 
Andrew Lee Lissitz
all.from.nj_at_gmail.com
Blogs and organic groups at http://www.ccie.net