From: joe_astorino@comcast.net
Date: Fri Jan 23 2009 - 17:05:42 ARST
Hey guys,
1. show ip cef 4.4.4.4 internal, to check if adjacency is gleaning somehow
2. ip ospf network point-to-point on loopback that has 4.4.4.4, just in case there is a prolem with /32
3. try turning off redistribution on R6
4. try static route to 4.4.4.4 on R6, with combination of int+next hop.
R4 is not running OSPF. It is running RIP. 4.4.4.4 gets redistributed into OSPF.
I have looked at CEF, results below. I have also tried turning off redistribution on R6 completely, and adding a static route 4.4.4.4 255.255.255.255 141.41.26.5 to no avail.
R6#sh ip cef 4.4.4.4
4.4.4.4/32, version 50, epoch 0, cached adjacency 141.41.26.5
0 packets, 0 bytes
via 141.41.26.5, Serial2/0, 1 dependency
next hop 141.41.26.5, Serial2/0
valid cached adjacency
R6#sh ip cef adjacency s2/0 141.41.26.5 | i 4.4.4.4
4.4.4.4/32 141.41.26.5 Serial2/0
R6#sh ip cef internal | beg 4.4.4.4/32, version
4.4.4.4/32, version 50, epoch 0, cached adjacency 141.41.26.5
0 packets, 0 bytes
via 141.41.26.5, Serial2/0, 1 dependency
next hop 141.41.26.5, Serial2/0
valid cached adjacency
refcount 5
Regarding IOS, R6 is running c3640-ik9o3s-mz.124-19.bin. I suppose I could try loading the code you suggested, but I will likely have to wait until I get home from work.
----- Original Message -----
From: "Pavel Bykov" <slidersv@gmail.com>
To: "joe astorino" <joe_astorino@comcast.net>
Cc: ccielab@groupstudy.com
Sent: Friday, January 23, 2009 1:38:42 PM GMT -05:00 US/Canada Eastern
Subject: Re: Frame-Relay Encapsulation Failed -- Driving me mad
no FIB because R6 is using control plane for translation.
On Fri, Jan 23, 2009 at 7:21 PM, < joe_astorino@comcast.net > wrote:
Interesting .... if I look at the debug of the successful ping to 4.4.4.4 from R6's ethernet address I see something I do NOT see when it fails encap.
R6(config-if)#do ping 4.4.4.4 so e0/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
Packet sent with a source address of 141.41.67.6
!
*Jan 23 18:13:03.869: IP: tableid=0, s=141.41.67.6 (local), d=4.4.4.4 (Serial2/0), routed via FIB
*Jan 23 18:13:03.873: IP: s=141.41.67.6 (local), d=4.4.4.4 (Serial2/0), len 100, sending
*Jan 23 18:13:03.873: ICMP type=8, code=0
Notice that on the first line it says "routed via FIB". When it fails I do not see that. hmmmmmmm.............
----- Original Message -----
From: "joe astorino" < joe_astorino@comcast.net >
To: "Pavel Bykov" < slidersv@gmail.com >
Cc: ccielab@groupstudy.com
Sent: Friday, January 23, 2009 12:35:38 PM GMT -05:00 US/Canada Eastern
Subject: Re: Frame-Relay Encapsulation Failed -- Driving me mad
Hi guys,
Thanks to everybody that is chipping in on this one. It sure is a frustrating issue! Yes, I have tried pinging 4.4.4.4 sourced from R6's ethernet (141.41.67.6) and it works just fine. That was actually one of the first things I tried. Scott, no I have not turned off any services or done anything "strange." The startup configurations are pretty straight forward, just interfaces and IP addresses, and I have carefully looked them over again just to be sure.
Pavel -- I also thought of it possibly being related to CEF so I did try making sure ip cef was on and also ip route-cache cef on all interfaces. I can even see the 4.4.4.4 entry in the cef table pointing to the correct next hop....and like I said I can ping it fine even sourced from R6's ethernet.
Just for fun, here are my frame configurations from R6, and some logs. maybe you guys see something I don't!
interface Serial2/0
ip address 141.41.26.6 255.255.255.0
ip pim sparse-mode
encapsulation frame-relay
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 cisco
ip ospf priority 0
serial restart-delay 0
frame-relay map ip 141.41.26.2 602 broadcast
frame-relay map ip 141.41.26.5 602 broadcast
frame-relay map ip 141.41.26.6 602 broadcast
no frame-relay inverse-arp
frame-relay lmi-type cisco
R6#sh frame map
Serial2/0 (up): ip 141.41.26.2 dlci 602(0x25A,0x94A0), static,
broadcast,
CISCO, status defined, active
Serial2/0 (up): ip 141.41.26.5 dlci 602(0x25A,0x94A0), static,
broadcast,
CISCO, status defined, active
Serial2/0 (up): ip 141.41.26.6 dlci 602(0x25A,0x94A0), static,
broadcast,
CISCO, status defined, active
Ok looks good.....lets ping 4.4.4.4 sourced from our loopback and e0/0
R6#ping 4.4.4.4 so lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
Packet sent with a source address of 6.6.6.6
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 196/205/236 ms
R6#ping 4.4.4.4 so e0/0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
Packet sent with a source address of 141.41.67.6
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 196/197/200 ms
OK how about a traceroute for good measure???
R6#trace 4.4.4.4
Type escape sequence to abort.
Tracing the route to 4.4.4.4
1 141.41.26.2 36 msec 32 msec 36 msec
2 141.41.26.5 76 msec 64 msec 68 msec
3 141.141.45.4 96 msec * 96 msec
exactly as expected....it goes up to the frame hub, down to R5, then over to R4 through the PPPoFR virtual-template.
Now, at this point R8 has an interface set for "ip dhcp" on the 141.41.67.0/24 subnet. Here is a debug ip packet of what is going on.
R6#sh access-list 101
Extended IP access list 101
10 permit udp any any eq bootpc
20 permit udp any any eq bootps
30 permit udp any any
R6#debug ip packet 101 det
IP packet debugging is on (detailed) for access list 101
*Jan 23 17:22:51.464: IP: s=0.0.0.0 (Ethernet0/0), d=255.255.255.255, len 604, rcvd 2
*Jan 23 17:22:51.464: UDP src=68, dst=67
*Jan 23 17:22:51.468: IP: s=141.41.67.6 (local), d=4.4.4.4 (Serial2/0), len 604, sending
*Jan 23 17:22:51.468: UDP src=67, dst=67
*Jan 23 17:22:51.468: IP: s=141.41.67.6 (local), d=4.4.4.4 (Serial2/0), len 604, encapsulation failed
*Jan 23 17:22:51.468: UDP src=67, dst=67
Again, we see the DHCP broadcast come in from R8. We attempt to unicast a DHCP request packet to 4.4.4.4 and it poops.
Now, we add a frame-map to 4.4.4.4 which DOESN'T MAKE ANY SENSE and it works!
R6(config)#int s2/0
R6(config-if)#frame map ip 4.4.4.4 602
R6(config-if)#do sh frame map
Serial2/0 (up): ip 4.4.4.4 dlci 602(0x25A,0x94A0), static,
CISCO, status defined, active
Serial2/0 (up): ip 141.41.26.2 dlci 602(0x25A,0x94A0), static,
broadcast,
CISCO, status defined, active
Serial2/0 (up): ip 141.41.26.5 dlci 602(0x25A,0x94A0), static,
broadcast,
CISCO, status defined, active
Serial2/0 (up): ip 141.41.26.6 dlci 602(0x25A,0x94A0), static,
broadcast,
CISCO, status defined, active
*Jan 23 17:27:53.324: IP: s=0.0.0.0 (Ethernet0/0), d=255.255.255.255, len 604, rcvd 2
*Jan 23 17:27:53.324: UDP src=68, dst=67
*Jan 23 17:27:53.328: IP: s=141.41.67.6 (local), d=4.4.4.4 (Serial2/0), len 604, sending
*Jan 23 17:27:53.328: UDP src=67, dst=67
R6(config-if)#
*Jan 23 17:27:56.152: IP: tableid=0, s=141.141.45.4 (Serial2/0), d=141.41.67.6 (Ethernet0/0), routed via RIB
*Jan 23 17:27:56.152: IP: s=141.141.45.4 (Serial2/0), d=141.41.67.6, len 328, rcvd 4
*Jan 23 17:27:56.152: UDP src=67, dst=67
*Jan 23 17:27:56.156: IP: s=141.41.67.6 (local), d=255.255.255.255 (Ethernet0/0), len 328, sending broad/multicast
*Jan 23 17:27:56.156: UDP src=67, dst=68
*Jan 23 17:27:56.160: IP: s=0.0.0.0 (Ethernet0/0), d=255.255.255.255, len 604, rcvd 2
*Jan 23 17:27:56.160: UDP src=68, dst=67
*Jan 23 17:27:56.164: IP: s=141.41.67.6 (local), d=4.4.4.4 (Serial2/0), len 604, sending
*Jan 23 17:27:56.164: UDP src=67, dst=67
R6(config-if)#
*Jan 23 17:27:56.992: IP: tableid=0, s=141.141.45.4 (Serial2/0), d=141.41.67.6 (Ethernet0/0), routed via RIB
*Jan 23 17:27:56.992: IP: s=141.141.45.4 (Serial2/0), d=141.41.67.6, len 328, rcvd 4
*Jan 23 17:27:56.992: UDP src=67, dst=67
*Jan 23 17:27:56.992: IP: s=141.41.67.6 (local), d=255.255.255.255 (Ethernet0/0), len 328, sending broad/multicast
*Jan 23 17:27:56.996: UDP src=67, dst=68
R8(config-if)#do sh ip int brie
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 141.41.67.27 YES DHCP up up
Am I going insane or what?!
- Joe
----- Original Message -----
From: "Pavel Bykov" < slidersv@gmail.com >
To: "joe astorino" < joe_astorino@comcast.net >
Cc: ccielab@groupstudy.com
Sent: Friday, January 23, 2009 11:44:41 AM GMT -05:00 US/Canada Eastern
Subject: Re: Frame-Relay Encapsulation Failed -- Driving me mad
Looks like control plane / data plane issue to me
Try "ip cef" on all routers and "ip route-cache cef" on all interfaces.
On Fri, Jan 23, 2009 at 9:01 AM, < joe_astorino@comcast.net > wrote:
Hello everybody. I am hoping somebody might be able to explain to me an issue I am having regarding frame-relay that is driving me mad in the practice lab. Here is the scenario:
R2, R5, R6 are in a frame-relay hub/spoke topology whereby R2 is the hub. DLCIs are X0Y where X is the source router number and Y is the destination router number. Additionally, R5 has a PPPoFR link to R4. The R2,R5,R6 cloud is the 141.41.26.0/24 network. The R5/R4 P2P is the 141.141.45.0/24 network.
I have full ip reachability, all my routing is working fine and I run into this IOS services task.
R4 has an ethernet interface on the 141.141.200.0/24 subnet. R6, which is on the other end of the frame-relay cloud from R4 has an ethernet interface on the 141.41.67.0/24 subnet. The task requires that you configure R4 to be a DHCP server to hand out addresses on the 141.41.67.0/24 subnet. In other words, you need a helper address on R6's ethernet. In order to verify your configuration, you need to bring up R8's ethernet interface and set it for dhcp.
Here is where it gets weird. On R6 I have the following frame mappings as you would expect of a spoke: A static mapping to the hub, the other spoke, and itself.
R6(config-if)#do sh frame map
Serial2/0 (up): ip 141.41.26.2 dlci 602(0x25A,0x94A0), static,
broadcast,
CISCO, status defined, active
Serial2/0 (up): ip 141.41.26.5 dlci 602(0x25A,0x94A0), static,
CISCO, status defined, active
Serial2/0 (up): ip 141.41.26.6 dlci 602(0x25A,0x94A0), static,
CISCO, status defined, active
After I have everything setup, R8 is not pulling a DHCP address, so I turn on some debugging and see this:
.Jan 23 07:52:32.443: IP: s=0.0.0.0 (Ethernet0/0), d=255.255.255.255, len 604, rcvd 2
.Jan 23 07:52:32.443: UDP src=68, dst=67
.Jan 23 07:52:32.443: IP: s=141.41.67.6 (local), d=4.4.4.4 (Serial2/0), len 604, sending
.Jan 23 07:52:32.443: UDP src=67, dst=67
.Jan 23 07:52:32.447: IP: s=141.41.67.6 (local), d=4.4.4.4 (Serial2/0), len 604, encapsulation failed
.Jan 23 07:52:32.447: UDP src=67, dst=67
So, I see the DHCP broadcast come in from R8, thats good. Then I see R6 attempt to unicast to R4 because it has ip helper-address 4.4.4.4 configured (R4's loopback) so thats good ....... and then boom encapsulation failed. First off I don't understand why this is happening. I have a route to 4.4.4.4, which R6 learnes via ospf from R5 from across the frame. I would think that like ANY other route it has that it would first lookup 4.4.4.4 in the routing table, see that the next hop is 141.41.26.5, then send the packet source 141.41.67.6 , destination 4.4.4.4 with a DLCI of 602 as it is destined for R5. Here is the route entry on R6.
Routing entry for 4.4.4.4/32
Known via "ospf 1", distance 110, metric 20, type extern 2, forward metric 64
Redistributing via eigrp 679
Advertised by eigrp 679 metric 1 1 1 1 1
Last update from 141.41.26.5 on Serial2/0, 01:37:43 ago
Routing Descriptor Blocks:
* 141.41.26.5, from 5.5.5.5, 01:37:43 ago, via Serial2/0
Route metric is 20, traffic share count is 1
No, here is the really insane part! If I change absolutely NOTHING and ping 4.4.4.4 from R6 it works just fine. If I ping ANY other route in my lab network, it works just fine. I am stumped, hoping for some help!
Here is the first packet sent, and first received: So why is it that if I send an icmp packet it works, but if I send a UDP packet to the exact same ip address it fails encapsulation? FYI there are no ACLs or filtering involved. Also if I do a frame map to 141.141.45.4 out DLCI 602 it works. I just don't get why I would need that at all, since as I said above I think it should recursively look up the route, and send it out the proper DLCI to the next hop address.
R6(config-if)#do ping 4.4.4.4
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 4.4.4.4, timeout is 2 seconds:
!!
.Jan 23 07:57:37.352: IP: tableid=0, s=141.41.26.6 (local), d=4.4.4.4 (Serial2/0), routed via FIB
.Jan 23 07:57:37.352: IP: s=141.41.26.6 (local), d=4.4.4.4 (Serial2/0), len 100, sending
.Jan 23 07:57:37.352: ICMP type=8, code=0
.Jan 23 07:57:37.548: IP: tableid=0, s=4.4.4.4 (Serial2/0), d=141.41.26.6 (Serial2/0), routed via RIB
.Jan 23 07:57:37.548: IP: s=4.4.4.4 (Serial2/0), d=141.41.26.6 (Serial2/0), len 100, rcvd 3
.Jan 23 07:57:37.548: ICMP type=0, code=0
Thanks for ANY help on this one!
- Joe
Blogs and organic groups at http://www.ccie.net
This archive was generated by hypermail 2.1.4 : Sun Mar 01 2009 - 09:43:39 ARST