From: Jonathan V Hays (jhays@jtan.com)
Date: Wed Aug 27 2003 - 13:56:00 GMT-3
This little scenario was indeed VERY instructive. I took the trouble to
recreate this scenario in my lab and learned quite a bit, in addition to
what I had learned from the groupstudy thread.
I have a couple of general comments and then there is some detailed
analysis.
Both John Matijevic and Dave Swink taught me something in this thread.
But they both also made some incorrect assumptions about what is
happening that I would like to clear up.
GENERAL COMMENT 1:
Until recently I had adopted the habit of configuring 'neighbor
update-source lo0' (and 'neighbor ebgp-multihop' if the neighbor was an
eBGP peer). I then encountered a difficult scenario wherein the
underlying IGP (I think it was OSPF) was taking different routes to
various routes injected into BGP than BGP was taking. This cause
horrendous problems.
My current opinion for _lab scenarios_ is DON'T USE LOOPBACKS FOR BGP
PEERING unless you are *required* to do so. Use directly connected
interfaces for BGP peering if at all possible. Yes, I've read Halabi and
Doyle and I know the advantages of peering with loopbacks. But in real
life the ISPs will probably use static routes to the loopbacks. The
major problem is that (depending on the scenario) there can be multiple
ways to get to a loopback which means possible routing loops. If you
need it for fault tolerance, fine. There is (usually) only one best
route to a directly connected neighbor's address.
GENERAL COMMENT 2:
Also, this is the first scenario I've seen where the peering interfaces
(loopback 0 in each case) are also the routes injected into BGP.
Anything is possible, but I'll go out on a limb and say it's very
unlikely you'll see anything like this on the CCIE lab exam. Usually it
will be another interface that gets injected into BGP. This was
certainly instructive, but now that we understand the problem don't
spend a lot of time creating similar scenarios. Create additional
loopbacks for injecting routes into BGP for your scenarios.
FULL CONFIGS PLEASE!
BTW Larry, I vote for just copying and pasting *complete* configs and
routing tables. As Dave pointed out (and you admitted) there was a lot
missing, which made it more difficult to recreate and decipher the
problem. Don't worry about being concise when you post. IMHO, the vast
majority of groupstudy posts have the opposite problem - they leave out
too much information.
LOSS OF BGP PEERING
Many thanks to Dave Swink giving us an overview of the routing loop
mechanism. I don't think I would have figured it out just by looking at
the configs - at least not without a staring at it for a long time.
(Maybe that's why Dave is a CCIE and I am not. ;-) However, using debug
(see below) and constantly checking "sh ip bgp summary" I found that in
fact the BGP peering did not actually break at any time during the route
flapping. And the routes do not disappear from the BGP table.
What *does* happen is that the routes for the loopbacks in the routing
table switch back and forth between BGP routes and EIGRP routes. My
debug output below show this in more detail.
DISTANCE COMMAND.
On R3, the AD for 1.1.1.0 and 2.2.2.0 are both 170, not 90. (I did not
investigate specific AD issues on the other routers - just R3.) When I
entered the command 'distance 91 200 200' it did not resolve the routing
problem on R3. Larry's config may have differed from mine in some
essential aspect. But the 91 implies he wants to change the default
External BGP AD of 20 to something greater than EIGRP's distance of 90.
But on R3 routes are being installed with a BGP AD of 20.
R3(config-router)#distance bgp ?
<1-255> Distance for routes external to the AS
R3(config-router)#distance bgp 171 ?
<1-255> Distance for routes internal to the AS
R3(config-router)#distance bgp 171 91 ?
<1-255> Distance for local routes
So I used 'distance bgp 171 91 91' on R3 and the problem was fixed.
Unless there is a reason to leave the internal and local distances at
200, a command like 'distance bgp 171 91 91' would probably be a better
reminder to oneself that the distances in conflict are those of EIGRP.
NEXT-HOP-SELF
I investigated John Matijevic's theory that the use of 'next-hop-self'
may have been part of the cause of this problem. I did not use
'next-hop-self' anywhere and was able to recreate the problem exactly as
shown. Use of the 'neighbor next-hop-self' command has little effect in
this simple scenario of two point-to-point links. The next-hop normally
advertised to eBGP neighbors is the eBGP peer's outgoing interface,
which in each case is already loopback 0. Thus, I suspect
'next-hop-self' had little or nothing to do with this problem.
DEBUG IP ROUTING
The 'debug ip routing' command is the first tool that should come to
mind in looking for routing loops. Since the R1-R2 has the same basic
problem as the R2-R3 link, I focused on one router (R3).
The "recursion error routing 2.2.2.2" message appears when BGP shows up
in the routing table as the source for 2.2.2.0, which I think means
that a recursive lookup for the route is missing. Some list members may
not know that a 'recursive routing table lookup' means that if you have
a route to 2.2.2.0 via 2.2.2.2 then the routing table must also have a
route table entry for 2.2.2.2 (i.e., the router must have a physical
interface associated with the path to 2.2.2.2).
This fact is at the heart of the problem experienced here.
=======
R3#debug ip routing
18:40:49: RT: recursion error routing 2.2.2.2 - probable routing loop
R3#
18:41:07: RT: del 1.1.1.0/24 via 2.2.2.2, bgp metric [20/0]
18:41:07: RT: delete subnet route to 1.1.1.0/24
18:41:07: RT: del 2.2.2.0/24 via 2.2.2.2, bgp metric [20/0]
18:41:07: RT: delete subnet route to 2.2.2.0/24
18:41:07: RT: delete network route to 2.0.0.0
18:41:07: RT: add 2.2.2.0/24 via 10.1.3.2, eigrp metric [170/2560002816]
R3#
18:42:07: RT: network 1.0.0.0 is now variably masked
18:42:07: RT: add 1.1.1.0/24 via 2.2.2.2, bgp metric [20/0]
18:42:07: RT: closer admin distance for 2.2.2.0, flushing 1 routes
18:42:07: RT: add 2.2.2.0/24 via 2.2.2.2, bgp metric [20/0]
R3#
18:42:13: RT: recursion error routing 2.2.2.2 - probable routing loop
R3#
=======
DEBUG BGP IP UPDATES
Looking at the output of 'debug ip bgp updates' is also very
instructive. You can see the routes being added or removed from the
routing table, depending on whether or not BGP has a path to those
routes.
=======
R3#sh ip route | exclude -
Gateway of last resort is not set
1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
D EX 1.1.1.1/32 [170/2560002816] via 10.1.3.2, 01:46:57,
FastEthernet0/0
B 1.1.1.0/24 [20/0] via 2.2.2.2, 00:00:07
2.0.0.0/24 is subnetted, 1 subnets
B 2.2.2.0 [20/0] via 2.2.2.2, 00:00:07
3.0.0.0/24 is subnetted, 1 subnets
C 3.3.3.0 is directly connected, Loopback0
10.0.0.0/24 is subnetted, 4 subnets
C 10.1.3.0 is directly connected, FastEthernet0/0
D EX 10.1.2.0 [170/2560002816] via 10.1.3.2, 01:46:58,
FastEthernet0/0
D EX 10.1.1.0 [170/2560002816] via 10.1.3.2, 01:46:58,
FastEthernet0/0
C 10.1.4.0 is directly connected, Loopback10
R3#
19:21:09: BGP(0): no valid path for 1.1.1.0/24
19:21:09: BGP(0): no valid path for 2.2.2.0/24
19:21:09: BGP(0): nettable_walker 1.1.1.0/24 no best path
19:21:09: BGP(0): nettable_walker 2.2.2.0/24 no best path
19:21:09: BGP(0): 2.2.2.2 computing updates, afi 0, neighbor version
128, table version 130, starting at 0.0.0.0
19:21:09: BGP(0): 2.2.2.2 update run completed, afi 0, ran for 0ms,
neighbor version 128, start version 130, throttled to 130
R3#sh ip route | exclude -
Gateway of last resort is not set
1.0.0.0/32 is subnetted, 1 subnets
D EX 1.1.1.1 [170/2560002816] via 10.1.3.2, 01:47:58, FastEthernet0/0
2.0.0.0/24 is subnetted, 1 subnets
D EX 2.2.2.0 [170/2560002816] via 10.1.3.2, 00:00:09, FastEthernet0/0
3.0.0.0/24 is subnetted, 1 subnets
C 3.3.3.0 is directly connected, Loopback0
10.0.0.0/24 is subnetted, 4 subnets
C 10.1.3.0 is directly connected, FastEthernet0/0
D EX 10.1.2.0 [170/2560002816] via 10.1.3.2, 01:47:58,
FastEthernet0/0
D EX 10.1.1.0 [170/2560002816] via 10.1.3.2, 01:47:59,
FastEthernet0/0
C 10.1.4.0 is directly connected, Loopback10
R3#
R3#
19:30:09: BGP(0): Revise route installing 1.1.1.0/24 -> 2.2.2.2 to main
IP table
19:30:09: BGP(0): Revise route installing 2.2.2.0/24 -> 2.2.2.2 to main
IP table
19:30:09: BGP(0): 2.2.2.2 computing updates, afi 0, neighbor version
146, table version 148, starting at 0.0.0.0
19:30:09: BGP(0): 2.2.2.2 update run completed, afi 0, ran for 0ms,
neighbor version 146, start version 148, throttled to 148
19:30:09: BGP(0): route 2.2.2.0/24 down
R3#sh ip route | exclude -
Gateway of last resort is not set
1.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
D EX 1.1.1.1/32 [170/2560002816] via 10.1.3.2, 01:56:56,
FastEthernet0/0
B 1.1.1.0/24 [20/0] via 2.2.2.2, 00:00:06
2.0.0.0/24 is subnetted, 1 subnets
B 2.2.2.0 [20/0] via 2.2.2.2, 00:00:06
3.0.0.0/24 is subnetted, 1 subnets
C 3.3.3.0 is directly connected, Loopback0
10.0.0.0/24 is subnetted, 4 subnets
C 10.1.3.0 is directly connected, FastEthernet0/0
D EX 10.1.2.0 [170/2560002816] via 10.1.3.2, 01:56:57,
FastEthernet0/0
D EX 10.1.1.0 [170/2560002816] via 10.1.3.2, 01:56:57,
FastEthernet0/0
C 10.1.4.0 is directly connected, Loopback10
R3#
=======
I hope this is of some use.
Jonathan
This archive was generated by hypermail 2.1.4 : Tue Sep 02 2003 - 18:54:07 GMT-3