BGP Multipath (maximum-paths)

What BGP multipath is

Even when several routes exist for the same prefix, BGP selects exactly one best path and installs it in the routing table. Section 9.1.2.2 of RFC 4271 states how equally preferred routes are handled:

The local speaker can select only one of these routes for inclusion in the associated Loc-RIB.

Multipath is a feature in which the implementation relaxes this limit and installs several routes at once for load sharing. On IOS XR it is enabled with maximum-paths.

Multipath is not standardized in any RFC. The BGP specification has no rule for using several routes for forwarding; each implementation adds it on its own. The candidate conditions, limits and command names therefore differ between vendors.

What multipath changes is only the contents of the forwarding table.

Item	Changed by multipath?
Result of best-path selection	No. There is still one best path
Routes advertised to peers	No. Only the best path is advertised, and nothing is sent to peers
Routing table (forwarding)	Yes. Several routes equal to the best path are installed, and packets are spread across them per flow (a hash of source and destination addresses and so on)

The spreading is per flow, not per packet. The next hop is chosen by a hash of the source and destination addresses and so on, so traffic between the same source and destination always takes the same path. Conversely, different sources can land on the same path. The hash table is rebuilt for each number of paths, so when the number changes, the distribution changes too.

Three kinds of multipath

Multipath is divided into three kinds by which routes are bundled. On IOS XR the kind is chosen with a maximum-paths keyword.

Kind	Command	What is bundled
eBGP multipath	`maximum-paths ebgp <n>`	Routes learned over eBGP
iBGP multipath	`maximum-paths ibgp <n>`	Routes learned over iBGP
eiBGP multipath	`maximum-paths eibgp <n>`	eBGP and iBGP routes may be mixed

What qualifies as a candidate

A route joins the multipath set when all of the following attributes match the best path. In other words, the candidates are the routes that stayed tied through best-path selection.

Condition	Detail
`WEIGHT`	Same as the best path
`LOCAL_PREF`	Same as the best path
`AS_PATH` length	Same as the best path
`ORIGIN`	Same as the best path
`MULTI_EXIT_DISC`	Same as the best path
`AS_PATH` contents	Identical to the best path (the same sequence of AS numbers)

Each kind adds its own condition.

Kind	Additional condition
eBGP	Learned from an external (or confederation-external) peer, and the IGP metric to the `NEXT_HOP` equals that of the best path
iBGP	Learned from an internal peer, and the IGP metric to the `NEXT_HOP` equals that of the best path
eiBGP	On IOS XR the routes are bundled even when the IGP metrics differ, such as a directly connected eBGP route and an iBGP route several hops away

eiBGP multipath was built for the PE of an MPLS VPN. Cisco’s guide describes it as “BGP Multipath Load Sharing for Both eBGP and iBGP in an MPLS-VPN”, a feature that can be configured only under the IPv4 VRF address family. A PE bundles the eBGP route from its CE with the iBGP route from the remote PE, and packets on the iBGP side are carried by labels to the remote PE, so the routers in between (P) never consult BGP. It was later extended so that it can also be configured outside a VRF (plain IPv4 / IPv6), but the same guide warns that it “should be configured only on one PE device at the customer site” and that with more than one, “some parts of the traffic may loop between the PE devices”. In plain IPv4, packets on the iBGP side follow the IGP to an intermediate router, and that router decides where to forward them from its own BGP table. If the intermediate router’s best path goes back through the original router, the packet is sent back and loops. R1 in this verification is that case: the next hop of its iBGP path (R2’s Loopback0) is reached through R3, whose best path for 192.168.4.0/24 is via R1.

`multipath-relax`: ignoring the contents of AS_PATH

The requirement that the AS_PATH contents match is a common problem in practice. When the same destination is received from two upstream ISPs, the AS_PATH lengths are equal but the ASes differ, so by default the routes are not bundled. Enabling multipath-relax loosens the condition to the same length is enough.

Setting	Condition on `AS_PATH`
Default	Identical contents (`65001 65004` bundles only with `65001 65004`)
With `multipath-relax`	Same length is enough (`65001 65004` also bundles with `65003 65004`)

Difference from ADD-PATH (RFC 7911)

ADD-PATH (RFC 7911) is another mechanism that deals with several routes, but the two operate at different layers. ADD-PATH is about how many routes are advertised to a peer; multipath is about how many are installed in the local forwarding table.

Item	Multipath	ADD-PATH (RFC 7911)
Standardized	No (per-implementation extension)	RFC 7911
What it changes	The number of routes in the forwarding table	The number of routes advertised to peers
As seen by a peer	No difference (only the best path is advertised)	Several routes for the same prefix arrive
Mechanism	Local configuration only	Capability exchange and Path Identifier required

Configuration and verification on IOS XR

Item	Detail
`maximum-paths ebgp <n>`	eBGP multipath, configured per `address-family`. The accepted range is 2 to 1024 (XRd 26.1.1). Without it the value is 1 (no multipath)
`maximum-paths ibgp <n>` / `eibgp <n>`	iBGP routes only / a mix of eBGP and iBGP
`bgp bestpath as-path multipath-relax`	Loosens the `AS_PATH` condition to length only
`show bgp <prefix>`	Routes in the multipath set are marked `multipath`
`show route <prefix>`	Several next hops are listed as `BGP multi path`. This is where forwarding actually changes
`show cef exact-route <source> <destination>`	Looks up which next hop the flow with that source and destination takes, using the forwarding table’s hash

The actual configuration is shown with show configuration commit changes in each STEP.

Verification on real devices

Verified with seven XRd 26.1.1 routers.

In AS 65001, R1 and R2 are the border routers and R3 is internal (iBGP full mesh, next-hop-self, OSPF area 0). AS 65002 is R4, AS 65003 is R5, AS 65004 is R6 and AS 65005 is R7
Each router advertises 192.168.N.0/24 on Loopback1 with a network statement. R1 and R3 also advertise Loopback2, and R7 advertises Loopback2 to Loopback4, which are used as different source addresses
R7 is the traffic source placed behind R4. Traffic from R7 to 192.168.6.0/24 always passes through R4, where R4’s multipath spreads it
Outbound and inbound policies have a different name per neighbor (all just pass); every neighbor has soft-reconfiguration inbound always

The three kinds of multipath are verified on different routers. The observation points are fixed at these three.

Observation point	Prefix	Routes received	Kind verified
R4	`192.168.6.0/24`	Via R1 and via R2 (both `AS_PATH 65001 65004`, identical), and via R5 (`65003 65004`, same length only)	eBGP multipath and `multipath-relax`
R3	`192.168.4.0/24`	Two iBGP routes via R1 and via R2 (same IGP metric)	iBGP multipath
R1	`192.168.4.0/24`	eBGP from R4 and iBGP from R2	eiBGP multipath

Overview of the verification

STEP	Change	What to check
0	No multipath	R4 holds three routes but one best path. One next hop, and all four of R7’s sources take the same path
1	`maximum-paths ebgp 2` on R4	The two identical routes are bundled, and R7’s sources are split between the R1 path and the R2 path. The advertisement to R5 is unchanged and no UPDATE is sent
2	`multipath-relax` and `ebgp 3` on R4	The R5 path with different `AS_PATH` contents joins, making three
3	`LOCAL_PREF` 90 on the R5 path inbound at R4	The route with a differing attribute leaves the set, back to two. The best path is unchanged
4	`maximum-paths ibgp 2` on R3	iBGP routes are bundled: R1’s and R2’s Loopback0 become the two next hops
5	`maximum-paths eibgp 2` on R1	eBGP and iBGP routes are bundled. But R3, which sits on the iBGP path, forwards by its own BGP routes, so some flows loop between R1 and R3
6	Revert STEP 1-5 (final state)	The `multipath` marks disappear and the next hop is back to one

All show output in the following sections is taken on the observation-point routers. The path of each source is confirmed both with show cef exact-route (the answer of the forwarding table) and with a traceroute from that source (the actual forwarding). traceroute sends three probes per hop, and an intermediate router sometimes does not answer only the second one (the router rate-limits its ICMP replies; it is not a difference in path).

STEP 0: No multipath

R4 holds three routes, but only the best path is used for forwarding. In show bgp only the path via R1 (10.1.4.1) is best, and nothing is marked multipath.

R4 show bgp 192.168.6.0/24 (STEP 0)
RP/0/RP0/CPU0:R4#show bgp 192.168.6.0/24
Mon Sep 14 21:13:24.256 UTC
BGP routing table entry for 192.168.6.0/24
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 60           60
Last Modified: Sep 14 21:08:56.023 for 00:04:28
Paths: (3 available, best #1)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.4.7.7        10.4.5.5        
  Path #1: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.4.7.7        10.4.5.5        
  65001 65004, (received & used)
    10.1.4.1 from 10.1.4.1 (10.0.0.1)
      Origin IGP, localpref 100, valid, external, best, group-best
      Received Path ID 0, Local Path ID 1, version 54
      Origin-AS validity: (disabled)
  Path #2: Received by speaker 0
  Not advertised to any peer
  65001 65004, (received & used)
    10.2.4.2 from 10.2.4.2 (10.0.0.2)
      Origin IGP, localpref 100, valid, external
      Received Path ID 0, Local Path ID 0, version 0
      Origin-AS validity: (disabled)
  Path #3: Received by speaker 0
  Not advertised to any peer
  65003 65004, (received & used)
    10.4.5.5 from 10.4.5.5 (10.0.0.5)
      Origin IGP, localpref 100, valid, external, group-best
      Received Path ID 0, Local Path ID 0, version 0
      Origin-AS validity: (disabled)

The routing table has a single next hop.

R4 show route 192.168.6.0/24 (STEP 0)
RP/0/RP0/CPU0:R4#show route 192.168.6.0/24
Mon Sep 14 21:13:24.863 UTC

Routing entry for 192.168.6.0/24
  Known via "bgp 65002", distance 20, metric 0
  Tag 65001, type external
  Installed Sep 14 21:08:55.779 for 00:04:29
  Routing Descriptor Blocks
    10.1.4.1, from 10.1.4.1, BGP external
      Route metric is 0, Wt is 1
  No advertising protos. 

A traceroute from each of R7’s four sources to 192.168.6.1 shows R1 (10.1.4.1) as the second hop for all of them.

R7 traceroute (STEP 0, source 192.168.7.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.7.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:16:06.561 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 62 msec  4 msec  4 msec 
 2  10.1.4.1 7 msec  14 msec  6 msec 
 3  10.1.6.6 32 msec  *  11 msec 

R7 traceroute (STEP 0, source 192.168.71.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.71.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:16:10.364 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 5 msec  5 msec  5 msec 
 2  10.1.4.1 8 msec  8 msec  7 msec 
 3  10.1.6.6 11 msec  *  14 msec 

R7 traceroute (STEP 0, source 192.168.75.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.75.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:16:14.148 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 6 msec  7 msec  6 msec 
 2  10.1.4.1 9 msec  48 msec  9 msec 
 3  10.1.6.6 13 msec  *  12 msec 

R7 traceroute (STEP 0, source 192.168.79.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.79.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:16:18.042 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 6 msec  5 msec  5 msec 
 2  10.1.4.1 9 msec  10 msec  8 msec 
 3  10.1.6.6 13 msec  *  13 msec 

R3 and R1 likewise hold two routes with one best path (R3: via R1, 10.0.0.1; R1: the directly connected eBGP path 10.1.4.4).

R3 show bgp 192.168.4.0/24 (STEP 0)
RP/0/RP0/CPU0:R3#show bgp 192.168.4.0/24
Mon Sep 14 21:12:26.450 UTC
BGP routing table entry for 192.168.4.0/24
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 28           28
Last Modified: Sep 14 21:09:05.023 for 00:03:21
Paths: (2 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Not advertised to any peer
  65002, (received & used)
    10.0.0.1 (metric 2) from 10.0.0.1 (10.0.0.1)
      Origin IGP, metric 0, localpref 100, valid, internal, best, group-best
      Received Path ID 0, Local Path ID 1, version 28
  Path #2: Received by speaker 0
  Not advertised to any peer
  65002, (received & used)
    10.0.0.2 (metric 2) from 10.0.0.2 (10.0.0.2)
      Origin IGP, metric 0, localpref 100, valid, internal
      Received Path ID 0, Local Path ID 0, version 0

R1 show bgp 192.168.4.0/24 (STEP 0)
RP/0/RP0/CPU0:R1#show bgp 192.168.4.0/24
Mon Sep 14 21:10:39.491 UTC
BGP routing table entry for 192.168.4.0/24
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 27           27
Last Modified: Sep 14 21:08:19.023 for 00:02:20
Paths: (2 available, best #2)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.1.6.6        10.0.0.3        10.0.0.2        
  Path #1: Received by speaker 0
  Not advertised to any peer
  65002, (received & used)
    10.0.0.2 (metric 3) from 10.0.0.2 (10.0.0.2)
      Origin IGP, metric 0, localpref 100, valid, internal
      Received Path ID 0, Local Path ID 0, version 0
  Path #2: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.1.6.6        10.0.0.3        10.0.0.2        
  65002, (received & used)
    10.1.4.4 from 10.1.4.4 (10.0.0.4)
      Origin IGP, metric 0, localpref 100, valid, external, best, group-best
      Received Path ID 0, Local Path ID 1, version 27
      Origin-AS validity: (disabled)

STEP 1: `maximum-paths ebgp 2` on R4

The two routes via R1 and via R2, whose AS_PATH is identical, are bundled; the route via R5 is not. The configuration is this single line.

Configuration committed on R4 in STEP 1
RP/0/RP0/CPU0:R4#show configuration commit changes last 1
Mon Sep 14 21:16:46.478 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
router bgp 65002
 address-family ipv4 unicast
  maximum-paths ebgp 2
 !
!
end

In show bgp the paths via R1 and via R2 are marked multipath, and the best path is still the one via R1. The path via R5 is not a candidate because its AS_PATH contents (65003 65004) differ.

R4 show bgp 192.168.6.0/24 (STEP 1)
RP/0/RP0/CPU0:R4#show bgp 192.168.6.0/24
Mon Sep 14 21:21:36.760 UTC
BGP routing table entry for 192.168.6.0/24
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 74           74
Last Modified: Sep 14 21:16:48.023 for 00:04:48
Paths: (3 available, best #1)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.4.7.7        10.4.5.5        
  Path #1: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.4.7.7        10.4.5.5        
  65001 65004, (received & used)
    10.1.4.1 from 10.1.4.1 (10.0.0.1)
      Origin IGP, localpref 100, valid, external, best, group-best, multipath
      Received Path ID 0, Local Path ID 1, version 74
      Origin-AS validity: (disabled)
  Path #2: Received by speaker 0
  Not advertised to any peer
  65001 65004, (received & used)
    10.2.4.2 from 10.2.4.2 (10.0.0.2)
      Origin IGP, localpref 100, valid, external, multipath
      Received Path ID 0, Local Path ID 0, version 0
      Origin-AS validity: (disabled)
  Path #3: Received by speaker 0
  Not advertised to any peer
  65003 65004, (received & used)
    10.4.5.5 from 10.4.5.5 (10.0.0.5)
      Origin IGP, localpref 100, valid, external, group-best
      Received Path ID 0, Local Path ID 0, version 0
      Origin-AS validity: (disabled)

The routing table now has two next hops marked BGP multi path.

R4 show route 192.168.6.0/24 (STEP 1)
RP/0/RP0/CPU0:R4#show route 192.168.6.0/24
Mon Sep 14 21:21:37.403 UTC

Routing entry for 192.168.6.0/24
  Known via "bgp 65002", distance 20, metric 0
  Tag 65001, type external
  Installed Sep 14 21:16:47.563 for 00:04:49
  Routing Descriptor Blocks
    10.1.4.1, from 10.1.4.1, BGP external, BGP multi path
      Route metric is 0
    10.2.4.2, from 10.2.4.2, BGP external, BGP multi path
      Route metric is 0
  No advertising protos. 

The CEF table shows which next hop each hash value uses, as Load distribution: 0 1 0 1. Flows are distributed by this table.

R4 show cef 192.168.6.0/24 (STEP 1)
RP/0/RP0/CPU0:R4#show cef 192.168.6.0/24
Mon Sep 14 21:21:37.722 UTC
192.168.6.0/24, version 165, internal 0x5000001 0x40 (ptr 0x88529f80) [1], 0x0 (0x0), 0x0 (0x0)
 Updated Sep 14 21:16:47.658 
 local adjacency to GigabitEthernet0/0/0/0

 Prefix Len 24, traffic index 0, precedence n/a, priority 4
  gateway array (0x8938bc68) reference count 6, flags 0x2010, source rib (7), 0 backups
                [1 type 3 flags 0x48501 (0x8942e380) ext 0x0 (0x0) (collapsed)]
  LW-LDI[type=0, refc=0, ptr=0x0, sh-ldi=0x0]
  gateway array update type-time 1 Sep 14 21:16:47.568
 LDI Update time Sep 14 21:16:47.658
   via 10.1.4.1/32, 2 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60a0]
    path-idx 0 NHID 0x0 [0x8852af40 0x0]
    next hop 10.1.4.1/32 via 10.1.4.1/32
   via 10.2.4.2/32, 2 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60a0]
    path-idx 1 NHID 0x0 [0x8852ae60 0x0]
    next hop 10.2.4.2/32 via 10.2.4.2/32

    Load distribution: 0 1 0 1 (refcount 1)

    Hash  OK  Interface                 Address
    0     Y   GigabitEthernet0/0/0/0    10.1.4.1       
    1     Y   GigabitEthernet0/0/0/1    10.2.4.2       
    2     Y   GigabitEthernet0/0/0/0    10.1.4.1       
    3     Y   GigabitEthernet0/0/0/1    10.2.4.2       

Looked up per source, 192.168.7.1 and 192.168.71.1 go via R1, and 192.168.75.1 and 192.168.79.1 go via R2.

R4 show cef exact-route 192.168.7.1 192.168.6.1 (STEP 1)
RP/0/RP0/CPU0:R4#show cef exact-route 192.168.7.1 192.168.6.1
Mon Sep 14 21:21:39.556 UTC
192.168.6.0/24, version 165, internal 0x5000001 0x40 (ptr 0x88529f80) [1], 0x0 (0x0), 0x0 (0x0)
 Updated Sep 14 21:16:47.658 
 local adjacency to GigabitEthernet0/0/0/0

 Prefix Len 24, traffic index 0, precedence n/a, priority 4
   via GigabitEthernet0/0/0/0
   via 10.1.4.1/32, 2 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60a0]
    path-idx 0 NHID 0x0 [0x8852af40 0x0]
    next hop 10.1.4.1/32 via 10.1.4.1/32

R4 show cef exact-route 192.168.71.1 192.168.6.1 (STEP 1)
RP/0/RP0/CPU0:R4#show cef exact-route 192.168.71.1 192.168.6.1
Mon Sep 14 21:21:39.741 UTC
192.168.6.0/24, version 165, internal 0x5000001 0x40 (ptr 0x88529f80) [1], 0x0 (0x0), 0x0 (0x0)
 Updated Sep 14 21:16:47.658 
 local adjacency to GigabitEthernet0/0/0/0

 Prefix Len 24, traffic index 0, precedence n/a, priority 4
   via GigabitEthernet0/0/0/0
   via 10.1.4.1/32, 2 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60a0]
    path-idx 0 NHID 0x0 [0x8852af40 0x0]
    next hop 10.1.4.1/32 via 10.1.4.1/32

R4 show cef exact-route 192.168.75.1 192.168.6.1 (STEP 1)
RP/0/RP0/CPU0:R4#show cef exact-route 192.168.75.1 192.168.6.1
Mon Sep 14 21:21:39.972 UTC
192.168.6.0/24, version 165, internal 0x5000001 0x40 (ptr 0x88529f80) [1], 0x0 (0x0), 0x0 (0x0)
 Updated Sep 14 21:16:47.657 
 local adjacency to GigabitEthernet0/0/0/1

 Prefix Len 24, traffic index 0, precedence n/a, priority 4
   via GigabitEthernet0/0/0/1
   via 10.2.4.2/32, 2 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60a0]
    path-idx 1 NHID 0x0 [0x8852ae60 0x0]
    next hop 10.2.4.2/32 via 10.2.4.2/32

R4 show cef exact-route 192.168.79.1 192.168.6.1 (STEP 1)
RP/0/RP0/CPU0:R4#show cef exact-route 192.168.79.1 192.168.6.1
Mon Sep 14 21:21:40.291 UTC
192.168.6.0/24, version 165, internal 0x5000001 0x40 (ptr 0x88529f80) [1], 0x0 (0x0), 0x0 (0x0)
 Updated Sep 14 21:16:47.658 
 local adjacency to GigabitEthernet0/0/0/1

 Prefix Len 24, traffic index 0, precedence n/a, priority 4
   via GigabitEthernet0/0/0/1
   via 10.2.4.2/32, 2 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60a0]
    path-idx 1 NHID 0x0 [0x8852ae60 0x0]
    next hop 10.2.4.2/32 via 10.2.4.2/32

The traceroute from R7 splits the same way. The second hop is R1 (10.1.4.1) for two sources and R2 (10.2.4.2) for the other two — the proof that multipath changed forwarding.

R7 traceroute (STEP 1, source 192.168.7.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.7.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:24:18.641 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 6 msec  5 msec  5 msec 
 2  10.1.4.1 9 msec  8 msec  7 msec 
 3  10.1.6.6 12 msec  *  12 msec 

R7 traceroute (STEP 1, source 192.168.71.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.71.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:24:22.443 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 7 msec  5 msec  5 msec 
 2  10.1.4.1 9 msec  8 msec  8 msec 
 3  10.1.6.6 13 msec  *  14 msec 

R7 traceroute (STEP 1, source 192.168.75.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.75.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:24:26.346 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 6 msec  5 msec  5 msec 
 2  10.2.4.2 9 msec  50 msec  8 msec 
 3  10.2.3.3 12 msec  11 msec  12 msec 
 4  10.1.3.1 12 msec  11 msec  11 msec 
 5  10.1.6.6 17 msec  *  17 msec 

R7 traceroute (STEP 1, source 192.168.79.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.79.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:24:30.241 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 7 msec  50 msec  6 msec 
 2  10.2.4.2 9 msec  8 msec  23 msec 
 3  10.2.3.3 12 msec  11 msec  11 msec 
 4  10.1.3.1 12 msec  11 msec  11 msec 
 5  10.1.6.6 15 msec  *  18 msec 

On the other hand, the advertisement to peers does not change. The advertised-routes to R5 are the same as in STEP 0, and the capture on the R4-R5 link contains only KEEPALIVEs (not a single UPDATE).

R4 show bgp neighbors 10.4.5.5 advertised-routes (STEP 1)
RP/0/RP0/CPU0:R4#show bgp neighbors 10.4.5.5 advertised-routes
Mon Sep 14 21:21:42.361 UTC
Network            Next Hop        From            AS Path
192.168.1.0/24     10.4.5.4        10.1.4.1        65002 65001i
192.168.2.0/24     10.4.5.4        10.1.4.1        65002 65001i
192.168.3.0/24     10.4.5.4        10.1.4.1        65002 65001i
192.168.4.0/24     10.4.5.4        Local           65002i
192.168.6.0/24     10.4.5.4        10.1.4.1        65002 65001 65004i
192.168.7.0/24     10.4.5.4        10.4.7.7        65002 65005i
192.168.13.0/24    10.4.5.4        10.1.4.1        65002 65001i
192.168.35.0/24    10.4.5.4        10.1.4.1        65002 65001i
192.168.71.0/24    10.4.5.4        10.4.7.7        65002 65005i
192.168.75.0/24    10.4.5.4        10.4.7.7        65002 65005i
192.168.79.0/24    10.4.5.4        10.4.7.7        65002 65005i

Processed 11 prefixes, 11 paths

Capture on the R4-R5 link during STEP 1 (BGP only)
$ tshark -r bgp-multipath-step1-r4r5.pcap -Y bgp
 0.000000     10.4.5.4 → 10.4.5.5     BGP 73 KEEPALIVE Message
 9.319897     10.4.5.5 → 10.4.5.4     BGP 73 KEEPALIVE Message
59.999511     10.4.5.4 → 10.4.5.5     BGP 73 KEEPALIVE Message
69.320302     10.4.5.5 → 10.4.5.4     BGP 73 KEEPALIVE Message
119.999457     10.4.5.4 → 10.4.5.5     BGP 73 KEEPALIVE Message
129.320784     10.4.5.5 → 10.4.5.4     BGP 73 KEEPALIVE Message
180.000764     10.4.5.4 → 10.4.5.5     BGP 73 KEEPALIVE Message
189.321086     10.4.5.5 → 10.4.5.4     BGP 73 KEEPALIVE Message
240.000615     10.4.5.4 → 10.4.5.5     BGP 73 KEEPALIVE Message
249.321227     10.4.5.5 → 10.4.5.4     BGP 73 KEEPALIVE Message
300.000736     10.4.5.4 → 10.4.5.5     BGP 73 KEEPALIVE Message
309.321196     10.4.5.5 → 10.4.5.4     BGP 73 KEEPALIVE Message
360.001210     10.4.5.4 → 10.4.5.5     BGP 73 KEEPALIVE Message
369.322027     10.4.5.5 → 10.4.5.4     BGP 73 KEEPALIVE Message
420.001005     10.4.5.4 → 10.4.5.5     BGP 73 KEEPALIVE Message
429.322356     10.4.5.5 → 10.4.5.4     BGP 73 KEEPALIVE Message

STEP 2: `multipath-relax` and `ebgp 3` on R4

The path via R5, whose AS_PATH contents differ, now joins the set, making three.

Configuration committed on R4 in STEP 2
RP/0/RP0/CPU0:R4#show configuration commit changes last 1
Mon Sep 14 21:24:46.493 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
router bgp 65002
 bgp bestpath as-path multipath-relax
 address-family ipv4 unicast
  maximum-paths ebgp 3
 !
!
end

R4 show bgp 192.168.6.0/24 (STEP 2)
RP/0/RP0/CPU0:R4#show bgp 192.168.6.0/24
Mon Sep 14 21:29:36.100 UTC
BGP routing table entry for 192.168.6.0/24
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 81           81
Last Modified: Sep 14 21:24:48.023 for 00:04:48
Paths: (3 available, best #1)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.4.7.7        10.4.5.5        
  Path #1: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.4.7.7        10.4.5.5        
  65001 65004, (received & used)
    10.1.4.1 from 10.1.4.1 (10.0.0.1)
      Origin IGP, localpref 100, valid, external, best, group-best, multipath
      Received Path ID 0, Local Path ID 1, version 81
      Origin-AS validity: (disabled)
  Path #2: Received by speaker 0
  Not advertised to any peer
  65001 65004, (received & used)
    10.2.4.2 from 10.2.4.2 (10.0.0.2)
      Origin IGP, localpref 100, valid, external, multipath
      Received Path ID 0, Local Path ID 0, version 0
      Origin-AS validity: (disabled)
  Path #3: Received by speaker 0
  Not advertised to any peer
  65003 65004, (received & used)
    10.4.5.5 from 10.4.5.5 (10.0.0.5)
      Origin IGP, localpref 100, valid, external, multipath
      Received Path ID 0, Local Path ID 0, version 0
      Origin-AS validity: (disabled)

R4 show route 192.168.6.0/24 (STEP 2)
RP/0/RP0/CPU0:R4#show route 192.168.6.0/24
Mon Sep 14 21:29:36.462 UTC

Routing entry for 192.168.6.0/24
  Known via "bgp 65002", distance 20, metric 0
  Tag 65001, type external
  Installed Sep 14 21:24:47.761 for 00:04:48
  Routing Descriptor Blocks
    10.1.4.1, from 10.1.4.1, BGP external, BGP multi path
      Route metric is 0
    10.2.4.2, from 10.2.4.2, BGP external, BGP multi path
      Route metric is 0
    10.4.5.5, from 10.4.5.5, BGP external, BGP multi path
      Route metric is 0
  No advertising protos. 

With three next hops the hash table is rebuilt, so the per-source distribution is not the same as in STEP 1. 192.168.71.1 moves to the path via R5 (10.4.5.5), and the other three go via R1.

R7 traceroute (STEP 2, source 192.168.7.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.7.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:32:18.313 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 7 msec  6 msec  5 msec 
 2  10.1.4.1 10 msec  10 msec  8 msec 
 3  10.1.6.6 14 msec  *  16 msec 

R7 traceroute (STEP 2, source 192.168.71.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.71.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:32:21.918 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 8 msec  5 msec  5 msec 
 2  10.4.5.5 10 msec  7 msec  7 msec 
 3  10.5.6.6 11 msec  *  14 msec 

R7 traceroute (STEP 2, source 192.168.75.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.75.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:32:25.433 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 6 msec  5 msec  24 msec 
 2  10.1.4.1 9 msec  9 msec  8 msec 
 3  10.1.6.6 12 msec  *  12 msec 

R7 traceroute (STEP 2, source 192.168.79.1)
RP/0/RP0/CPU0:R7#traceroute 192.168.6.1 source 192.168.79.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:32:29.102 UTC

Type escape sequence to abort.
Tracing the route to 192.168.6.1

 1  10.4.7.4 5 msec  4 msec  5 msec 
 2  10.1.4.1 8 msec  9 msec  7 msec 
 3  10.1.6.6 10 msec  *  14 msec 

STEP 3: `LOCAL_PREF` 90 on the R5 path inbound at R4

A route leaves the set as soon as one attribute differs from the best path. Setting LOCAL_PREF to 90 in the inbound policy from R5 removes the multipath mark from the R5 path, back to two. The best path (via R1) is unchanged.

Configuration committed on R4 in STEP 3
RP/0/RP0/CPU0:R4#show configuration commit changes last 1
Mon Sep 14 21:32:45.693 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
route-policy FROM-R5
  set local-preference 90
  pass
end-policy
!
end

Paths: (4 available) appears because soft-reconfiguration inbound always keeps the route exactly as received (received-only, localpref 100) alongside. The one used for forwarding is the localpref 90 copy, which no longer matches the best path and so is not a candidate.

R4 show bgp 192.168.6.0/24 (STEP 3)
RP/0/RP0/CPU0:R4#show bgp 192.168.6.0/24
Mon Sep 14 21:37:33.225 UTC
BGP routing table entry for 192.168.6.0/24
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 87           87
Last Modified: Sep 14 21:32:47.023 for 00:04:46
Paths: (4 available, best #1)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.4.7.7        10.4.5.5        
  Path #1: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.4.7.7        10.4.5.5        
  65001 65004, (received & used)
    10.1.4.1 from 10.1.4.1 (10.0.0.1)
      Origin IGP, localpref 100, valid, external, best, group-best, multipath
      Received Path ID 0, Local Path ID 1, version 81
      Origin-AS validity: (disabled)
  Path #2: Received by speaker 0
  Not advertised to any peer
  65001 65004, (received & used)
    10.2.4.2 from 10.2.4.2 (10.0.0.2)
      Origin IGP, localpref 100, valid, external, multipath
      Received Path ID 0, Local Path ID 0, version 0
      Origin-AS validity: (disabled)
  Path #3: Received by speaker 0
  Not advertised to any peer
  65003 65004
    10.4.5.5 from 10.4.5.5 (10.0.0.5)
      Origin IGP, localpref 90, valid, external
      Received Path ID 0, Local Path ID 0, version 0
      Origin-AS validity: (disabled)
  Path #4: Received by speaker 0
  Not advertised to any peer
  65003 65004, (received-only)
    10.4.5.5 from 10.4.5.5 (10.0.0.5)
      Origin IGP, localpref 100, valid, external
      Received Path ID 0, Local Path ID 0, version 0
      Origin-AS validity: (disabled)

R4 show route 192.168.6.0/24 (STEP 3)
RP/0/RP0/CPU0:R4#show route 192.168.6.0/24
Mon Sep 14 21:37:33.799 UTC

Routing entry for 192.168.6.0/24
  Known via "bgp 65002", distance 20, metric 0
  Tag 65001, type external
  Installed Sep 14 21:32:47.057 for 00:04:46
  Routing Descriptor Blocks
    10.1.4.1, from 10.1.4.1, BGP external, BGP multi path
      Route metric is 0
    10.2.4.2, from 10.2.4.2, BGP external, BGP multi path
      Route metric is 0
  No advertising protos. 

There is no UPDATE in the R4-R5 capture, and the advertisement to R5 is unchanged. On the R1-R4 link a single re-advertisement of 192.168.5.0/24, received from R5, does appear: re-evaluating the inbound policy changed the version of that route, so it was re-sent with the same contents as before.

STEP 4: `maximum-paths ibgp 2` on R3

iBGP routes are bundled under the same conditions. R3 receives the routes via R1 and via R2 over iBGP, and the IGP metric to each NEXT_HOP (Loopback0 of R1 and R2) is 2 for both.

Configuration committed on R3 in STEP 4
RP/0/RP0/CPU0:R3#show configuration commit changes last 1
Mon Sep 14 21:40:42.595 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
router bgp 65001
 address-family ipv4 unicast
  maximum-paths ibgp 2
 !
!
end

R3 show bgp 192.168.4.0/24 (STEP 4)
RP/0/RP0/CPU0:R3#show bgp 192.168.4.0/24
Mon Sep 14 21:44:37.129 UTC
BGP routing table entry for 192.168.4.0/24
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 37           37
Last Modified: Sep 14 21:40:44.023 for 00:03:53
Paths: (2 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Not advertised to any peer
  65002, (received & used)
    10.0.0.1 (metric 2) from 10.0.0.1 (10.0.0.1)
      Origin IGP, metric 0, localpref 100, valid, internal, best, group-best, multipath
      Received Path ID 0, Local Path ID 1, version 37
  Path #2: Received by speaker 0
  Not advertised to any peer
  65002, (received & used)
    10.0.0.2 (metric 2) from 10.0.0.2 (10.0.0.2)
      Origin IGP, metric 0, localpref 100, valid, internal, multipath
      Received Path ID 0, Local Path ID 0, version 0

The routing table holds both R1’s Loopback0 (10.0.0.1) and R2’s Loopback0 (10.0.0.2).

R3 show route 192.168.4.0/24 (STEP 4)
RP/0/RP0/CPU0:R3#show route 192.168.4.0/24
Mon Sep 14 21:44:38.487 UTC

Routing entry for 192.168.4.0/24
  Known via "bgp 65001", distance 200, metric 0
  Tag 65002, type internal
  Installed Sep 14 21:40:43.864 for 00:03:54
  Routing Descriptor Blocks
    10.0.0.1, from 10.0.0.1, BGP multi path
      Route metric is 0
    10.0.0.2, from 10.0.0.2, BGP multi path
      Route metric is 0
  No advertising protos. 

A traceroute from R3’s own two sources shows the first hop split between R1 (10.1.3.1) and R2 (10.2.3.2).

R3 traceroute (STEP 4, source 192.168.3.1)
RP/0/RP0/CPU0:R3#traceroute 192.168.4.1 source 192.168.3.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:45:00.949 UTC

Type escape sequence to abort.
Tracing the route to 192.168.4.1

 1  10.1.3.1 6 msec  5 msec  5 msec 
 2  10.1.4.4 9 msec  *  11 msec 

R3 traceroute (STEP 4, source 192.168.35.1)
RP/0/RP0/CPU0:R3#traceroute 192.168.4.1 source 192.168.35.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:45:04.816 UTC

Type escape sequence to abort.
Tracing the route to 192.168.4.1

 1  10.2.3.2 5 msec  5 msec  5 msec 
 2  10.2.4.4 8 msec  *  8 msec 

STEP 5: `maximum-paths eibgp 2` on R1

The eBGP and iBGP routes are bundled, but in plain IPv4 the iBGP side loops.

Configuration committed on R1 in STEP 5
RP/0/RP0/CPU0:R1#show configuration commit changes last 1
Mon Sep 14 21:48:47.524 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
router bgp 65001
 address-family ipv4 unicast
  maximum-paths eibgp 2
 !
!
end

Both the directly connected eBGP route (10.1.4.4) and the iBGP route from R2 (10.0.0.2, IGP metric 3) are marked multipath. The iBGP side also carries backup and add-path, but these are not marks of a path that is left unused.

R1 show bgp 192.168.4.0/24 (STEP 5)
RP/0/RP0/CPU0:R1#show bgp 192.168.4.0/24
Mon Sep 14 21:50:55.020 UTC
BGP routing table entry for 192.168.4.0/24
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 37           37
Last Modified: Sep 14 21:48:49.023 for 00:02:06
Paths: (2 available, best #2)
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.1.6.6        10.0.0.3        10.0.0.2        
  Path #1: Received by speaker 0
  Not advertised to any peer
  65002, (received & used)
    10.0.0.2 (metric 3) from 10.0.0.2 (10.0.0.2)
      Origin IGP, metric 0, localpref 100, valid, internal, multipath, backup, add-path
      Received Path ID 0, Local Path ID 3, version 37
  Path #2: Received by speaker 0
  Advertised IPv4 Unicast paths to peers (in unique update groups):
    10.1.6.6        10.0.0.3        10.0.0.2        
  65002, (received & used)
    10.1.4.4 from 10.1.4.4 (10.0.0.4)
      Origin IGP, metric 0, localpref 100, valid, external, best, group-best, multipath
      Received Path ID 0, Local Path ID 1, version 37
      Origin-AS validity: (disabled)

Both next hops are installed in the routing table and in CEF, and in the hash table the iBGP side sends packets out to R3 (10.1.3.3).

R1 show route 192.168.4.0/24 (STEP 5)
RP/0/RP0/CPU0:R1#show route 192.168.4.0/24
Mon Sep 14 21:50:56.138 UTC

Routing entry for 192.168.4.0/24
  Known via "bgp 65001", distance 20, metric 0
  Tag 65002
  Number of pic paths 1 , type internal and external
  Installed Sep 14 21:48:48.786 for 00:02:07
  Routing Descriptor Blocks
    10.0.0.2, from 10.0.0.2, BGP backup path, BGP multi path
      Route metric is 0
    10.1.4.4, from 10.1.4.4, BGP external, BGP multi path
      Route metric is 0
  No advertising protos. 

R1 show cef 192.168.4.0/24 (STEP 5)
RP/0/RP0/CPU0:R1#show cef 192.168.4.0/24
Mon Sep 14 21:50:56.479 UTC
192.168.4.0/24, version 75, internal 0x5000001 0x40 (ptr 0x8851a0b0) [1], 0x0 (0x0), 0x0 (0x0)
 Updated Sep 14 21:48:48.790 
 local adjacency to GigabitEthernet0/0/0/0

 Prefix Len 24, traffic index 0, precedence n/a, priority 4
  gateway array (0x896e4640) reference count 6, flags 0x102810, source rib (7), 0 backups
                [1 type 3 flags 0x48501 (0x897886c8) ext 0x0 (0x0) (collapsed)]
  LW-LDI[type=0, refc=0, ptr=0x0, sh-ldi=0x0]
  gateway array update type-time 1 Sep 14 21:48:48.790
 LDI Update time Sep 14 21:48:48.892
   via 10.0.0.2/32, 3 dependencies, recursive, bgp-multipath, backup [flags 0x6180]
    path-idx 0 NHID 0x0 [0x8851a5f0 0x0]
    next hop 10.0.0.2/32 via 10.0.0.2/32
   via 10.1.4.4/32, 2 dependencies, recursive, bgp-ext, bgp-multipath [flags 0x60a0]
    path-idx 1 NHID 0x0 [0x8851ac10 0x0], Internal 0x8a94a0a0
    next hop 10.1.4.4/32 via 10.1.4.4/32

    Load distribution: 0 1 0 1 (refcount 1)

    Hash  OK  Interface                 Address
    0     Y   GigabitEthernet0/0/0/0    10.1.3.3       
    1     Y   GigabitEthernet0/0/0/1    10.1.4.4       
    2     Y   GigabitEthernet0/0/0/0    10.1.3.3       
    3     Y   GigabitEthernet0/0/0/1    10.1.4.4       

Both of R1’s sources land on the iBGP side and are sent to R3. R3, with its multipath from STEP 4, spreads them between the path via R1 and the path via R2. The flow from 192.168.1.1 lands on the R2 side at R3 and reaches R4, but the flow from 192.168.13.1 lands on the R1 side at R3, R1 sends it to R3 again, and it bounces between R1 and R3 until the TTL runs out (ping 0/50).

R1 traceroute (STEP 5, source 192.168.1.1)
RP/0/RP0/CPU0:R1#traceroute 192.168.4.1 source 192.168.1.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:51:18.816 UTC

Type escape sequence to abort.
Tracing the route to 192.168.4.1

 1  10.1.3.3 6 msec  5 msec  5 msec 
 2  10.2.3.2 8 msec  8 msec  8 msec 
 3  10.2.4.4 19 msec  *  13 msec 

R1 ping / traceroute (STEP 5, source 192.168.13.1)
RP/0/RP0/CPU0:R1#ping 192.168.4.1 source 192.168.13.1 count 50 timeout 1
Mon Sep 14 21:51:20.831 UTC
Type escape sequence to abort.
Sending 50, 100-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
..................................................
Success rate is 0 percent (0/50)
RP/0/RP0/CPU0:R1#traceroute 192.168.4.1 source 192.168.13.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:52:34.527 UTC

Type escape sequence to abort.
Tracing the route to 192.168.4.1

 1  10.1.3.3 7 msec  5 msec  5 msec 
 2  10.1.3.1 6 msec  50 msec  4 msec 
 3  10.1.3.3 7 msec  7 msec  7 msec 
 4  10.1.3.1 22 msec  8 msec  8 msec 
 5  10.1.3.3 11 msec  12 msec  11 msec 
 6  10.1.3.1 12 msec  57 msec  13 msec 

R3’s own traffic is caught too. The flow from 192.168.3.1, which worked in STEP 4, is sent by R3 to R1, and R1 sends it back to R3 on the iBGP side, so it loops in the same way.

R3 ping / traceroute (STEP 5, source 192.168.3.1)
RP/0/RP0/CPU0:R3#ping 192.168.4.1 source 192.168.3.1 count 50 timeout 1
Mon Sep 14 21:54:14.260 UTC
Type escape sequence to abort.
Sending 50, 100-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
..................................................
Success rate is 0 percent (0/50)
RP/0/RP0/CPU0:R3#traceroute 192.168.4.1 source 192.168.3.1 probe 3 timeout 1 maxttl 6
Mon Sep 14 21:55:26.451 UTC

Type escape sequence to abort.
Tracing the route to 192.168.4.1

 1  10.1.3.1 6 msec  59 msec  5 msec 
 2  10.1.3.3 5 msec  5 msec  5 msec 
 3  10.1.3.1 52 msec  7 msec  7 msec 
 4  10.1.3.3 8 msec  9 msec  8 msec 
 5  10.1.3.1 10 msec  11 msec  9 msec 
 6  10.1.3.3 9 msec  9 msec  10 msec 

This is the phenomenon Cisco’s guide warns about when it says traffic “may loop between the PE devices” if the feature is configured on more than one PE. With labels carrying the packet to the remote PE in an MPLS VPN, the routers in between (P) never consult BGP, so nothing is sent back.

STEP 6: Revert STEP 1-5 (final state)

Once the three routers are reverted, the multipath marks disappear and the routing tables are back to the single next hop of STEP 0.

R4 show route 192.168.6.0/24 (STEP 6)
RP/0/RP0/CPU0:R4#show route 192.168.6.0/24
Mon Sep 14 22:04:26.012 UTC

Routing entry for 192.168.6.0/24
  Known via "bgp 65002", distance 20, metric 0
  Tag 65001, type external
  Installed Sep 14 21:59:19.870 for 00:05:06
  Routing Descriptor Blocks
    10.1.4.1, from 10.1.4.1, BGP external
      Route metric is 0, Wt is 1
  No advertising protos. 

R3 show route 192.168.4.0/24 (STEP 6)
RP/0/RP0/CPU0:R3#show route 192.168.4.0/24
Mon Sep 14 22:03:28.455 UTC

Routing entry for 192.168.4.0/24
  Known via "bgp 65001", distance 200, metric 0
  Tag 65002, type internal
  Installed Sep 14 21:59:26.478 for 00:04:02
  Routing Descriptor Blocks
    10.0.0.1, from 10.0.0.1
      Route metric is 0, Wt is 1
  No advertising protos. 

R1 show route 192.168.4.0/24 (STEP 6)
RP/0/RP0/CPU0:R1#show route 192.168.4.0/24
Mon Sep 14 22:01:40.729 UTC

Routing entry for 192.168.4.0/24
  Known via "bgp 65001", distance 20, metric 0
  Tag 65002, type external
  Installed Sep 14 21:59:33.214 for 00:02:07
  Routing Descriptor Blocks
    10.1.4.4, from 10.1.4.4, BGP external
      Route metric is 0, Wt is 1
  No advertising protos. 

Verification config and show output

In each STEP the following files were collected from all seven routers, one set per router. The verification config is ..._run.txt (the final state is the one from STEP 6).

File	Contents
`..._show.txt`	`show version` / `show interface description` / `show route` / `show route bgp` / `show bgp summary` / `show bgp` / `show bgp <prefix>` (`192.168.1.0/24` to `192.168.7.0/24`) / `show route` and `show cef` for `192.168.6.0/24` and `192.168.4.0/24` / `show bgp neighbors` / `show bgp update-group` / `show rpl route-policy` / `advertised-routes` / `routes` / `received routes` per peer. R1-R3 also `show ospf neighbor`; the observation points (R4, R3, R1) also `show cef exact-route`
`..._log.txt`	`show logging` limited to that STEP. A `logmsg` marker is put at the start of each STEP and its time is given to `show logging start`
`..._run.txt`	`show running-config` at that STEP (= the verification config of the STEP)
`..._ping.txt`	`ping` (50 packets, timeout 1 s) and `traceroute` (probe 3) from R7’s four sources to `192.168.6.1`, and from R3’s and R1’s Loopback1/Loopback2 to `192.168.4.1`
`..._trace.txt`	`show bgp trace neighbor` / `show bgp trace error`
`..._commit.cfg`	Only the configuration actually committed in that STEP. Only for routers whose configuration changed

STEP 0: No multipath

Router	show output	syslog	running-config	ping	trace	commit
R1	show	log	run	ping	trace	-
R2	show	log	run	-	trace	-
R3	show	log	run	ping	trace	-
R4	show	log	run	-	trace	-
R5	show	log	run	-	trace	-
R6	show	log	run	-	trace	-
R7	show	log	run	ping	trace	-

STEP 1: maximum-paths ebgp 2 on R4

Router	show output	syslog	running-config	ping	trace	commit
R1	show	log	run	ping	trace	-
R2	show	log	run	-	trace	-
R3	show	log	run	ping	trace	-
R4	show	log	run	-	trace	cfg
R5	show	log	run	-	trace	-
R6	show	log	run	-	trace	-
R7	show	log	run	ping	trace	-

STEP 2: multipath-relax and ebgp 3 on R4

Router	show output	syslog	running-config	ping	trace	commit
R1	show	log	run	ping	trace	-
R2	show	log	run	-	trace	-
R3	show	log	run	ping	trace	-
R4	show	log	run	-	trace	cfg
R5	show	log	run	-	trace	-
R6	show	log	run	-	trace	-
R7	show	log	run	ping	trace	-

STEP 3: LOCAL_PREF 90 on the R5 path inbound at R4

Router	show output	syslog	running-config	ping	trace	commit
R1	show	log	run	ping	trace	-
R2	show	log	run	-	trace	-
R3	show	log	run	ping	trace	-
R4	show	log	run	-	trace	cfg
R5	show	log	run	-	trace	-
R6	show	log	run	-	trace	-
R7	show	log	run	ping	trace	-

STEP 4: maximum-paths ibgp 2 on R3

Router	show output	syslog	running-config	ping	trace	commit
R1	show	log	run	ping	trace	-
R2	show	log	run	-	trace	-
R3	show	log	run	ping	trace	cfg
R4	show	log	run	-	trace	-
R5	show	log	run	-	trace	-
R6	show	log	run	-	trace	-
R7	show	log	run	ping	trace	-

STEP 5: maximum-paths eibgp 2 on R1

Router	show output	syslog	running-config	ping	trace	commit
R1	show	log	run	ping	trace	cfg
R2	show	log	run	-	trace	-
R3	show	log	run	ping	trace	-
R4	show	log	run	-	trace	-
R5	show	log	run	-	trace	-
R6	show	log	run	-	trace	-
R7	show	log	run	ping	trace	-

STEP 6: Revert STEP 1-5 (final state)

Router	show output	syslog	running-config	ping	trace	commit
R1	show	log	run	ping	trace	cfg
R2	show	log	run	-	trace	-
R3	show	log	run	ping	trace	cfg
R4	show	log	run	-	trace	cfg
R5	show	log	run	-	trace	-
R6	show	log	run	-	trace	-
R7	show	log	run	ping	trace	-

Packet captures were taken per STEP on the R4-R5 link (to confirm the advertisement to R5 does not change) and on the R1-R4 link.

STEP	R4-R5	R1-R4
0	pcap	pcap
1	pcap	pcap
2	pcap	pcap
3	pcap	pcap
4	pcap	pcap
5	pcap	pcap
6	pcap	pcap

References

Source	Title	Summary
RFC 4271	A Border Gateway Protocol 4 (BGP-4)	9.1.2.2: only one route is installed in the Loc-RIB
RFC 7911	Advertisement of Multiple Paths in BGP	ADD-PATH, which operates at a different layer from multipath
Cisco IOS XE 17.x IP Routing Configuration Guide	BGP Multipath Load Sharing for Both eBGP and iBGP in an MPLS-VPN	The original eiBGP multipath feature. For MPLS VPN; can be configured only under the IPv4 VRF address family
Cisco IOS XE 17.x IP Routing Configuration Guide	eiBGP Multipath for Non-VRF Interfaces (IPv4/IPv6)	The extension outside VRFs (IOS 15.4(1)T and later). “Should be configured only on one PE device at the customer site”; with more than one, traffic “may loop between the PE devices”
Cisco ASR 9000 Routing Command Reference, IOS XR 6.4.x	maximum-paths (BGP)	On IOS XR, `eibgp` can be configured in both the VRF and non-VRF IPv4 / IPv6 address families
Cisco 8000 BGP Configuration Guide, IOS XR	EIBGP policy-based multipath with equal-cost multipath	eiBGP in the default VRF on IOS XR 7.10.1 and later, which assumes a route-policy narrowing the candidates