MPLS VPN MP-BGP (Propagating VPNv4 Routes)

What is MP-BGP in an MPLS VPN?

In an MPLS VPN, the PEs exchange customer routes with MP-BGP (Multiprotocol BGP). MP-BGP is the extension that lets a single BGP session carry routes other than IPv4, defined in RFC 4760.

Route types are told apart by the pair of AFI (Address Family Identifier) and SAFI (Subsequent Address Family Identifier). MPLS VPN routes (VPNv4) are AFI 1 / SAFI 128 (RFC 4364 section 4.3.4).

Address family	AFI	SAFI	NLRI contents
IPv4 unicast	1	1	IPv4 prefix
VPNv4 unicast	1	128	Label + RD + IPv4 prefix

Several address families can share one session. Which ones flow is settled when the session comes up.

Capability exchange decides what flows

At session start BGP exchanges OPEN messages, and their Capabilities tell the peer which address families are supported. The MP-BGP capability has code 1 and length 4, and its value is the AFI (2 bytes), a reserved byte and the SAFI (1 byte) (RFC 4760 section 8).

RFC 4760 section 8 sets the condition for exchange:

To have a bi-directional exchange of routing information for a particular <AFI, SAFI> between a pair of BGP speakers, each such speaker MUST advertise to the other (via the Capability Advertisement mechanism) the capability to support that particular <AFI, SAFI> route.

Only the address families that both sides advertise are exchanged. If one side leaves an address family out, it is not exchanged.

Because capabilities are exchanged only in OPEN, IOS XR re-establishes the session when an address family is added to or removed from a neighbor. It sends a NOTIFICATION with Cease (6), subcode 6 “Other Configuration Change” (RFC 4486 section 4). The other address families on the same session also go down while the session is re-established.

OPEN carries other capabilities too. SAFI 128 also appears inside Extended Next Hop Encoding (code 5), but that capability is about how the next hop is written. What decides which address families are carried is code 1.

Inside a VPNv4 UPDATE

VPNv4 routes are carried in an attribute of the UPDATE: the MP_REACH_NLRI attribute (Type 14). The original BGP NLRI field and NEXT_HOP attribute are for IPv4 unicast, so VPNv4 can only be carried in this attribute. IOS XR in fact sends IPv4 unicast routes in MP_REACH_NLRI as well.

The next hop is the PE’s own address with an RD of 0, 12 bytes in all. RFC 4364 section 4.3.2 says:

When a PE router distributes a VPN-IPv4 route via BGP, it uses its own address as the “BGP next hop”. This address is encoded as a VPN-IPv4 address with an RD of 0.

RFC 4760 requires the next hop to be in the same address family as the NLRI, so it takes the VPN-IPv4 form rather than a plain IPv4 address.

The NLRI holds the label, the RD and the prefix, in that order. The leading length field counts bits (RFC 8277 section 2.2). The label field is 3 bytes (20 bits of label, 3 reserved bits and the S bit), so a /24 route has a length of 24 + 64 + 24 = 112. How labels are assigned is covered in MPLS VPN label allocation, and the RD bytes in MPLS VPN Route Distinguisher (RD).

Withdrawal and End-of-RIB

Routes are withdrawn with the MP_UNREACH_NLRI attribute (Type 15). It holds the AFI, the SAFI and the routes being withdrawn, with no next hop. The routes are written the same way as when advertised, except that the label field is called Compatibility and the sender puts 0x800000 in it (RFC 8277 section 2.4).

End-of-RIB tells the peer that the initial routes after session establishment have all been sent. For address families other than IPv4 unicast, End-of-RIB is an UPDATE carrying only an MP_UNREACH_NLRI with no withdrawn routes (RFC 4724 section 2). For IPv4 unicast it is the minimum UPDATE (23 bytes) with no withdrawals and no attributes. IOS XR sends End-of-RIB even without Graceful Restart configured.

Configuring MP-BGP on IOS XR

On IOS XR, the address families in use are written both directly under router bgp and under the neighbor. What is written under the neighbor is what gets advertised as capabilities in the OPEN to that peer.

MP-BGP configuration on PE1 (IOS XR)
router bgp 65001
 bgp router-id 1.1.1.1
 address-family ipv4 unicast
  network 198.51.100.1/32
 !
 address-family vpnv4 unicast
 !
 neighbor 2.2.2.2
  remote-as 65001
  update-source Loopback0
  address-family ipv4 unicast
  !
  address-family vpnv4 unicast
  !
 !
 vrf CUST-A
  rd 65001:1
  address-family ipv4 unicast
   redistribute static
  !
 !
!

VRF routes enter BGP through redistribute static under vrf CUST-A, get the RD and RT attached, and are sent to neighbor 2.2.2.2 as VPNv4 routes. RT configuration is covered in MPLS VPN Route Target (RT).

Lab setup

Five XRd routers, with PE1 and PE2 serving the two sites of customer A. The iBGP session between the PEs carries both ipv4 unicast and vpnv4 unicast, so that removing only one of them shows the difference.

Node	Role
PE1 / PE2	vrf CUST-A (RD `65001:1`, RT `65001:100`). Lo1 (`198.51.100.1` / `.2`) is advertised only by BGP ipv4 unicast (not in OSPF)
P1	Core only (OSPF area 0 + LDP). It runs no BGP
CE-A1 / CE-A2	Customer A sites 1 / 2. LAN `10.1.1.0/24` / `10.1.2.0/24`

STEP	Change	What it shows
0	Initial state	Both VPNv4 and IPv4 are exchanged. P1 runs no BGP
1	Remove `address-family vpnv4 unicast` from PE2’s neighbor PE1	The session is re-established and only VPNv4 stops being exchanged. IPv4 remains
2	Put it back	OPEN carries the SAFI 128 capability, then UPDATEs and End-of-RIB flow
3	Remove the `10.1.1.0/24` static route from PE1’s VRF	The withdrawal arrives in `MP_UNREACH_NLRI`
4	Put it back	The route is advertised again

One session carries two kinds of route (STEP 0)

This is the result of the capability exchange with PE2, seen from PE1. IPv4 unicast and VPNv4 unicast are both advertised and received, so both are exchanged.

STEP 0 PE1, show bgp neighbors 2.2.2.2 (excerpt)
  Multi-protocol capability received
  Neighbor capabilities:
    Route refresh: advertised (old + new) and received (old + new)
    4-byte AS: advertised and received
    Address family IPv4 Unicast: advertised and received
    Address family VPNv4 Unicast: advertised and received

P1 runs no BGP. VPNv4 routes are exchanged only between PE1 and PE2.

STEP 0 P1, show bgp vpnv4 unicast
RP/0/RP0/CPU0:P1#show bgp vpnv4 unicast
Tue Sep 22 07:23:26.227 UTC
% BGP instance 'default' not active

When one side does not advertise it, only VPNv4 stops (STEP 1)

At STEP 1, removing address-family vpnv4 unicast from PE2’s neighbor PE1 made PE2 re-establish the session.

STEP 1 PE2 syslog
RP/0/RP0/CPU0:Sep 22 07:24:26.709 UTC: bgp[1084]: %ROUTING-BGP-5-ADJCHANGE : neighbor 1.1.1.1 Down - Address family removed (CEASE notification sent - configuration change) (VRF: default) (AS: 65001) 
RP/0/RP0/CPU0:Sep 22 07:24:26.710 UTC: bgp[1084]: %ROUTING-BGP-5-NSR_STATE_CHANGE : Changed state to NSR-Ready 
RP/0/RP0/CPU0:Sep 22 07:24:28.006 UTC: ssh_syslog_proxy[1188]: %SECURITY-SSHD_SYSLOG_PRX-6-INFO_GENERAL : sshd[5917]: Received disconnect from 10.20.1.1 port 33910:11: disconnected by user 
RP/0/RP0/CPU0:Sep 22 07:24:28.006 UTC: ssh_syslog_proxy[1188]: %SECURITY-SSHD_SYSLOG_PRX-6-INFO_GENERAL : sshd[5917]: Disconnected from user cisco 10.20.1.1 port 33910 
RP/0/RP0/CPU0:Sep 22 07:24:54.951 UTC: bgp[1084]: %ROUTING-BGP-5-ADJCHANGE : neighbor 1.1.1.1 Up (VRF: default) (AS: 65001) 

The NOTIFICATION PE2 sent before re-establishing is No.9 in the attached STEP 1 capture.

STEP 1 No.9 NOTIFICATION (PE2 to PE1), tshark -V
Border Gateway Protocol - NOTIFICATION Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 21
    Type: NOTIFICATION Message (3)
    Major error Code: Cease (6)
    Minor error Code (Cease): Other Configuration Change (6)

Download the pcap of the packet in the tshark output above (No.9 NOTIFICATION)

In the capability exchange after re-establishment, VPNv4 unicast is only advertised. PE1 advertised it, but PE2 did not advertise it back.

STEP 1 PE1, show bgp neighbors 2.2.2.2 (excerpt)
  Multi-protocol capability received
  Neighbor capabilities:
    Route refresh: advertised (old + new) and received (old + new)
    4-byte AS: advertised and received
    Address family IPv4 Unicast: advertised and received
    Address family VPNv4 Unicast: advertised

As a result, PE2’s route (10.1.2.0/24) disappeared from PE1’s VPNv4 table, leaving only PE1’s own route. On the IPv4 unicast side, PE2’s 198.51.100.2/32 is still received.

STEP 1 PE1, show bgp vpnv4 unicast (excerpt)
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65001:1 (default for vrf CUST-A)
Route Distinguisher Version: 9
*> 10.1.1.0/24        172.16.1.2               0         32768 ?

Processed 1 prefixes, 1 paths

STEP 1 PE1, show bgp ipv4 unicast (excerpt)
   Network            Next Hop            Metric LocPrf Weight Path
*> 198.51.100.1/32    0.0.0.0                  0         32768 i
*>i198.51.100.2/32    2.2.2.2                  0    100      0 i

Processed 2 prefixes, 2 paths

Reachability splits the same way. The VRF destination is unreachable (U is an ICMP unreachable reply), while the destination learned through IPv4 unicast is reachable.

STEP 1 PE1 ping
RP/0/RP0/CPU0:PE1#ping vrf CUST-A 10.1.2.1 count 50 timeout 1
Tue Sep 22 07:26:01.416 UTC
Type escape sequence to abort.
Sending 50, 100-byte ICMP Echos to 10.1.2.1 timeout is 1 seconds:
UUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUUU
Success rate is 0 percent (0/50)
RP/0/RP0/CPU0:PE1#ping 198.51.100.2 source 198.51.100.1 count 20 timeout 1
Tue Sep 22 07:26:52.573 UTC
Type escape sequence to abort.
Sending 20, 100-byte ICMP Echos to 198.51.100.2 timeout is 1 seconds:
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (20/20), round-trip min/avg/max = 7/11/49 ms

OPEN advertises the capability, then UPDATEs and End-of-RIB follow (STEP 2)

At STEP 2 the address family is put back. PE2 re-established the session once more and included the SAFI 128 capability in its OPEN, No.33 in the attached STEP 2 capture.

STEP 2 No.33 OPEN (PE2 to PE1), tshark -V (excerpt)
Border Gateway Protocol - OPEN Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 83
    Type: OPEN Message (1)
    Version: 4
    My AS: 65001
    Hold Time: 180
    BGP Identifier: 2.2.2.2
    Optional Parameters Length: 54
    Optional Parameters
        Optional Parameter: Capability
            Parameter Type: Capability (2)
            Parameter Length: 6
            Capability: Multiprotocol extensions capability
                Type: Multiprotocol extensions capability (1)
                Length: 4
                AFI: IPv4 (1)
                Reserved: 00
                SAFI: Unicast (1)
        Optional Parameter: Capability
            Parameter Type: Capability (2)
            Parameter Length: 6
            Capability: Multiprotocol extensions capability
                Type: Multiprotocol extensions capability (1)
                Length: 4
                AFI: IPv4 (1)
                Reserved: 00
                SAFI: Labeled VPN Unicast (128)

Download the pcap of the packet in the tshark output above (No.33 OPEN)

PE2 then sent its VPNv4 route and End-of-RIB in one TCP segment (No.38). In the MP_REACH_NLRI of the first UPDATE, the next hop is PE2’s address with RD 0:0 and the NLRI length is 112.

STEP 2 No.38 MP_REACH_NLRI of the first UPDATE (tshark -V excerpt)
        Path Attribute - MP_REACH_NLRI
            Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
                1... .... = Optional: Set
                .0.. .... = Transitive: Not set
                ..0. .... = Partial: Not set
                ...1 .... = Extended-Length: Set
                .... 0000 = Unused: 0x0
            Type Code: MP_REACH_NLRI (14)
            Length: 32
            Address family identifier (AFI): IPv4 (1)
            Subsequent address family identifier (SAFI): Labeled VPN Unicast (128)
            Next hop:  RD=0:0 IPv4=2.2.2.2
                Route Distinguisher: 0:0
                IPv4 Address: 2.2.2.2
            Number of Subnetwork points of attachment (SNPA): 0
            Network Layer Reachability Information (NLRI)
                BGP Prefix
                    Prefix Length: 112
                    Label Stack: 24003 (bottom)
                    Route Distinguisher: 65001:1
                    MP Reach NLRI IPv4 prefix: 10.1.2.0

The second UPDATE is the VPNv4 End-of-RIB. The MP_UNREACH_NLRI has a length of 3 (AFI and SAFI only), with no routes withdrawn.

STEP 2 No.38 second UPDATE (End-of-RIB), tshark -V
Border Gateway Protocol - UPDATE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 29
    Type: UPDATE Message (2)
    Withdrawn Routes Length: 0
    Total Path Attribute Length: 6
    Path attributes
        Path Attribute - MP_UNREACH_NLRI
            Flags: 0x80, Optional, Non-transitive, Complete
                1... .... = Optional: Set
                .0.. .... = Transitive: Not set
                ..0. .... = Partial: Not set
                ...0 .... = Extended-Length: Not set
                .... 0000 = Unused: 0x0
            Type Code: MP_UNREACH_NLRI (15)
            Length: 3
            Address family identifier (AFI): IPv4 (1)
            Subsequent address family identifier (SAFI): Labeled VPN Unicast (128)
            Withdrawn Routes

Download the pcap of the packet in the tshark output above (No.38 UPDATE)

Withdrawing with MP_UNREACH_NLRI (STEP 3-4)

At STEP 3, removing the 10.1.1.0/24 static route from PE1’s VRF made PE1 send a withdrawal, No.6 in the attached STEP 3 capture.

STEP 3 No.6 UPDATE (PE1 to PE2), tshark -V
Border Gateway Protocol - UPDATE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 45
    Type: UPDATE Message (2)
    Withdrawn Routes Length: 0
    Total Path Attribute Length: 22
    Path attributes
        Path Attribute - MP_UNREACH_NLRI
            Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
                1... .... = Optional: Set
                .0.. .... = Transitive: Not set
                ..0. .... = Partial: Not set
                ...1 .... = Extended-Length: Set
                .... 0000 = Unused: 0x0
            Type Code: MP_UNREACH_NLRI (15)
            Length: 18
            Address family identifier (AFI): IPv4 (1)
            Subsequent address family identifier (SAFI): Labeled VPN Unicast (128)
            Withdrawn Routes
                BGP Prefix
                    Prefix Length: 112
                    Label Stack: 0 (withdrawn)
                    Route Distinguisher: 65001:1
                    MP Unreach NLRI IPv4 prefix: 10.1.1.0

tshark shows the label field as Label Stack: 0 (withdrawn), but the bytes on the wire are 80 00 00 (Compatibility 0x800000). They are the three bytes right after 80 (SAFI 128) and 70 (length 112) on the 0x0050 row.

STEP 3 No.6 raw bytes (tshark -x)
52 54 00 48 6a f7 52 54 00 d4 50 0d 88 47 05 dc   RT.Hj.RT..P..G..
1d ff 45 c0 00 68 c1 79 00 00 ff 06 f3 50 01 01   ..E..h.y.....P..
01 01 02 02 02 02 eb 82 00 b3 eb 78 93 ab a3 89   ...........x....
07 99 50 18 7e 6e 4c ee 00 00 ff ff ff ff ff ff   ..P.~nL.........
ff ff ff ff ff ff ff ff ff ff 00 2d 02 00 00 00   ...........-....
16 90 0f 00 12 00 01 80 70 80 00 00 00 00 fd e9   ........p.......
00 00 00 01 0a 01 01 ff ff ff ff ff ff ff ff ff   ................
ff ff ff ff ff ff ff 00 13 04                     ..........

Download the pcap of the packet in the tshark output above (No.6 UPDATE)

10.1.1.0/24 disappeared from the VPNv4 table of PE2, which received the withdrawal, and came back when STEP 4 restored it.

STEP 3 PE2, show bgp vpnv4 unicast (excerpt)
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65001:1 (default for vrf CUST-A)
Route Distinguisher Version: 12
*> 10.1.2.0/24        172.16.2.2               0         32768 ?

Processed 1 prefixes, 1 paths

STEP 4 PE2, show bgp vpnv4 unicast (excerpt)
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65001:1 (default for vrf CUST-A)
Route Distinguisher Version: 14
*>i10.1.1.0/24        1.1.1.1                  0    100      0 ?
*> 10.1.2.0/24        172.16.2.2               0         32768 ?

Processed 2 prefixes, 2 paths

References

Standard	Title	What this article used
RFC 4760	Multiprotocol Extensions for BGP-4	3 (MP_REACH_NLRI), 4 (MP_UNREACH_NLRI), 8 (the capability and the exchange condition)
RFC 4364	BGP/MPLS IP Virtual Private Networks (VPNs)	4.3.2 (next hop as VPN-IPv4 with RD 0), 4.3.4 (AFI 1 / SAFI 128)
RFC 8277	Using BGP to Bind MPLS Labels to Address Prefixes	2.2 (length of labeled NLRI), 2.4 (Compatibility in withdrawals)
RFC 4724	Graceful Restart Mechanism for BGP	2 (End-of-RIB)
RFC 4486	Subcodes for BGP Cease Notification Message	4 (Other Configuration Change)

Test environment: Cisco IOS XRd 26.1.1 (Cisco Modeling Labs)

Configurations and show output

The following files were captured from all five routers at every STEP, one file per router. The configurations under test are these ..._run.txt files (the final state is the one from the last STEP).

File	Contents
`..._show.txt`	`show version`, `show bgp neighbors`, `show bgp ipv4 unicast`, `show bgp vpnv4 unicast`, `show bgp vpnv4 unicast rd <RD> <prefix>`, the three-command set (`advertised-routes` / `routes` / `received routes`, for both ipv4 and vpnv4), `show bgp vrf CUST-A`, `show mpls forwarding` and others
`..._log.txt`	`show logging` narrowed to that STEP. A marker is written with `logmsg` at the start of each STEP and its timestamp passed to `show logging start`
`..._run.txt`	`show running-config` at that STEP (the configuration under test)
`..._ping.txt`	The `ping` and `traceroute` results for that STEP
`..._trace.txt`	The PE’s `show bgp trace`, filtered by the peer address, last 150 lines
`..._commit.cfg`	Only the configuration committed in that STEP, for the routers that were changed

The final state (STEP 4) is back to the same configuration as STEP 0.

STEP 0: Initial state

Router	show	syslog	running-config	ping	trace	committed config
PE1	show	log	run	ping	trace	-
PE2	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
CE-A1	show	log	run	ping	-	-
CE-A2	show	log	run	ping	-	-

STEP 1: Remove address-family vpnv4 unicast from PE2’s neighbor PE1

Router	show	syslog	running-config	ping	trace	committed config
PE1	show	log	run	ping	trace	-
PE2	show	log	run	ping	trace	commit
P1	show	log	run	ping	-	-
CE-A1	show	log	run	ping	-	-
CE-A2	show	log	run	ping	-	-

STEP 2: Put it back

Router	show	syslog	running-config	ping	trace	committed config
PE1	show	log	run	ping	trace	-
PE2	show	log	run	ping	trace	commit
P1	show	log	run	ping	-	-
CE-A1	show	log	run	ping	-	-
CE-A2	show	log	run	ping	-	-

STEP 3: Remove the 10.1.1.0/24 static route from PE1’s VRF

Router	show	syslog	running-config	ping	trace	committed config
PE1	show	log	run	ping	trace	commit
PE2	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
CE-A1	show	log	run	ping	-	-
CE-A2	show	log	run	ping	-	-

STEP 4: Put it back (final state)

Router	show	syslog	running-config	ping	trace	committed config
PE1	show	log	run	ping	trace	commit
PE2	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
CE-A1	show	log	run	ping	-	-
CE-A2	show	log	run	ping	-	-
Packet captures were taken per STEP.

STEP	PE1-P1
0	pcap
1	pcap
2	pcap
3	pcap
4	pcap