MPLS VPN PE-CE Routing with Static Routes

PE-CE Routing in MPLS VPN

For each attachment circuit leading to a CE, the PE has to know which addresses are reachable over it. Section 7 of RFC 4364 puts it this way.

The PE routers that attach to a particular VPN need to know, for each attachment circuit leading to that VPN, which of the VPN’s addresses should be reached over that attachment circuit.

The PE turns those into VPN-IPv4 routes and feeds them to BGP. The site’s routes never enter the IGP of the backbone.

Routes from a VPN site are NOT leaked into the backbone’s IGP.

How the PE learns them is what PE-CE routing means, and RFC 4364 lists static routing, RIP, OSPF and eBGP. This article covers the simplest of them, static routing. The CE holds a single default route toward the PE and runs neither MPLS nor BGP.

Static Routing Suits a stub VPN

Section 7 of RFC 4364 is explicit about where static routing applies.

1. Static routing (i.e., configuration) may be used. (This is likely to be useful only in stub VPNs.)

What separates a transit VPN from a stub VPN is whether the site holds a router that receives routes from outside the VPN and redistributes them to the PE.

A “transit VPN” is one that contains a router that receives routes from a “third party” (i.e., from a router that is not in the VPN, but is not a PE router) and that redistributes those routes to a PE router. A VPN that is not a transit VPN is a “stub VPN”.

When all routing is closed inside the site, the list of prefixes to configure on the PE is fixed. In a site that relays routes coming from outside, that list moves, and static routes cannot follow it. RFC 4364 also says that the vast majority of VPNs, including corporate enterprise networks, are stubs.

Configuration on IOS XR

There are two parts: writing the static routes per VRF, and feeding them to MP-BGP.

router static
 vrf CUST-A
  address-family ipv4 unicast
   10.1.2.0/24 172.16.2.2
   10.1.3.0/24 172.16.2.2
  !
 !
!
router bgp 65001
 vrf CUST-A
  rd 65001:1
  address-family ipv4 unicast
   redistribute static
   redistribute connected
  !
 !
!

Writing them in the VRF alone does not get them to the other PE. The VRF routing table and MP-BGP are separate, and redistribute is the explicit bridge between them.

The CE only needs a default route toward the PE; the prefixes of the other sites are not configured there.

router static
 address-family ipv4 unicast
  0.0.0.0/0 172.16.2.1
 !
!

Naming the Outgoing Interface Ties the Route to It

A next hop is not the only thing that can follow the destination.

Naming the interface ties the route to the state of that interface. That makes it possible to invalidate the route by shutting the interface alone, so the traffic falls back to a backup route. It is the usual lever for planned maintenance on one link, or for pushing traffic onto the backup during a fault.

The backup is the same destination with a larger administrative distance, a floating static route.

router static
 vrf CUST-A
  address-family ipv4 unicast
   10.1.2.0/24 GigabitEthernet0/0/0/0 172.16.2.2
   10.1.2.0/24 GigabitEthernet0/0/0/2 172.16.4.2 200
  !
 !
!

Only the primary, with the lower distance, is installed; when its interface goes down the backup takes its place. The prefix seen from the VPN does not change, so nothing happens at the other PE: the VPNv4 advertisement continues and only the exit inside PE2 moves.

Whether to Add redistribute connected

redistribute connected advertises the subnets of the interfaces that belong to the VRF, which means the PE-CE links. Without it, the other site cannot reach the addresses of the PE-CE link.

The choice follows from the addressing plan and from how much of the provider network the customer should see.

What Static Routing Cannot Do

A static route says nothing about whether the destination is reachable. As long as it is configured it stays in the table, and the PE keeps advertising it. It disappears only when the next hop can no longer be resolved, or when an outgoing interface named in it goes down.

So when a CE dies quietly, the PE keeps advertising the route and the packets are dropped on the PE-CE link. With eBGP or OSPF the session or the adjacency goes down and the route follows; static routing has no such mechanism.

Lab Setup

Five XRd routers. PE2 and CE-A2 are joined by two links so that shutting the primary interface can show the failover to the backup.

At day 0 the PEs have neither VRF static routes nor redistribute. The VRF, the IGP, LDP and MP-BGP are up, and the steps add the rest one at a time.

STEP 1: Writing Them in the VRF Is Not Enough

Two static routes go on PE2. 10.1.2.0/24 names the outgoing interface and 10.1.3.0/24 gives only the next hop.

Gateway of last resort is not set

S    10.1.2.0/24 [1/0] via 172.16.2.2, 00:03:00, GigabitEthernet0/0/0/0
S    10.1.3.0/24 [1/0] via 172.16.2.2, 00:03:00
C    172.16.2.0/24 is directly connected, 00:08:35, GigabitEthernet0/0/0/0
L    172.16.2.1/32 is directly connected, 00:08:35, GigabitEthernet0/0/0/0
C    172.16.4.0/24 is directly connected, 00:08:35, GigabitEthernet0/0/0/2
L    172.16.4.1/32 is directly connected, 00:08:35, GigabitEthernet0/0/0/2

The difference between the two forms shows in the output: the 10.1.2.0/24 entry carries the outgoing interface (GigabitEthernet0/0/0/0) and the 10.1.3.0/24 entry does not.

VPNv4 on PE1, meanwhile, holds nothing.

RP/0/RP0/CPU0:PE1#show bgp vpnv4 unicast
Wed Sep 23 15:39:04.036 UTC

Not a single line. The VRF routing table and MP-BGP are separate, and nothing crosses between them without redistribute.

STEP 2: redistribute static Advertises Them

redistribute static goes on both PEs.

Route Distinguisher: 65001:1 (default for vrf CUST-A)
Route Distinguisher Version: 9
*> 10.1.1.0/24        172.16.1.2               0         32768 ?
*>i10.1.2.0/24        2.2.2.2                  0    100      0 ?
*>i10.1.3.0/24        2.2.2.2                  0    100      0 ?

Processed 3 prefixes, 3 paths

The two routes from PE2 are there, along with PE1’s own 10.1.1.0/24. From here the sites can ping each other.

STEP 3: redistribute connected Adds the PE-CE Links

Route Distinguisher: 65001:1 (default for vrf CUST-A)
Route Distinguisher Version: 14
*> 10.1.1.0/24        172.16.1.2               0         32768 ?
*>i10.1.2.0/24        2.2.2.2                  0    100      0 ?
*>i10.1.3.0/24        2.2.2.2                  0    100      0 ?
*> 172.16.1.0/24      0.0.0.0                  0         32768 ?
*>i172.16.2.0/24      2.2.2.2                  0    100      0 ?
*>i172.16.4.0/24      2.2.2.2                  0    100      0 ?

Processed 6 prefixes, 6 paths

Three PE-CE link subnets bring the total to six. The difference is visible in a ping from CE-A1 to a link address.

RP/0/RP0/CPU0:CE-A1#ping 172.16.2.1 source 10.1.1.1 count 10 timeout 1
Wed Sep 23 15:42:16.717 UTC
Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 172.16.2.1 timeout is 1 seconds:
..........
Success rate is 0 percent (0/10)

RP/0/RP0/CPU0:CE-A1#ping 172.16.2.1 source 10.1.1.1 count 10 timeout 1
Wed Sep 23 15:46:25.434 UTC
Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 172.16.2.1 timeout is 1 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 12/14/29 ms

STEP 4 and 5: A Static Route Does Not Watch the Destination

STEP 4 shuts down the LAN (Loopback1) on CE-A2 and STEP 5 shuts down its interface toward the PE. Neither changes PE2’s routing table.

Gateway of last resort is not set

B    10.1.1.0/24 [200/0] via 1.1.1.1 (nexthop in vrf default), 00:10:13
S    10.1.2.0/24 [1/0] via 172.16.2.2, 00:14:42, GigabitEthernet0/0/0/0
S    10.1.3.0/24 [1/0] via 172.16.2.2, 00:14:42
B    172.16.1.0/24 [200/0] via 1.1.1.1 (nexthop in vrf default), 00:06:04
C    172.16.2.0/24 is directly connected, 00:20:17, GigabitEthernet0/0/0/0
L    172.16.2.1/32 is directly connected, 00:20:17, GigabitEthernet0/0/0/0
C    172.16.4.0/24 is directly connected, 00:20:17, GigabitEthernet0/0/0/2
L    172.16.4.1/32 is directly connected, 00:20:17, GigabitEthernet0/0/0/2

10.1.2.0/24 is still there and the VPNv4 advertisement is still six routes. Only the ping fails.

RP/0/RP0/CPU0:CE-A1#ping 10.1.2.1 source 10.1.1.1 count 10 timeout 1
Wed Sep 23 15:50:00.716 UTC
Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 10.1.2.1 timeout is 1 seconds:
..........
Success rate is 0 percent (0/10)
RP/0/RP0/CPU0:CE-A1#ping 10.1.3.1 source 10.1.1.1 count 10 timeout 1
Wed Sep 23 15:50:15.082 UTC
Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 10.1.3.1 timeout is 1 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 10/15/38 ms

10.1.2.1 fails while 10.1.3.1, behind the same CE, still answers. Shutting the CE’s own interface in STEP 5 gives the same result, because the interface on the PE2 side stays up: the PE never learns that the CE is gone.

STEP 6: The Route Is Withdrawn When the PE Side Goes Down

Shutting Gi0/0/0/0 on PE2 removes the connected route, the next hop can no longer be resolved, and the static routes leave the RIB.

Gateway of last resort is not set

B    10.1.1.0/24 [200/0] via 1.1.1.1 (nexthop in vrf default), 00:19:33
B    172.16.1.0/24 [200/0] via 1.1.1.1 (nexthop in vrf default), 00:15:25
C    172.16.4.0/24 is directly connected, 00:29:38, GigabitEthernet0/0/0/2
L    172.16.4.1/32 is directly connected, 00:29:38, GigabitEthernet0/0/0/2

Both 10.1.2.0/24 and 10.1.3.0/24 are gone, and VPNv4 on PE1 is down to three routes.

Route Distinguisher: 65001:1 (default for vrf CUST-A)
Route Distinguisher Version: 17
*> 10.1.1.0/24        172.16.1.2               0         32768 ?
*> 172.16.1.0/24      0.0.0.0                  0         32768 ?
*>i172.16.4.0/24      2.2.2.2                  0    100      0 ?

Processed 3 prefixes, 3 paths

The UPDATE that PE2 sent is in the capture, as No.31 of the STEP 6 file.

Border Gateway Protocol - UPDATE Message
    Marker: ffffffffffffffffffffffffffffffff
    Length: 75
    Type: UPDATE Message (2)
    Withdrawn Routes Length: 0
    Total Path Attribute Length: 52
    Path attributes
        Path Attribute - MP_UNREACH_NLRI
            Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
                1... .... = Optional: Set
                .0.. .... = Transitive: Not set
                ..0. .... = Partial: Not set
                ...1 .... = Extended-Length: Set
                .... 0000 = Unused: 0x0
            Type Code: MP_UNREACH_NLRI (15)
            Length: 48
            Address family identifier (AFI): IPv4 (1)
            Subsequent address family identifier (SAFI): Labeled VPN Unicast (128)
            Withdrawn Routes
                BGP Prefix
                    Prefix Length: 112
                    Label Stack: 0 (withdrawn)
                    Route Distinguisher: 65001:1
                    MP Unreach NLRI IPv4 prefix: 10.1.3.0
                BGP Prefix
                    Prefix Length: 112
                    Label Stack: 0 (withdrawn)
                    Route Distinguisher: 65001:1
                    MP Unreach NLRI IPv4 prefix: 10.1.2.0
                BGP Prefix
                    Prefix Length: 112
                    Label Stack: 0 (withdrawn)
                    Route Distinguisher: 65001:1
                    MP Unreach NLRI IPv4 prefix: 172.16.2.0

One MP_UNREACH_NLRI withdraws all three at once.

STEP 7 and 8: Naming the Interface Makes the Traffic Fail Over

The primary link comes back and a backup route is added with a distance of 200.

interface GigabitEthernet0/0/0/0
 no shutdown
!
router static
 vrf CUST-A
  address-family ipv4 unicast
   10.1.2.0/24 GigabitEthernet0/0/0/2 172.16.4.2 200
   10.1.3.0/24 172.16.4.2 200
  !
 !
!
end

The backup does not appear in the table; only the primary, with the lower distance, is installed.

Gateway of last resort is not set

B    10.1.1.0/24 [200/0] via 1.1.1.1 (nexthop in vrf default), 00:23:22
S    10.1.2.0/24 [1/0] via 172.16.2.2, 00:02:22, GigabitEthernet0/0/0/0
S    10.1.3.0/24 [1/0] via 172.16.2.2, 00:02:22
B    172.16.1.0/24 [200/0] via 1.1.1.1 (nexthop in vrf default), 00:19:14
C    172.16.2.0/24 is directly connected, 00:02:22, GigabitEthernet0/0/0/0
L    172.16.2.1/32 is directly connected, 00:02:22, GigabitEthernet0/0/0/0
C    172.16.4.0/24 is directly connected, 00:33:27, GigabitEthernet0/0/0/2
L    172.16.4.1/32 is directly connected, 00:33:27, GigabitEthernet0/0/0/2

Now both ends of the primary link go down. As STEP 5 showed, shutting one end leaves the interface up at the other, so a link failure has to be made at both ends. The primary route, tied to its outgoing interface, becomes invalid and the backup takes over.

Gateway of last resort is not set

B    10.1.1.0/24 [200/0] via 1.1.1.1 (nexthop in vrf default), 00:27:21
S    10.1.2.0/24 [200/0] via 172.16.4.2, 00:02:28, GigabitEthernet0/0/0/2
S    10.1.3.0/24 [200/0] via 172.16.4.2, 00:02:28
B    172.16.1.0/24 [200/0] via 1.1.1.1 (nexthop in vrf default), 00:23:12
C    172.16.4.0/24 is directly connected, 00:37:25, GigabitEthernet0/0/0/2
L    172.16.4.1/32 is directly connected, 00:37:25, GigabitEthernet0/0/0/2

The distance is now [200/0] and the exit is GigabitEthernet0/0/0/2, the backup. Connectivity never stopped.

RP/0/RP0/CPU0:CE-A1#traceroute 10.1.2.1 source 10.1.1.1 timeout 1 probe 2 maxttl 6
Wed Sep 23 16:07:24.228 UTC

Type escape sequence to abort.
Tracing the route to 10.1.2.1

 1  172.16.1.1 5 msec  4 msec 
 2  10.0.13.3 [MPLS: Labels 24001/24003 Exp 0] 25 msec  17 msec 
 3  10.0.23.2 [MPLS: Label 24003 Exp 0] 22 msec  15 msec 
 4  172.16.4.2 18 msec  * 

The last hop is 172.16.4.2, the CE end of the backup link. The detour is visible in the path itself.

What changed as seen from the VPN becomes clear next to STEP 6.

$ tshark -r mpls-vpn-pece-static-step6-pe1p1.pcap -Y 'bgp.type==2' \
    -T fields -e frame.number -e ip.src \
    -e bgp.mp_unreach_nlri_ipv4_prefix -e bgp.mp_reach_nlri_ipv4_prefix
31	2.2.2.2	10.1.3.0,10.1.2.0,172.16.2.0	

$ tshark -r mpls-vpn-pece-static-step8-pe1p1.pcap -Y 'bgp.type==2' \
    -T fields -e frame.number -e ip.src \
    -e bgp.mp_unreach_nlri_ipv4_prefix -e bgp.mp_reach_nlri_ipv4_prefix
7	2.2.2.2	172.16.2.0	
9	2.2.2.2		10.1.3.0,10.1.2.0

STEP 6 withdrew the two customer routes and one link subnet. STEP 8 withdrew only 172.16.2.0/24, the link that really went away. 10.1.2.0/24 and 10.1.3.0/24 were not withdrawn; they were simply re-advertised, so from the other PE the routes stayed alive and only the exit inside PE2 moved.

STEP 9: Restoring the Primary

Bringing the primary link back installs the lower-distance route again and the state returns to that of STEP 7. The backup configuration stays in place.

References

Book: Luc De Ghein, MPLS Fundamentals (Cisco Press, 2006), Chapter 7

Verification Configs and show Output

Every STEP was captured on all five routers, one file per router and kind. The verification config is the ..._run.txt file (the final state is the one from the last STEP).

STEP 0: Initial state (no VRF static routes, no redistribute)

STEP 1: Add the VRF static routes on the PEs

STEP 2: redistribute static

STEP 3: redistribute connected

STEP 4: Shut down Loopback1 on CE-A2

STEP 5: Bring Lo1 back up and shut down Gi0/0/0/0 on CE-A2

STEP 6: Bring CE-A2 Gi0/0/0/0 back up and shut down Gi0/0/0/0 on PE2

STEP 7: Bring PE2 Gi0/0/0/0 back up and add the backup route (AD 200)

STEP 8: Shut down both ends of the primary link

STEP 9: Restore (final state)

Packet captures were taken per STEP.

Related Articles

• What Is MPLS VPN (L3VPN)
• MPLS VPN VRF (Virtual Routing and Forwarding)
• MPLS VPN RD (Route Distinguisher)
• MPLS VPN Route Target (RT)
• MPLS VPN MP-BGP (Propagating VPNv4 Routes)
• MPLS VPN Label Allocation (per-prefix / per-CE / per-VRF)
• MPLS VPN Forwarding with Two Labels (Transport Label and VPN Label)
• MPLS VPN PE-CE Routing with Static Routes
• MPLS VPN PE-CE Routing with eBGP
• When Several MPLS VPN Sites Share One AS Number (as-override)