Sham links in MPLS VPN (dealing with a backdoor link)

Sham links in MPLS VPN (dealing with a backdoor link)

Two sites joined by an MPLS VPN are sometimes also joined by a direct link between them: a leased line kept as a backup, or a circuit that was there before the VPN. That is a backdoor link.

With OSPF as the PE-CE protocol, this layout makes traffic between the sites take the backdoor link, no matter how much better the path through the VPN is.

An intra-area route always beats an inter-area route

The reason is in the OSPF decision process. OSPF always prefers an intra-area route over an inter-area route, and never compares their costs.

The backdoor link joins the two sites directly in OSPF, so a route across it is intra-area. A route through the VPN is advertised by the PE in a Type 3 LSA, which makes it inter-area (PE-CE routing with OSPF in MPLS VPN). Setting the backdoor cost to 100 does not change the outcome.

RFC 4577 section 4.2.7.1 first puts this in perspective.

It is possible to treat these two sites as a single VPN site that just happens to be multihomed to the backbone. This is in fact the simplest thing to do and is perfectly adequate, provided that the preferred route between the two sites is via the intra-area OSPF link (a “backdoor link”), rather than via the VPN backbone.

If the backdoor is the path you want, that is fine. The problem is the other case.

If it is desired to have OSPF prefer the routes through the backbone over the routes through the backdoor link, then the routes through the backbone must be appear to be intra-area routes. To make a route through the backbone appear to be an intra-area route, it is necessary to make it appear as if there is an intra-area link connecting the two PE routers. This is what we refer to as a “sham link”.

Make it look as if there is an intra-area link between the two PEs. That is a sham link, and RFC 4577 marks it as an OPTIONAL feature.

Rules for the endpoint address

A sham link is a relation between two VRFs. Each VRF is given one Sham Link Endpoint Address.

The Sham Link Endpoint Address is an address in the VPN’s address space, not the SP’s address space.

Two rules go with that address.

The Sham Link Endpoint Address MUST be distributed by BGP as a VPN-IPv4 address whose IPv4 address prefix part is 32 bits long. The Sham Link Endpoint Address MUST NOT be advertised by OSPF; if there is no BGP route to the Sham Link Endpoint Address, that address is to appear unreachable, so that the sham link appears to be down.

The configuration is done at both ends; there is no procedure for bringing up a sham link configured on one side only (4.2.7.2).

OSPF on a sham link travels as MPLS

A sham link is a logical link whose substance is the VPN backbone.

An OSPF protocol packet sent on a Sham Link from one PE to another must have as its IP source address the Sham Link Endpoint Address of the sender, and as its IP destination address the Sham Link Endpoint Address of the receiver. The packet will travel from one PE router to the other over the VPN backbone, which means that it can be expected to traverse multiple hops.

So OSPF packets between the two endpoint addresses cross the core with labels on them. Forwarding has its own rule (4.2.7.4): the OSPF route is the preferred one, but a route whose next hop is the sham link is forwarded according to the corresponding BGP route.

The same section also fixes what a sham link is: an unnumbered point-to-point intra-area link, advertised as a type 1 link in a type 1 LSA. On the metric:

The OSPF metric associated with a sham link MUST be configurable (and there MUST be a configurable default). Whether traffic between the sites flows via a backdoor link or via the VPN backbone (i.e., via the sham link) depends on the settings of the OSPF link metrics.

Once the sham link is there, the metric decides which way the traffic goes.

A sham link is treated as a demand circuit

Section 4.2.7.3 has one more rule.

Sham links SHOULD be treated by OSPF as OSPF Demand Circuits. This means that LSAs will be flooded over them, but periodic refresh traffic is avoided.

Demand circuits come from RFC 1793: LSAs are still flooded, but the periodic refreshes are not sent. When both ends agree by setting the DC bit in their hellos and database description packets, the hellos themselves are suppressed as well (RFC 1793 section 3.2.1), and the LSAs carry the DoNotAge bit so they are not aged in the database. The DC bit is one of the options; its position in the field is covered in the OSPF options field.

The reason for keeping the flooding is spelled out too.

Note that, as long as the backdoor link is up, flooding the LSAs over the sham link serves no purpose. However, if the backdoor link goes down, OSPF does not have mechanisms enabling the routers in one site to rapidly flush the LSAs from the other site. Therefore, it is still necessary to maintain synchronization among the LSA databases at the two sites, hence the flooding over the sham link.

It is pointless while the backdoor is up, but when the backdoor goes down OSPF has no way to flush the other site’s LSAs quickly. So the databases are kept in sync regardless.

The lab

The two sites are joined through PE1 and PE2 by the MPLS VPN, and CE-A1 and CE-A2 are also joined by a direct backdoor link. The PE-CE links and the backdoor are all in OSPF area 1, and the backdoor has a cost of 100. The sham link endpoints are /32 loopbacks in the VRF of each PE (10.0.101.1 and 10.0.102.1).

The backdoor wins even at a high cost

Looking at 10.1.2.1/32 from CE-A1, the next hop is 10.1.9.2, the backdoor, at a cost of 100.

RP/0/RP0/CPU0:CE-A1#show route 10.1.2.1/32
Mon Oct 12 09:58:04.552 UTC

Routing entry for 10.1.2.1/32
  Known via "ospf 1", distance 110, metric 101, type intra area
  Installed Oct 12 09:53:04.175 for 00:05:00
  Routing Descriptor Blocks
    10.1.9.2, from 12.12.12.12, via GigabitEthernet0/0/0/1
      Route metric is 101
  No advertising protos. 

The traceroute crosses the direct link in one hop.

RP/0/RP0/CPU0:CE-A1#traceroute 10.1.2.1 source 10.1.1.1 timeout 1 probe 2 maxttl 6
Mon Oct 12 09:55:40.242 UTC

Type escape sequence to abort.
Tracing the route to 10.1.2.1

 1  10.1.9.2 6 msec  * 

How the endpoint address is held

Before adding the sham link, check that the endpoint address follows the rules. The remote endpoint 10.0.102.1/32 arrives over BGP.

RP/0/RP0/CPU0:PE1#show route vrf CUST-A 10.0.102.1/32
Mon Oct 12 09:56:25.044 UTC

Routing entry for 10.0.102.1/32
  Known via "bgp 65001", distance 200, metric 0, type internal
  Installed Oct 12 09:54:15.829 for 00:02:09
  Routing Descriptor Blocks
    2.2.2.2, from 2.2.2.2
      Nexthop in Vrf: "default", Table: "default", IPv4 Unicast, Table Id: 0xe0000000
      Route metric is 0, Wt is 1
  No advertising protos. 

It is not among the OSPF routes of the VRF, because it is never advertised in OSPF.

RP/0/RP0/CPU0:PE1#show route vrf CUST-A ospf
Mon Oct 12 09:56:25.251 UTC

O    10.1.1.1/32 [110/2] via 172.16.1.2, 00:03:30, GigabitEthernet0/0/0/0
O    10.1.2.1/32 [110/102] via 172.16.1.2, 00:03:16, GigabitEthernet0/0/0/0
O    10.1.9.0/24 [110/101] via 172.16.1.2, 00:03:30, GigabitEthernet0/0/0/0
O    172.16.2.0/24 [110/102] via 172.16.1.2, 00:03:16, GigabitEthernet0/0/0/0

Adding the sham link

Both PEs are configured with their own endpoint and the remote one.

router ospf 100
 vrf CUST-A
  area 1
   sham-link 10.0.101.1 10.0.102.1
    cost 1
   !
  !
 !
!
end

The sham link comes up. State POINT_TO_POINT and Adjacency State FULL: the adjacency forms just as it would on an ordinary OSPF link.

RP/0/RP0/CPU0:PE1#show ospf 100 vrf CUST-A sham-links
Mon Oct 12 10:01:26.797 UTC

Sham Links for OSPF 100, VRF CUST-A

Sham Link OSPF_SL0 to address 10.0.102.1 is up
Area 1, source address 10.0.101.1
IfIndex = 2
  Run as demand circuit
  DoNotAge LSA allowed., Cost of using 1
  Transmit Delay is 1 sec, State POINT_TO_POINT,
  Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
    Hello due in 00:00:03:693
    Adjacency State FULL (Hello suppressed)
    Number of DBD retrans during last exchange 0
    Index 2/2, retransmission queue length 0, number of retransmission 0
    First 0(0)/0(0) Next 0(0)/0(0)
    Last retransmission scan length is 0, maximum is 0
    Last retransmission scan time is 0 msec, maximum is 0 msec

The output matches what RFC 4577 prescribes. It shows Run as demand circuit and DoNotAge LSA allowed, and once the adjacency is up the hellos are suppressed (Hello suppressed), so seeing FULL with no hellos going by is normal here.

The route seen by CE-A1 switches to the VPN, with PE1 (172.16.1.1) as the next hop.

RP/0/RP0/CPU0:CE-A1#show route 10.1.2.1/32
Mon Oct 12 10:03:11.048 UTC

Routing entry for 10.1.2.1/32
  Known via "ospf 1", distance 110, metric 4, type intra area
  Installed Oct 12 09:59:08.780 for 00:04:02
  Routing Descriptor Blocks
    172.16.1.1, from 12.12.12.12, via GigabitEthernet0/0/0/0
      Route metric is 4
  No advertising protos. 

The traceroute now goes through the provider network.

RP/0/RP0/CPU0:CE-A1#traceroute 10.1.2.1 source 10.1.1.1 timeout 1 probe 2 maxttl 6
Mon Oct 12 10:00:44.280 UTC

Type escape sequence to abort.
Tracing the route to 10.1.2.1

 1  172.16.1.1 7 msec  4 msec 
 2  10.0.13.3 [MPLS: Labels 24001/24005 Exp 0] 14 msec  11 msec 
 3  10.0.23.2 [MPLS: Label 24005 Exp 0] 12 msec  23 msec 
 4  172.16.2.2 20 msec  * 

OSPF on the sham link travels with labels

In the capture of the core (the PE1-P1 link), OSPF packets between the two endpoint addresses cross with a two-label stack. This is No.3 of the attached STEP 1 capture.

MultiProtocol Label Switching Header, Label: 24001, Exp: 6, S: 0, TTL: 255
    0000 0101 1101 1100 0001 .... .... .... = MPLS Label: 24001 (0x05dc1)
    .... .... .... .... .... 110. .... .... = MPLS Experimental Bits: 6
    .... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0
    .... .... .... .... .... .... 1111 1111 = MPLS TTL: 255
MultiProtocol Label Switching Header, Label: 24003, Exp: 6, S: 1, TTL: 255
    0000 0101 1101 1100 0011 .... .... .... = MPLS Label: 24003 (0x05dc3)
    .... .... .... .... .... 110. .... .... = MPLS Experimental Bits: 6
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1111 = MPLS TTL: 255
Internet Protocol Version 4, Src: 10.0.101.1, Dst: 10.0.102.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0xc0 (DSCP: CS6, ECN: Not-ECT)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (48)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 76
    Identification: 0x0033 (51)
    000. .... = Flags: 0x0
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 255
    Protocol: OSPF IGP (89)
    Header Checksum: 0xdb63 [validation disabled]
    [Header checksum status: Unverified]
    Source Address: 10.0.101.1
    Destination Address: 10.0.102.1
    [Stream index: 2]
Open Shortest Path First
    OSPF Header
        Version: 2
        Message Type: Hello Packet (1)

The options of that same hello carry (DC) Demand Circuits. That is the negotiation of RFC 1793 section 3.2.1: once both ends agree on this bit, the hellos are suppressed. In the attached capture the database description packets of the adjacency carry it too.

    OSPF Hello Packet
        Network Mask: 0.0.0.0
        Hello Interval [sec]: 10
        Options: 0x32, (DC) Demand Circuits, (L) LLS Data block, (E) External Routing

From here on, cost decides

The sham link cost is raised to 200, above the backdoor.

router ospf 100
 vrf CUST-A
  area 1
   sham-link 10.0.101.1 10.0.102.1
    cost 200
   !
  !
 !
!
end

The backdoor takes over again. Both paths are intra-area now, so the lower cost wins.

RP/0/RP0/CPU0:CE-A1#show route 10.1.2.1/32
Mon Oct 12 10:08:17.309 UTC

Routing entry for 10.1.2.1/32
  Known via "ospf 1", distance 110, metric 101, type intra area
  Installed Oct 12 10:04:05.788 for 00:04:11
  Routing Descriptor Blocks
    10.1.9.2, from 12.12.12.12, via GigabitEthernet0/0/0/1
      Route metric is 101
  No advertising protos. 

Putting the cost back to 1 returns the traffic to the VPN.

RP/0/RP0/CPU0:CE-A1#show route 10.1.2.1/32
Mon Oct 12 10:13:25.079 UTC

Routing entry for 10.1.2.1/32
  Known via "ospf 1", distance 110, metric 4, type intra area
  Installed Oct 12 10:09:09.989 for 00:04:15
  Routing Descriptor Blocks
    172.16.1.1, from 12.12.12.12, via GigabitEthernet0/0/0/0
      Route metric is 4
  No advertising protos. 

Verification configs and show output

For every STEP the following files were collected from all five routers, separated per router. The verification config is ..._run.txt (the final state is the one from the last STEP).

STEP 0: Initial state (no sham link)

STEP 1: A sham link on PE1 and PE2 (cost 1)

STEP 2: Raise the sham link cost to 200 (above the backdoor cost of 100)

STEP 3: Put the sham link cost back to 1

STEP 4: Remove the sham link (final state)

Packet captures were taken per STEP.

Related articles

• What Is MPLS VPN (L3VPN)
• MPLS VPN VRF (Virtual Routing and Forwarding)
• MPLS VPN RD (Route Distinguisher)
• MPLS VPN Route Target (RT)
• MPLS VPN MP-BGP (Propagating VPNv4 Routes)
• MPLS VPN Label Allocation (per-prefix / per-CE / per-VRF)
• MPLS VPN Forwarding with Two Labels (Transport Label and VPN Label)
• MPLS VPN PE-CE Routing with Static Routes
• MPLS VPN PE-CE Routing with eBGP
• When Several MPLS VPN Sites Share One AS Number (as-override)
• Site of Origin (SoO) in MPLS VPN
• PE-CE routing with OSPF in MPLS VPN
• The OSPF DN bit and the domain tag in MPLS VPN
• Sham links in MPLS VPN (dealing with a backdoor link)
• Controlling routes with RTs in MPLS VPN (extranet, import and export maps)
• Hub and spoke in MPLS VPN