↓ Skip to main content
KAZULOG.INFO
  1. Network Articles/
  2. MPLS Articles/

Steering Traffic into MPLS TE Tunnels

Table of Contents

Steering Traffic into MPLS TE Tunnels

An MPLS TE tunnel that is up still carries no traffic. Packets enter it only when the routing table picks the tunnel as the next hop. This article explains three ways to put traffic into a tunnel (autoroute announce, forwarding adjacency and a static route), then uses an IOS XR (XRd) lab to see whose routing table changes. Which path the tunnel itself takes is covered in MPLS TE CSPF and Path Constraints.

Three steering methods

Method How it works Routers whose routing table changes
autoroute announce The head-end uses the tunnel as a next hop in its own SPF (IGP shortcut) Head-end only
forwarding adjacency The tunnel is advertised as a link in the IGP Every router in the area
Static route Points only the specified destination at the tunnel Head-end only (specified destination only)

autoroute announce

RFC 3906 describes using a tunnel as the next hop when the head-end reaches the tunnel’s tail-end during SPF. As a result, traffic to the tail-end, and also traffic to nodes downstream of the tail-end, flows over the tunnel. Nodes before the tail-end are not affected.

The tunnel is not advertised, so other routers know nothing about it. The metric carries over the IGP cost to the tail-end, so the metric in the routing table stays the same and only the next hop becomes the tunnel.

forwarding adjacency

The tunnel is advertised as an IGP link and enters the SPF of every router in the area. This concept is defined as a forwarding adjacency (FA) in RFC 4206.

  • In OSPF it appears in the Router-LSA as a point-to-point link. No routing adjacency (Hello) runs over the tunnel
  • An IGP link is used by SPF only when it is bidirectional, so the reverse tunnel needs the same configuration
  • IOS XR advertises the FA link with metric 1 (changeable with metric under forwarding-adjacency). That is smaller than the physical links, so traffic from other routers is pulled toward the head-end too

Cisco’s IOS XR configuration guide lists IS-IS as the IGP that supports FA. In this article’s lab (XRd 26.1.1), the FA also appeared in the OSPF Router-LSA and was used in route computation.

Static route

Points a specified destination at the tunnel. With administrative distance 1 it wins over the IGP route, and only that destination changes.

VPN traffic follows the route to the BGP next hop

A VPN route (a VRF prefix) is resolved recursively through its BGP next hop (the remote PE’s loopback). Once the route to PE2’s loopback (9.9.9.9/32) points at the tunnel, every VPN route with that next hop goes through the tunnel. There is no need to point VRF prefixes at the tunnel one by one.

Lab topology

There are three paths with the same hop count between PE1 and PE2. The IGP shortest path is the middle one. The tunnel (PE1 → PE2) is pinned to the lower path with an affinity. Whether traffic entered the tunnel shows up as the P router names in the CE1-to-CE2 traceroute.

Router Lo0 Role
CE1 / CE2 1.1.1.1 / 10.10.10.10 AS 65101 / AS 65102 (VRF CUST-A)
PE1 2.2.2.2 head-end
P1 / P2 3.3.3.3 / 4.4.4.4 Upper path (OSPF cost 30 per link)
P3 / P4 5.5.5.5 / 6.6.6.6 Middle path (10 per link, IGP shortest)
P5 / P6 7.7.7.7 / 8.8.8.8 Lower path (20 per link, used by the tunnel)
PE2 9.9.9.9 tail-end, BGP next hop of CE2’s route

Overview of the test

STEP Change What it shows
0 Create the tunnel (lower path, no steering) The tunnel is up, but CE1 to CE2 stays on the middle path
1 autoroute announce Only PE1’s routing table changes. Destinations beyond the tail-end also use the tunnel
2 Remove autoroute and add forwarding-adjacency to tunnels in both directions A PE1 - PE2 link appears in the Router-LSA, and the routes on P1, P3 and P5 change too
3 Remove FA and point a static route for 9.9.9.9/32 at the tunnel VPN traffic follows the BGP next hop alone. Other routers return to normal
4 Remove everything (final state) Back to the STEP 0 state

STEP 0: an up tunnel is not used

Configuration committed on PE1 in STEP 0
RP/0/RP0/CPU0:PE1#show configuration commit changes last 1
Sun Sep 13 08:29:50.770 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
interface tunnel-te0
 ipv4 unnumbered Loopback0
 destination 9.9.9.9
 record-route
 affinity 0x2 mask 0x2
 path-option 10 dynamic
!
end
STEP 0 PE1 show mpls traffic-eng tunnels (path)
  Path info (OSPF 1 area 0):
  Hop0: 10.2.7.7
  Hop1: 10.7.8.8
  Hop2: 10.8.9.9
  Hop3: 9.9.9.9

The tunnel is up on the lower path (P5 → P6). The next hop of CE2’s route, 9.9.9.9, is still resolved over the IGP middle path.

STEP 0 PE1 show cef vrf CUST-A 192.168.2.0/24 detail (excerpt)
   via 9.9.9.9/32, 3 dependencies, recursive [flags 0x6000]
    path-idx 0 NHID 0x0 [0x8856e168 0x0]
    recursion-via-/32
    next hop VRF - 'default', table - 0xe0000000
    next hop 9.9.9.9/32 via 24013/0/21
     next hop 10.2.5.5/32 GigabitEthernet0/0/0/2 labels imposed {24013 24013}
STEP 0 CE1 traceroute 192.168.2.1
RP/0/RP0/CPU0:CE1#traceroute 192.168.2.1 source 192.168.1.1
Sun Sep 13 08:36:58.683 UTC

Type escape sequence to abort.
Tracing the route to 192.168.2.1

 1  10.1.2.2 6 msec  4 msec  31 msec 
 2  10.2.5.5 [MPLS: Labels 24013/24013 Exp 0] 18 msec  17 msec  22 msec 
 3  10.5.6.6 [MPLS: Labels 24013/24013 Exp 0] 18 msec  16 msec  19 msec 
 4  10.6.9.9 [MPLS: Label 24013 Exp 0] 18 msec  16 msec  32 msec 
 5  10.9.10.10 23 msec  *  16 msec

CE1 to CE2 goes through P3 and P4. Both labels are 24013; the outer one is LDP and the inner one is the VPN label.

STEP 1: autoroute announce

Configuration committed on PE1 in STEP 1
RP/0/RP0/CPU0:PE1#show configuration commit changes last 1
Sun Sep 13 08:39:22.638 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
interface tunnel-te0
 autoroute announce
 !
!
end
STEP 1 PE1 show route (loopback routes, excerpt)
O    3.3.3.3/32 [110/31] via 10.2.3.3, 00:25:51, GigabitEthernet0/0/0/1
O    4.4.4.4/32 [110/61] via 9.9.9.9, 00:05:21, tunnel-te0
                [110/61] via 10.2.3.3, 00:05:21, GigabitEthernet0/0/0/1
O    5.5.5.5/32 [110/11] via 10.2.5.5, 00:25:58, GigabitEthernet0/0/0/2
O    6.6.6.6/32 [110/21] via 10.2.5.5, 00:25:58, GigabitEthernet0/0/0/2
O    7.7.7.7/32 [110/21] via 10.2.7.7, 00:25:59, GigabitEthernet0/0/0/3
O    8.8.8.8/32 [110/41] via 10.2.7.7, 00:25:59, GigabitEthernet0/0/0/3
O    9.9.9.9/32 [110/31] via 9.9.9.9, 00:05:21, tunnel-te0

The next hop of 9.9.9.9/32 became tunnel-te0. The metric stays 31; only the next hop changed. P2’s loopback beyond PE2 (4.4.4.4/32) also gained tunnel-te0 as a path, forming two-way ECMP with the upper path. Destinations downstream of the tail-end use the tunnel too. P5 (7.7.7.7) and P6 (8.8.8.8), which lie along the tunnel, and P4 (6.6.6.6) on the middle path stay on physical interfaces.

STEP 1 PE1 show cef vrf CUST-A 192.168.2.0/24 detail (excerpt)
   via 9.9.9.9/32, 3 dependencies, recursive [flags 0x6000]
    path-idx 0 NHID 0x0 [0x8856e168 0x0]
    recursion-via-/32
    next hop VRF - 'default', table - 0xe0000000
    next hop 9.9.9.9/32 via 24013/0/21
     next hop 9.9.9.9/32 tunnel-te0   labels imposed {ImplNull 24013}

The VPN route is now resolved through the tunnel via the next hop 9.9.9.9/32. The outer label is shown as ImplNull; the tunnel label is pushed on the output interface (tunnel-te0) side.

The routing tables of other routers did not change.

STEP 1 P1 show route 9.9.9.9/32
RP/0/RP0/CPU0:P1#show route 9.9.9.9/32
Sun Sep 13 08:40:10.494 UTC

Routing entry for 9.9.9.9/32
  Known via "ospf 1", distance 110, metric 61, type intra area
  Installed Sep 13 08:21:51.141 for 00:18:19
  Routing Descriptor Blocks
    10.2.3.2, from 9.9.9.9, via GigabitEthernet0/0/0/0
      Route metric is 61
    10.3.4.4, from 9.9.9.9, via GigabitEthernet0/0/0/1
      Route metric is 61
  No advertising protos.
STEP 1 P3 show route 9.9.9.9/32
RP/0/RP0/CPU0:P3#show route 9.9.9.9/32
Sun Sep 13 08:41:42.196 UTC

Routing entry for 9.9.9.9/32
  Known via "ospf 1", distance 110, metric 21, type intra area
  Installed Sep 13 08:21:51.143 for 00:19:51
  Routing Descriptor Blocks
    10.5.6.6, from 9.9.9.9, via GigabitEthernet0/0/0/1
      Route metric is 21
  No advertising protos.
STEP 1 P5 show route 9.9.9.9/32
RP/0/RP0/CPU0:P5#show route 9.9.9.9/32
Sun Sep 13 08:43:11.597 UTC

Routing entry for 9.9.9.9/32
  Known via "ospf 1", distance 110, metric 41, type intra area
  Installed Sep 13 08:21:49.530 for 00:21:22
  Routing Descriptor Blocks
    10.7.8.8, from 9.9.9.9, via GigabitEthernet0/0/0/1
      Route metric is 41
  No advertising protos.
STEP 1 CE1 traceroute 192.168.2.1
RP/0/RP0/CPU0:CE1#traceroute 192.168.2.1 source 192.168.1.1
Sun Sep 13 08:46:46.179 UTC

Type escape sequence to abort.
Tracing the route to 192.168.2.1

 1  10.1.2.2 6 msec  16 msec  5 msec 
 2  10.2.7.7 [MPLS: Labels 24014/24013 Exp 0] 18 msec  15 msec  15 msec 
 3  10.7.8.8 [MPLS: Labels 24014/24013 Exp 0] 25 msec  16 msec  43 msec 
 4  10.8.9.9 [MPLS: Label 24013 Exp 0] 16 msec  16 msec  18 msec 
 5  10.9.10.10 17 msec  *  17 msec

CE1 to CE2 moved to the lower path (P5 and P6). The outer label 24014 is the tunnel (RSVP) and the inner 24013 is the VPN label.

STEP 2: forwarding adjacency

FA is configured on both ends. PE2 gets a reverse tunnel (PE2 → PE1), also pinned to the lower path.

Configuration committed on PE1 in STEP 2
RP/0/RP0/CPU0:PE1#show configuration commit changes last 1
Sun Sep 13 08:49:01.869 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
interface tunnel-te0
 no autoroute announce
 forwarding-adjacency
 !
!
end
Configuration committed on PE2 in STEP 2
RP/0/RP0/CPU0:PE2#show configuration commit changes last 1
Sun Sep 13 08:48:56.033 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
interface tunnel-te0
 ipv4 unnumbered Loopback0
 destination 2.2.2.2
 record-route
 affinity 0x2 mask 0x2
 forwarding-adjacency
 !
 path-option 10 dynamic
!
end

A point-to-point link to PE2 (9.9.9.9) with metric 1 was added at the top of PE1’s Router-LSA.

STEP 2 PE1 show ospf database router self-originate
RP/0/RP0/CPU0:PE1#show ospf database router self-originate
Sun Sep 13 08:55:46.906 UTC


            OSPF Router with ID (2.2.2.2) (Process ID 1)

		Router Link States (Area 0)

  LS age: 239
  Options: (No TOS-capability, DC)
  LS Type: Router Links
  Link State ID: 2.2.2.2
  Advertising Router: 2.2.2.2
  LS Seq Number: 80000007
  Checksum: 0xb2a9
  Length: 120
   Number of Links: 8

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 9.9.9.9
     (Link Data) Router Interface address: 0.0.0.9
      Number of TOS metrics: 0
       TOS 0 Metrics: 1

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 2.2.2.2
     (Link Data) Network Mask: 255.255.255.255
      Number of TOS metrics: 0
       TOS 0 Metrics: 1

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 3.3.3.3
     (Link Data) Router Interface address: 10.2.3.2
      Number of TOS metrics: 0
       TOS 0 Metrics: 30

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.2.3.0
     (Link Data) Network Mask: 255.255.255.0
      Number of TOS metrics: 0
       TOS 0 Metrics: 30

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 5.5.5.5
     (Link Data) Router Interface address: 10.2.5.2
      Number of TOS metrics: 0
       TOS 0 Metrics: 10

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.2.5.0
     (Link Data) Network Mask: 255.255.255.0
      Number of TOS metrics: 0
       TOS 0 Metrics: 10

    Link connected to: another Router (point-to-point)
     (Link ID) Neighboring Router ID: 7.7.7.7
     (Link Data) Router Interface address: 10.2.7.2
      Number of TOS metrics: 0
       TOS 0 Metrics: 20

    Link connected to: a Stub Network
     (Link ID) Network/subnet number: 10.2.7.0
     (Link Data) Network Mask: 255.255.255.0
      Number of TOS metrics: 0
       TOS 0 Metrics: 20

In the attached capture between PE1 and P3, No.1777 is the LS Update that carried this Router-LSA.

STEP 2 No.1777 LS Update (PE1 to P3, tshark -V excerpt)
Internet Protocol Version 4, Src: 10.2.5.2, Dst: 224.0.0.5
        Message Type: LS Update (4)
        Source OSPF Router: 2.2.2.2
            LS Type: Router-LSA (1)
            Link State ID: 2.2.2.2
            Advertising Router: 2.2.2.2
            Sequence Number: 0x80000006
            Number of Links: 8
            Type: PTP      ID: 9.9.9.9         Data: 0.0.0.9         Metric: 1
            Type: Stub     ID: 2.2.2.2         Data: 255.255.255.255 Metric: 1
            Type: PTP      ID: 3.3.3.3         Data: 10.2.3.2        Metric: 30
            Type: Stub     ID: 10.2.3.0        Data: 255.255.255.0   Metric: 30
            Type: PTP      ID: 5.5.5.5         Data: 10.2.5.2        Metric: 10
            Type: Stub     ID: 10.2.5.0        Data: 255.255.255.0   Metric: 10
            Type: PTP      ID: 7.7.7.7         Data: 10.2.7.2        Metric: 20
            Type: Stub     ID: 10.2.7.0        Data: 255.255.255.0   Metric: 20
Download the pcap of the packet in the tshark output above (No.1777 LS Update)

In PE1’s routing table, even the loopbacks of P2, P4 and P6 now go through the tunnel, because the cost to PE2 is only metric 1.

STEP 2 PE1 show route (loopback routes, excerpt)
O    3.3.3.3/32 [110/31] via 10.2.3.3, 00:36:47, GigabitEthernet0/0/0/1
O    4.4.4.4/32 [110/32] via 0.0.0.0, 00:06:37, tunnel-te0
O    5.5.5.5/32 [110/11] via 10.2.5.5, 00:36:54, GigabitEthernet0/0/0/2
O    6.6.6.6/32 [110/12] via 0.0.0.0, 00:06:37, tunnel-te0
O    7.7.7.7/32 [110/21] via 10.2.7.7, 00:36:55, GigabitEthernet0/0/0/3
O    8.8.8.8/32 [110/22] via 0.0.0.0, 00:06:37, tunnel-te0
O    9.9.9.9/32 [110/2] via 0.0.0.0, 00:06:37, tunnel-te0

Other routers’ routing tables changed as well. On all three, the next hop of 9.9.9.9/32 points toward PE1.

STEP 2 P1 show route 9.9.9.9/32
RP/0/RP0/CPU0:P1#show route 9.9.9.9/32
Sun Sep 13 08:51:02.825 UTC

Routing entry for 9.9.9.9/32
  Known via "ospf 1", distance 110, metric 32, type intra area
  Installed Sep 13 08:49:01.460 for 00:02:01
  Routing Descriptor Blocks
    10.2.3.2, from 9.9.9.9, via GigabitEthernet0/0/0/0
      Route metric is 32
  No advertising protos.
STEP 2 P3 show route 9.9.9.9/32
RP/0/RP0/CPU0:P3#show route 9.9.9.9/32
Sun Sep 13 08:52:36.673 UTC

Routing entry for 9.9.9.9/32
  Known via "ospf 1", distance 110, metric 12, type intra area
  Installed Sep 13 08:49:01.460 for 00:03:35
  Routing Descriptor Blocks
    10.2.5.2, from 9.9.9.9, via GigabitEthernet0/0/0/0
      Route metric is 12
  No advertising protos.
STEP 2 P5 show route 9.9.9.9/32
RP/0/RP0/CPU0:P5#show route 9.9.9.9/32
Sun Sep 13 08:54:07.791 UTC

Routing entry for 9.9.9.9/32
  Known via "ospf 1", distance 110, metric 22, type intra area
  Installed Sep 13 08:49:01.460 for 00:05:06
  Routing Descriptor Blocks
    10.2.7.2, from 9.9.9.9, via GigabitEthernet0/0/0/0
      Route metric is 22
  No advertising protos.

P3 used P4 (metric 21) through STEP 1, but going back to PE1 and into the tunnel (10 + 1 + 1 = 12) is now shorter. P5’s traceroute goes up to PE1 and then comes back through P5 itself (10.2.7.7) in the tunnel.

STEP 2 P5 traceroute 9.9.9.9
RP/0/RP0/CPU0:P5#traceroute 9.9.9.9 source 7.7.7.7
Sun Sep 13 08:54:26.626 UTC

Type escape sequence to abort.
Tracing the route to 9.9.9.9

 1  10.2.7.2 [MPLS: Label 24013 Exp 0] 15 msec  13 msec  14 msec 
 2  10.2.7.7 [MPLS: Label 24014 Exp 0] 27 msec  12 msec  14 msec 
 3  10.7.8.8 [MPLS: Label 24014 Exp 0] 14 msec  13 msec  13 msec 
 4  10.8.9.9 14 msec  *  16 msec

STEP 3: static route

Configuration committed on PE1 in STEP 3
RP/0/RP0/CPU0:PE1#show configuration commit changes last 1
Sun Sep 13 08:59:58.517 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
interface tunnel-te0
 no forwarding-adjacency
!
router static
 address-family ipv4 unicast
  9.9.9.9/32 tunnel-te0
 !
!
end
Configuration committed on PE2 in STEP 3
RP/0/RP0/CPU0:PE2#show configuration commit changes last 1
Sun Sep 13 08:59:52.143 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
no interface tunnel-te0
end

router static is added per prefix.

STEP 3 PE1 show route 9.9.9.9/32
RP/0/RP0/CPU0:PE1#show route 9.9.9.9/32
Sun Sep 13 09:05:31.342 UTC

Routing entry for 9.9.9.9/32
  Known via "static", distance 1, metric 0 (connected)
  Installed Sep 13 08:59:57.812 for 00:05:33
  Routing Descriptor Blocks
    directly connected, via tunnel-te0
      Route metric is 0
  No advertising protos.
STEP 3 PE1 show route (loopback routes, excerpt)
O    3.3.3.3/32 [110/31] via 10.2.3.3, 00:46:39, GigabitEthernet0/0/0/1
O    4.4.4.4/32 [110/61] via 10.2.3.3, 00:05:39, GigabitEthernet0/0/0/1
                [110/61] via 10.2.5.5, 00:05:39, GigabitEthernet0/0/0/2
O    5.5.5.5/32 [110/11] via 10.2.5.5, 00:46:46, GigabitEthernet0/0/0/2
O    6.6.6.6/32 [110/21] via 10.2.5.5, 00:05:39, GigabitEthernet0/0/0/2
O    7.7.7.7/32 [110/21] via 10.2.7.7, 00:46:47, GigabitEthernet0/0/0/3
O    8.8.8.8/32 [110/41] via 10.2.7.7, 00:05:39, GigabitEthernet0/0/0/3
S    9.9.9.9/32 is directly connected, 00:05:33, tunnel-te0

Only 9.9.9.9/32 points at tunnel-te0 as a static route; the other loopbacks are back on IGP routes. The VPN route is still resolved through the tunnel.

STEP 3 PE1 show cef vrf CUST-A 192.168.2.0/24 detail (excerpt)
   via 9.9.9.9/32, 3 dependencies, recursive [flags 0x6000]
    path-idx 0 NHID 0x0 [0x8856e168 0x0]
    recursion-via-/32
    next hop VRF - 'default', table - 0xe0000000
    next hop 9.9.9.9/32 via 24013/0/21
     next hop 0.0.0.0/32 tunnel-te0   labels imposed {ImplNull 24013}

With FA removed, the link to PE2 disappeared from PE1’s Router-LSA (No.2411).

STEP 3 No.2411 LS Update (PE1 to P3, tshark -V excerpt)
Internet Protocol Version 4, Src: 10.2.5.2, Dst: 224.0.0.5
        Message Type: LS Update (4)
        Source OSPF Router: 2.2.2.2
            LS Type: Router-LSA (1)
            Link State ID: 2.2.2.2
            Advertising Router: 2.2.2.2
            Sequence Number: 0x80000008
            Number of Links: 7
            Type: Stub     ID: 2.2.2.2         Data: 255.255.255.255 Metric: 1
            Type: PTP      ID: 3.3.3.3         Data: 10.2.3.2        Metric: 30
            Type: Stub     ID: 10.2.3.0        Data: 255.255.255.0   Metric: 30
            Type: PTP      ID: 5.5.5.5         Data: 10.2.5.2        Metric: 10
            Type: Stub     ID: 10.2.5.0        Data: 255.255.255.0   Metric: 10
            Type: PTP      ID: 7.7.7.7         Data: 10.2.7.2        Metric: 20
            Type: Stub     ID: 10.2.7.0        Data: 255.255.255.0   Metric: 20
Download the pcap of the packet in the tshark output above (No.2411 LS Update)
STEP 3 P3 show route 9.9.9.9/32
RP/0/RP0/CPU0:P3#show route 9.9.9.9/32
Sun Sep 13 09:02:27.028 UTC

Routing entry for 9.9.9.9/32
  Known via "ospf 1", distance 110, metric 21, type intra area
  Installed Sep 13 08:59:51.374 for 00:02:35
  Routing Descriptor Blocks
    10.5.6.6, from 9.9.9.9, via GigabitEthernet0/0/0/1
      Route metric is 21
  No advertising protos.
STEP 3 CE1 traceroute 192.168.2.1
RP/0/RP0/CPU0:CE1#traceroute 192.168.2.1 source 192.168.1.1
Sun Sep 13 09:07:32.982 UTC

Type escape sequence to abort.
Tracing the route to 192.168.2.1

 1  10.1.2.2 23 msec  6 msec  5 msec 
 2  10.2.7.7 [MPLS: Labels 24014/24013 Exp 0] 63 msec  17 msec  17 msec 
 3  10.7.8.8 [MPLS: Labels 24014/24013 Exp 0] 19 msec  18 msec  18 msec 
 4  10.8.9.9 [MPLS: Label 24013 Exp 0] 18 msec  18 msec  17 msec 
 5  10.9.10.10 19 msec  *  16 msec

P3 is back on P4, and CE1 to CE2 stays on the lower path.

STEP 4: remove everything

Configuration committed on PE1 in STEP 4
RP/0/RP0/CPU0:PE1#show configuration commit changes last 1
Sun Sep 13 09:09:46.338 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
no interface tunnel-te0
router static
 address-family ipv4 unicast
  no 9.9.9.9/32 tunnel-te0
 !
!
end
STEP 4 CE1 traceroute 192.168.2.1
RP/0/RP0/CPU0:CE1#traceroute 192.168.2.1 source 192.168.1.1
Sun Sep 13 09:17:09.673 UTC

Type escape sequence to abort.
Tracing the route to 192.168.2.1

 1  10.1.2.2 18 msec  5 msec  5 msec 
 2  10.2.5.5 [MPLS: Labels 24013/24013 Exp 0] 16 msec  17 msec  17 msec 
 3  10.5.6.6 [MPLS: Labels 24013/24013 Exp 0] 67 msec  18 msec  17 msec 
 4  10.6.9.9 [MPLS: Label 24013 Exp 0] 19 msec  32 msec  17 msec 
 5  10.9.10.10 18 msec  *  20 msec

CE1 to CE2 is back on the middle path.

Comparing the three methods

STEP Method 9.9.9.9/32 on PE1 9.9.9.9/32 on P3 CE1 → CE2
0 None Middle path (Gi0/0/0/2) Via P4 Middle
1 autoroute announce tunnel-te0 Via P4 Lower
2 forwarding adjacency tunnel-te0 (metric 2) Via PE1 Lower
3 Static route tunnel-te0 (static) Via P4 Lower
4 None Middle path Via P4 Middle

Design notes

  • autoroute and static routes are one-way. Carrying return traffic needs a tunnel and configuration on PE2 as well
  • FA pulls traffic in. Adding a metric-1 link makes some routers take a detour, like P5. Set the FA metric to match the cost of the real path
  • FA is used by SPF only once tunnels exist in both directions
  • To carry VPN traffic, the destination to point at the tunnel is the remote PE’s loopback (the BGP next hop)

Lab configuration and show outputs

In each STEP the following files were collected from all ten routers, one file per router. The lab configuration is the ..._run.txt (the final state is in the last STEP).

File Contents
..._show.txt show version / show route / show ospf database router / show mpls traffic-eng tunnels / show cef vrf CUST-A (PEs only) and more
..._log.txt show logging limited to that STEP
..._run.txt show running-config at that STEP (the lab configuration of that STEP)
..._ping.txt ping (50 packets, 1-second timeout) and traceroute in that STEP
..._oam.txt ping / traceroute mpls traffic-eng tunnel-te 0 and ping / traceroute mpls ipv4 (PE1 and PE2)
..._trace.txt show mpls traffic-eng trace head-end / link and show rsvp trace signalling (eight core routers)
..._commit.cfg Only the configuration actually committed in that STEP, for the routers that changed

STEP 0: Create the tunnel (lower path, no steering)

Router show output syslog running-config ping OAM trace commit
CE1 show log run ping - - -
PE1 show log run ping oam trace cfg
P1 show log run ping - trace -
P2 show log run - - trace -
P3 show log run ping - trace -
P4 show log run - - trace -
P5 show log run ping - trace -
P6 show log run - - trace -
PE2 show log run ping oam trace -
CE2 show log run ping - - -

STEP 1: autoroute announce

Router show output syslog running-config ping OAM trace commit
CE1 show log run ping - - -
PE1 show log run ping oam trace cfg
P1 show log run ping - trace -
P2 show log run - - trace -
P3 show log run ping - trace -
P4 show log run - - trace -
P5 show log run ping - trace -
P6 show log run - - trace -
PE2 show log run ping oam trace -
CE2 show log run ping - - -

STEP 2: Remove autoroute and add forwarding-adjacency to tunnels in both directions

Router show output syslog running-config ping OAM trace commit
CE1 show log run ping - - -
PE1 show log run ping oam trace cfg
P1 show log run ping - trace -
P2 show log run - - trace -
P3 show log run ping - trace -
P4 show log run - - trace -
P5 show log run ping - trace -
P6 show log run - - trace -
PE2 show log run ping oam trace cfg
CE2 show log run ping - - -

STEP 3: Remove FA and point a static route for 9.9.9.9/32 at the tunnel

Router show output syslog running-config ping OAM trace commit
CE1 show log run ping - - -
PE1 show log run ping oam trace cfg
P1 show log run ping - trace -
P2 show log run - - trace -
P3 show log run ping - trace -
P4 show log run - - trace -
P5 show log run ping - trace -
P6 show log run - - trace -
PE2 show log run ping oam trace cfg
CE2 show log run ping - - -

STEP 4: Remove everything (final state)

Router show output syslog running-config ping OAM trace commit
CE1 show log run ping - - -
PE1 show log run ping oam trace cfg
P1 show log run ping - trace -
P2 show log run - - trace -
P3 show log run ping - trace -
P4 show log run - - trace -
P5 show log run ping - trace -
P6 show log run - - trace -
PE2 show log run ping oam trace -
CE2 show log run ping - - -

Full captures of the three links from PE1 (toward P1, P3 and P5). They contain the OSPF and RSVP packets of all STEPs.

Download the full capture between PE1 and P1 (upper path)

Download the full capture between PE1 and P3 (middle path)

Download the full capture between PE1 and P5 (lower path)

References

Source Sections used
RFC 3906 Calculating Interior Gateway Protocol (IGP) Routes Over Traffic Engineering Tunnels 2 (using the tunnel tail-end as the next hop in SPF; destinations downstream of the tail-end also use the tunnel), 4.1 (absolute and relative metrics)
RFC 4206 Label Switched Paths (LSP) Hierarchy with Generalized Multi-Protocol Label Switching (GMPLS) Traffic Engineering (TE) 1 (definition of FA; nodes connected by an FA would not usually have a routing adjacency), 3.1.1 (link type is point-to-point)
Cisco IOS XR MPLS Configuration Guide, “Implementing MPLS Traffic Engineering” Restrictions of MPLS-TE Forwarding Adjacency (bidirectional configuration, IS-IS as the supported IGP)

The FA metric of 1 and the behavior with OSPF are observations on the lab router (XRd 26.1.1).