MPLS OAM (LSP Ping and LSP Traceroute)

MPLS OAM (LSP Ping and LSP Traceroute)

An ordinary ping only tests the IP path. It pushes no label, so it succeeds even when the LSP is broken. The tool that tests the MPLS data plane is LSP Ping (RFC 8029). This article explains how it works, then observes the actual packets in an IOS XR (XRd) lab. Label distribution is covered in What Is LDP.

Why IP ping is not enough

When MPLS carries a packet, the ingress PE pushes a transport label. An IP ping, by contrast, is sent without a label, so it tests the IGP path and nothing else. If label forwarding is broken, the IP ping still succeeds and the routing table shows nothing wrong.

As the title of RFC 8029 — “Detecting Multiprotocol Label Switched (MPLS) Data-Plane Failures” — says, the mechanism targets data-plane failures: the case where the control plane (LDP) claims the LSP exists while forwarding is broken.

How LSP Ping works

LSP Ping sends an MPLS echo request with the label of the LSP under test. The receiving router answers with an MPLS echo reply.

The Target FEC Stack is the key part. Forwarding on the label alone cannot reveal that a transit LSR has confused it with the label of another FEC. Carrying “the FEC this label is supposed to serve” alongside the packet lets the receiver detect the mismatch.

LSP Traceroute

LSP Traceroute increments the TTL in the label stack one at a time. The LSR where the TTL expires answers, so the state of each hop becomes visible. The reply carries the label used at that hop and the MRU (maximum receive unit).

An ordinary IP traceroute shows neither which label each core LSR uses nor where PHP happens.

Lab topology

A non-redundant chain. OSPF and LDP run in the core (PE1 - P1 - P2 - PE2), and CE1 and CE2 attach to the PEs through VRF CUST-A, reachable over VPNv4. LSP Ping starts at PE1 and tests the FEC of PE2’s loopback (5.5.5.5/32).

  CE1 ------ PE1 ------ P1 ------ P2 ------ PE2 ------ CE2
(AS 65101)    |    ↑ LDP dropped here in STEP 2  |   (AS 65102)
           vrf CUST-A                        vrf CUST-A
              └────── iBGP vpnv4 (between Lo0) ──────┘

Overview of the test

STEP 0: not available by default on IOS XR

Without it enabled, ping mpls never runs.

RP/0/RP0/CPU0:PE1#ping mpls ipv4 5.5.5.5/32
Sat Sep 12 08:40:54.133 UTC
% MPLS Embedded Management Subsystem is not running.
To enable, use 'mpls oam' global config command.

The subsystem that handles OAM is not running. The configuration is a single line, applied to all four core routers (the responders need it too).

RP/0/RP0/CPU0:PE1#show configuration commit changes last 1
Sat Sep 12 08:41:28.848 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
mpls oam
!
end

STEP 1: testing a healthy LSP

Once enabled it succeeds. Adding verbose prints the Return Code for each reply. The legend of result characters is printed every time, and it is what to read when something fails.

RP/0/RP0/CPU0:PE1#ping mpls ipv4 5.5.5.5/32 repeat 2 verbose
Sat Sep 12 08:49:30.733 UTC

Sending 2, 100-byte MPLS Echos to 5.5.5.5/32,
      timeout is 2 seconds, send interval is 0 msec:

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface, 
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label, 
  'P' - no rx intf label prot, 'p' - premature termination of LSP, 
  'R' - transit router, 'I' - unknown upstream index,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.

!      size 100, reply addr 10.4.5.5, return code 3 
!      size 100, reply addr 10.4.5.5, return code 3 

Success rate is 100 percent (2/2), round-trip min/avg/max = 15/16/17 ms

return code 3 is RFC 8029’s “the replying router is an egress for that FEC”.

Traceroute shows the label at each hop

Traceroute prints the same legend and then one line per hop. L marks a hop that forwards with a label, ! marks arrival.

RP/0/RP0/CPU0:PE1#traceroute mpls ipv4 5.5.5.5/32
Sat Sep 12 08:49:30.974 UTC

Tracing MPLS Label Switched Path to 5.5.5.5/32, timeout is 2 seconds

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface, 
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label, 
  'P' - no rx intf label prot, 'p' - premature termination of LSP, 
  'R' - transit router, 'I' - unknown upstream index,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.

  0 10.2.3.2 MRU 1500 [Labels: 24003 Exp: 0]
L 1 10.2.3.3 MRU 1500 [Labels: 24003 Exp: 0] 10 ms
L 2 10.3.4.4 MRU 1500 [Labels: implicit-null Exp: 0] 12 ms
! 3 10.4.5.5 13 ms

Hop 2 shows implicit-null, which is P2 popping the label with PHP, readable straight from the output. The MRU is shown for each hop as well.

Inside the packets

No.912 in the attached capture is the echo request PE1 sent.

    Type: MPLS label switched packet (0x8847)
MultiProtocol Label Switching Header, Label: 24003, Exp: 0, S: 1, TTL: 255
Internet Protocol Version 4, Src: 10.2.3.2, Dst: 127.0.0.1
User Datagram Protocol, Src Port: 3503, Dst Port: 3503
    Message Type: MPLS Echo Request (1)
    Reply Mode: Reply via an IPv4/IPv6 UDP packet (2)
    Return Code: No return code (0)
        Type: Target FEC Stack (1)
        FEC Element 1: LDP IPv4 prefix
            Type: LDP IPv4 prefix (1)
            IPv4 Prefix: 5.5.5.5
            Prefix Length: 32

The destination is 127.0.0.1, the UDP source and destination ports are both 3503, and the packet carries label 24003 with TTL 255. The Target FEC Stack holds the FEC under test (5.5.5.5/32). The Return Code is 0 because this is a request.

Here is the echo reply that came back (No.913).

Internet Protocol Version 4, Src: 10.4.5.5, Dst: 10.2.3.2
User Datagram Protocol, Src Port: 3503, Dst Port: 3503
    Message Type: MPLS Echo Reply (2)
    Reply Mode: Reply via an IPv4/IPv6 UDP packet (2)
    Return Code: Replying router is an egress for the FEC at stack depth RSC (3)
    Return Subcode: 1

The reply carries no label; it comes back as plain IP. Reply Mode 2 is doing exactly what it says. The Return Code is 3, matching what the CLI verbose printed.

The traceroute TTLs are visible too. LSP Ping sends with TTL 255, while traceroute increments 1, 2, 3.

STEP 2: what a broken LSP looks like

An ACL on PE1’s interface towards P1 drops LDP only. OSPF and IP reachability are untouched.

RP/0/RP0/CPU0:PE1#show configuration commit changes last 1
Sat Sep 12 08:49:57.370 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
ipv4 access-list BLOCK-LDP
 10 deny udp any any eq ldp
 20 deny tcp any any eq ldp
 30 deny tcp any eq ldp any
 40 permit ipv4 any any
!
interface GigabitEthernet0/0/0/1
 ipv4 access-group BLOCK-LDP ingress
!
end

The LDP session drops and the label is lost, so LSP Ping cannot even send the echo request.

RP/0/RP0/CPU0:PE1#ping mpls ipv4 5.5.5.5/32 repeat 2 verbose
Sat Sep 12 09:00:01.017 UTC

Sending 2, 100-byte MPLS Echos to 5.5.5.5/32,
      timeout is 2 seconds, send interval is 0 msec:

Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
  'L' - labeled output interface, 'B' - unlabeled output interface, 
  'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
  'M' - malformed request, 'm' - unsupported tlvs, 'N' - no rx label, 
  'P' - no rx intf label prot, 'p' - premature termination of LSP, 
  'R' - transit router, 'I' - unknown upstream index,
  'X' - unknown return code, 'x' - return code 0

Type escape sequence to abort.

Q      size 100, Unable to send echo request packet
Q      size 100, Unable to send echo request packet

Success rate is 0 percent (0/2)

Q is “request not sent” from the legend. At the same time, the IP ping is lossless.

RP/0/RP0/CPU0:PE1#ping 5.5.5.5 source 2.2.2.2 count 50 timeout 1
Sat Sep 12 08:54:03.788 UTC
Type escape sequence to abort.
Sending 50, 100-byte ICMP Echos to 5.5.5.5 timeout is 1 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (50/50), round-trip min/avg/max = 9/21/76 ms

To the same destination, IP ping is 100% and LSP Ping is 0%. The VPN traffic carried over MPLS (CE1 - CE2) is fully lost at this point as well.

A data-plane-only failure cannot be built in a lab

What this test produced is a control-plane failure: the LDP session is down, so show mpls ldp neighbor reveals the cause. The state LSP Ping really targets — control plane healthy, forwarding broken — could not be created by configuration.

Using static labels on IOS XR to override the label LDP allocated (24003 on P1) does commit, but the entry ends up as Discrepancy and is never programmed into the LFIB.

Label   VRF             Type         Prefix           RW Configured   Status   
------- --------------- ------------ ---------------- --------------- -------- 
24003   default         X-Connect    N/A              Yes             Discrepancy

XR detects the conflict with the LDP-managed label and refuses it. A misconfiguration therefore cannot drive the control plane and the data plane apart. The failures LSP Ping finds originate outside the configuration, such as a forwarding entry that failed to be written to hardware — which is exactly why a tool is needed for faults that show cannot reveal.

Design notes

• On IOS XR, mpls oam must already be there. Enabling it after trouble starts is too late, so make it part of the standard core configuration
• The responders need it too. Enabling it only on the initiator leaves transit LSRs and the egress unable to answer
• Q (request not sent) means the label is missing locally. Nothing reached the peer, so start with your own show mpls ldp neighbor and show mpls forwarding
• A successful LSP Ping does not guarantee VPN reachability. LSP Ping tests the transport LSP; the VPN label is pushed separately

Verification configs and show output

At every STEP the following was collected from all six routers, split per router. The verification config is the ..._run.txt file (the final state is the one from the last STEP).

STEP 0: initial state (mpls oam not enabled)

STEP 1: enable mpls oam on all four core routers

STEP 2: drop LDP between PE1 and P1 with an ACL

STEP 3: remove everything (final state)

The full capture between PE1 and P1, containing the MPLS echo requests and replies from every STEP.

References

UDP port 3503 is a value measured in the capture, not quoted from the RFC text. The lab ran on XRd 26.1.1 under Cisco Modeling Labs.

• What Is MPLS
• MPLS Labels and the Label Stack
• MPLS Label Operations (push / swap / pop) and PHP
• MPLS TTL Processing and MTU
• What Is LDP
• LDP Label Distribution Modes and Label Spaces
• LDP-IGP Synchronization and LDP Session Protection
• LDP Label Advertisement Control (Filtering)
• LDP Session Authentication (TCP MD5)
• MPLS OAM (LSP Ping and LSP Traceroute)
• What Is MPLS TE (RSVP-TE)
• MPLS TE CSPF and Path Constraints (Bandwidth, Affinity, TE Metric)
• Steering Traffic into MPLS TE Tunnels