MPLS Label Operations (push / swap / pop) and PHP

An MPLS router (LSR) performs three operations on a packet: push to add a label, swap to replace the top label, and pop to remove the top label. This article explains which table each router consults and what it does as a single packet travels from the ingress PE to the egress PE, and what PHP (Penultimate Hop Popping) is, then confirms it hop by hop in an IOS XR (XRd) lab. The label format is covered in MPLS Labels and the Label Stack.

Three operations and two tables

Operation	LSR that performs it	What happens
push	Ingress LSR (PE1)	Receives an unlabelled IP packet and adds the label of the FEC chosen by the destination.
swap	Transit LSR (P1, P2)	Replaces the top label with the label received from the next LSR.
pop	Penultimate LSR (P3) or egress LSR	Removes the top label. If no label remains, the packet is handled as IP.

Which table is consulted depends on whether the received packet carries a label.

Received packet	Table	IOS XR command
No label (IP packet)	FIB. The destination IP address is looked up by longest match and the label attached to the result is pushed (the FTN of RFC 3031)	`show cef <prefix> detail` (`labels imposed {…}`)
Labelled	LFIB. The top label value indexes the operation (swap / pop) and the outgoing interface (the ILM of the same RFC)	`show mpls forwarding` (Local / Outgoing columns)

A transit LSR never looks at the IP header. It consults the top label only, and either replaces it with the value in Outgoing Label (swap) or removes it when the entry says Pop.

The Outgoing Label column of show mpls forwarding shows the following in addition to a label value.

Display	Meaning
A number	Swap to that value.
`Pop`	Remove the top label. Shown when the next LSR advertised implicit-null (3) for this prefix.
`Exp-Null-v4`	Swap the top label to 0 (IPv4 Explicit NULL). Shown when the next LSR advertised explicit-null.
`Unlabelled`	Remove the label and send the packet as IP. Shown when the next hop does not speak MPLS (a CE, for example) or has not advertised a label for that prefix.
`Aggregate`	Remove the label and perform a fresh IP lookup. It appears when a single label points at a whole VRF table and therefore does not determine a next hop: when VPN labels are allocated one per VRF (per-VRF), or for a label bound to an aggregated prefix. The lab in this article has no route that needs an aggregate label (such as a connected route put into BGP with `redistribute connected`), so this value does not appear in its output. The allocation modes are covered in MPLS VPN Label Allocation.

A label is replaced at every hop

A label is only meaningful between two adjacent routers, so its value changes each time the packet passes an LSR. In the figure above, the 30005 that PE1 pushed (the value P1 assigned to 6.6.6.6/32) is replaced by P1 with 40005 (the value P2 assigned to the same prefix), and by P2 with 50005 (P3’s value). How the values are chosen is covered in What Is LDP. Each LSR assigns its local labels independently, so two neighbours may happen to pick the same value. The swap still takes place; only the values coincide. The assignment also changes when a router reboots, so the label value itself carries no meaning.

PHP (Penultimate Hop Popping)

An egress LSR advertises implicit-null (3) for the routes connected to itself, such as its loopback. The LSR one hop before it removes the label from packets destined for that prefix (pop) before forwarding them. That behaviour is PHP.

The purpose of PHP is to reduce the number of lookups at the egress LSR (RFC 3031 section 3.16). If the egress LSR received the packet still labelled, it would first consult the LFIB to decide that the label is its own and must be removed, then consult the FIB with the IP header, which is two lookups. Removing the label one hop earlier lets the egress LSR receive an IP packet and do a single FIB lookup.

To avoid PHP, the egress LSR is made to advertise explicit-null (0). The penultimate LSR swaps the label to 0 and forwards it, and the egress LSR removes label 0 before doing the IP lookup. This is used when the TC field (the QoS class) of the label needs to reach the egress LSR. On IOS XR it is selected with label local advertise explicit-null under mpls ldp.

Value the egress LSR advertises	Operation at the penultimate LSR	Lookups at the egress LSR
implicit-null (3, default)	pop	One IP lookup
explicit-null (0)	Swap to 0	A label lookup plus an IP lookup

PHP only removes the last label one hop early. When there are two labels, as in an MPLS VPN, the penultimate LSR removes only the top transport label, and the VPN label reaches the egress PE.

Lab verification

Lab setup

CE1, PE1, P1, P2, P3, PE2, and CE2 are connected in a row. PE1 / P1 / P2 / P3 / PE2 in AS 65001 form the MPLS network, with OSPF (area 0, every link network point-to-point) distributing routes and LDP distributing labels. CE1 and CE2 exchange routes with the PEs over eBGP, and PE1 and PE2 peer over iBGP between their Loopback0 addresses with next-hop-self. CE3 / CE4 are customers in VRF CUST-A and are used to confirm that PHP removes only the top entry of the two-label VPN stack.

The transit routers P1 / P2 / P3 boot with their dynamic label ranges split into 30000-, 40000- and 50000- (mpls label range table 0). Each LSR assigns labels on its own, so with the default range two neighbours can pick the same value and a swap would not show up as a change of value. Splitting the ranges does not change the behaviour.

Router	Role	Lo0	Dynamic label range
CE1	Customer side (AS 65101), advertises 192.168.1.0/24	1.1.1.1/32	—
PE1	Ingress LSR (push)	2.2.2.2/32	Default (24000-)
P1	Transit LSR (swap)	3.3.3.3/32	30000-39999
P2	Transit LSR (swap)	4.4.4.4/32	40000-49999
P3	Penultimate hop (pop)	5.5.5.5/32	50000-59999
PE2	Egress LSR	6.6.6.6/32	Default (24000-)
CE2	Customer side (AS 65102), advertises 192.168.2.0/24	7.7.7.7/32	—
CE3 / CE4	Customers in VRF CUST-A, 192.168.3.0/24 / 192.168.4.0/24	8.8.8.8 / 9.9.9.9	—

CE1 pings 192.168.2.1 (CE2) and CE3 pings 192.168.4.1 (CE4) while all four core links are captured. The BGP next hop of 192.168.2.0/24 is PE2’s Loopback0 (6.6.6.6), so every LSR looks up its entry for 6.6.6.6/32.

Verification steps

STEP	Action	What to confirm
0	Initial state (default = PHP)	PE1 pushes, P1 and P2 swap, P3 pops. The pcaps of the four links are matched against each router’s FIB / LFIB. With the two-label VPN stack only the top entry is removed
1	`label local advertise explicit-null` on PE2	The pop moves from P3 to PE2, and P3’s operation becomes a swap to `Exp-Null-v4`
2	Remove STEP 1 (final state)	Back to the same state as STEP 0

Push: the ingress PE1 adds the label

PE1 learned 192.168.2.0/24 with next hop 6.6.6.6, so the label to push is in the FIB entry for 6.6.6.6/32. labels imposed {30005} is that value: the label P1 assigned to 6.6.6.6/32.

PE1 show cef 6.6.6.6/32 detail (STEP 0)
RP/0/RP0/CPU0:PE1#show cef 6.6.6.6/32 detail
Tue Sep 22 04:11:39.891 UTC
6.6.6.6/32, version 17, internal 0x1000001 0x10 (ptr 0x88519ae0) [2], 0x600 (0x894f2018), 0xa28 (0x899723f8)
 Updated Sep 22 00:53:32.311
 local adjacency to GigabitEthernet0/0/0/1

 Prefix Len 32, traffic index 0, precedence n/a, priority 1
  gateway array (0x8938b638) reference count 15, flags 0x68, source lsd (5), 1 backups
                [6 type 5 flags 0x8401 (0x899ab460) ext 0x0 (0x0)]
  LW-LDI[type=5, refc=3, ptr=0x894f2018, sh-ldi=0x899ab460]
  gateway array update type-time 1 Sep 22 00:53:32.252
 LDI Update time Sep 22 00:53:32.253
 LW-LDI-TS Sep 22 00:53:32.311
   via 10.2.3.3/32, GigabitEthernet0/0/0/1, 6 dependencies, weight 0, class 0 [flags 0x0]
    path-idx 0 NHID 0x4 [0x8a7776e0 0x0]
    next hop 10.2.3.3/32
    local adjacency
     local label 24006      labels imposed {30005}

    Load distribution: 0 (refcount 6)

    Hash  OK  Interface                 Address
    0     Y   GigabitEthernet0/0/0/1    10.2.3.3

No.33 of the PE1 - P1 capture is the echo request from CE1. The EtherType has become 0x8847 and label 30005 is attached.

No.33 Echo request (PE1 → P1) tshark -V
    Type: MPLS label switched packet (0x8847)
    [Stream index: 4]
MultiProtocol Label Switching Header, Label: 30005, Exp: 0, S: 1, TTL: 254
    0000 0111 0101 0011 0101 .... .... .... = MPLS Label: 30005 (0x07535)
    .... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
Internet Protocol Version 4, Src: 192.168.1.1, Dst: 192.168.2.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 100
    Identification: 0x0000 (0)
    000. .... = Flags: 0x0
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 254

Download the pcap of the packet in the tshark output above (No.33 Echo request)

Swap: the transit P1 and P2 replace the label

P1 and P2 do not look at the IP header; they look up the received label in the LFIB. P1 replaces 30005 with 40005 (the value P2 assigned), and P2 replaces 40005 with 50005 (the value P3 assigned).

P1 show mpls forwarding (STEP 0)
RP/0/RP0/CPU0:P1#show mpls forwarding
Tue Sep 22 04:12:38.601 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes
Label  Label       or ID              Interface                    Switched
------ ----------- ------------------ ------------ --------------- ------------
30000  Pop         2.2.2.2/32         Gi0/0/0/0    10.2.3.2        38122
30001  Pop         4.4.4.4/32         Gi0/0/0/1    10.3.4.4        212
30002  Pop         10.4.5.0/24        Gi0/0/0/1    10.3.4.4        0
30003  40003       5.5.5.5/32         Gi0/0/0/1    10.3.4.4        0
30004  40004       10.5.6.0/24        Gi0/0/0/1    10.3.4.4        0
30005  40005       6.6.6.6/32         Gi0/0/0/1    10.3.4.4        37598

P2 show mpls forwarding (STEP 0)
RP/0/RP0/CPU0:P2#show mpls forwarding
Tue Sep 22 04:13:04.101 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes
Label  Label       or ID              Interface                    Switched
------ ----------- ------------------ ------------ --------------- ------------
40000  30000       2.2.2.2/32         Gi0/0/0/0    10.3.4.3        38553
40001  Pop         3.3.3.3/32         Gi0/0/0/0    10.3.4.3        274
40002  Pop         10.2.3.0/24        Gi0/0/0/0    10.3.4.3        0
40003  Pop         5.5.5.5/32         Gi0/0/0/1    10.4.5.5        318
40004  Pop         10.5.6.0/24        Gi0/0/0/1    10.4.5.5        0
40005  50005       6.6.6.6/32         Gi0/0/0/1    10.4.5.5        38553

Following the same echo request across the four links, the label value changes at every hop and the label TTL drops by one each time.

The same echo request on the four core links (STEP 0)
$ tshark -r - -Y 'icmp.type==8 && ip.src==192.168.1.1' -T fields -e frame.number -e mpls.label -e mpls.ttl -e ip.ttl \
    < mpls-label-operation-step0-pe1p1.pcap
33	30005	254	254
$ tshark -r - -Y 'icmp.type==8 && ip.src==192.168.1.1' -T fields -e frame.number -e mpls.label -e mpls.ttl -e ip.ttl \
    < mpls-label-operation-step0-p1p2.pcap
31	40005	253	254
$ tshark -r - -Y 'icmp.type==8 && ip.src==192.168.1.1' -T fields -e frame.number -e mpls.label -e mpls.ttl -e ip.ttl \
    < mpls-label-operation-step0-p2p3.pcap
34	50005	252	254
$ tshark -r - -Y 'icmp.type==8 && ip.src==192.168.1.1' -T fields -e frame.number -e mpls.label -e mpls.ttl -e ip.ttl \
    < mpls-label-operation-step0-p3pe2.pcap
35			251

A traceroute from CE1 also shows the label each hop received.

CE1 traceroute (STEP 0)
RP/0/RP0/CPU0:CE1#traceroute 192.168.2.1 source 192.168.1.1 timeout 1 probe 2 maxttl 7
Tue Sep 22 04:10:51.791 UTC

Type escape sequence to abort.
Tracing the route to 192.168.2.1

 1  10.1.2.2 6 msec  5 msec
 2  10.2.3.3 [MPLS: Label 30005 Exp 0] 18 msec  18 msec
 3  10.3.4.4 [MPLS: Label 40005 Exp 0] 18 msec  17 msec
 4  10.4.5.5 [MPLS: Label 50005 Exp 0] 19 msec  17 msec
 5  10.5.6.6 19 msec  19 msec
 6  10.6.7.7 36 msec  *

Pop (PHP): the penultimate P3 removes the label

PE2 distributes implicit-null (3) for its own Loopback0 (6.6.6.6/32). In P3’s bindings the value from PE2 is ImpNull, and the outgoing operation in its LFIB is Pop.

P3 show mpls ldp bindings 6.6.6.6/32 (STEP 0)
RP/0/RP0/CPU0:P3#show mpls ldp bindings 6.6.6.6/32
Tue Sep 22 04:13:27.552 UTC
6.6.6.6/32, rev 18
	Local binding: label: 50005
	Remote bindings: (2 peers)
	    Peer                Label
	    -----------------   ---------
	    4.4.4.4:0           40005
	    6.6.6.6:0           ImpNull

P3 show mpls forwarding (STEP 0)
RP/0/RP0/CPU0:P3#show mpls forwarding
Tue Sep 22 04:13:28.219 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes
Label  Label       or ID              Interface                    Switched
------ ----------- ------------------ ------------ --------------- ------------
50000  40000       2.2.2.2/32         Gi0/0/0/0    10.4.5.4        37705
50001  40001       3.3.3.3/32         Gi0/0/0/0    10.4.5.4        0
50002  Pop         4.4.4.4/32         Gi0/0/0/0    10.4.5.4        380
50003  40002       10.2.3.0/24        Gi0/0/0/0    10.4.5.4        0
50004  Pop         10.3.4.0/24        Gi0/0/0/0    10.4.5.4        0
50005  Pop         6.6.6.6/32         Gi0/0/0/1    10.5.6.6        38345

No.35 on P3 - PE2 is the echo request after the label was removed. The EtherType is 0x0800; PE2 receives an IP packet and forwards it with a single IP lookup.

No.35 Echo request (P3 → PE2) tshark -V
    Type: IPv4 (0x0800)
    [Stream index: 4]
Internet Protocol Version 4, Src: 192.168.1.1, Dst: 192.168.2.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 100
    Identification: 0x0000 (0)
    000. .... = Flags: 0x0
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 251

Download the pcap of the packet in the tshark output above (No.35 Echo request)

Two labels for VPN: PHP removes only the top one

On the path from CE3 to CE4, PE1 pushes two labels: 30005 on top to reach PE2, and the VPN label 24007 that PE2 assigned underneath.

PE1 show cef vrf CUST-A 192.168.4.0/24 detail (STEP 0)
RP/0/RP0/CPU0:PE1#show cef vrf CUST-A 192.168.4.0/24 detail
Tue Sep 22 04:11:44.681 UTC
192.168.4.0/24, version 7, internal 0x5000001 0x30 (ptr 0x885194c0) [1], 0x0 (0x0), 0x208 (0x89972398)
 Updated Sep 22 00:54:44.354
 Prefix Len 24, traffic index 0, precedence n/a, priority 3
  gateway array (0x8938bc68) reference count 1, flags 0x2038, source rib (7), 0 backups
                [1 type 1 flags 0x48441 (0x899ab4b8) ext 0x0 (0x0)]
  LW-LDI[type=0, refc=0, ptr=0x0, sh-ldi=0x0]
  gateway array update type-time 1 Sep 22 00:54:44.354
 LDI Update time Sep 22 00:54:44.354
   via 6.6.6.6/32, 3 dependencies, recursive [flags 0x6000]
    path-idx 0 NHID 0x0 [0x8856ecc8 0x0]
    recursion-via-/32
    next hop VRF - 'default', table - 0xe0000000
    next hop 6.6.6.6/32 via 24006/0/21
     next hop 10.2.3.3/32 GigabitEthernet0/0/0/1 labels imposed {30005 24007}

    Load distribution: 0 (refcount 1)

    Hash  OK  Interface                 Address
    0     Y   recursive                 24006/0

No.521 on PE1 - P1 is the echo request with two labels.

No.521 Echo request (PE1 → P1, two labels) tshark -V
    Type: MPLS label switched packet (0x8847)
    [Stream index: 4]
MultiProtocol Label Switching Header, Label: 30005, Exp: 0, S: 0, TTL: 254
    0000 0111 0101 0011 0101 .... .... .... = MPLS Label: 30005 (0x07535)
    .... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0
    .... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
MultiProtocol Label Switching Header, Label: 24007, Exp: 0, S: 1, TTL: 254
    0000 0101 1101 1100 0111 .... .... .... = MPLS Label: 24007 (0x05dc7)
    .... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
Internet Protocol Version 4, Src: 192.168.3.1, Dst: 192.168.4.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 100
    Identification: 0x0000 (0)
    000. .... = Flags: 0x0
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 254

Download the pcap of the packet in the tshark output above (No.521 Echo request, two labels)

P3 removes only the top entry 50005; No.522 on P3 - PE2 still carries the VPN label 24007.

No.522 Echo request (P3 → PE2, VPN label only) tshark -V
    Type: MPLS label switched packet (0x8847)
    [Stream index: 4]
MultiProtocol Label Switching Header, Label: 24007, Exp: 0, S: 1, TTL: 251
    0000 0101 1101 1100 0111 .... .... .... = MPLS Label: 24007 (0x05dc7)
    .... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1011 = MPLS TTL: 251
Internet Protocol Version 4, Src: 192.168.3.1, Dst: 192.168.4.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 100
    Identification: 0x0000 (0)
    000. .... = Flags: 0x0
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 254

Download the pcap of the packet in the tshark output above (No.522 Echo request)

PE2 looks up 24007 in its LFIB and sends the IP packet to CE4 in VRF CUST-A.

PE2 show mpls forwarding (STEP 0)
RP/0/RP0/CPU0:PE2#show mpls forwarding
Tue Sep 22 04:12:08.823 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes
Label  Label       or ID              Interface                    Switched
------ ----------- ------------------ ------------ --------------- ------------
24000  50000       2.2.2.2/32         Gi0/0/0/0    10.5.6.5        36750
24001  50001       3.3.3.3/32         Gi0/0/0/0    10.5.6.5        0
24002  50002       4.4.4.4/32         Gi0/0/0/0    10.5.6.5        0
24003  Pop         5.5.5.5/32         Gi0/0/0/0    10.5.6.5        212
24004  50003       10.2.3.0/24        Gi0/0/0/0    10.5.6.5        0
24005  50004       10.3.4.0/24        Gi0/0/0/0    10.5.6.5        0
24006  Pop         10.4.5.0/24        Gi0/0/0/0    10.5.6.5        0
24007  Unlabelled  192.168.4.0/24[V]  Gi0/0/0/2    10.6.9.9        13088

The traceroute from CE3 likewise shows the top label being swapped and only the VPN label left on the last hop.

CE3 traceroute (STEP 0)
RP/0/RP0/CPU0:CE3#traceroute 192.168.4.1 source 192.168.3.1 timeout 1 probe 2 maxttl 8
Tue Sep 22 04:11:20.459 UTC

Type escape sequence to abort.
Tracing the route to 192.168.4.1

 1  10.2.8.2 6 msec  5 msec
 2  10.2.3.3 [MPLS: Labels 30005/24007 Exp 0] 66 msec  22 msec
 3  10.3.4.4 [MPLS: Labels 40005/24007 Exp 0] 24 msec  23 msec
 4  10.4.5.5 [MPLS: Labels 50005/24007 Exp 0] 23 msec  31 msec
 5  10.5.6.6 [MPLS: Label 24007 Exp 0] 20 msec  20 msec
 6  10.6.9.9 22 msec  *

Explicit-null: the pop moves to PE2

PE2 is made to distribute explicit-null.

STEP 1 configuration committed on PE2
RP/0/RP0/CPU0:PE2#show configuration commit changes last 1
Tue Sep 22 04:15:05.290 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
mpls ldp
 address-family ipv4
  label
   local
    advertise
     explicit-null
    !
   !
  !
 !
!
end

The value P3 receives from PE2 becomes ExpNullv4, and the LFIB operation changes from Pop to Exp-Null-v4 (a swap to label 0).

P3 show mpls ldp bindings 6.6.6.6/32 (STEP 1)
RP/0/RP0/CPU0:P3#show mpls ldp bindings 6.6.6.6/32
Tue Sep 22 04:19:21.900 UTC
6.6.6.6/32, rev 18
	Local binding: label: 50005
	Remote bindings: (2 peers)
	    Peer                Label
	    -----------------   ---------
	    4.4.4.4:0           40005
	    6.6.6.6:0           ExpNullv4

P3 show mpls forwarding (STEP 1)
RP/0/RP0/CPU0:P3#show mpls forwarding
Tue Sep 22 04:19:22.563 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes
Label  Label       or ID              Interface                    Switched
------ ----------- ------------------ ------------ --------------- ------------
50000  40000       2.2.2.2/32         Gi0/0/0/0    10.4.5.4        37713
50001  40001       3.3.3.3/32         Gi0/0/0/0    10.4.5.4        0
50002  Pop         4.4.4.4/32         Gi0/0/0/0    10.4.5.4        380
50003  40002       10.2.3.0/24        Gi0/0/0/0    10.4.5.4        0
50004  Pop         10.3.4.0/24        Gi0/0/0/0    10.4.5.4        0
50005  Exp-Null-v4 6.6.6.6/32         Gi0/0/0/1    10.5.6.6        39781

In No.92 on P3 - PE2, the echo request from CE1 carries label 0. PE2 removes label 0 and then performs an IP lookup, so it does two lookups.

No.92 Echo request (P3 → PE2, explicit-null) tshark -V
    Type: MPLS label switched packet (0x8847)
    [Stream index: 5]
MultiProtocol Label Switching Header, Label: 0 (IPv4 Explicit-Null), Exp: 0, S: 1, TTL: 251
    0000 0000 0000 0000 0000 .... .... .... = MPLS Label: IPv4 Explicit-Null (0)
    .... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1011 = MPLS TTL: 251
Internet Protocol Version 4, Src: 192.168.1.1, Dst: 192.168.2.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 100
    Identification: 0x0000 (0)
    000. .... = Flags: 0x0
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 254

Download the pcap of the packet in the tshark output above (No.92 Echo request)

For VPN traffic the stack has two entries, label 0 with the VPN label 24007 under it (No.584). The traceroute also shows Labels 0/24007 at hop 5.

No.584 Echo request (P3 → PE2, explicit-null plus VPN label) tshark -V
    Type: MPLS label switched packet (0x8847)
    [Stream index: 5]
MultiProtocol Label Switching Header, Label: 0 (IPv4 Explicit-Null), Exp: 0, S: 0, TTL: 251
    0000 0000 0000 0000 0000 .... .... .... = MPLS Label: IPv4 Explicit-Null (0)
    .... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0
    .... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0
    .... .... .... .... .... .... 1111 1011 = MPLS TTL: 251
MultiProtocol Label Switching Header, Label: 24007, Exp: 0, S: 1, TTL: 254
    0000 0101 1101 1100 0111 .... .... .... = MPLS Label: 24007 (0x05dc7)
    .... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
Internet Protocol Version 4, Src: 192.168.3.1, Dst: 192.168.4.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 100
    Identification: 0x0000 (0)
    000. .... = Flags: 0x0
        0... .... = Reserved bit: Not set
        .0.. .... = Don't fragment: Not set
        ..0. .... = More fragments: Not set
    ...0 0000 0000 0000 = Fragment Offset: 0
    Time to Live: 254

Download the pcap of the packet in the tshark output above (No.584 Echo request, two labels)

CE3 traceroute (STEP 1)
RP/0/RP0/CPU0:CE3#traceroute 192.168.4.1 source 192.168.3.1 timeout 1 probe 2 maxttl 8
Tue Sep 22 04:17:16.610 UTC

Type escape sequence to abort.
Tracing the route to 192.168.4.1

 1  10.2.8.2 6 msec  5 msec
 2  10.2.3.3 [MPLS: Labels 30005/24007 Exp 0] 25 msec  24 msec
 3  10.3.4.4 [MPLS: Labels 40005/24007 Exp 0] 21 msec  21 msec
 4  10.4.5.5 [MPLS: Labels 50005/24007 Exp 0] 21 msec  22 msec
 5  10.5.6.6 [MPLS: Labels 0/24007 Exp 0] 22 msec  23 msec
 6  10.6.9.9 25 msec  *

Removing the configuration in STEP 2 brings P3’s LFIB back to Pop.

P3 show mpls forwarding (STEP 2)
RP/0/RP0/CPU0:P3#show mpls forwarding
Tue Sep 22 04:25:10.263 UTC
Local  Outgoing    Prefix             Outgoing     Next Hop        Bytes
Label  Label       or ID              Interface                    Switched
------ ----------- ------------------ ------------ --------------- ------------
50000  40000       2.2.2.2/32         Gi0/0/0/0    10.4.5.4        37705
50001  40001       3.3.3.3/32         Gi0/0/0/0    10.4.5.4        0
50002  Pop         4.4.4.4/32         Gi0/0/0/0    10.4.5.4        380
50003  40002       10.2.3.0/24        Gi0/0/0/0    10.4.5.4        0
50004  Pop         10.3.4.0/24        Gi0/0/0/0    10.4.5.4        0
50005  Pop         6.6.6.6/32         Gi0/0/0/1    10.5.6.6        38239

Reachability

The label operations change, but traffic between the CEs keeps flowing in every STEP.

CE1 to CE2 (global, STEP 0)
RP/0/RP0/CPU0:CE1#ping 192.168.2.1 source 192.168.1.1 count 50 timeout 1
Tue Sep 22 04:10:49.733 UTC
Type escape sequence to abort.
Sending 50, 100-byte ICMP Echos to 192.168.2.1 timeout is 1 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (50/50), round-trip min/avg/max = 18/25/79 ms

CE3 to CE4 (VPN, STEP 0)
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 count 50 timeout 1
Tue Sep 22 04:11:18.275 UTC
Type escape sequence to abort.
Sending 50, 100-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (50/50), round-trip min/avg/max = 17/23/61 ms

Verification config and show output

For every STEP the following files were collected from all nine routers, one file per router. The verification config is the ..._run.txt (the final state is the one from STEP 2). Each STEP was collected in the order “clear the counters, ping, then show”, so Bytes Switched in show mpls forwarding covers only the pings sent during that STEP.

File	Contents
`..._clear.txt`	The counters cleared at the start of the STEP (`clear counters interface` and `clear mpls forwarding counters`)
`..._ping.txt`	`ping` (50 packets) and `traceroute`: CE1 - CE2 (global), CE3 - CE4 (VPN), PE1 - PE2 (the core LSP)
`..._show.txt`	`show version` / `show interface description` / `show route` / OSPF / LDP / `show mpls forwarding` / `show mpls label range` / `show cef` and more; the PEs add BGP and VPN output
`..._log.txt`	`show logging` limited to the range of that STEP
`..._run.txt`	`show running-config` at that STEP (that is, the verification config for the STEP)
`..._trace.txt`	The five core routers only: `show mpls ldp trace binding` / `peer` / `discovery` / `forwarding` and `show mpls lsd trace`
`..._commit.cfg`	The configuration actually committed in that STEP (only the routers that were changed)

STEP 0: Initial state (PHP)

Router	show	ping	clear	syslog	running-config	trace	committed config
CE1	show	ping	clear	log	run	—	—
PE1	show	ping	clear	log	run	trace	—
P1	show	—	clear	log	run	trace	—
P2	show	—	clear	log	run	trace	—
P3	show	—	clear	log	run	trace	—
PE2	show	ping	clear	log	run	trace	—
CE2	show	ping	clear	log	run	—	—
CE3	show	ping	clear	log	run	—	—
CE4	show	ping	clear	log	run	—	—

STEP 1: label local advertise explicit-null on PE2

Router	show	ping	clear	syslog	running-config	trace	committed config
CE1	show	ping	clear	log	run	—	—
PE1	show	ping	clear	log	run	trace	—
P1	show	—	clear	log	run	trace	—
P2	show	—	clear	log	run	trace	—
P3	show	—	clear	log	run	trace	—
PE2	show	ping	clear	log	run	trace	cfg
CE2	show	ping	clear	log	run	—	—
CE3	show	ping	clear	log	run	—	—
CE4	show	ping	clear	log	run	—	—

STEP 2: Remove explicit-null (final state)

Router	show	ping	clear	syslog	running-config	trace	committed config
CE1	show	ping	clear	log	run	—	—
PE1	show	ping	clear	log	run	trace	—
P1	show	—	clear	log	run	trace	—
P2	show	—	clear	log	run	trace	—
P3	show	—	clear	log	run	trace	—
PE2	show	ping	clear	log	run	trace	cfg
CE2	show	ping	clear	log	run	—	—
CE3	show	ping	clear	log	run	—	—
CE4	show	ping	clear	log	run	—	—

Packet captures were taken per STEP on all four core links.

STEP	PE1 - P1	P1 - P2	P2 - P3	P3 - PE2
0	pcap	pcap	pcap	pcap
1	pcap	pcap	pcap	pcap
2	pcap	pcap	pcap	pcap

References

RFC	Title	Summary
RFC 3031	Multiprotocol Label Switching Architecture	NHLFE / ILM / FTN (sections 3.10 to 3.12), label swapping (section 3.13) and PHP (section 3.16).
RFC 3032	MPLS Label Stack Encoding	Reserved labels (implicit-null 3, explicit-null 0).
RFC 4182	Removing a Restriction on the use of MPLS Explicit NULL	The revision that allows Explicit NULL outside the bottom of the stack.

Book: Luc De Ghein, MPLS Fundamentals (Cisco Press, 2006), Chapter 3