MPLS TTL Processing and MTU

An MPLS label is attached to a packet as a 4-byte label stack entry, of which the TTL field is 8 bits. Whether the ingress LSR copies the IP TTL into that field decides if the core routers show up in a traceroute. Separately, each label stack entry costs 4 bytes, so the IP packet that fits inside a given link MTU shrinks by that much. This article explains both points and then confirms them in an IOS XR (XRv9000 and XRd) lab. The label format is covered in MPLS Labels and the Label Stack.

The label TTL and the IP TTL

The MPLS TTL is a separate field from the IP TTL. While a packet is forwarded with a label, an LSR decrements only the TTL of the top label and never touches the IP header inside.

What the ingress LSR puts in that field splits the behaviour in two. RFC 3443 organises this as the uniform model and the pipe / short pipe models.

Model	TTL placed in the label at the ingress	At the egress
Uniform (section 3.1, the IOS XR default)	A copy of the IP TTL minus one	The label TTL is written back into the IP header
Short pipe (section 3.2)	A value set by the operator (255 on IOS XR)	The label TTL is not used
Pipe (section 3.3)	Same as short pipe	Same. For TTL it is treated like short pipe without PHP; the difference is which header (the inner IP or the outer label) the egress uses to forward

Under the uniform model the IP TTL drops by the number of core hops. A traceroute probe raises the TTL one at a time, so the core LSRs answer with a TTL-exceeded ICMP and the core P router (P1 in this lab) appears as a hop. With propagation off the label TTL starts at 255 and never runs out inside the core. The core sends no ICMP, and from CE1 the span between the PEs looks like a single hop.

On IOS XR the switch is mpls ip-ttl-propagate disable. It is configured only on the PE that acts as the ingress, and it takes effect per direction. With it on PE1 alone, a traceroute from CE1 to CE2 hides the core while a traceroute from CE2 to CE1 still shows it.

Operators who do not want to expose the core topology to customers turn propagation off. The trade-off is that troubleshooting a path becomes harder.

A label consumes MTU

Each label is 4 bytes. The link MTU does not change, so the IP packet that fits shrinks by the size of the label stack.

The mtu of an IOS XR interface is the L2 MTU and includes the 14-byte Ethernet header. The default of 1514 means an IP MTU of 1500. By default the MPLS MTU is the L2 MTU minus 14.

With M as the L2 MTU of the core links, the largest IP packet that gets through is as follows.

Labels	Used for	Limit	Default (M = 1514)
One	Transport label only	M − 14 − 4	1496 bytes
Two	MPLS VPN (transport + VPN)	M − 14 − 8	1492 bytes

When a packet above the limit reaches the ingress PE, the DF bit in the IP header decides what happens. Without DF the PE fragments it and forwards the pieces. With DF the packet cannot be forwarded, so it is dropped and an ICMP Type 3 Code 4 (Fragmentation Needed) goes back to the sender. The MTU advertised in that message already has the label subtracted (RFC 3032 section 3.4), so Path MTU Discovery converges on a workable size.

In production the MPLS section’s MTU is raised so a customer’s 1500-byte IP packet passes with its labels. The L2 MTU needed is 1500 plus 4 per label plus 14, which is 1522 for the two labels of an MPLS VPN. To leave room for three or more labels and for L2VPN (which carries the customer’s whole Ethernet frame), core links are commonly sized in the 9000 range, such as 9216. Any switch along the way must be configured to pass frames of the same size.

Lab verification

Lab setup

CE1, PE1, P1, PE2, and CE2 are connected in a row, and the VRF CUST-A customers CE3 / CE4 hang off PE1 and PE2. PE1 / P1 / PE2 in AS 65001 form the MPLS network, with OSPF (area 0, every link network point-to-point) distributing routes and LDP distributing labels. The routes of CE1 / CE2 travel over iBGP between the PEs (next-hop-self), and the routes of CE3 / CE4 travel as VPNv4. A packet from CE1 to CE2 carries one label; a packet from CE3 to CE4 carries two (transport plus VPN).

PE1 / P1 / PE2 are XRv9000 and the CEs are XRd. Between XRd instances (IOS XR running as a container) frames larger than 1500 bytes do not arrive even with a larger mtu, so only the MPLS section, where the MTU is raised, uses XRv9000. The CE-PE links stay at the default MTU (1514), and the CEs send IP packets of up to 1500 bytes.

Router	Type	Role	Lo0	Links
CE1	XRd	Customer side (AS 65101), 192.168.1.0/24	1.1.1.1/32	Gi0/0/0/0 10.1.2.1
PE1	XRv9000	Ingress / egress LSR	2.2.2.2/32	Gi0/0/0/0 10.1.2.2 (CE1) / Gi0/0/0/1 10.2.3.2 (P1) / Gi0/0/0/2 10.2.6.2 (CE3, VRF)
P1	XRv9000	Transit LSR	3.3.3.3/32	Gi0/0/0/0 10.2.3.3 / Gi0/0/0/1 10.3.4.3
PE2	XRv9000	Ingress / egress LSR	4.4.4.4/32	Gi0/0/0/0 10.3.4.4 (P1) / Gi0/0/0/1 10.4.5.4 (CE2) / Gi0/0/0/2 10.4.7.4 (CE4, VRF)
CE2	XRd	Customer side (AS 65102), 192.168.2.0/24	5.5.5.5/32	Gi0/0/0/0 10.4.5.5
CE3	XRd	Customer in VRF CUST-A (AS 65103), 192.168.3.0/24	6.6.6.6/32	Gi0/0/0/0 10.2.6.6
CE4	XRd	Customer in VRF CUST-A (AS 65104), 192.168.4.0/24	7.7.7.7/32	Gi0/0/0/0 10.4.7.7

Each STEP was captured on four links: CE1 - PE1, CE3 - PE1, PE1 - P1, and P1 - PE2. The MTU is changed on the four MPLS-side interfaces (PE1 Gi0/0/0/1, both of P1’s, and PE2 Gi0/0/0/0).

Verification steps

STEP	Action	What to confirm
0	Initial state (MPLS-side MTU 1514, TTL propagation on)	The largest packet that passes with DF is 1496 with one label and 1492 with two. PE1 returns ICMP Type 3 Code 4, and a 1500-byte packet without DF is fragmented. P1 appears in the traceroute
1	`mtu 1522` on the four MPLS-side interfaces	The 1500-byte L3VPN packet (two labels) passes with DF set. Frames of 1522 bytes cross the MPLS section
2	`mtu 9216` on the four MPLS-side interfaces	Frames above 1522 also pass (9000 bytes between PE1 and PE2)
3	`mpls ip-ttl-propagate disable` on PE1	P1 disappears from the traceroute from CE1 to CE2 but stays visible in the reverse direction
4	Remove STEP 1 to 3 (final state)	Back to the same state as STEP 0

With the default MTU a 1500-byte L3VPN packet does not pass

With the MPLS section at the default 1514, the largest packet that goes from CE3 to CE4 with DF set is 1492 bytes; 1493 and 1500 do not arrive. Without DF, 1500 arrives as well.

CE3 to CE4 (L3VPN, two labels, STEP 0)
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1492 df-bit count 5 timeout 1
Mon Sep 28 12:42:14.351 UTC
Type escape sequence to abort.
Sending 5, 1492-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 19/19/21 ms
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1493 df-bit count 5 timeout 1
Mon Sep 28 12:42:15.544 UTC
Type escape sequence to abort.
Sending 5, 1493-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
.....
Success rate is 0 percent (0/5)
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1500 df-bit count 5 timeout 1
Mon Sep 28 12:42:21.880 UTC
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
.....
Success rate is 0 percent (0/5)
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1500 count 5 timeout 1
Mon Sep 28 12:42:28.080 UTC
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 25/30/42 ms

From CE1 to CE2, with one label, the limit is 4 bytes higher at 1496.

CE1 to CE2 (global, one label, STEP 0)
RP/0/RP0/CPU0:CE1#ping 192.168.2.1 source 192.168.1.1 size 1496 df-bit count 5 timeout 1
Mon Sep 28 12:40:34.301 UTC
Type escape sequence to abort.
Sending 5, 1496-byte ICMP Echos to 192.168.2.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 19/20/22 ms
RP/0/RP0/CPU0:CE1#ping 192.168.2.1 source 192.168.1.1 size 1497 df-bit count 5 timeout 1
Mon Sep 28 12:40:35.437 UTC
Type escape sequence to abort.
Sending 5, 1497-byte ICMP Echos to 192.168.2.1 timeout is 1 seconds:
.....
Success rate is 0 percent (0/5)
RP/0/RP0/CPU0:CE1#ping 192.168.2.1 source 192.168.1.1 size 1500 df-bit count 5 timeout 1
Mon Sep 28 12:40:41.302 UTC
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.168.2.1 timeout is 1 seconds:
.....
Success rate is 0 percent (0/5)

When PE1 receives the 1493-byte packet with DF, it does not forward it and returns ICMP Type 3 Code 4 (No.64 on CE3 - PE1). The advertised MTU (MTU of next hop) is 1492, which already has the 8 bytes of the two labels subtracted. The same ICMP sent back to CE1 carried 1496.

No.64 ICMP Type 3 Code 4 (PE1 → CE3) tshark -V
Internet Control Message Protocol
    Type: Destination unreachable (3)
        [Expert Info (Note/Response): Type indicates an error]
            [Type indicates an error]
            [Severity level: Note]
            [Group: Response]
    Code: 4 (Fragmentation needed)
    Checksum: 0xa902 [correct]
    [Checksum Status: Good]
    Unused: 00
    Length: 17
    [Length of original datagram: 68]
    MTU of next hop: 1492
    Internet Protocol Version 4, Src: 192.168.3.1, Dst: 192.168.4.1
        0100 .... = Version: 4
        .... 0101 = Header Length: 20 bytes (5)
        Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
            0000 00.. = Differentiated Services Codepoint: Default (0)
            .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
        Total Length: 1493
        Identification: 0x0000 (0)
        010. .... = Flags: 0x2, Don't fragment
            0... .... = Reserved bit: Not set
            .1.. .... = Don't fragment: Set
            ..0. .... = More fragments: Not set
        ...0 0000 0000 0000 = Fragment Offset: 0

Download the pcap of the packet in the tshark output above (No.64 ICMP Type 3 Code 4)

The 1500-byte packet without DF is fragmented by PE1 before the labels are pushed. On PE1 - P1 it becomes two pieces of 1492 and 28 bytes, each with the two labels. The first frame is 1514 bytes, filling the MTU.

1500 bytes without DF (PE1 → P1, STEP 0)
$ tshark -r - -Y 'frame.number==445 || frame.number==449' -T fields -e frame.number -e frame.len -e mpls.label -e ip.len -e ip.flags.mf -e ip.frag_offset \
    < mpls-ttl-mtu-step0-pe1p1.pcap
445	50	24001,24003	28	False	184
449	50	24001,24003	28	False	184

A 1522-byte MPLS MTU lets the 1500-byte L3VPN packet through

A customer’s 1500 bytes plus two labels (8 bytes) plus the Ethernet header (14 bytes) is 1522. The mtu at both ends of every MPLS link is set to 1522; on P1 it goes on both interfaces.

Configuration committed on P1 in STEP 1
RP/0/RP0/CPU0:P1#show configuration commit changes last 1
Sun Oct  4 03:51:32.767 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
interface GigabitEthernet0/0/0/0
 mtu 1522
!
interface GigabitEthernet0/0/0/1
 mtu 1522
!
end

IP gets 1508 bytes after subtracting 14, which holds a 1500-byte packet with two labels exactly.

P1 show ipv4 interface GigabitEthernet0/0/0/0 (STEP 1)
RP/0/RP0/CPU0:P1#show ipv4 interface GigabitEthernet0/0/0/0
Sun Oct  4 04:04:54.019 UTC
GigabitEthernet0/0/0/0 is Up, ipv4 protocol is Up
  Vrf is default (vrfid 0x60000000)
  Internet address is 10.2.3.3/24
  MTU is 1522 (1508 is available to IP)
  Helper address is not set
  Multicast reserved groups joined: 224.0.0.2 224.0.0.5 224.0.0.6
  Directed broadcast forwarding is disabled
  Outgoing access list is not set
  Inbound  common access list is not set, access list is not set
  Proxy ARP is disabled
  ICMP redirects are never sent
  ICMP unreachables are always sent
  ICMP mask replies are never sent
  Table Id is 0xe0000000

Packets from CE3 to CE4 now pass with DF set up to 1500 bytes.

CE3 to CE4 (L3VPN, two labels, STEP 1)
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1492 df-bit count 5 timeout 1
Sun Oct  4 04:00:07.118 UTC
Type escape sequence to abort.
Sending 5, 1492-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 15/16/19 ms
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1493 df-bit count 5 timeout 1
Sun Oct  4 04:00:08.121 UTC
Type escape sequence to abort.
Sending 5, 1493-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 15/15/17 ms
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1500 df-bit count 5 timeout 1
Sun Oct  4 04:00:09.225 UTC
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 15/18/23 ms
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1500 count 5 timeout 1
Sun Oct  4 04:00:10.240 UTC
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 17/17/20 ms

No.4611 on PE1 - P1 is the 1500-byte echo request with DF. The frame is 1522 bytes long, and the 1500-byte IP packet sits unchanged under the two labels.

No.4611 Echo request (PE1 → P1, 1500 bytes plus two labels) tshark -V
    Frame Length: 1522 bytes (12176 bits)
    Capture Length: 1522 bytes (12176 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:mpls:icmp:data]
    Character encoding: ASCII (0)
Ethernet II, Src: 52:54:00:ee:89:fb (52:54:00:ee:89:fb), Dst: 52:54:00:05:93:9b (52:54:00:05:93:9b)
    Destination: 52:54:00:05:93:9b (52:54:00:05:93:9b)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Source: 52:54:00:ee:89:fb (52:54:00:ee:89:fb)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: MPLS label switched packet (0x8847)
    [Stream index: 4]
MultiProtocol Label Switching Header, Label: 24001, Exp: 0, S: 0, TTL: 254
    0000 0101 1101 1100 0001 .... .... .... = MPLS Label: 24001 (0x05dc1)
    .... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0
    .... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
MultiProtocol Label Switching Header, Label: 24003, Exp: 0, S: 1, TTL: 254
    0000 0101 1101 1100 0011 .... .... .... = MPLS Label: 24003 (0x05dc3)
    .... .... .... .... .... 000. .... .... = MPLS Experimental Bits: 0
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
Internet Protocol Version 4, Src: 192.168.3.1, Dst: 192.168.4.1
    0100 .... = Version: 4
    .... 0101 = Header Length: 20 bytes (5)
    Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
        0000 00.. = Differentiated Services Codepoint: Default (0)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 1500
    Identification: 0x0000 (0)
    010. .... = Flags: 0x2, Don't fragment
        0... .... = Reserved bit: Not set
        .1.. .... = Don't fragment: Set

Download the pcap of the packet in the tshark output above (No.4611 Echo request)

CE1 to CE2, with one label, also passes up to 1500.

CE1 to CE2 (global, one label, STEP 1)
RP/0/RP0/CPU0:CE1#ping 192.168.2.1 source 192.168.1.1 size 1496 df-bit count 5 timeout 1
Sun Oct  4 03:58:42.865 UTC
Type escape sequence to abort.
Sending 5, 1496-byte ICMP Echos to 192.168.2.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 14/15/18 ms
RP/0/RP0/CPU0:CE1#ping 192.168.2.1 source 192.168.1.1 size 1497 df-bit count 5 timeout 1
Sun Oct  4 03:58:43.795 UTC
Type escape sequence to abort.
Sending 5, 1497-byte ICMP Echos to 192.168.2.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 14/15/18 ms
RP/0/RP0/CPU0:CE1#ping 192.168.2.1 source 192.168.1.1 size 1500 df-bit count 5 timeout 1
Sun Oct  4 03:58:44.702 UTC
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.168.2.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 13/14/16 ms

A ping from PE1 to PE2 (4.4.4.4) confirms that the limit is exactly 1522. PE1 pushes the single label assigned by P1, so the limit is 1504 + 4 + 14 = 1522 bytes: 1504 passes and 1505 does not. PE1’s ping file has terminal line-editing control characters in the command line, so only the reply lines are shown and the command is given in the title.

PE1 ping 4.4.4.4 source 2.2.2.2 size 1504 df-bit count 5 timeout 1 (STEP 1)
Sun Oct  4 03:59:09.866 UTC
Type escape sequence to abort.
Sending 5, 1504-byte ICMP Echos to 4.4.4.4 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 27/40/54 ms

PE1 ping 4.4.4.4 source 2.2.2.2 size 1505 df-bit count 5 timeout 1 (STEP 1)
Sun Oct  4 03:59:12.021 UTC
Type escape sequence to abort.
Sending 5, 1505-byte ICMP Echos to 4.4.4.4 timeout is 1 seconds:
.....
Success rate is 0 percent (0/5)

An interface whose mtu changes goes down briefly, so the OSPF adjacencies are re-established (the STEP 1 ..._log.txt of each router). In P1’s log, the adjacency with PE1 sat in EXSTART while PE1 had already changed and the MTUs did not match.

Customer traffic is lost while they re-establish. A ping from CE1 that ran across the change lost 80 packets in STEP 1 and 79 in STEP 2 (2000 each), while the TTL change in STEP 3, which does not touch an interface, lost 0.

STEP	Action	Ping across the change
1	`mtu 1522` on the four MPLS-side interfaces	1920/2000 (80 lost)
2	`mtu 9216` on the same interfaces	1921/2000 (79 lost)
3	`mpls ip-ttl-propagate disable` on PE1	2000/2000 (none lost)

Changing the MTU restarts the interface, so on a production device the convergence has to be budgeted for. Switching TTL propagation does not stop forwarding. The loss count only resolves times coarser than the ping’s granularity (interval 100 timeout 1), so this is treated as the difference between losing packets and not. Changing only one end leaves the MTUs mismatched and OSPF stalls in EXSTART. Change both ends together.

Raising the MPLS MTU to 9216

1522 fits a 1500-byte packet with two labels exactly. Real MPLS networks size the core MTU in the 9000 range to leave room for three or more labels, L2VPN (which carries the customer’s whole Ethernet frame), and customer jumbo frames. In STEP 2 the four interfaces were set to 9216.

Configuration committed on P1 in STEP 2
RP/0/RP0/CPU0:P1#show configuration commit changes last 1
Sun Oct  4 04:08:50.913 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
interface GigabitEthernet0/0/0/0
 mtu 9216
!
interface GigabitEthernet0/0/0/1
 mtu 9216
!
end

P1 show ipv4 interface GigabitEthernet0/0/0/0 (STEP 2)
RP/0/RP0/CPU0:P1#show ipv4 interface GigabitEthernet0/0/0/0
Sun Oct  4 04:21:59.545 UTC
GigabitEthernet0/0/0/0 is Up, ipv4 protocol is Up
  Vrf is default (vrfid 0x60000000)
  Internet address is 10.2.3.3/24
  MTU is 9216 (9202 is available to IP)
  Helper address is not set
  Multicast reserved groups joined: 224.0.0.2 224.0.0.5 224.0.0.6
  Directed broadcast forwarding is disabled
  Outgoing access list is not set
  Inbound  common access list is not set, access list is not set
  Proxy ARP is disabled
  ICMP redirects are never sent
  ICMP unreachables are always sent
  ICMP mask replies are never sent
  Table Id is 0xe0000000

The 1505-byte and 9000-byte pings that failed in STEP 1 now reach PE2 from PE1.

PE1 ping 4.4.4.4 source 2.2.2.2 size 1505 df-bit count 5 timeout 1 (STEP 2)
Sun Oct  4 04:16:24.812 UTC
Type escape sequence to abort.
Sending 5, 1505-byte ICMP Echos to 4.4.4.4 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 29/34/39 ms

PE1 ping 4.4.4.4 source 2.2.2.2 size 9000 df-bit count 5 timeout 1 (STEP 2)
Sun Oct  4 04:16:27.278 UTC
Type escape sequence to abort.
Sending 5, 9000-byte ICMP Echos to 4.4.4.4 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 37/45/61 ms

On PE1 - P1 a labelled frame of 9018 bytes (9000 + 4 + 14) crosses the link. P1 removes the label by PHP, so the reply from PE2 is an unlabelled 9014-byte frame.

9000-byte echo request / reply (PE1 - P1, STEP 2)
$ tshark -r - -Y 'frame.number==4398 || frame.number==4399' -T fields -e frame.number -e frame.len -e mpls.label -e ip.src -e ip.dst -e ip.len \
    < mpls-ttl-mtu-step2-pe1p1.pcap
4398	9018	24001	2.2.2.2	4.4.4.4	9000
4399	9014		4.4.4.4	2.2.2.2	9000

The CE-PE links stay at 1514, so a CE can still send only up to 1500 bytes. With a large MPLS section, the customer’s 1500 bytes are no longer dropped inside it however many labels are pushed.

Turning off TTL propagation hides P1

In STEP 0, with TTL propagation on, P1 (10.2.3.3) appears at hop 2 of the traceroute from CE1 to CE2.

CE1 traceroute (STEP 0)
RP/0/RP0/CPU0:CE1#traceroute 192.168.2.1 source 192.168.1.1 timeout 1 probe 2 maxttl 5
Mon Sep 28 12:40:47.375 UTC

Type escape sequence to abort.
Tracing the route to 192.168.2.1

 1  10.1.2.2 42 msec  41 msec
 2  10.2.3.3 [MPLS: Label 24001 Exp 0] 82 msec  25 msec
 3  10.3.4.4 39 msec  61 msec
 4  10.4.5.5 22 msec  *

Following the same echo request across three links, PE1 copies the IP TTL (255 minus one, 254) into the label, and P1 decrements the label TTL before removing the label. The IP TTL on P1 - PE2 is 253.

TTL of CE1's echo request (STEP 0)
$ tshark -r - -Y 'icmp.type==8 && ip.src==192.168.1.1' -T fields -e frame.number -e mpls.label -e mpls.ttl -e ip.ttl \
    < mpls-ttl-mtu-step0-ce1pe1.pcap
5			255
$ tshark -r - -Y 'icmp.type==8 && ip.src==192.168.1.1' -T fields -e frame.number -e mpls.label -e mpls.ttl -e ip.ttl \
    < mpls-ttl-mtu-step0-pe1p1.pcap
55	24001	254	254
$ tshark -r - -Y 'icmp.type==8 && ip.src==192.168.1.1' -T fields -e frame.number -e mpls.label -e mpls.ttl -e ip.ttl \
    < mpls-ttl-mtu-step0-p1pe2.pcap
55			253

mpls ip-ttl-propagate disable goes on PE1.

Configuration committed on PE1 in STEP 3
RP/0/RP0/CPU0:PE1#show configuration commit changes last 1
Sun Oct  4 04:25:17.409 UTC
!! Building configuration...
!! IOS XR Configuration 26.1.1
mpls ip-ttl-propagate disable
end

The label TTL now starts at 255 and never runs out at P1. P1 sends no ICMP for the second probe, and the traceroute shrinks to three hops.

CE1 traceroute (STEP 3)
RP/0/RP0/CPU0:CE1#traceroute 192.168.2.1 source 192.168.1.1 timeout 1 probe 2 maxttl 5
Sun Oct  4 04:32:01.652 UTC

Type escape sequence to abort.
Tracing the route to 192.168.2.1

 1  10.1.2.2 18 msec  28 msec
 2  10.3.4.4 39 msec  26 msec
 3  10.4.5.5 25 msec  *

TTL of CE1's echo request (STEP 3)
$ tshark -r - -Y 'icmp.type==8 && ip.src==192.168.1.1' -T fields -e frame.number -e mpls.label -e mpls.ttl -e ip.ttl \
    < mpls-ttl-mtu-step3-ce1pe1.pcap
3			255
$ tshark -r - -Y 'icmp.type==8 && ip.src==192.168.1.1' -T fields -e frame.number -e mpls.label -e mpls.ttl -e ip.ttl \
    < mpls-ttl-mtu-step3-pe1p1.pcap
4	24001	254	254
$ tshark -r - -Y 'icmp.type==8 && ip.src==192.168.1.1' -T fields -e frame.number -e mpls.label -e mpls.ttl -e ip.ttl \
    < mpls-ttl-mtu-step3-p1pe2.pcap
2			253

The IP TTL on P1 - PE2 is 254, decremented only by PE1. When P1 removes the label, it leaves the smaller of the label TTL (254) and the IP TTL (254) in the IP header.

Only the ingress PE1 was configured, so in the reverse direction from CE2 to CE1 the ingress is PE2 and P1 (10.3.4.3) stays visible.

CE2 traceroute (STEP 3)
RP/0/RP0/CPU0:CE2#traceroute 192.168.1.1 source 192.168.2.1 timeout 1 probe 2 maxttl 5
Sun Oct  4 04:32:58.802 UTC

Type escape sequence to abort.
Tracing the route to 192.168.1.1

 1  10.4.5.4 19 msec  21 msec
 2  10.3.4.3 [MPLS: Label 24000 Exp 0] 27 msec  50 msec
 3  10.2.3.2 33 msec  31 msec
 4  10.1.2.1 20 msec  *

Reverting

Removing the STEP 1 to 3 configuration in STEP 4 brings the DF limit from CE3 to CE4 back to 1492.

CE3 to CE4 (L3VPN, two labels, STEP 4)
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1492 df-bit count 5 timeout 1
Sun Oct  4 04:47:58.045 UTC
Type escape sequence to abort.
Sending 5, 1492-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 10/12/15 ms
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1493 df-bit count 5 timeout 1
Sun Oct  4 04:47:59.216 UTC
Type escape sequence to abort.
Sending 5, 1493-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
.....
Success rate is 0 percent (0/5)
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1500 df-bit count 5 timeout 1
Sun Oct  4 04:48:05.385 UTC
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
.....
Success rate is 0 percent (0/5)
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 size 1500 count 5 timeout 1
Sun Oct  4 04:48:11.259 UTC
Type escape sequence to abort.
Sending 5, 1500-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 24/31/45 ms

Reachability

Whatever the MTU and TTL settings, pings of normal size arrive in every STEP.

CE1 to CE2 (global, STEP 0)
RP/0/RP0/CPU0:CE1#ping 192.168.2.1 source 192.168.1.1 count 20 timeout 1
Mon Sep 28 12:40:32.886 UTC
Type escape sequence to abort.
Sending 20, 100-byte ICMP Echos to 192.168.2.1 timeout is 1 seconds:
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (20/20), round-trip min/avg/max = 18/20/23 ms

CE3 to CE4 (L3VPN, STEP 0)
RP/0/RP0/CPU0:CE3#ping 192.168.4.1 source 192.168.3.1 count 20 timeout 1
Mon Sep 28 12:42:12.772 UTC
Type escape sequence to abort.
Sending 20, 100-byte ICMP Echos to 192.168.4.1 timeout is 1 seconds:
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (20/20), round-trip min/avg/max = 18/18/20 ms

Verification config and show output

For every STEP the following files were collected from all seven routers, one file per router. The verification config is the ..._run.txt (the final state is the one from STEP 4). Each STEP was collected in the order “clear the counters, ping, then show”.

File	Contents
`..._clear.txt`	The counters cleared at the start of the STEP (`clear counters interface`, plus `clear mpls forwarding counters` on the PEs and P)
`..._ping.txt`	`ping` and `traceroute`. The CEs send DF pings of several sizes (CE1: 1496 / 1497 / 1500; CE3: 1492 / 1493 / 1500 and 1500 without DF). PE1 sends 1504 / 1505 / 9000 in the global table. The 1500 / 1501 pings PE1 sends from its VRF get no reply in any STEP, because the source 10.2.6.0/24 is not in CE4’s routing table (unrelated to the MTU)
`..._show.txt`	`show version` / `show interface description` / `show route` / OSPF / `show interfaces` / `show ipv4 interface` / LDP / `show mpls forwarding` / `show cef` and more; the PEs add BGP and VPN output
`..._log.txt`	`show logging` limited to the range of that STEP
`..._run.txt`	`show running-config` at that STEP (that is, the verification config for the STEP)
`..._trace.txt`	PE1 / P1 / PE2 only: `show mpls ldp trace binding` / `peer` and `show mpls lsd trace`
`..._commit.cfg`	The configuration actually committed in that STEP (only the routers that were changed)

STEP 0: Initial state (MPLS-side MTU 1514, TTL propagation on)

Router	show	ping	clear	syslog	running-config	trace	committed config	outage ping
CE1	show	ping	clear	log	run	—	—	—
PE1	show	ping	clear	log	run	trace	—	—
P1	show	—	clear	log	run	trace	—	—
PE2	show	ping	clear	log	run	trace	—	—
CE2	show	ping	clear	log	run	—	—	—
CE3	show	ping	clear	log	run	—	—	—
CE4	show	ping	clear	log	run	—	—	—

STEP 1: mtu 1522 on the four MPLS-side interfaces

Router	show	ping	clear	syslog	running-config	trace	committed config	outage ping
CE1	show	ping	clear	log	run	—	—	outage
PE1	show	ping	clear	log	run	trace	cfg	—
P1	show	—	clear	log	run	trace	cfg	—
PE2	show	ping	clear	log	run	trace	cfg	—
CE2	show	ping	clear	log	run	—	—	—
CE3	show	ping	clear	log	run	—	—	—
CE4	show	ping	clear	log	run	—	—	—

STEP 2: mtu 9216 on the four MPLS-side interfaces

Router	show	ping	clear	syslog	running-config	trace	committed config	outage ping
CE1	show	ping	clear	log	run	—	—	outage
PE1	show	ping	clear	log	run	trace	cfg	—
P1	show	—	clear	log	run	trace	cfg	—
PE2	show	ping	clear	log	run	trace	cfg	—
CE2	show	ping	clear	log	run	—	—	—
CE3	show	ping	clear	log	run	—	—	—
CE4	show	ping	clear	log	run	—	—	—

STEP 3: mpls ip-ttl-propagate disable on PE1

Router	show	ping	clear	syslog	running-config	trace	committed config	outage ping
CE1	show	ping	clear	log	run	—	—	outage
PE1	show	ping	clear	log	run	trace	cfg	—
P1	show	—	clear	log	run	trace	—	—
PE2	show	ping	clear	log	run	trace	—	—
CE2	show	ping	clear	log	run	—	—	—
CE3	show	ping	clear	log	run	—	—	—
CE4	show	ping	clear	log	run	—	—	—

STEP 4: Remove STEP 1 to 3 (final state)

Router	show	ping	clear	syslog	running-config	trace	committed config	outage ping
CE1	show	ping	clear	log	run	—	—	—
PE1	show	ping	clear	log	run	trace	cfg	—
P1	show	—	clear	log	run	trace	cfg	—
PE2	show	ping	clear	log	run	trace	cfg	—
CE2	show	ping	clear	log	run	—	—	—
CE3	show	ping	clear	log	run	—	—	—
CE4	show	ping	clear	log	run	—	—	—

Packet captures were taken per STEP.

STEP	CE1 - PE1	CE3 - PE1	PE1 - P1	P1 - PE2
0	pcap	pcap	pcap	pcap
1	pcap	pcap	pcap	pcap
2	pcap	pcap	pcap	pcap
3	pcap	pcap	pcap	pcap
4	pcap	pcap	pcap	pcap

References

RFC	Title	Summary
RFC 3443	Time To Live (TTL) Processing in Multi-Protocol Label Switching (MPLS) Networks	The uniform (3.1), short pipe (3.2) and pipe (3.3) models.
RFC 3032	MPLS Label Stack Encoding	The 32 bits of a label stack entry and the TTL field (section 2.4); fragmentation and Path MTU Discovery (section 3).

Book: Luc De Ghein, MPLS Fundamentals (Cisco Press, 2006), Chapter 3