MPLS TTL Processing and MTU

MPLS TTL Processing and MTU

Eight of the 32 bits in an MPLS label carry a TTL. Whether the ingress LSR copies the IP TTL into that field decides if the core routers show up in a traceroute. Separately, each label costs 4 bytes, so the IP packet that fits inside a given link MTU shrinks by that much. This article explains both points and then confirms them in an IOS XR (XRd) lab. The label format is covered in MPLS Labels and the Label Stack.

The label TTL and the IP TTL

The MPLS TTL is a separate field from the IP TTL. While a packet is forwarded with a label, an LSR decrements only the TTL of the top label and never touches the IP header inside.

What the ingress LSR puts in that field splits the behaviour in two. RFC 3443 organises this as the uniform model and the pipe / short pipe models.

Under the uniform model the IP TTL drops by the number of core hops. A traceroute probe raises the TTL one at a time, so the core LSRs answer with a TTL-exceeded ICMP and P1 and P2 appear as hops. With propagation off the label TTL starts at 255 and never runs out inside the core. The core sends no ICMP, and from CE1 the span between the PEs looks like a single hop.

On IOS XR the switch is mpls ip-ttl-propagate disable. It is configured only on the PE that acts as the ingress, and it takes effect per direction. With it on PE1 alone, a traceroute from CE1 to CE2 hides the core while a traceroute from CE2 to CE1 still shows it.

Operators who do not want to expose the core topology to customers turn propagation off. The trade-off is that troubleshooting a path becomes harder.

A label consumes MTU

Each label is 4 bytes. The link MTU does not change, so the IP packet that fits shrinks by the size of the label stack.

The mtu of an IOS XR interface is the L2 MTU and includes the 14-byte Ethernet header. The default of 1514 means an IP MTU of 1500. The MPLS MTU is not configured separately; it is the L2 MTU minus 14.

With M as the L2 MTU of the core links, the largest IP packet that gets through is as follows.

When a packet above the limit reaches the ingress PE, the DF bit in the IP header decides what happens. Without DF the PE fragments it and forwards the pieces. With DF the packet cannot be forwarded, so it is dropped and an ICMP Type 3 Code 4 (Fragmentation Needed) goes back to the sender. The MTU advertised in that message already has the label subtracted, so Path MTU Discovery converges on a workable size.

In production the core MTU is raised above 1500 so customers can pass a full 1500 bytes untouched. A frame that exceeds 1500 by a small amount is called a baby giant, and the switches and routers in the core need to be configured to accept it.

Lab verification

Lab setup

CE1 — PE1 — P1 — P2 — PE2 — CE2 are connected in a line. PE1 / P1 / P2 / PE2 in AS 65001 form the MPLS network, distributing routes with OSPF (area 0, network point-to-point on every link) and labels with LDP. CE1 and CE2 exchange routes with the PEs over eBGP, and PE1 and PE2 run iBGP between their Loopback0 addresses with next-hop-self. CE3 and CE4 are customers in VRF CUST-A, used for the two-label case.

The three links CE1 - PE1, PE1 - P1 and P2 - PE2 are captured to read the TTL values, the frame lengths and the ICMP contents.

Verification steps

STEP 0: Initial state (core MTU 1514, TTL propagation on)

A traceroute from CE1 to CE2. The core routers P1 (10.2.3.3) and P2 (10.3.4.4) appear as hops.

 1  10.1.2.2 5 msec  4 msec  4 msec 
 2  10.2.3.3 [MPLS: Label 24003 Exp 0] 13 msec  11 msec  13 msec 
 3  10.3.4.4 [MPLS: Label 24003 Exp 0] 12 msec  12 msec  12 msec 
 4  10.4.5.5 36 msec  13 msec  12 msec 
 5  10.5.6.6 73 msec  *  17 msec 

Reading the TTLs out of the captures shows the IP TTL being copied into the label at the ingress and the core hops being written back into the IP header at the egress.

PE1 copies 254, one less than 255, into the label. P1 and P2 each decrement the label TTL. When P2 pops the label, that value of 252 is written back into the IP header.

The MTU limit is measured with a DF-set ping. 1496 bytes gets through and 1497 does not.

RP/0/RP0/CPU0:CE1#ping 192.168.6.1 source 192.168.1.1 size 1496 df-bit count 5
Thu Sep 10 07:53:32.159 UTC
Type escape sequence to abort.
Sending 5, 1496-byte ICMP Echos to 192.168.6.1 timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 15/16/22 ms
RP/0/RP0/CPU0:CE1#ping 192.168.6.1 source 192.168.1.1 size 1497 df-bit count 5
Thu Sep 10 07:53:32.926 UTC
Type escape sequence to abort.
Sending 5, 1497-byte ICMP Echos to 192.168.6.1 timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

The CLI only prints dots, but the capture on CE1 - PE1 holds the ICMP that PE1 sent back.

In VRF CUST-A there are two labels, so the limit is another 4 bytes lower at 1492 bytes.

STEP 1: Lower the MTU of the four core links to 1400

mtu 1400 goes on the four core links. Because the mtu of an IOS XR interface is the L2 MTU, the IP and MPLS MTU become 1386.

  MTU 1400 bytes, BW 1000000 Kbit (Max: 1000000 Kbit)

The limit drops to 1382 with one label and 1378 with two. Clearing DF lets the same size through.

RP/0/RP0/CPU0:CE1#ping 192.168.6.1 source 192.168.1.1 size 1382 df-bit count 5
Thu Sep 10 12:37:40.256 UTC
Type escape sequence to abort.
Sending 5, 1382-byte ICMP Echos to 192.168.6.1 timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 13/14/18 ms
RP/0/RP0/CPU0:CE1#ping 192.168.6.1 source 192.168.1.1 size 1383 df-bit count 5
Thu Sep 10 12:37:41.222 UTC
Type escape sequence to abort.
Sending 5, 1383-byte ICMP Echos to 192.168.6.1 timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
RP/0/RP0/CPU0:CE1#ping 192.168.6.1 source 192.168.1.1 size 1383 count 5
Thu Sep 10 12:37:52.171 UTC
Type escape sequence to abort.
Sending 5, 1383-byte ICMP Echos to 192.168.6.1 timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 14/16/21 ms

Putting the two MTUs side by side makes the 4 bytes per label visible in the numbers.

The MTU carried in the ICMP changes to 1382 as well. Because the label is already subtracted, the sender converges on a workable size without knowing that labels exist.

On the PE1 - P1 capture the largest frame is 1400 bytes. The L2 MTU is the ceiling.

STEP 2: mpls ip-ttl-propagate disable on PE1

Only PE1 is configured. The traceroute from CE1 to CE2 falls from five hops to three and the core disappears.

 1  10.1.2.2 6 msec  4 msec  4 msec 
 2  10.4.5.5 12 msec  10 msec  10 msec 
 3  10.5.6.6 14 msec  *  16 msec 

A traceroute from CE2 to CE1 taken at the same time still shows five hops, because PE2, the ingress for that direction, is untouched. Only the configured direction changes.

 1  10.5.6.5 5 msec  4 msec  4 msec 
 2  10.4.5.4 [MPLS: Label 24002 Exp 0] 11 msec  11 msec  12 msec 
 3  10.3.4.3 [MPLS: Label 24002 Exp 0] 12 msec  12 msec  11 msec 
 4  10.2.3.2 12 msec  12 msec  12 msec 
 5  10.1.2.1 17 msec  *  16 msec 

CE3 to CE4 in the VRF drops to three hops too. The Label 24005 still shown on the second hop is the VPN label; PHP removes only the transport label on top.

 1  10.2.7.2 6 msec  5 msec  5 msec 
 2  10.4.5.5 [MPLS: Label 24005 Exp 0] 52 msec  21 msec  13 msec 
 3  10.5.8.8 16 msec  *  16 msec 

The TTLs in the captures differ from STEP 0. The label TTL starts at 255 and the IP TTL does not drop across the core.

With propagation on the packet arrives at the egress with 252, without it with 253. The difference of one is the two core hops less the single ingress hop. The core never touches the IP TTL, so the customer sees one hop between the PEs.

The MTU limit stays at 1382, the same as STEP 1. The TTL setting has nothing to do with packet size.

STEP 3: Remove STEP 1 and STEP 2 (final state)

With both removed, the MTU is back to 1514, the traceroute back to five hops and the limits back to 1496 and 1492.

 1  10.1.2.2 7 msec  4 msec  4 msec 
 2  10.2.3.3 [MPLS: Label 24003 Exp 0] 15 msec  12 msec  14 msec 
 3  10.3.4.4 [MPLS: Label 24003 Exp 0] 16 msec  12 msec  13 msec 
 4  10.4.5.5 16 msec  14 msec  14 msec 
 5  10.5.6.6 19 msec  *  16 msec 

Changing mtu bounces the interface, so the OSPF adjacency is rebuilt. That is why Neighbor Down: interface down or detached appears in the ..._log.txt of STEP 1 and STEP 3. Changing only one end leaves the MTUs mismatched and OSPF stalls in EXSTART or EXCHANGE, so change both ends together.

A note on baby giants

Raising the core MTU above 1500 so a customer’s full 1500 bytes passes untouched was not observed in this lab. XRd accepts mtu 1600 and a 1518-byte frame does leave PE1, but it never reaches P1, and P1 counts no giants and no input errors. The virtual link in CML tops out at 1514 bytes, which no router-side configuration can raise.

Captures

Each STEP captures the three links CE1 - PE1, PE1 - P1 and P2 - PE2 at the same time, with no filter because the traffic contains MPLS.

STEP 0 CE1 - PE1

STEP 0 PE1 - P1

STEP 0 P2 - PE2

STEP 1 CE1 - PE1

STEP 1 PE1 - P1

STEP 1 P2 - PE2

STEP 2 CE1 - PE1

STEP 2 PE1 - P1

STEP 2 P2 - PE2

STEP 3 CE1 - PE1

STEP 3 PE1 - P1

STEP 3 P2 - PE2

Verification config and show output

The following file types were collected from all eight routers at every STEP, split per router. The verification config is these ..._run.txt files (the final state is the one from the last STEP).

STEP 0: Initial state (core MTU 1514, TTL propagation on)

STEP 1: Lower the MTU of the four core links to 1400

STEP 2: mpls ip-ttl-propagate disable on PE1

STEP 3: Remove STEP 1 and STEP 2 (final state)

References

Book: Luc De Ghein, MPLS Fundamentals (Cisco Press, 2006), Chapter 3

Related articles

• What Is MPLS
• MPLS Labels and the Label Stack
• MPLS Label Operations (push / swap / pop) and PHP
• MPLS TTL Processing and MTU
• What Is LDP
• LDP Label Distribution Modes and Label Spaces