MPLS QoS (EXP and Uniform / Pipe / Short-Pipe)

MPLS QoS (EXP and Uniform / Pipe / Short-Pipe)

An MPLS label carries a 3-bit EXP field, written by the router that pushes the label. It is separate from the IP DSCP and only has meaning while the label is on the packet, so an operator can run its own QoS on EXP without touching the customer DSCP. This article explains where EXP sits, what happens by default, how the ingress PE and a transit P router rewrite it, and the three egress PE behaviours (uniform / pipe / short-pipe), then confirms all of it with captures from an IOS XR (XRv9000) lab.

Label format and PHP are covered in MPLS Labels and the Label Stack, and the two-label stack of an L3VPN in MPLS Label Operations (push / swap / pop).

The 3 bits of EXP

A label is attached to a packet as a 4-byte (32-bit) label stack entry, made up of a 20-bit Label (the label value), 3 bits of EXP, 1 bit of S (bottom of stack) and 8 bits of TTL.

RFC 3032 defined this field as EXP (Experimental Use) and RFC 5462 renamed it TC (Traffic Class). Neither its size nor its position changed. IOS XR configuration (set mpls experimental), the show column (Exp) and the Wireshark field (MPLS Experimental Bits) all still say EXP, so this article calls it EXP.

The standard does not assign meanings to the values. Which of 0 to 7 maps to which service is up to the operator. With no configuration, IP precedence (the top 3 bits of DSCP) is copied into EXP. DSCP 46 (EF) is precedence 5, so it becomes EXP 5; DSCP 24 (CS3) becomes EXP 3 (observed on the XRv9000 26.1.1 used for this article and on the XRd 26.1.1 used for the CEs).

imposition and topmost

IOS XR has two commands for writing EXP, and they act on different labels.

An L3VPN stacks two labels (outer LDP, inner VPN), so the difference shows up directly. imposition sets both to the same value; topmost changes only the outer one. When PHP pops the outer label, the value written by topmost disappears with it and the inner value becomes visible.

The three egress PE modes

RFC 3270 defines three tunnel behaviours for carrying DiffServ over MPLS. They differ in what the egress PE looks at to pick a queue and whether it rewrites the customer DSCP.

Uniform reflects the treatment inside the network back to the customer, pipe keeps it inside the network, and short-pipe defers to the customer value at the egress. IOS XR does not name the modes: the way you write the policy-map decides which one you get. In this lab, copying EXP into qos-group and using set dscp on output behaved as uniform, using qos-group only to pick a queue behaved as pipe, and matching on match dscp at the customer-facing output behaved as short-pipe.

Lab setup

Five routers: CE1 - PE1 - P1 - PE2 - CE2. The core runs OSPF + LDP, the PEs carry VRF CUST-A over iBGP vpnv4, and each CE peers with its PE over eBGP. A packet from CE1 to CE2 gets two labels at PE1, and P1 pops the outer one with PHP, so the P1 - PE2 segment carries a single label.

PE1, P1 and PE2 are XRv9000; CE1 and CE2 are XRd. XRd (Control Plane) does not install service policies in its data plane, so QoS is verified on XRv9000. CE1 sends pings with the type option to vary ToS, and the results are compared for DSCP 0, 24 and 46.

Overview of the steps

Default behaviour (STEP 0)

Nothing is configured. The traceroute from CE1 to CE2 shows two labels and Exp 0.

RP/0/RP0/CPU0:CE1#traceroute 192.168.2.1 source 192.168.1.1 timeout 1 probe 2 maxttl 5
Sun Oct  4 10:36:53.929 UTC

Type escape sequence to abort.
Tracing the route to 192.168.2.1

 1  10.11.101.11 71 msec  15 msec
 2  10.1.11.1 [MPLS: Labels 24001/24003 Exp 0] 88 msec  30 msec
 3  10.1.22.22 [MPLS: Label 24003 Exp 0] 70 msec  30 msec
 4  10.22.102.102 23 msec  *

With DSCP 46 the label EXP is 5. This is No.172 of the PE1 - P1 capture.

MultiProtocol Label Switching Header, Label: 24001, Exp: 5, S: 0, TTL: 254
    0000 0101 1101 1100 0001 .... .... .... = MPLS Label: 24001 (0x05dc1)
    .... .... .... .... .... 101. .... .... = MPLS Experimental Bits: 5
    .... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
MultiProtocol Label Switching Header, Label: 24003, Exp: 5, S: 1, TTL: 254
    0000 0101 1101 1100 0011 .... .... .... = MPLS Label: 24003 (0x05dc3)
    .... .... .... .... .... 101. .... .... = MPLS Experimental Bits: 5
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254

DSCP 0 gave EXP 0 and DSCP 24 gave EXP 3: precedence is copied as is. The IP DSCP is unchanged on every segment.

Setting EXP at the ingress PE (STEP 1 and 2)

A policy-map that writes EXP 5 is applied to the input of the customer-facing interface on PE1.

policy-map PE1-CE-IN
 class class-default
  set mpls experimental imposition 5
 !
 end-policy-map
!
interface GigabitEthernet0/0/0/0
 service-policy input PE1-CE-IN
!

Even a packet with DSCP 0 now carries EXP 5 in both labels, while its DSCP stays 0.

MultiProtocol Label Switching Header, Label: 24001, Exp: 5, S: 0, TTL: 254
    0000 0101 1101 1100 0001 .... .... .... = MPLS Label: 24001 (0x05dc1)
    .... .... .... .... .... 101. .... .... = MPLS Experimental Bits: 5
    .... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
MultiProtocol Label Switching Header, Label: 24003, Exp: 5, S: 1, TTL: 254

STEP 2 adds set mpls experimental topmost 6 on the core-side output. On PE1 - P1 the outer label is 6 and the inner one is still 5.

MultiProtocol Label Switching Header, Label: 24001, Exp: 6, S: 0, TTL: 254
    0000 0101 1101 1100 0001 .... .... .... = MPLS Label: 24001 (0x05dc1)
    .... .... .... .... .... 110. .... .... = MPLS Experimental Bits: 6
    .... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
MultiProtocol Label Switching Header, Label: 24003, Exp: 5, S: 1, TTL: 254

After P1 pops the outer label, the P1 - PE2 segment shows EXP 5. The 6 written by topmost is gone with the label that carried it.

MultiProtocol Label Switching Header, Label: 24003, Exp: 5, S: 1, TTL: 253
    0000 0101 1101 1100 0011 .... .... .... = MPLS Label: 24003 (0x05dc3)
    .... .... .... .... .... 101. .... .... = MPLS Experimental Bits: 5
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1101 = MPLS TTL: 253

Rewriting EXP on a P router (STEP 3)

A transit router can rewrite it too. On P1, input traffic matching match mpls experimental topmost 5 is put into qos-group 5, and on output that group gets set mpls experimental topmost 2. A qos-group is an internal marker and never appears on the wire.

The P1 - PE2 segment now carries EXP 2: the output policy acts on the VPN label that remains after PHP.

MultiProtocol Label Switching Header, Label: 24003, Exp: 2, S: 1, TTL: 253
    0000 0101 1101 1100 0011 .... .... .... = MPLS Label: 24003 (0x05dc3)
    .... .... .... .... .... 010. .... .... = MPLS Experimental Bits: 2
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1101 = MPLS TTL: 253

The three egress modes (STEP 4, 5 and 6)

STEP 4 to 6 keep imposition 5 on the ingress PE and only swap the configuration on PE2.

Uniform: reflect EXP into DSCP (STEP 4)

On the core side, match mpls experimental topmost 5 is copied into qos-group 5; on the customer-facing output, that group gets set dscp cs5. Every packet leaving PE2 towards CE2 now carries DSCP 40 (CS5), whether it started as 0, 24 or 46.

    Differentiated Services Field: 0xa0 (DSCP: CS5, ECN: Not-ECT)
        1010 00.. = Differentiated Services Codepoint: Class Selector 5 (40)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 100

Pipe: pick the queue from EXP, leave DSCP alone (STEP 5)

The customer-facing output becomes a parent shape average 300 kbps with a child class for qos-group 5 given bandwidth remaining percent 80. There is no set dscp.

Sending 1000 packets each of DSCP 46 (ToS 184) and DSCP 0 from CE1, four seconds apart, both got through 1000/1000 with an average RTT of 70 ms. They land in the same EXP-selected queue, so there is no difference between them. The DSCP values arrive unchanged as 46 and 0.

GigabitEthernet0/0/0/1 output: PE2-CE-PIPE

Class class-default
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                1999/2834582              301
    Transmitted         :                1999/2834582              301
    Total Dropped       :                   0/0                    0

  Policy PE2-CE-PIPE-CHILD Class QG5
    Classification statistics          (packets/bytes)     (rate - kbps)
      Matched             :                1999/2834582              301
      Transmitted         :                1999/2834582              301
      Total Dropped       :                   0/0                    0
    Queueing statistics
      Queue ID                             : 32
      High watermark                       : N/A
      Inst-queue-len                       : N/A
      Avg-queue-len                        : N/A
      Taildropped(packets/bytes)           : 0/0
      Queue(conform)      :                1999/2834582              301
      RED random drops(packets/bytes)      : 0/0


  Policy PE2-CE-PIPE-CHILD Class class-default
    Classification statistics          (packets/bytes)     (rate - kbps)
      Matched             :                   0/0                    0

Both loads (1000 packets each) landed in the QG5 class and left at about 301 kbps, as the shaper allows; the count reads 1999 because one packet was still in flight when the counters were read. The child class-default stays at 0, so only the class picked by EXP is used.

Short-pipe: pick the queue from the customer DSCP (STEP 6)

The core-side policy is removed and the customer-facing output becomes a class matching match dscp ef with priority level 1 and police rate 100 kbps. Classification now follows the customer DSCP instead of the operator EXP.

Under the same load, DSCP 46 got 667/1000 with an average RTT of 9 ms, while DSCP 0 got 1000/1000 with 37 ms. EF goes into the priority queue, so its latency is low, and the policer drops what exceeds 100 kbps.

GigabitEthernet0/0/0/1 output: PE2-CE-SHORTPIPE

Class class-default
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                2000/2836000              30
    Transmitted         :                1667/2363806              21
    Total Dropped       :                 333/472194               9

  Policy PE2-CE-SHORTPIPE-CHILD Class EF
    Classification statistics          (packets/bytes)     (rate - kbps)
      Matched             :                1000/1418000              30
      Transmitted         :                 667/945806               21
      Total Dropped       :                 333/472194               9
    Policing statistics                (packets/bytes)     (rate - kbps)
      Policed(conform)    :                 667/945806               21
      Policed(exceed)     :                 333/472194               9

The DSCP is not rewritten. Packets leaving PE2 towards CE2 still carry 46.

    Differentiated Services Field: 0xb8 (DSCP: EF, ECN: Not-ECT)
        1011 10.. = Differentiated Services Codepoint: Expedited Forwarding (46)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 1400

Design notes

• Confusing imposition with topmost gives different results depending on how many labels are on the packet. Where a stack of two is normal, as in an L3VPN, decide first whether you want every label set at the ingress or only the outer one changed
• The egress PE classifies on different fields on each side. The core-side input still sees labels, so it uses match mpls experimental; the customer-facing output no longer has them, so it uses match dscp. Carry EXP across with a qos-group
• The counters in show policy-map interface only count what matched that policy on that interface. The traffic PE2 forwards after popping the VPN label does not appear in show mpls forwarding (the Unlabelled row stays at 0), so use a capture or show interface instead

Verification configs and show output

For every STEP the following files were collected per router. The verification config is the ..._run.txt file (the final state is the one from the last STEP).

The show policy-map interface in ..._show.txt and the one in ..._pmap-after.txt show different numbers. The former only counts the few dozen ping packets of that STEP, sent after the counters were cleared; the latter counts the load (500 or 1000 packets). Where the text needs the load figures, it quotes ..._pmap-after.txt.

Counters are cleared, then the pings are sent, and only then is the show output taken, so the numbers in each STEP belong to that STEP alone.

STEP 0: no QoS configuration

STEP 1: set EXP 5 on pushed labels at PE1 (imposition)

STEP 2: set EXP 6 on the outer label on the core side (topmost)

STEP 3: rewrite EXP on the P router (via qos-group)

STEP 4: reflect EXP into DSCP at the egress PE (uniform)

STEP 5: pick the queue from EXP at the egress PE (pipe)

STEP 6: pick the queue from the customer DSCP (short-pipe, final state) (final state)

Loads sent from CE1 (ping ... size 1400 interval 10). The policy-map counters right after each load are the policy-map after load column above.

Packet captures were taken per STEP on all four links.

References

Related articles

• What Is MPLS
• MPLS Labels and the Label Stack
• MPLS Label Operations (push / swap / pop) and PHP
• MPLS TTL Processing and MTU
• What Is LDP
• LDP Label Distribution Modes and Label Spaces
• LDP-IGP Synchronization and LDP Session Protection
• LDP label advertisement control (filtering)
• LDP session authentication (TCP MD5)
• MPLS OAM (LSP ping and LSP traceroute)
• What Is MPLS TE (RSVP-TE)
• MPLS TE CSPF and Path Constraints (Bandwidth, Affinity, TE Metric)
• Steering Traffic into MPLS TE Tunnels
• MPLS TE Fast Reroute (Link and Node Protection)
• MPLS TE Fast Reroute (Auto-Tunnel Backup and SRLG)
• MPLS QoS (EXP and Uniform / Pipe / Short-Pipe)
• MPLS QoS Rate Control and Queuing (policer / shaper / priority)