MPLS QoS (Traffic Class and Uniform / Pipe / Short-Pipe)

MPLS QoS (Traffic Class and Uniform / Pipe / Short-Pipe)

An MPLS label carries a 3-bit TC (Traffic Class) field, written by the router that pushes the label. It is separate from the IP DSCP and only has meaning while the label is on the packet, so an operator can run its own QoS on TC without touching the customer DSCP. This article explains where TC sits, what happens by default, how the ingress PE and a transit P router rewrite it, and the three egress PE behaviours (uniform / pipe / short-pipe), then confirms all of it with captures from an IOS XR (XRv9000) lab.

Label format and PHP are covered in MPLS Labels and the Label Stack, and the two-label stack of an L3VPN in MPLS Label Operations (push / swap / pop).

The 3 bits of TC (formerly EXP)

A label is 32 bits: 20 bits of label value, 3 bits of TC, 1 bit of S (bottom of stack) and 8 bits of TTL. RFC 5462 renamed this field, which used to be called EXP (Experimental Use). Neither its size nor its position changed, and the IOS XR command is still mpls experimental.

The standard does not assign meanings to the values. Which of 0 to 7 maps to which service is up to the operator. With no configuration, IP precedence (the top 3 bits of DSCP) is copied into TC. DSCP 46 (EF) is precedence 5, so it becomes TC 5; DSCP 24 (CS3) becomes TC 3 (observed on the XRv9000 26.1.1 used for this article and on the XRd 26.1.1 used for the CEs).

imposition and topmost

IOS XR has two commands for writing TC, and they act on different labels.

An L3VPN stacks two labels (outer LDP, inner VPN), so the difference shows up directly. imposition sets both to the same value; topmost changes only the outer one. When PHP pops the outer label, the value written by topmost disappears with it and the inner value becomes visible.

The three egress PE modes

RFC 3270 defines three tunnel behaviours for carrying DiffServ over MPLS. They differ in what the egress PE looks at to pick a queue and whether it rewrites the customer DSCP.

Uniform reflects the treatment inside the network back to the customer, pipe keeps it inside the network, and short-pipe defers to the customer value at the egress. IOS XR does not name the modes: the way you write the policy-map decides which one you get. In this lab, copying TC into qos-group and using set dscp on output behaved as uniform, using qos-group only to pick a queue behaved as pipe, and matching on match dscp at the customer-facing output behaved as short-pipe.

Lab setup

Five routers: CE1 - PE1 - P1 - PE2 - CE2. The core runs OSPF + LDP, the PEs carry VRF CUST-A over iBGP vpnv4, and each CE peers with its PE over eBGP. A packet from CE1 to CE2 gets two labels at PE1, and P1 pops the outer one with PHP, so the P1 - PE2 segment carries a single label.

PE1, P1 and PE2 are XRv9000; CE1 and CE2 are XRd. XRd (Control Plane) does not install service policies in its data plane, so QoS is verified on XRv9000. CE1 sends pings with the type option to vary ToS, and the results are compared for DSCP 0, 24 and 46.

Overview of the steps

Default behaviour (STEP 0)

Nothing is configured. The traceroute from CE1 to CE2 shows two labels and Exp 0.

RP/0/RP0/CPU0:CE1#traceroute 192.168.2.1 source 192.168.1.1
Wed Sep 16 07:22:52.607 UTC

Type escape sequence to abort.
Tracing the route to 192.168.2.1

 1  10.11.101.11 19 msec  18 msec  53 msec 
 2  10.1.11.1 [MPLS: Labels 24001/24003 Exp 0] 81 msec  44 msec  54 msec 

With DSCP 46 the label TC is 5. This is No.445 of the PE1 - P1 capture.

MultiProtocol Label Switching Header, Label: 24001, Exp: 5, S: 0, TTL: 254
    0000 0101 1101 1100 0001 .... .... .... = MPLS Label: 24001 (0x05dc1)
    .... .... .... .... .... 101. .... .... = MPLS Experimental Bits: 5
    .... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
MultiProtocol Label Switching Header, Label: 24003, Exp: 5, S: 1, TTL: 254
    0000 0101 1101 1100 0011 .... .... .... = MPLS Label: 24003 (0x05dc3)
    .... .... .... .... .... 101. .... .... = MPLS Experimental Bits: 5
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254

DSCP 0 gave TC 0 and DSCP 24 gave TC 3: precedence is copied as is. The IP DSCP is unchanged on every segment.

Setting TC at the ingress PE (STEP 1 and 2)

A policy-map that writes TC 5 is applied to the input of the customer-facing interface on PE1.

policy-map PE1-CE-IN
 class class-default
  set mpls experimental imposition 5
 ! 
 end-policy-map
! 
interface GigabitEthernet0/0/0/0
 service-policy input PE1-CE-IN
!

Even a packet with DSCP 0 now carries TC 5 in both labels, while its DSCP stays 0.

MultiProtocol Label Switching Header, Label: 24001, Exp: 5, S: 0, TTL: 254
    0000 0101 1101 1100 0001 .... .... .... = MPLS Label: 24001 (0x05dc1)
    .... .... .... .... .... 101. .... .... = MPLS Experimental Bits: 5
    .... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
MultiProtocol Label Switching Header, Label: 24003, Exp: 5, S: 1, TTL: 254

STEP 2 adds set mpls experimental topmost 6 on the core-side output. On PE1 - P1 the outer label is 6 and the inner one is still 5.

MultiProtocol Label Switching Header, Label: 24001, Exp: 6, S: 0, TTL: 254
    0000 0101 1101 1100 0001 .... .... .... = MPLS Label: 24001 (0x05dc1)
    .... .... .... .... .... 110. .... .... = MPLS Experimental Bits: 6
    .... .... .... .... .... ...0 .... .... = MPLS Bottom Of Label Stack: 0
    .... .... .... .... .... .... 1111 1110 = MPLS TTL: 254
MultiProtocol Label Switching Header, Label: 24003, Exp: 5, S: 1, TTL: 254

After P1 pops the outer label, the P1 - PE2 segment shows TC 5. The 6 written by topmost is gone with the label that carried it.

MultiProtocol Label Switching Header, Label: 24003, Exp: 5, S: 1, TTL: 253
    0000 0101 1101 1100 0011 .... .... .... = MPLS Label: 24003 (0x05dc3)
    .... .... .... .... .... 101. .... .... = MPLS Experimental Bits: 5
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1101 = MPLS TTL: 253

Rewriting TC on a P router (STEP 3)

A transit router can rewrite it too. On P1, input traffic matching match mpls experimental topmost 5 is put into qos-group 5, and on output that group gets set mpls experimental topmost 2. A qos-group is an internal marker and never appears on the wire.

The P1 - PE2 segment now carries TC 2: the output policy acts on the VPN label that remains after PHP.

MultiProtocol Label Switching Header, Label: 24003, Exp: 2, S: 1, TTL: 253
    0000 0101 1101 1100 0011 .... .... .... = MPLS Label: 24003 (0x05dc3)
    .... .... .... .... .... 010. .... .... = MPLS Experimental Bits: 2
    .... .... .... .... .... ...1 .... .... = MPLS Bottom Of Label Stack: 1
    .... .... .... .... .... .... 1111 1101 = MPLS TTL: 253

The three egress modes (STEP 4, 5 and 6)

STEP 4 to 6 keep imposition 5 on the ingress PE and only swap the configuration on PE2.

Uniform: reflect TC into DSCP (STEP 4)

On the core side, match mpls experimental topmost 5 is copied into qos-group 5; on the customer-facing output, that group gets set dscp cs5. Every packet leaving PE2 towards CE2 now carries DSCP 40 (CS5), whether it started as 0, 24 or 46.

    Differentiated Services Field: 0xa0 (DSCP: CS5, ECN: Not-ECT)
        1010 00.. = Differentiated Services Codepoint: Class Selector 5 (40)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 100

Pipe: pick the queue from TC, leave DSCP alone (STEP 5)

The customer-facing output becomes a parent shape average 300 kbps with a child class for qos-group 5 given bandwidth remaining percent 80. There is no set dscp.

Sending 1000 packets each of DSCP 46 (ToS 184) and DSCP 0 from CE1, four seconds apart, both got through 1000/1000 with an average RTT of 70 ms. They land in the same TC-selected queue, so there is no difference between them. The DSCP values arrive unchanged as 46 and 0.

GigabitEthernet0/0/0/1 output: PE2-CE-PIPE

Class class-default
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                2000/2836000              301
  Policy PE2-CE-PIPE-CHILD Class QG5
    Classification statistics          (packets/bytes)     (rate - kbps)
      Matched             :                2000/2836000              301

All 2000 packets of both flows landed in the QG5 class and left at about 301 kbps, as the shaper allows.

Short-pipe: pick the queue from the customer DSCP (STEP 6)

The core-side policy is removed and the customer-facing output becomes a class matching match dscp ef with priority level 1 and police rate 100 kbps. Classification now follows the customer DSCP instead of the operator TC.

Under the same load, DSCP 46 got 667/1000 with an average RTT of 9 ms, while DSCP 0 got 1000/1000 with 37 ms. EF goes into the priority queue, so its latency is low, and the policer drops what exceeds 100 kbps.

GigabitEthernet0/0/0/1 output: PE2-CE-SHORTPIPE

Class class-default
  Classification statistics          (packets/bytes)     (rate - kbps)
    Matched             :                2000/2836000              0
    Transmitted         :                1669/2366642              0
  Policy PE2-CE-SHORTPIPE-CHILD Class EF
    Classification statistics          (packets/bytes)     (rate - kbps)
      Matched             :                1000/1418000              0
      Transmitted         :                 669/948642               0
      Policed(exceed)     :                 331/469358               0

The DSCP is not rewritten. Packets leaving PE2 towards CE2 still carry 46.

    Differentiated Services Field: 0xb8 (DSCP: EF PHB, ECN: Not-ECT)
        1011 10.. = Differentiated Services Codepoint: Expedited Forwarding (46)
        .... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
    Total Length: 1400

Design notes

• Confusing imposition with topmost gives different results depending on how many labels are on the packet. Where a stack of two is normal, as in an L3VPN, decide first whether you want every label set at the ingress or only the outer one changed
• The egress PE classifies on different fields on each side. The core-side input still sees labels, so it uses match mpls experimental; the customer-facing output no longer has them, so it uses match dscp. Carry TC across with a qos-group
• The counters in show policy-map interface only count what matched that policy on that interface. The traffic PE2 forwards after popping the VPN label does not appear in show mpls forwarding (the Unlabelled row stays at 0), so use a capture or show interface instead

Verification configs and show output

For every STEP the following files were collected per router. The verification config is the ..._run.txt file (the final state is the one from the last STEP).

Counters are cleared, then the pings are sent, and only then is the show output taken, so the numbers in each STEP belong to that STEP alone.

STEP 0: no QoS configuration

STEP 1: set TC 5 on pushed labels at PE1 (imposition)

STEP 2: set TC 6 on the outer label on the core side (topmost)

STEP 3: rewrite TC on the P router (via qos-group)

STEP 4: reflect TC into DSCP at the egress PE (uniform)

STEP 5: pick the queue from TC at the egress PE (pipe)

The load is in step5_ce1_load_tos184.txt / step5_ce1_load_tos0.txt, and the counters taken after it in step5_pe2_pmap-after.txt.

STEP 6: pick the queue from the customer DSCP (short-pipe, final state)

The load is in step6_ce1_load_tos184.txt / step6_ce1_load_tos0.txt. The counters after the load come from step6r_pe2_pmap-after.txt, taken by reproducing the same configuration after the run, because the SSH session to PE2 dropped during the original collection (its load results are step6r_ce1_load_tos*.txt).

Packet captures were taken per STEP on all four links.

References

Related articles

• What Is MPLS
• MPLS Labels and the Label Stack
• MPLS Label Operations (push / swap / pop) and PHP
• MPLS TTL Processing and MTU
• What Is LDP
• LDP Label Distribution Modes and Label Spaces
• LDP-IGP Synchronization and LDP Session Protection
• LDP Label Advertisement Control (Filtering)
• LDP Session Authentication (TCP MD5)
• MPLS OAM (LSP Ping and LSP Traceroute)
• What Is MPLS TE (RSVP-TE)
• MPLS TE CSPF and Path Constraints (Bandwidth, Affinity, TE Metric)
• Steering Traffic into MPLS TE Tunnels
• MPLS TE Fast Reroute (Link and Node Protection)
• MPLS TE Fast Reroute (Auto-Tunnel Backup and SRLG)
• MPLS QoS (Traffic Class and Uniform / Pipe / Short-Pipe)
• MPLS QoS Rate Control and Queuing (policer / shaper / priority)