Site of Origin (SoO) in MPLS VPN

SoO marks which site a route came in from. A PE attaches it to the routes it receives from a CE, and uses it to make sure a route carrying that mark is never advertised back to that site.

RFC 4364 section 7 states what a PE does before it redistributes a VPN-IPv4 route:

Before a PE can redistribute a VPN-IPv4 route learned from a site, it must assign a Route Target attribute (see Section 4.3.1) to the route, and it may assign a Site of Origin attribute to the route.

The RT is mandatory; the SoO is optional. The two have different jobs.

Attribute	What it says	Who acts on it
RT (Route Target)	Which VRF to import into	The PE that receives the route (Route Target (RT))
SoO (Site of Origin)	Which site the route came from	The PE that advertises the route to a CE

Both travel in the same attribute, Extended Communities. SoO is the Route Origin community (Sub-Type 0x03).

The Site of Origin attribute, if used, is encoded as a Route Origin Extended Community [BGP-EXTCOMM].

The format of the attribute itself (the 8-byte structure, Type / Sub-Type, whether it is transitive) is covered in BGP Extended Community.

At a multihomed site, a route the site originated comes back

When one site connects to two PEs (multihoming), a route that the site originated can come back to the same site through the other PE. RFC 4364 section 7 gives this as the reason the attribute exists:

This attribute is needed in some cases to ensure that a route learned from a particular site via a particular PE/CE connection is not distributed back to the site through a different PE/CE connection.

Normally BGP loop detection stops it. With eBGP, a route whose AS_PATH already contains the peer’s AS number is not sent to that peer. As long as each site has its own AS number, the returning route is dropped on its own.

It stops working when every site uses the same AS number. In that design the routes of the other sites are dropped for the same reason, so as-override is configured on the PE to rewrite the customer AS number in the AS_PATH (Sites sharing one AS number in MPLS VPN). That also removes the loop protection. RFC 4364 section 7 describes SoO in exactly this context:

If a site is not in a transit VPN, note that it need not have a unique Autonomous System Number (ASN). Every CE whose site is not in a transit VPN can use the same ASN. This can be chosen from the private ASN space, and it will be stripped out by the PE. Routing loops are prevented by use of the Site of Origin attribute (see below).

The returning route beats the route inside the site

The real problem is not that the route comes back, but that the returning route wins over the path inside the site.

A CE has two ways to reach a destination at its own site: the route learned from the IGP running inside the site, and the BGP route received from the PE. In IOS XR the administrative distance is 20 for eBGP and 110 for OSPF, so BGP wins. Traffic between two routers at the same site takes a round trip through the provider network instead of the direct link.

What the egress PE does

RFC 4364 section 8 defines how a route carrying an SoO must be treated:

We add one more restriction on the distribution of routes from PE to CE: if a route’s Site of Origin attribute identifies a particular site, that route must never be redistributed to any CE at that site.

In IOS XR, site-of-origin goes under the address family of the PE-CE neighbor. One line does two things.

Direction	Behaviour
Receiving from the CE	Attaches the SoO to the route
Advertising to the CE	Removes routes that carry the same SoO

Configuring SoO in IOS XR (on the PE)
router bgp 65001
 vrf CUST-A
  neighbor 172.16.1.2
   address-family ipv4 unicast
    site-of-origin 65001:11

The value identifies a site. For a multihomed site, the same value goes on the neighbors of every PE the site connects to. The check happens on the PE that advertises the route, so if only one of the PEs has it, the route still comes back through the other one.

The lab

Site 1 is multihomed to PE1 and PE2 through two routers, CE-A1a and CE-A1b. The two are connected inside the site and run OSPF (area 0), and each CE advertises only its own Lo1 to its PE over BGP. Each learns the other’s Lo1 through OSPF. Site 2 (CE-A2) has a single link to PE3. All three customer routers are in AS 65102.

STEP	Change	What it shows
0	Initial state (no `as-override`, no SoO)	It does not come back. CE-A1b reaches `10.1.11.1/32` over the OSPF path inside the site
1	`as-override` on PE1, PE2 and PE3	It comes back. CE-A1b takes the eBGP route and traceroute goes through the provider network
2	`site-of-origin 65001:11` on the site 1 neighbors of PE1 and PE2	It stops. CE-A1b is back on the OSPF route; the VPNv4 route carries `SoO:65001:11`
3	`site-of-origin 65001:12` on the CE-A2 neighbor of PE3	A different SoO does not stop anything. The routes between sites keep flowing
4	Remove the SoO from the site 1 neighbor of PE2 only	It comes back again. The check uses the SoO configured on the outgoing neighbor
5	Put the SoO back on PE2 (final state)	Back to the state of STEP 2 and 3

In the initial state the route does not come back

What PE2 advertises to CE-A1b is only the three PE-CE subnets that the PEs originate with redistribute connected. The 10.1.11.1/32 of site 1 is not there: its AS_PATH contains AS 65102, the AS of CE-A1b, so PE2 drops it before sending.

STEP 0: what PE2 advertises to CE-A1b
RP/0/RP0/CPU0:PE2#show bgp vrf CUST-A neighbors 172.16.2.2 advertised-routes
Mon Oct 12 00:26:16.098 UTC
Network            Next Hop        From            AS Path
Route Distinguisher: 65001:1 (default for vrf CUST-A)
Route Distinguisher Version: 16
172.16.1.0/24      172.16.2.1      1.1.1.1         65001?
172.16.2.0/24      172.16.2.1      Local           65001?
172.16.3.0/24      172.16.2.1      4.4.4.4         65001?

Processed 3 prefixes, 3 paths

CE-A1b reaches 10.1.11.1/32 over OSPF inside the site.

STEP 0: 10.1.11.1/32 as seen by CE-A1b
RP/0/RP0/CPU0:CE-A1b#show route 10.1.11.1/32
Mon Oct 12 00:28:03.772 UTC

Routing entry for 10.1.11.1/32
  Known via "ospf 1", distance 110, metric 2, type intra area
  Installed Oct 12 00:21:06.625 for 00:06:57
  Routing Descriptor Blocks
    10.1.1.1, from 11.11.11.11, via GigabitEthernet0/0/0/1
      Route metric is 2
  No advertising protos. 

The traceroute is a single hop to the neighbouring CE-A1a.

STEP 0: traceroute from CE-A1b to 10.1.11.1
RP/0/RP0/CPU0:CE-A1b#traceroute 10.1.11.1 source 10.1.12.1 timeout 1 probe 2 maxttl 6
Mon Oct 12 00:24:54.420 UTC

Type escape sequence to abort.
Tracing the route to 10.1.11.1

 1  10.1.1.1 6 msec  * 

With as-override the route comes back

Because every site uses the same AS number, as-override is required for the routes between sites to get through. It goes on the CE-facing neighbor of PE1, PE2 and PE3.

STEP 1: configuration committed on PE2
router bgp 65001
 vrf CUST-A
  neighbor 172.16.2.2
   address-family ipv4 unicast
    as-override
   !
  !
 !
!
end

10.1.11.1/32 now reaches CE-A1b. The AS_PATH reads 65001 65001: the customer AS 65102 has been rewritten with the AS number of the PE.

STEP 1: 10.1.11.1/32 in the BGP table of CE-A1b
RP/0/RP0/CPU0:CE-A1b#show bgp 10.1.11.1/32
Mon Oct 12 00:34:08.362 UTC
BGP routing table entry for 10.1.11.1/32
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                  8            8
Last Modified: Oct 12 00:29:09.023 for 00:04:59
Paths: (1 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Not advertised to any peer
  65001 65001, (received & used)
    172.16.2.1 from 172.16.2.1 (2.2.2.2)
      Origin IGP, localpref 100, valid, external, best, group-best
      Received Path ID 0, Local Path ID 1, version 8
      Origin-AS validity: (disabled)

The route arrives over eBGP (distance 20), so it pushes the OSPF route inside the site (distance 110) out of the RIB.

STEP 1: 10.1.11.1/32 as seen by CE-A1b
RP/0/RP0/CPU0:CE-A1b#show route 10.1.11.1/32
Mon Oct 12 00:34:07.801 UTC

Routing entry for 10.1.11.1/32
  Known via "bgp 65102", distance 20, metric 0
  Tag 65001, type external
  Installed Oct 12 00:29:09.251 for 00:04:58
  Routing Descriptor Blocks
    172.16.2.1, from 172.16.2.1, BGP external
      Route metric is 0, Wt is 1
  No advertising protos. 

Forwarding changes with it. What used to be one hop to CE-A1a now goes PE2 to P1 to PE1 through the MPLS VPN before reaching CE-A1a: traffic inside the site makes a round trip through the provider network.

STEP 1: traceroute from CE-A1b to 10.1.11.1
RP/0/RP0/CPU0:CE-A1b#traceroute 10.1.11.1 source 10.1.12.1 timeout 1 probe 2 maxttl 6
Mon Oct 12 00:31:20.729 UTC

Type escape sequence to abort.
Tracing the route to 10.1.11.1

 1  172.16.2.1 6 msec  5 msec 
 2  10.0.23.3 [MPLS: Labels 24001/24005 Exp 0] 67 msec  15 msec 
 3  10.0.13.1 [MPLS: Label 24005 Exp 0] 16 msec  14 msec 
 4  172.16.1.2 51 msec  * 

The packets show the same thing. In No.1 of the attached STEP 1 capture, PE2 sends 10.1.11.1/32 to CE-A1b with the AS_PATH 65001 65001.

STEP 1 No.1 UPDATE (PE2 to CE-A1b), NLRI and AS_PATH (tshark -V excerpt)
            Network Layer Reachability Information (NLRI)
                10.1.2.1/32
                    MP Reach NLRI prefix length: 32
                    MP Reach NLRI IPv4 prefix: 10.1.2.1
                10.1.11.1/32
                    MP Reach NLRI prefix length: 32
                    MP Reach NLRI IPv4 prefix: 10.1.11.1
        Path Attribute - ORIGIN: IGP
            Flags: 0x40, Transitive, Well-known, Complete
                0... .... = Optional: Not set
                .1.. .... = Transitive: Set
                ..0. .... = Partial: Not set
                ...0 .... = Extended-Length: Not set
                .... 0000 = Unused: 0x0
            Type Code: ORIGIN (1)
            Length: 1
            Origin: IGP (0)
        Path Attribute - AS_PATH: 65001 65001 
            Flags: 0x40, Transitive, Well-known, Complete
                0... .... = Optional: Not set
                .1.. .... = Transitive: Set
                ..0. .... = Partial: Not set
                ...0 .... = Extended-Length: Not set
                .... 0000 = Unused: 0x0
            Type Code: AS_PATH (2)
            Length: 10
            AS Path segment: 65001 65001

Download the pcap of the packet in the tshark output above (No.1 UPDATE, the returning route)

SoO stops it

The same value goes on the site 1 neighbor of both PE1 and PE2, the two PEs the site connects to.

STEP 2: configuration committed on PE2
router bgp 65001
 vrf CUST-A
  neighbor 172.16.2.2
   address-family ipv4 unicast
    site-of-origin 65001:11
   !
  !
 !
!
end

The VPNv4 route held by PE2 now carries SoO:65001:11, next to the RT in the same Extended Communities attribute.

STEP 2: 10.1.11.1/32 (VPNv4) on PE2
RP/0/RP0/CPU0:PE2#show bgp vpnv4 unicast rd 65001:1 10.1.11.1/32 detail
Mon Oct 12 00:38:15.729 UTC
BGP routing table entry for 10.1.11.1/32, Route Distinguisher: 65001:1
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 17           17
    Flags: 0x00043001+0x00000000+0x00000000 
Last Modified: Oct 12 00:35:03.023 for 00:03:12
Paths: (1 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Flags: 0x2000000085068005+0x00, import: 0x39f
  Not advertised to any peer
  65102, (received & used)
    1.1.1.1 (metric 3) from 1.1.1.1 (1.1.1.1), if-handle 0x00000000
      Received Label 24005 
      Origin IGP, metric 0, localpref 100, valid, internal, best, group-best, import-candidate, imported
      Received Path ID 0, Local Path ID 1, version 17
      Extended community: SoO:65001:11 RT:65001:100 
      Source AFI: VPNv4 Unicast, Source VRF: CUST-A, Source Route Distinguisher: 65001:1

In the packets it appears as a Route Origin community (Sub-Type 0x03). This is No.7 of the attached STEP 2 capture of the PE1-P1 link.

STEP 2 No.7 VPNv4 UPDATE (PE1 to P1), Extended Communities (tshark -V excerpt)
        Path Attribute - EXTENDED_COMMUNITIES
            Flags: 0xc0, Optional, Transitive, Complete
                1... .... = Optional: Set
                .1.. .... = Transitive: Set
                ..0. .... = Partial: Not set
                ...0 .... = Extended-Length: Not set
                .... 0000 = Unused: 0x0
            Type Code: EXTENDED_COMMUNITIES (16)
            Length: 16
            Carried extended communities: (2 communities)
                Route Origin: 65001:11 [Transitive 2-Octet AS-Specific]
                    Type: Transitive 2-Octet AS-Specific (0x00)
                        0... .... = IANA Authority: Allocated on First Come First Serve Basis
                        .0.. .... = Transitive across ASes: Transitive
                    Subtype (AS2): Route Origin (0x03)
                    2-Octet AS: 65001
                    4-Octet AN: 11
                Route Target: 65001:100 [Transitive 2-Octet AS-Specific]
                    Type: Transitive 2-Octet AS-Specific (0x00)
                        0... .... = IANA Authority: Allocated on First Come First Serve Basis
                        .0.. .... = Transitive across ASes: Transitive
                    Subtype (AS2): Route Target (0x02)
                    2-Octet AS: 65001
                    4-Octet AN: 100

Download the pcap of the packet in the tshark output above (No.7 VPNv4 UPDATE with SoO)

10.1.11.1/32 is gone from what PE2 advertises to CE-A1b. 10.1.2.1/32 from site 2 is still there: only the route that came from the same site was stopped.

STEP 2: what PE2 advertises to CE-A1b
RP/0/RP0/CPU0:PE2#show bgp vrf CUST-A neighbors 172.16.2.2 advertised-routes
Mon Oct 12 00:38:13.026 UTC
Network            Next Hop        From            AS Path
Route Distinguisher: 65001:1 (default for vrf CUST-A)
Route Distinguisher Version: 18
10.1.2.1/32        172.16.2.1      4.4.4.4         65001 65102i
172.16.1.0/24      172.16.2.1      1.1.1.1         65001?
172.16.2.0/24      172.16.2.1      Local           65001?
172.16.3.0/24      172.16.2.1      4.4.4.4         65001?

Processed 4 prefixes, 4 paths

The withdrawal is in the packets too: No.3 of the STEP 2 capture of the same link.

STEP 2 No.3 UPDATE (PE2 to CE-A1b), Withdrawn Routes (tshark -V excerpt)
    Withdrawn Routes Length: 5
    Withdrawn Routes
        10.1.11.1/32
            Withdrawn route prefix length: 32
            Withdrawn prefix: 10.1.11.1

Download the pcap of the packet in the tshark output above (No.3 WITHDRAW)

CE-A1b goes back to the OSPF route inside the site.

STEP 2: 10.1.11.1/32 as seen by CE-A1b
RP/0/RP0/CPU0:CE-A1b#show route 10.1.11.1/32
Mon Oct 12 00:40:03.007 UTC

Routing entry for 10.1.11.1/32
  Known via "ospf 1", distance 110, metric 2, type intra area
  Installed Oct 12 00:35:13.752 for 00:04:49
  Routing Descriptor Blocks
    10.1.1.1, from 11.11.11.11, via GigabitEthernet0/0/0/1
      Route metric is 2
  No advertising protos. 

STEP 2: traceroute from CE-A1b to 10.1.11.1
RP/0/RP0/CPU0:CE-A1b#traceroute 10.1.11.1 source 10.1.12.1 timeout 1 probe 2 maxttl 6
Mon Oct 12 00:37:13.649 UTC

Type escape sequence to abort.
Tracing the route to 10.1.11.1

 1  10.1.1.1 15 msec  * 

A different SoO does not stop anything

SoO only stops routes carrying the same value. Putting a different value (65001:12) on the CE-A2 neighbor of PE3 tags the routes of site 2 with that value.

STEP 3: 10.1.2.1/32 (VPNv4) on PE3
RP/0/RP0/CPU0:PE3#show bgp vpnv4 unicast rd 65001:1 10.1.2.1/32 detail
Mon Oct 12 00:44:35.403 UTC
BGP routing table entry for 10.1.2.1/32, Route Distinguisher: 65001:1
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 20           20
    Local Label: 24005 (no rewrite); 
    Flags: 0x013e3001+0x00000000+0x00000000 
Last Modified: Oct 12 00:40:57.023 for 00:03:38
Paths: (2 available, best #1)
  Advertised to update-groups (with more than one peer):
    0.2 
  Path #1: Received by speaker 0
  Flags: 0x300000000d060001+0x00, import: 0x33f
  Advertised to update-groups (with more than one peer):
    0.2 
  65102
    172.16.3.2 from 172.16.3.2 (13.13.13.13), if-handle 0x00000000
      Origin IGP, metric 0, localpref 100, valid, external, best, group-best, import-candidate
      Received Path ID 0, Local Path ID 1, version 20
      Extended community: SoO:65001:12 RT:65001:100 
  Path #2: Received by speaker 0
  Flags: 0x3000000000024001+0x00, import: 0x020
  Not advertised to any peer
  65102, (received-only)
    172.16.3.2 from 172.16.3.2 (13.13.13.13), if-handle 0x00000000
      Origin IGP, metric 0, localpref 100, valid, external
      Received Path ID 0, Local Path ID 0, version 0

Site 1 keeps using the route to site 2.

STEP 3: 10.1.2.1/32 as seen by CE-A1b
RP/0/RP0/CPU0:CE-A1b#show route 10.1.2.1/32
Mon Oct 12 00:45:44.947 UTC

Routing entry for 10.1.2.1/32
  Known via "bgp 65102", distance 20, metric 0
  Tag 65001, type external
  Installed Oct 12 00:29:09.251 for 00:16:35
  Routing Descriptor Blocks
    172.16.2.1, from 172.16.2.1, BGP external
      Route metric is 0, Wt is 1
  No advertising protos. 

Removing it from one PE brings the route back

The SoO is removed from the site 1 neighbor of PE2 only; PE1 keeps its configuration.

STEP 4: configuration committed on PE2
router bgp 65001
 vrf CUST-A
  neighbor 172.16.2.2
   address-family ipv4 unicast
    no site-of-origin 65001:11
   !
  !
 !
!
end

The route is back.

STEP 4: 10.1.11.1/32 as seen by CE-A1b
RP/0/RP0/CPU0:CE-A1b#show route 10.1.11.1/32
Mon Oct 12 00:51:28.395 UTC

Routing entry for 10.1.11.1/32
  Known via "bgp 65102", distance 20, metric 0
  Tag 65001, type external
  Installed Oct 12 00:46:37.715 for 00:04:50
  Routing Descriptor Blocks
    172.16.2.1, from 172.16.2.1, BGP external
      Route metric is 0, Wt is 1
  No advertising protos. 

STEP 4: traceroute from CE-A1b to 10.1.11.1
RP/0/RP0/CPU0:CE-A1b#traceroute 10.1.11.1 source 10.1.12.1 timeout 1 probe 2 maxttl 6
Mon Oct 12 00:48:39.357 UTC

Type escape sequence to abort.
Tracing the route to 10.1.11.1

 1  172.16.2.1 6 msec  5 msec 
 2  10.0.23.3 [MPLS: Labels 24001/24005 Exp 0] 22 msec  14 msec 
 3  10.0.13.1 [MPLS: Label 24005 Exp 0] 14 msec  21 msec 
 4  172.16.1.2 15 msec  * 

The SoO:65001:11 that PE1 attached is still on the route. What stopped working is that the advertising PE2 no longer has a value to compare with. The check pairs the value carried by the route with the value configured on the outgoing neighbor, so a multihomed site needs the same value on every PE it connects to.

Verification configs and show output

For every STEP the following files were collected from all seven routers, separated per router. The verification config is ..._run.txt (the final state is the one from the last STEP).

File	Contents
`..._show.txt`	`show version` / `show interface description` / the OSPF, LDP, BGP and VRF show commands
`..._log.txt`	`show logging` limited to that STEP
`..._run.txt`	`show running-config` at that STEP (the verification config)
`..._ping.txt`	`ping` and `traceroute` for that STEP
`..._trace.txt`	`show bgp trace` (PEs only)
`..._commit.cfg`	The configuration actually committed in that STEP (only on routers that changed)

STEP 0: Initial state (no as-override, no SoO)

Router	show	syslog	running-config	ping	trace	committed config
CE-A1a	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	-
CE-A1b	show	log	run	ping	-	-
PE3	show	log	run	ping	trace	-
CE-A2	show	log	run	ping	-	-

STEP 1: as-override on PE1, PE2 and PE3

Router	show	syslog	running-config	ping	trace	committed config
CE-A1a	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	commit
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	commit
CE-A1b	show	log	run	ping	-	-
PE3	show	log	run	ping	trace	commit
CE-A2	show	log	run	ping	-	-

STEP 2: site-of-origin 65001:11 on the site 1 neighbors of PE1 and PE2

Router	show	syslog	running-config	ping	trace	committed config
CE-A1a	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	commit
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	commit
CE-A1b	show	log	run	ping	-	-
PE3	show	log	run	ping	trace	-
CE-A2	show	log	run	ping	-	-

STEP 3: site-of-origin 65001:12 on the CE-A2 neighbor of PE3

Router	show	syslog	running-config	ping	trace	committed config
CE-A1a	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	-
CE-A1b	show	log	run	ping	-	-
PE3	show	log	run	ping	trace	commit
CE-A2	show	log	run	ping	-	-

STEP 4: Remove the SoO from the site 1 neighbor of PE2 only

Router	show	syslog	running-config	ping	trace	committed config
CE-A1a	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	commit
CE-A1b	show	log	run	ping	-	-
PE3	show	log	run	ping	trace	-
CE-A2	show	log	run	ping	-	-

STEP 5: Put the SoO back on PE2 (final state)

Router	show	syslog	running-config	ping	trace	committed config
CE-A1a	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	commit
CE-A1b	show	log	run	ping	-	-
PE3	show	log	run	ping	trace	-
CE-A2	show	log	run	ping	-	-

Packet captures were taken per STEP.

STEP	PE2-CE-A1b	PE1-P1
0	pcap	pcap
1	pcap	pcap
2	pcap	pcap
3	pcap	pcap
4	pcap	pcap
5	pcap	pcap