When Several MPLS VPN Sites Share One AS Number (as-override)

When Several Sites Share One AS Number in MPLS VPN

Customers rarely assign a different AS number to each site; an enterprise network normally uses one AS number for all of them. Run eBGP between PE and CE just like that, though, and the routes of one site never reach the others.

The reason is the loop detection BGP does with the AS_PATH. A route from site 1 picks up the provider’s AS number on its way through the PEs, so it would arrive at the CE of another site with an AS_PATH of 65001 65102. That CE is in AS 65102, which means its own AS number is in the path.

Section 9.1.2 of RFC 4271 defines the check on the receiving side.

If the AS_PATH attribute of a BGP route contains an AS loop, the BGP route should be excluded from the Phase 2 decision function. AS loop detection is done by scanning the full AS path (as specified in the AS_PATH attribute), and checking that the autonomous system number of the local system does not appear in the AS path.

The Sending PE Drops It, Not the Receiving CE

Before deciding what to fix, it matters where the route is lost.

The rule quoted above is about the receiving side, but in practice the route never gets that far. An IOS XR PE does not send a route to an eBGP neighbour when that neighbour’s AS number is in the AS_PATH. Section 9.2 of RFC 4271 (the Update-Send Process) says nothing about suppressing on send, so this is implementation behaviour.

The distinction has a practical consequence. Configuring the CE to accept routes that contain its own AS changes nothing, because no such route arrives. The fix belongs on the PE.

as-override

as-override is configured on the PE and replaces the neighbour’s own AS number inside the AS_PATH with the PE’s AS number, just before sending to that CE.

On the PE (IOS XR)
router bgp 65001
 vrf CUST-A
  neighbor 172.16.1.2
   address-family ipv4 unicast
    as-override
   !
  !
 !
!

After the rewrite the AS_PATH no longer holds the receiving CE’s AS number, so it passes the suppression on send and the check on receipt.

Two things are worth keeping in mind.

Point	Detail
Per neighbour	It sits under `neighbor`, so it applies only when sending to that neighbour. Configuring one PE does not help the sites behind the others
Only the neighbour’s own AS number is replaced	It does not rewrite the whole AS_PATH. Another AS beyond that site stays as it is

The second point shows up when a site has another AS behind it. Say a router in AS 65009 sits behind the CE and a route carries the AS_PATH 65001 65102 65009. as-override replaces only the 65102, giving 65001 65001 65009; the 65009 remains.

What to Watch Out For

With as-override the AS_PATH no longer reflects reality. The AS numbers seen along a route differ from the ones it actually traversed, and troubleshooting with show bgp has to account for the rewrite.

The other point is that this disables the loop detection itself. With the neighbour’s AS number gone from the AS_PATH, a CE cannot tell that a route has come all the way back to it. In a multihomed site attached to two PEs, a real loop becomes possible. Stopping that is the job of SoO (Site of Origin), covered in MPLS VPN SoO.

Lab Setup

Eight XRd routers, with all three sites in AS 65102. Another AS, 65009 (CE9), hangs behind CE3 so that a route carries a third-party AS number in its AS_PATH.

Node	AS	What it advertises
CE1	65102	`10.1.1.0/24` (the observer)
CE2	65102	`10.1.2.0/24`
CE3	65102	`10.1.3.0/24`
CE9	65009	`10.1.9.0/24` (eBGP with CE3; it never touches a PE)

The CEs hold no default route, so a missing route shows up directly in a ping. All three PEs run redistribute connected, which means the PE-CE subnets are advertised by the PEs themselves — used later to isolate the cause.

STEP	Change	What it shows
0	Initial state (no as-override)	The PE is not sending (three routes in `advertised-routes`); the CE table has none of them
1	`as-override` on PE2 only	Only the configured direction works (CE2 receives, CE1 and CE3 do not yet)
2	`as-override` on PE1 and PE3 as well	Everything gets through; `10.1.9.0/24` reads `65001 65001 65009`
3	Remove `as-override` everywhere	Back to STEP 0
4	`allowas-in` on the CEs (not covered in this article)	(not covered in this article)
5	Remove `allowas-in` and put `as-override` back (final state)	Back to the state of STEP 2

STEP 0: The PE Holds the Routes but Does Not Send Them

PE1 has seven routes in the VRF.

PE1: the BGP table of vrf CUST-A (STEP 0)
RP/0/RP0/CPU0:PE1#show bgp vrf CUST-A
Sun Sep 27 16:18:55.337 UTC
BGP VRF CUST-A, state: Active
BGP Route Distinguisher: 65001:1
VRF ID: 0x60000002
BGP router identifier 1.1.1.1, local AS number 65001
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000002   RD version: 19
BGP main routing table version 19
BGP NSR Initial initsync version 6 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
Route Distinguisher: 65001:1 (default for vrf CUST-A)
Route Distinguisher Version: 19
*> 10.1.1.0/24        172.16.1.2               0             0 65102 i
*>i10.1.2.0/24        2.2.2.2                  0    100      0 65102 i
*>i10.1.3.0/24        4.4.4.4                  0    100      0 65102 i
*>i10.1.9.0/24        4.4.4.4                       100      0 65102 65009 i
*> 172.16.1.0/24      0.0.0.0                  0         32768 ?
*>i172.16.2.0/24      2.2.2.2                  0    100      0 ?
*>i172.16.3.0/24      4.4.4.4                  0    100      0 ?

Processed 7 prefixes, 7 paths

Both 10.1.2.0/24 and 10.1.9.0/24 are there. Yet only three routes are advertised to CE1.

What PE1 advertises to CE1 (STEP 0)
RP/0/RP0/CPU0:PE1#show bgp vrf CUST-A neighbors 172.16.1.2 advertised-routes
Sun Sep 27 16:18:57.786 UTC
Network            Next Hop        From            AS Path
Route Distinguisher: 65001:1 (default for vrf CUST-A)
Route Distinguisher Version: 19
172.16.1.0/24      172.16.1.1      Local           65001?
172.16.2.0/24      172.16.1.1      2.2.2.2         65001?
172.16.3.0/24      172.16.1.1      4.4.4.4         65001?

Processed 3 prefixes, 3 paths

The three are the 172.16.x.0/24 subnets, which the PE itself originates through redistribute connected and whose AS_PATH holds only 65001. Not one route originated by a CE — the ones whose AS_PATH contains 65102 — is sent.

The receiving side agrees.

The raw routes CE1 received (STEP 0)
RP/0/RP0/CPU0:CE1#show bgp neighbors 172.16.1.1 received routes
Sun Sep 27 16:20:47.773 UTC
BGP router identifier 11.11.11.11, local AS number 65102
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 6
BGP main routing table version 6
BGP NSR Initial initsync version 3 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*> 172.16.1.0/24      172.16.1.1               0             0 65001 ?
*> 172.16.2.0/24      172.16.1.1                             0 65001 ?
*> 172.16.3.0/24      172.16.1.1                             0 65001 ?

Processed 3 prefixes, 3 paths

received routes shows the routes before any inbound policy, and there are still only three. The PE stops sending before anything reaches the CE.

So the CE’s BGP table has none of the remote sites either.

The BGP table of CE1 (STEP 0)
RP/0/RP0/CPU0:CE1#show bgp
Sun Sep 27 16:20:46.907 UTC
BGP router identifier 11.11.11.11, local AS number 65102
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 6
BGP main routing table version 6
BGP NSR Initial initsync version 3 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*> 10.1.1.0/24        0.0.0.0                  0         32768 i
*> 172.16.1.0/24      172.16.1.1               0             0 65001 ?
*> 172.16.2.0/24      172.16.1.1                             0 65001 ?
*> 172.16.3.0/24      172.16.1.1                             0 65001 ?

Processed 4 prefixes, 4 paths

Only its own 10.1.1.0/24 and the 172.16.x.0/24 subnets from the PEs. The ping fails.

ping from CE1 to 10.1.9.1 (STEP 0)
RP/0/RP0/CPU0:CE1#ping 10.1.9.1 source 10.1.1.1 count 10 timeout 1
Sun Sep 27 16:17:33.195 UTC
Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 10.1.9.1 timeout is 1 seconds:
..........
Success rate is 0 percent (0/10)

STEP 1: `as-override` Works per Neighbour

It goes on PE2 only.

Configuration committed on PE2 (STEP 1)
router bgp 65001
 vrf CUST-A
  neighbor 172.16.2.2
   address-family ipv4 unicast
    as-override
   !
  !
 !
!
end

CE2 starts receiving the routes; CE1 does not change.

The BGP table of CE2 (STEP 1)
RP/0/RP0/CPU0:CE2#show bgp
Sun Sep 27 16:27:38.538 UTC
BGP router identifier 12.12.12.12, local AS number 65102
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 9
BGP main routing table version 9
BGP NSR Initial initsync version 3 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*> 10.1.1.0/24        172.16.2.1                             0 65001 65001 i
*> 10.1.2.0/24        0.0.0.0                  0         32768 i
*> 10.1.3.0/24        172.16.2.1                             0 65001 65001 i
*> 10.1.9.0/24        172.16.2.1                             0 65001 65001 65009 i
*> 172.16.1.0/24      172.16.2.1                             0 65001 ?
*> 172.16.2.0/24      172.16.2.1               0             0 65001 ?
*> 172.16.3.0/24      172.16.2.1                             0 65001 ?

Processed 7 prefixes, 7 paths

The BGP table of CE1 (STEP 1)
RP/0/RP0/CPU0:CE1#show bgp
Sun Sep 27 16:27:19.933 UTC
BGP router identifier 11.11.11.11, local AS number 65102
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 6
BGP main routing table version 6
BGP NSR Initial initsync version 3 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*> 10.1.1.0/24        0.0.0.0                  0         32768 i
*> 172.16.1.0/24      172.16.1.1               0             0 65001 ?
*> 172.16.2.0/24      172.16.1.1                             0 65001 ?
*> 172.16.3.0/24      172.16.1.1                             0 65001 ?

Processed 4 prefixes, 4 paths

It sits under neighbor, so it only applies when sending to that neighbour. Every PE with a CE needs it for the sites to reach each other.

STEP 2: With Every PE Configured It Works, and 65009 Is Left Alone

PE1 and PE3 get it too, and PE1 now advertises six routes to CE1.

What PE1 advertises to CE1 (STEP 2)
RP/0/RP0/CPU0:PE1#show bgp vrf CUST-A neighbors 172.16.1.2 advertised-routes
Sun Sep 27 16:31:13.861 UTC
Network            Next Hop        From            AS Path
Route Distinguisher: 65001:1 (default for vrf CUST-A)
Route Distinguisher Version: 19
10.1.2.0/24        172.16.1.1      2.2.2.2         65001 65102i
10.1.3.0/24        172.16.1.1      4.4.4.4         65001 65102i
10.1.9.0/24        172.16.1.1      4.4.4.4         65001 65102 65009i
172.16.1.0/24      172.16.1.1      Local           65001?
172.16.2.0/24      172.16.1.1      2.2.2.2         65001?
172.16.3.0/24      172.16.1.1      4.4.4.4         65001?

Processed 6 prefixes, 6 paths

They arrive in CE1’s table.

The BGP table of CE1 (STEP 2)
RP/0/RP0/CPU0:CE1#show bgp
Sun Sep 27 16:33:04.073 UTC
BGP router identifier 11.11.11.11, local AS number 65102
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 9
BGP main routing table version 9
BGP NSR Initial initsync version 3 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*> 10.1.1.0/24        0.0.0.0                  0         32768 i
*> 10.1.2.0/24        172.16.1.1                             0 65001 65001 i
*> 10.1.3.0/24        172.16.1.1                             0 65001 65001 i
*> 10.1.9.0/24        172.16.1.1                             0 65001 65001 65009 i
*> 172.16.1.0/24      172.16.1.1               0             0 65001 ?
*> 172.16.2.0/24      172.16.1.1                             0 65001 ?
*> 172.16.3.0/24      172.16.1.1                             0 65001 ?

Processed 7 prefixes, 7 paths

Look at the AS_PATH. 10.1.2.0/24 and 10.1.3.0/24 read 65001 65001: the 65102 has become 65001. And 10.1.9.0/24 reads 65001 65001 65009, so the 65009 is still there. as-override rewrites only the neighbour’s own AS number.

The packets say the same. It is No.3 of the STEP 2 capture between PE1 and CE1.

PE1 to CE1, an UPDATE (tshark -V, the 10.1.9.0/24 part)
        Path Attribute - MP_REACH_NLRI
            Flags: 0x90, Optional, Extended-Length, Non-transitive, Complete
                1... .... = Optional: Set
                .0.. .... = Transitive: Not set
                ..0. .... = Partial: Not set
                ...1 .... = Extended-Length: Set
                .... 0000 = Unused: 0x0
            Type Code: MP_REACH_NLRI (14)
            Length: 13
            Address family identifier (AFI): IPv4 (1)
            Subsequent address family identifier (SAFI): Unicast (1)
            Next hop: 172.16.1.1
                IPv4 Address: 172.16.1.1
            Number of Subnetwork points of attachment (SNPA): 0
            Network Layer Reachability Information (NLRI)
                10.1.9.0/24
                    MP Reach NLRI prefix length: 24
                    MP Reach NLRI IPv4 prefix: 10.1.9.0
        Path Attribute - ORIGIN: IGP
            Flags: 0x40, Transitive, Well-known, Complete
                0... .... = Optional: Not set
                .1.. .... = Transitive: Set
                ..0. .... = Partial: Not set
                ...0 .... = Extended-Length: Not set
                .... 0000 = Unused: 0x0
            Type Code: ORIGIN (1)
            Length: 1
            Origin: IGP (0)
        Path Attribute - AS_PATH: 65001 65001 65009 
            Flags: 0x40, Transitive, Well-known, Complete
                0... .... = Optional: Not set
                .1.. .... = Transitive: Set
                ..0. .... = Partial: Not set
                ...0 .... = Extended-Length: Not set
                .... 0000 = Unused: 0x0
            Type Code: AS_PATH (2)
            Length: 14
            AS Path segment: 65001 65001 65009

Download the pcap of the packet in the tshark output above (No.3 UPDATE, the rewritten AS_PATH)

Connectivity follows.

ping from CE1 to 10.1.9.1 (STEP 2)
RP/0/RP0/CPU0:CE1#ping 10.1.9.1 source 10.1.1.1 count 10 timeout 1
Sun Sep 27 16:30:26.503 UTC
Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 10.1.9.1 timeout is 1 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 16/17/22 ms

STEP 3: Removing It Puts Things Back

With as-override gone everywhere, the PEs stop sending again and the state returns to that of STEP 0.

The BGP table of CE1 (STEP 3)
RP/0/RP0/CPU0:CE1#show bgp
Sun Sep 27 16:39:53.186 UTC
BGP router identifier 11.11.11.11, local AS number 65102
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000   RD version: 12
BGP main routing table version 12
BGP NSR Initial initsync version 3 (Reached)
BGP NSR/ISSU Sync-Group versions 0/0
BGP scan interval 60 secs

Status codes: s suppressed, d damped, h history, * valid, > best
              i - internal, r RIB-failure, S stale, N Nexthop-discard
Origin codes: i - IGP, e - EGP, ? - incomplete
   Network            Next Hop            Metric LocPrf Weight Path
*> 10.1.1.0/24        0.0.0.0                  0         32768 i
*> 172.16.1.0/24      172.16.1.1               0             0 65001 ?
*> 172.16.2.0/24      172.16.1.1                             0 65001 ?
*> 172.16.3.0/24      172.16.1.1                             0 65001 ?

Processed 4 prefixes, 4 paths

References

RFC	Title	Sections used
RFC 4271	A Border Gateway Protocol 4 (BGP-4)	9.1.2 (AS_PATH loop detection, the check on receipt), 9.2 (Update-Send Process, which does not specify suppression on send)
RFC 4364	BGP/MPLS IP Virtual Private Networks (VPNs)	7 (how PEs learn routes from CEs)

Book: Luc De Ghein, MPLS Fundamentals (Cisco Press, 2006), Chapter 7

Verification Configs and show Output

Every STEP was captured on all eight routers, one file per router and kind. The verification config is the ..._run.txt file (the final state is the one from the last STEP).

File	Contents
`..._show.txt`	The state of every protocol running on that router. On a PE that is OSPF, LDP, MPLS forwarding, MP-BGP and the VRF (`show ospf neighbor`, `show mpls ldp bindings`, `show mpls forwarding`, `show bgp vpnv4 unicast` and so on), plus `advertised-routes`, `routes` and `received routes` per neighbour
`..._log.txt`	`show logging` limited to that STEP
`..._run.txt`	`show running-config` at that STEP (the verification config)
`..._ping.txt`	`ping` and `traceroute` between the CEs
`..._trace.txt`	`show bgp trace` on the PEs
`..._commit.cfg`	The configuration actually committed in that STEP (only for the routers that changed)

STEP 0: Initial state (no as-override)

Router	show	syslog	running-config	ping	trace	committed config
CE1	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	-
CE2	show	log	run	ping	-	-
PE3	show	log	run	ping	trace	-
CE3	show	log	run	ping	-	-
CE9	show	log	run	ping	-	-

STEP 1: as-override on PE2 only

Router	show	syslog	running-config	ping	trace	committed config
CE1	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	commit
CE2	show	log	run	ping	-	-
PE3	show	log	run	ping	trace	-
CE3	show	log	run	ping	-	-
CE9	show	log	run	ping	-	-

STEP 2: as-override on PE1 and PE3 as well

Router	show	syslog	running-config	ping	trace	committed config
CE1	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	commit
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	-
CE2	show	log	run	ping	-	-
PE3	show	log	run	ping	trace	commit
CE3	show	log	run	ping	-	-
CE9	show	log	run	ping	-	-

STEP 3: Remove as-override everywhere

Router	show	syslog	running-config	ping	trace	committed config
CE1	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	commit
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	commit
CE2	show	log	run	ping	-	-
PE3	show	log	run	ping	trace	commit
CE3	show	log	run	ping	-	-
CE9	show	log	run	ping	-	-

STEP 4: allowas-in on the CEs (not covered in this article)

Router	show	syslog	running-config	ping	trace	committed config
CE1	show	log	run	ping	-	commit
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	-
CE2	show	log	run	ping	-	commit
PE3	show	log	run	ping	trace	-
CE3	show	log	run	ping	-	commit
CE9	show	log	run	ping	-	-

STEP 5: Remove allowas-in and put as-override back (final state)

Router	show	syslog	running-config	ping	trace	committed config
CE1	show	log	run	ping	-	commit
PE1	show	log	run	ping	trace	commit
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	commit
CE2	show	log	run	ping	-	commit
PE3	show	log	run	ping	trace	commit
CE3	show	log	run	ping	-	commit
CE9	show	log	run	ping	-	-

Packet captures were taken per STEP on the two PE-CE segments.

STEP	PE1-CE1	PE3-CE3
0	pcap	pcap
1	pcap	pcap
2	pcap	pcap
3	pcap	pcap
4	pcap	pcap
5	pcap	pcap