PE-CE routing with OSPF in MPLS VPN

When OSPF runs between PE and CE, the PE behaves as one of the OSPF routers of the site. The OSPF routes of a site are turned into VPN-IPv4 routes at the PE, carried by MP-BGP, and turned back into OSPF at the far PE before being advertised to the CE.

RFC 4577 defines how this works. A PE runs an independent OSPF instance per VRF (4.1.1) and redistributes the routes it learns there into BGP (4.1.2).

The PEs are treated as part of the OSPF backbone

The key decision is that a PE counts as an “area 0 router”.

PEs can thus be considered OSPF “area 0 routers”, i.e., they can be considered part of the “OSPF backbone”. Thus, they are allowed to distribute inter-area routes to the CE via Type 3 LSAs.

If the PE-CE link is in a non-zero area, the PE is an ABR for that area (4.1.4). The provider’s VPN backbone acts as an extension of area 0, so a route that crosses between sites is an inter-area route.

That leads to a result which looks surprising if you only read the configuration: putting both sites in the same area number does not make the routes intra-area. They arrive as inter-area routes in Type 3 LSAs, because area 0 (the VPN backbone) sits between the two sites. Making the sites look intra-area to each other requires a separate mechanism, the sham link.

Three extended communities carry the information

To carry an OSPF route over BGP and turn it back into OSPF at the other end, what kind of OSPF route it originally was has to travel with it. RFC 4577 section 4.2.6 defines three extended communities for that.

Extended community	Type	Contents
OSPF Domain Identifier	0005 / 0105 / 0205	Which OSPF domain the route belongs to. The default is NULL (4.2.4)
OSPF Route Type	0306	Area number (4 bytes) + route type (1 byte) + options (1 byte). Always present
OSPF Router ID	0107	The router ID of the PE, in the OSPF instance that installed the route. Optional

The route type values correspond to the kind of LSA the route came from.

Value	Meaning
1 / 2	Intra-area (from a Type 1 or Type 2 LSA)
3	Inter-area
5	External (the area number must be 0)
7	NSSA

The metric travels in the MED (by default the OSPF distance plus 1).

There are older type numbers too. RFC 4577 section 4.2.6 says that 8000 for the OSPF Route Type and 8001 for the OSPF Router ID must be accepted as well, for backward compatibility. Some implementations send those instead.

Which LSA the far site sees

The receiving PE first checks whether the Domain ID of the route matches a Domain ID of its own OSPF instance (4.2.4).

If R’s Domain Identifier is not one of I2’s Domain Identifiers, then, if R is redistributed into I2, R will be advertised as an AS-external route, no matter what its OSPF route type is.

If it matches, the original route type decides (4.1.3).

Domain ID of the route	Original route type	LSA advertised to the CE
Matches the receiver	Intra-area / inter-area (1 / 2 / 3)	Type 3 (inter-area)
Matches the receiver	External / NSSA (5 / 7)	Type 5 / Type 7
Does not match	Anything	Type 5 (AS-external)

The default Domain ID is NULL, and two NULL values count as the same domain. So with no configuration at all the result is Type 3; only an explicitly different value makes the route external.

The lab

Site 1 has two routers, CE-A1 and R-A1 behind it, so the link inside the site, 10.1.1.0/24, is an intra-area route. Site 2 is a single router, CE-A2. The PE-CE links and the link inside the site are all in area 1. Each PE runs router ospf 100 vrf CUST-A for the VRF and redistributes what it learns there into BGP.

STEP	Change	What it shows
0	Initial state (no `domain-id`, i.e. NULL)	An intra-area prefix of site 1 arrives at site 2 as `O IA` (Type 3); the VPNv4 route carries the OSPF Route Type and Router ID
1	The same `domain-id` on PE1 and PE2	Still Type 3. The Domain ID community is added to the VPNv4 route
2	A different `domain-id` on PE2 only	It becomes `O E2` (Type 5), and a Type-5 LSA appears in the LSDB
3	Put the original value back on PE2	Back to Type 3
4	Redistribute a static route (`10.9.9.0/24`) into OSPF on R-A1	Still Type 5 even though the domain matches. The Route Type has area number 0 and type 5
5	Remove that static route (final state)	Back to the previous state

The same area number does not make it intra-area

Looking at what CE-A1 learned through OSPF at site 1, the routes split in two: the ones from inside the site are O (intra-area), the ones from site 2 are O IA (inter-area).

STEP 0: OSPF routes on CE-A1 (site 1)
RP/0/RP0/CPU0:CE-A1#show route ospf
Mon Oct 12 02:52:37.855 UTC

O IA 10.1.2.1/32 [110/3] via 172.16.1.1, 00:04:08, GigabitEthernet0/0/0/0
O    10.1.11.1/32 [110/2] via 10.1.1.2, 00:05:36, GigabitEthernet0/0/0/1
O IA 172.16.2.0/24 [110/2] via 172.16.1.1, 00:04:08, GigabitEthernet0/0/0/0

Seen from CE-A2 at site 2, every prefix of site 1 is O IA. Both sites are in area 1, yet the routes arrive as ones that crossed an area boundary.

STEP 0: OSPF routes on CE-A2 (site 2)
RP/0/RP0/CPU0:CE-A2#show route ospf
Mon Oct 12 02:53:28.187 UTC

O IA 10.1.1.0/24 [110/3] via 172.16.2.1, 00:04:58, GigabitEthernet0/0/0/0
O IA 10.1.11.1/32 [110/4] via 172.16.2.1, 00:04:58, GigabitEthernet0/0/0/0
O IA 172.16.1.0/24 [110/2] via 172.16.2.1, 00:04:58, GigabitEthernet0/0/0/0

Looking at 10.1.1.0/24 on its own, it reads type inter area.

STEP 0: 10.1.1.0/24 as seen by CE-A2 (site 2)
RP/0/RP0/CPU0:CE-A2#show route 10.1.1.0/24
Mon Oct 12 02:53:28.331 UTC

Routing entry for 10.1.1.0/24
  Known via "ospf 1", distance 110, metric 3, type inter area
  Installed Oct 12 02:48:29.423 for 00:04:59
  Routing Descriptor Blocks
    172.16.2.1, from 100.2.2.2, via GigabitEthernet0/0/0/0
      Route metric is 3
  No advertising protos. 

The LSDB holds it as a Type 3 (summary) LSA. The advertising router is the router ID of PE2 (100.2.2.2), which shows the PE acting as the ABR for this area.

STEP 0: the Type 3 LSA on CE-A2
RP/0/RP0/CPU0:CE-A2#show ospf database summary
Mon Oct 12 02:53:26.578 UTC


            OSPF Router with ID (13.13.13.13) (Process ID 1)

		Summary Net Link States (Area 1)

  Routing Bit Set on this LSA
  LS age: 298
  Options: (No TOS-capability, DC, DN)
  LS Type: Summary Links (Network)
  Link State ID: 10.1.1.0 (Summary Network Number)
  Advertising Router: 100.2.2.2
  LS Seq Number: 80000001
  Checksum: 0xca79
  Length: 28
  Network Mask: /24
	TOS: 0 	Metric: 2 

The options carry DN, the mark on an LSA that came down from a PE to a CE, used for loop prevention. See The OSPF DN bit and the domain tag in MPLS VPN for the details.

What travels with the route

The route PE2 received over BGP carries an OSPF Route Type and an OSPF Router ID. 1:1:0x0 reads “area 1, route type 1 (intra-area), no options”. No domain-id is configured yet, so there is no Domain ID community.

STEP 0: 10.1.1.0/24 (VPNv4) held by PE2
RP/0/RP0/CPU0:PE2#show bgp vpnv4 unicast rd 65001:1 10.1.1.0/24 detail
Mon Oct 12 02:51:45.472 UTC
BGP routing table entry for 10.1.1.0/24, Route Distinguisher: 65001:1
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 13           13
    Flags: 0x00041001+0x00000000+0x00000000 
Last Modified: Oct 12 02:48:29.023 for 00:03:16
Paths: (1 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Flags: 0x2000000085068005+0x00, import: 0x39f
  Not advertised to any peer
  Local, (received & used)
    1.1.1.1 (metric 3) from 1.1.1.1 (1.1.1.1), if-handle 0x00000000
      Received Label 24003 
      Origin incomplete, metric 2, localpref 100, valid, internal, best, group-best, import-candidate, imported
      Received Path ID 0, Local Path ID 1, version 13
      Extended community: OSPF route-type:1:1:0x0 OSPF router-id:100.1.1.1 RT:65001:100 
      Source AFI: VPNv4 Unicast, Source VRF: CUST-A, Source Route Distinguisher: 65001:1

The same domain-id now goes on both PEs.

STEP 1: configuration committed on PE2
router ospf 100
 vrf CUST-A
  domain-id type 0005 value 000000000001
 !
!
end

The Domain ID community is added. The LSA type stays at Type 3, because the two values match and the instances count as one domain.

STEP 1: 10.1.1.0/24 (VPNv4) held by PE2
RP/0/RP0/CPU0:PE2#show bgp vpnv4 unicast rd 65001:1 10.1.1.0/24 detail
Mon Oct 12 02:57:17.310 UTC
BGP routing table entry for 10.1.1.0/24, Route Distinguisher: 65001:1
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 17           17
    Flags: 0x00043001+0x00000000+0x00000000 
Last Modified: Oct 12 02:53:57.023 for 00:03:20
Paths: (1 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Flags: 0x2000000085048005+0x00, import: 0x3bf
  Not advertised to any peer
  Local, (received & used)
    1.1.1.1 (metric 3) from 1.1.1.1 (1.1.1.1), if-handle 0x00000000
      Received Label 24003 
      Origin incomplete, metric 2, localpref 100, valid, internal, best, group-best, import-candidate, imported
      Received Path ID 0, Local Path ID 1, version 17
      Extended community: OSPF domain-id:0x5:0x000000000001 OSPF route-type:1:1:0x0 OSPF router-id:100.1.1.1 RT:65001:100 
      Source AFI: VPNv4 Unicast, Source VRF: CUST-A, Source Route Distinguisher: 65001:1

The packets show it as well, in No.4 of the attached STEP 1 capture of the PE1-P1 link.

STEP 1 No.4 VPNv4 UPDATE (PE1 to P1), Extended Communities (tshark -V excerpt)
        Path Attribute - EXTENDED_COMMUNITIES
            Flags: 0xc0, Optional, Transitive, Complete
                1... .... = Optional: Set
                .1.. .... = Transitive: Set
                ..0. .... = Partial: Not set
                ...0 .... = Extended-Length: Not set
                .... 0000 = Unused: 0x0
            Type Code: EXTENDED_COMMUNITIES (16)
            Length: 32
            Carried extended communities: (4 communities)
                OSPF Domain Identifier: 0:1 [Transitive 2-Octet AS-Specific]
                    Type: Transitive 2-Octet AS-Specific (0x00)
                        0... .... = IANA Authority: Allocated on First Come First Serve Basis
                        .0.. .... = Transitive across ASes: Transitive
                    Subtype (AS2): OSPF Domain Identifier (0x05)
                    2-Octet AS: 0
                    4-Octet AN: 1
                OSPF Route Type: Area: 0.0.0.1, Type: Router [Generic Transitive Extended Community]
                    Type: Generic Transitive Extended Community (0x80)
                        1... .... = IANA Authority: Allocated on Standard Action, Early Allocation or Experimental Basis
                        .0.. .... = Transitive across ASes: Transitive
                    Subtype (Experimental): OSPF Route Type (0x00)
                    Area ID: 0.0.0.1
                    Route type: Router (1)
                    Options: 0x00 (Metric: Type-1)
                        .... ...0 = Metric type: Type-1
                OSPF Router ID: 100.1.1.1 [Generic Transitive Extended Community]
                    Type: Generic Transitive Extended Community (0x80)
                        1... .... = IANA Authority: Allocated on Standard Action, Early Allocation or Experimental Basis
                        .0.. .... = Transitive across ASes: Transitive
                    Subtype (Experimental): OSPF Router ID (0x01)
                    Router ID: 100.1.1.1

Download the pcap of the packet in the tshark output above (No.4 VPNv4 UPDATE)

The OSPF Route Type and the OSPF Router ID are sent with the older type numbers (8000 and 8001). That is what tshark reads as Generic Transitive Extended Community (0x80), the form RFC 4577 keeps for backward compatibility. The Domain ID uses the current 0005.

A different Domain ID makes it Type 5

A different value goes on PE2 only.

STEP 2: configuration committed on PE2
router ospf 100
 vrf CUST-A
  domain-id type 0005 value 000000000002
 !
!
end

The route PE2 receives still carries Domain ID 0x000000000001, but PE2’s own OSPF instance is now 0x000000000002. They do not match, so PE2 advertises the route as AS-external.

STEP 2: 10.1.1.0/24 as seen by CE-A2
RP/0/RP0/CPU0:CE-A2#show route 10.1.1.0/24
Mon Oct 12 03:04:32.847 UTC

Routing entry for 10.1.1.0/24
  Known via "ospf 1", distance 110, metric 2
  Tag 3489725929, type extern 2
  Installed Oct 12 02:59:29.696 for 00:05:03
  Routing Descriptor Blocks
    172.16.2.1, from 100.2.2.2, via GigabitEthernet0/0/0/0
      Route metric is 2
  No advertising protos. 

The LSDB agrees.

STEP 2: the Type 5 LSA on CE-A2
RP/0/RP0/CPU0:CE-A2#show ospf database external
Mon Oct 12 03:04:31.882 UTC


            OSPF Router with ID (13.13.13.13) (Process ID 1)

		Type-5 AS External Link States

  Routing Bit Set on this LSA
  LS age: 303
  Options: (No TOS-capability, DC, DN)
  LS Type: AS External Link
  Link State ID: 10.1.1.0 (External Network Number)
  Advertising Router: 100.2.2.2
  LS Seq Number: 80000001
  Checksum: 0xec16
  Length: 36
  Network Mask: /24
	Metric Type: 2 (Larger than any link state path)
	TOS: 0 
	Metric: 2 
	Forward Address: 0.0.0.0
	External Route Tag: 3489725929

An originally external route stays Type 5, same domain or not

So far the rule was “a different domain makes it external”. A route that was external to begin with stays external even when the domain matches.

A single static route is added on R-A1 (redistribute static is there from the start).

STEP 4: configuration committed on R-A1
router static
 address-family ipv4 unicast
  10.9.9.0/24 Null0
 !
!
end

The Route Type that PE2 receives is 0:5:0x1: area number 0, route type 5, options 0x1 (type-2 metric). The Domain ID is the same value as the routes of site 1.

STEP 4: 10.9.9.0/24 (VPNv4) held by PE2
RP/0/RP0/CPU0:PE2#show bgp vpnv4 unicast rd 65001:1 10.9.9.0/24 detail
Mon Oct 12 03:13:30.968 UTC
BGP routing table entry for 10.9.9.0/24, Route Distinguisher: 65001:1
Versions:
  Process           bRIB/RIB   SendTblVer
  Speaker                 26           26
    Flags: 0x00041001+0x00000000+0x00000000 
Last Modified: Oct 12 03:10:25.023 for 00:03:06
Paths: (1 available, best #1)
  Not advertised to any peer
  Path #1: Received by speaker 0
  Flags: 0x2000000085068005+0x00, import: 0x39f
  Not advertised to any peer
  Local, (received & used)
    1.1.1.1 (metric 3) from 1.1.1.1 (1.1.1.1), if-handle 0x00000000
      Received Label 24006 
      Origin incomplete, metric 20, localpref 100, valid, internal, best, group-best, import-candidate, imported
      Received Path ID 0, Local Path ID 1, version 26
      Extended community: OSPF domain-id:0x5:0x000000000001 OSPF route-type:0:5:0x1 OSPF router-id:100.1.1.1 RT:65001:100 
      Source AFI: VPNv4 Unicast, Source VRF: CUST-A, Source Route Distinguisher: 65001:1

CE-A2 receives it as an external route.

STEP 4: 10.9.9.0/24 as seen by CE-A2
RP/0/RP0/CPU0:CE-A2#show route 10.9.9.0/24
Mon Oct 12 03:15:13.468 UTC

Routing entry for 10.9.9.0/24
  Known via "ospf 1", distance 110, metric 20
  Tag 3489725929, type extern 2
  Installed Oct 12 03:10:25.001 for 00:04:48
  Routing Descriptor Blocks
    172.16.2.1, from 100.2.2.2, via GigabitEthernet0/0/0/0
      Route metric is 20
  No advertising protos. 

Verification configs and show output

For every STEP the following files were collected from all six routers, separated per router. The verification config is ..._run.txt (the final state is the one from the last STEP).

File	Contents
`..._show.txt`	`show version` / `show interface description` / the OSPF, LDP, BGP and VRF show commands
`..._log.txt`	`show logging` limited to that STEP
`..._run.txt`	`show running-config` at that STEP (the verification config)
`..._ping.txt`	`ping` and `traceroute` for that STEP
`..._trace.txt`	`show bgp trace` (PEs only)
`..._commit.cfg`	The configuration actually committed in that STEP (only on routers that changed)

STEP 0: Initial state (no domain-id, i.e. NULL)

Router	show	syslog	running-config	ping	trace	committed config
R-A1	show	log	run	ping	-	-
CE-A1	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	-
CE-A2	show	log	run	ping	-	-

STEP 1: The same domain-id on PE1 and PE2

Router	show	syslog	running-config	ping	trace	committed config
R-A1	show	log	run	ping	-	-
CE-A1	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	commit
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	commit
CE-A2	show	log	run	ping	-	-

STEP 2: A different domain-id on PE2 only

Router	show	syslog	running-config	ping	trace	committed config
R-A1	show	log	run	ping	-	-
CE-A1	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	commit
CE-A2	show	log	run	ping	-	-

STEP 3: Put the original value back on PE2

Router	show	syslog	running-config	ping	trace	committed config
R-A1	show	log	run	ping	-	-
CE-A1	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	commit
CE-A2	show	log	run	ping	-	-

STEP 4: Redistribute a static route (10.9.9.0/24) into OSPF on R-A1

Router	show	syslog	running-config	ping	trace	committed config
R-A1	show	log	run	ping	-	commit
CE-A1	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	-
CE-A2	show	log	run	ping	-	-

STEP 5: Remove that static route (final state)

Router	show	syslog	running-config	ping	trace	committed config
R-A1	show	log	run	ping	-	commit
CE-A1	show	log	run	ping	-	-
PE1	show	log	run	ping	trace	-
P1	show	log	run	ping	-	-
PE2	show	log	run	ping	trace	-
CE-A2	show	log	run	ping	-	-

Packet captures were taken per STEP.

STEP	PE1-P1	PE2-CE-A2
0	pcap	pcap
1	pcap	pcap
2	pcap	pcap
3	pcap	pcap
4	pcap	pcap
5	pcap	pcap