EPVN MPLS label parsing error
Hi BIRDians, Been tinkering with EPVN in (built from git on Debian 13.1) hooked into an Arista vEOS-LAB network, with an IPv6 underlay. Have all the BGP sessions up and running, pushing MACs and their next hops, receiving traffic on the VXLAN interface, all good. But I'm not seeing the traffic coming out of the linux bridge to the respective VLAN subinterface for processing. Digging round I see this in the bridge FDB: <snip> 00:50:00:00:12:01 dev vxlan100 vlan 40 extern_learn master br1 42:a5:d1:11:88:4b dev vxlan100 vlan 40 master br1 permanent 42:a5:d1:11:88:4b dev vxlan100 master br1 permanent 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:ffff:1ad::403 vni 10040 self extern_learn permanent 00:50:00:00:12:01 dev vxlan100 dst 2001:db8:ffff:1ad::401 vni 627 self extern_learn permanent 627? the VNI is 10040. root@eve-deb:~# bridge vlan show port vlan-id br1 1 PVID Egress Untagged vxlan100 40 root@eve-deb:~# bridge vni show dev vni group/remote root@eve-deb:~# checking evpntab and etab: bird> sh route table evpntab Table evpntab: evpn mac 1004003:10040 0 fe:40:19:7e:2a:79 * unicast [SPINE1_EVPN 18:21:59.435 from 2001:db8:ffff:1aa::1] * (100/?) [AS4201004001i] via 2001:db8:ffff:1f3::8 on ens4 mpls 627 evpn imet 5001:10040 0 2001:db8:ffff:1ad::501 [evpn1 18:11:30.067] * (120) evpn imet 1004003:10040 0 2001:db8:ffff:1ad::401 [SPINE1_EVPN 18:07:31.985 from 2001:db8:ffff:1aa::1] * (100) [AS4201004001i] evpn imet 1004003:10040 0 2001:db8:ffff:1ad::403 [SPINE1_EVPN 18:07:31.987 from 2001:db8:ffff:1aa::1] * (100) [AS4201004003i] evpn mac 1004003:10040 0 00:50:00:00:12:01 * unicast [SPINE1_EVPN 18:21:43.590 from 2001:db8:ffff:1aa::1] * (100/?) [AS4201004001i] via 2001:db8:ffff:1f3::8 on ens4 mpls 627 bird> sh route table etab Table etab: 00:00:00:00:00:00 vlan 40 unicast [evpn1 18:11:30.067] * (80) via 2001:db8:ffff:1ad::403 on vxlan100 mpls 10040 42:a5:d1:11:88:4b unicast [bridge1 16:58:39.461] * L (0) dev vxlan100 fe:40:19:7e:2a:79 vlan 40 unicast [evpn1 18:21:59.434] * (80) via 2001:db8:ffff:1ad::401 on vxlan100 mpls 627 00:50:00:00:12:01 vlan 40 unicast [evpn1 18:21:43.590] * (80) via 2001:db8:ffff:1ad::401 on vxlan100 mpls 627 06:06:4f:2b:a5:e6 unicast [bridge1 16:54:40.358] * L (0) dev br1 bird> Where's it coming from? On the single (so far) attached Arista spine we see this: dc1-spine-1#sh bgp evpn BGP routing table information for VRF default Router identifier 192.168.1.10, local AS number 4201003001 Route status codes: * - valid, > - active, S - Stale, E - ECMP head, e - ECMP c - Contributing to ECMP, % - Pending best path selection Origin codes: i - IGP, e - EGP, ? - incomplete AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop Network Next Hop Metric LocPref Weight Path * > RD: 1004003:10040 mac-ip 0050.0000.1201 2001:db8:ffff:1ad::401 - 100 0 4201004001 i * > RD: 1004003:10040 imet 2001:db8:ffff:1ad::401 2001:db8:ffff:1ad::401 - 100 0 4201004001 i * > RD: 1004003:10040 imet 2001:db8:ffff:1ad::403 2001:db8:ffff:1ad::403 - 100 0 4201004003 i * > RD: 5001:10040 imet 2001:db8:ffff:1ad::501 2001:db8:ffff:1aa::501 - 100 0 4201005001 i dc1-spine-1# 2001:db8:ffff:1ad::501 is the loopback that the vxlan interface is using. Something going on weird with the RT processing. Having a look at a packet dump of an EVPN update on the wire: 18:21:59.434664 IP6 (class 0xc0, flowlabel 0x59dd9, hlim 3, next-header TCP (6) payload length: 151) 2001:db8:ffff:1aa::1.40949 > 2001:db8:ffff:1aa::501.179: Flags [P.], cksum 0x107d (correct), seq 4225498746:4225498865, ack 3651948816, win 498, options [nop,nop,TS val 3223710250 ecr 3856730090], length 119: BGP Update Message (2), length: 119 Origin (1), length: 1, Flags [T]: IGP 0x0000: 00 AS Path (2), length: 10, Flags [T]: 4201003001 4201004001 0x0000: 0202 fa66 37f9 fa66 3be1 Multi-Protocol Reach NLRI (14), length: 56, Flags [OE]: AFI: VPLS (25), SAFI: EVPN (70) no AFI 25 / SAFI 70 decoder 0x0000: 0019 4610 2001 0db8 ffff 01ad 0000 0000 0x0010: 0000 0401 0002 2100 0200 0f51 e327 3800 0x0020: 0000 0000 0000 0000 0000 0000 0030 fe40 0x0030: 197e 2a79 0000 2738 Extended Community (16), length: 16, Flags [OT]: target (0x0202), Flags [none]: 1004003:10040 encapsulation (0x030c), Flags [none]: Tunnel type: VXLAN 0x0000: 0202 000f 51e3 2738 030c 0000 0000 0008 The RT there is showing correctly. Hang on... 627 HEX is 273... looks like the 2 bytes of the latter half of the RT is being trimmed by 4 bits. I couldn't get a hang of the code to try and ID where it's playing up so sorry, no patch. Also, I'm not able to use 4-byte ASNs in the first part of the RT. The docs indicate that we should be able to, but the config barfs with out-of-range (was trying to use 1005001 - a subset of the ASNs being used). Config: filter SPINE_LO { if net.len = 128 then accept; reject; } evpn table evpntab; protocol bgp SPINE1_BGP { ipv6 { import all; export filter SPINE_LO; }; local 2001:db8:ffff:1f3::9 as 4201005001; neighbor 2001:db8:ffff:1f3::8 as 4201003001; } protocol bgp SPINE1_EVPN { evpn { import all; export all; }; local 2001:db8:ffff:1aa::501 as 4201005001; neighbor 2001:db8:ffff:1aa::1 as 4201003001; multihop; } eth table etab; protocol bridge { eth { table etab; export all; }; bridge device "br1"; debug all; }; protocol evpn { eth { table etab; }; evpn { }; rd 5001:10040; import target (rt, 1004001, 10040); import target (rt, 1004003, 10040); export target (rt, 5001, 10040); tunnel device "vxlan100"; router address 2001:db8:ffff:1ad::501; vni 10040; vid 40; debug all; }; Took a bit to work it all out based on a previous thread and the testing config, but got there in the end, then found this :( If you're able to spin up a patch I can re-compile with it and test. Regards, William -- This email has been checked for viruses by Avast antivirus software. www.avast.com
On Fri, Oct 10, 2025 at 06:52:28PM +1100, William via Bird-users wrote:
Hi BIRDians, Been tinkering with EPVN in (built from git on Debian 13.1) hooked into an Arista vEOS-LAB network, with an IPv6 underlay.
Hi I am glad to hear someone is playing with it. Do you use the 'evpn' branch?
Have all the BGP sessions up and running, pushing MACs and their next hops, receiving traffic on the VXLAN interface, all good.
But I'm not seeing the traffic coming out of the linux bridge to the respective VLAN subinterface for processing. Digging round I see this in the bridge FDB:
<snip> 00:50:00:00:12:01 dev vxlan100 vlan 40 extern_learn master br1 42:a5:d1:11:88:4b dev vxlan100 vlan 40 master br1 permanent 42:a5:d1:11:88:4b dev vxlan100 master br1 permanent 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:ffff:1ad::403 vni 10040 self extern_learn permanent 00:50:00:00:12:01 dev vxlan100 dst 2001:db8:ffff:1ad::401 vni 627 self extern_learn permanent
627? the VNI is 10040.
Also note that on IMET route, the VNI is decoded correctly, only for MAC route it is decoded incorrectly.
Having a look at a packet dump of an EVPN update on the wire: 18:21:59.434664 IP6 (class 0xc0, flowlabel 0x59dd9, hlim 3, next-header TCP (6) payload length: 151) 2001:db8:ffff:1aa::1.40949 > 2001:db8:ffff:1aa::501.179: Flags [P.], cksum 0x107d (correct), seq 4225498746:4225498865, ack 3651948816, win 498, options [nop,nop,TS val 3223710250 ecr 3856730090], length 119: BGP Update Message (2), length: 119 Origin (1), length: 1, Flags [T]: IGP 0x0000: 00 AS Path (2), length: 10, Flags [T]: 4201003001 4201004001 0x0000: 0202 fa66 37f9 fa66 3be1 Multi-Protocol Reach NLRI (14), length: 56, Flags [OE]: AFI: VPLS (25), SAFI: EVPN (70) no AFI 25 / SAFI 70 decoder 0x0000: 0019 4610 2001 0db8 ffff 01ad 0000 0000 0x0010: 0000 0401 0002 2100 0200 0f51 e327 3800 0x0020: 0000 0000 0000 0000 0000 0000 0030 fe40 0x0030: 197e 2a79 0000 2738 Extended Community (16), length: 16, Flags [OT]: target (0x0202), Flags [none]: 1004003:10040 encapsulation (0x030c), Flags [none]: Tunnel type: VXLAN 0x0000: 0202 000f 51e3 2738 030c 0000 0000 0008
The RT there is showing correctly.
Hang on... 627 HEX is 273... looks like the 2 bytes of the latter half of the RT is being trimmed by 4 bits.
First, note that the VNI is not decoded from RT, but from the MPLS field of Multi-Protocol Reach NLRI (14) - last 24 bits (00 2738). It is just a convention for RT to use the same value as VNI. IIRC there is a confusion about this field w.r.t. EVPN, as EVPN can use both MPLS and VXLAN as underlying transport, VXLAN VNIs are 24 bit long, but MPLS labels are 20 bit long and unfortunately padded from the LSB side. So it si read as MPLS label, ignoring the last four bits.
I couldn't get a hang of the code to try and ID where it's playing up so sorry, no patch.
Also, I'm not able to use 4-byte ASNs in the first part of the RT. The docs indicate that we should be able to, but the config barfs with out-of-range (was trying to use 1005001 - a subset of the ASNs being used).
In which context? From the config file below i see 'import target (rt, 1004001, 10040)', so 32bit ASN.
Took a bit to work it all out based on a previous thread and the testing config, but got there in the end, then found this :(
If you're able to spin up a patch I can re-compile with it and test.
Will look on both issues and make a fix, probably during weekend. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) "To err is human -- to blame it on a computer is even more so."
Hi Ondrej, Thanks for the fast reply! Just noticed the assorted typo's in the email. On 10/10/2025 11:42 pm, Ondrej Zajicek wrote:
On Fri, Oct 10, 2025 at 06:52:28PM +1100, William via Bird-users wrote:
Hi BIRDians, Been tinkering with EPVN in (built from git on Debian 13.1) hooked into an Arista vEOS-LAB network, with an IPv6 underlay. Hi
I am glad to hear someone is playing with it. Do you use the 'evpn' branch?
Yes I'm using the evpn branch: BIRD v2.13.1-161-gc5c9bd81-x ready.
Have all the BGP sessions up and running, pushing MACs and their next hops, receiving traffic on the VXLAN interface, all good.
But I'm not seeing the traffic coming out of the linux bridge to the respective VLAN subinterface for processing. Digging round I see this in the bridge FDB:
<snip> 00:50:00:00:12:01 dev vxlan100 vlan 40 extern_learn master br1 42:a5:d1:11:88:4b dev vxlan100 vlan 40 master br1 permanent 42:a5:d1:11:88:4b dev vxlan100 master br1 permanent 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:ffff:1ad::403 vni 10040 self extern_learn permanent 00:50:00:00:12:01 dev vxlan100 dst 2001:db8:ffff:1ad::401 vni 627 self extern_learn permanent
627? the VNI is 10040. Also note that on IMET route, the VNI is decoded correctly, only for MAC route it is decoded incorrectly.
Having a look at a packet dump of an EVPN update on the wire: 18:21:59.434664 IP6 (class 0xc0, flowlabel 0x59dd9, hlim 3, next-header TCP (6) payload length: 151) 2001:db8:ffff:1aa::1.40949 > 2001:db8:ffff:1aa::501.179: Flags [P.], cksum 0x107d (correct), seq 4225498746:4225498865, ack 3651948816, win 498, options [nop,nop,TS val 3223710250 ecr 3856730090], length 119: BGP Update Message (2), length: 119 Origin (1), length: 1, Flags [T]: IGP 0x0000: 00 AS Path (2), length: 10, Flags [T]: 4201003001 4201004001 0x0000: 0202 fa66 37f9 fa66 3be1 Multi-Protocol Reach NLRI (14), length: 56, Flags [OE]: AFI: VPLS (25), SAFI: EVPN (70) no AFI 25 / SAFI 70 decoder 0x0000: 0019 4610 2001 0db8 ffff 01ad 0000 0000 0x0010: 0000 0401 0002 2100 0200 0f51 e327 3800 0x0020: 0000 0000 0000 0000 0000 0000 0030 fe40 0x0030: 197e 2a79 0000 2738 Extended Community (16), length: 16, Flags [OT]: target (0x0202), Flags [none]: 1004003:10040 encapsulation (0x030c), Flags [none]: Tunnel type: VXLAN 0x0000: 0202 000f 51e3 2738 030c 0000 0000 0008
The RT there is showing correctly.
Hang on... 627 HEX is 273... looks like the 2 bytes of the latter half of the RT is being trimmed by 4 bits. First, note that the VNI is not decoded from RT, but from the MPLS field of Multi-Protocol Reach NLRI (14) - last 24 bits (00 2738). It is just a convention for RT to use the same value as VNI.
IIRC there is a confusion about this field w.r.t. EVPN, as EVPN can use both MPLS and VXLAN as underlying transport, VXLAN VNIs are 24 bit long, but MPLS labels are 20 bit long and unfortunately padded from the LSB side. So it si read as MPLS label, ignoring the last four bits.
Gotta love standards! So that brings an interesting side case I wouldn't have thought of - the "usable" VNI range is trimmed due to this?
I couldn't get a hang of the code to try and ID where it's playing up so sorry, no patch.
Also, I'm not able to use 4-byte ASNs in the first part of the RT. The docs indicate that we should be able to, but the config barfs with out-of-range (was trying to use 1005001 - a subset of the ASNs being used). In which context? From the config file below i see 'import target (rt, 1004001, 10040)', so 32bit ASN.
That was one of my typo's - I meant route descriptor, not route target.
Took a bit to work it all out based on a previous thread and the testing config, but got there in the end, then found this :(
If you're able to spin up a patch I can re-compile with it and test. Will look on both issues and make a fix, probably during weekend.
Thank you! Regards, William -- This email has been checked for viruses by Avast antivirus software. www.avast.com
On Sat, Oct 11, 2025 at 08:46:00AM +1100, William via Bird-users wrote:
Hi Ondrej,
Thanks for the fast reply! Just noticed the assorted typo's in the email.
On 10/10/2025 11:42 pm, Ondrej Zajicek wrote:
On Fri, Oct 10, 2025 at 06:52:28PM +1100, William via Bird-users wrote:
Hi BIRDians, Been tinkering with EPVN in (built from git on Debian 13.1) hooked into an Arista vEOS-LAB network, with an IPv6 underlay. Hi
I am glad to hear someone is playing with it. Do you use the 'evpn' branch?
Yes I'm using the evpn branch:
BIRD v2.13.1-161-gc5c9bd81-x ready.
You could also try more recent branch 'oz-evpn' (although the following patch related to VNIs is also not included there). There are some configuration changes in this branch, encapsulation-specific options are in its own subblock: protocol evpn { eth { table etab2; }; evpn; encapsulation vxlan { tunnel device "vxlan2"; router address 10.1.1.1; }; rd 1:12; route target (rt, 1, 0); tag 2; vni 12; };
Hang on... 627 HEX is 273... looks like the 2 bytes of the latter half of the RT is being trimmed by 4 bits. First, note that the VNI is not decoded from RT, but from the MPLS field of Multi-Protocol Reach NLRI (14) - last 24 bits (00 2738). It is just a convention for RT to use the same value as VNI.
IIRC there is a confusion about this field w.r.t. EVPN, as EVPN can use both MPLS and VXLAN as underlying transport, VXLAN VNIs are 24 bit long, but MPLS labels are 20 bit long and unfortunately padded from the LSB side. So it si read as MPLS label, ignoring the last four bits.
Gotta love standards! So that brings an interesting side case I wouldn't have thought of - the "usable" VNI range is trimmed due to this?
No, not really. It is just that EVPN/MPLS (RFC 7432) uses high 20 bits out of 24, while EVPN/VXLAN uses full 24 bits. But the BIRD EVPN BGP does not really care about whether it is MPLS or VXLAN (it could be any encapsulation if it is just an EVPN route reflector). The attached patch switches BGP code to use full 24 bits.
I couldn't get a hang of the code to try and ID where it's playing up so sorry, no patch.
Also, I'm not able to use 4-byte ASNs in the first part of the RT. The docs indicate that we should be able to, but the config barfs with out-of-range (was trying to use 1005001 - a subset of the ASNs being used). In which context? From the config file below i see 'import target (rt, 1004001, 10040)', so 32bit ASN.
That was one of my typo's - I meant route descriptor, not route target.
rd / route distinguisher? It works for me even there: protocol evpn { ... # rd 1:13; rd 1005001:10040; route target (rt, 1, 0); ... } -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) "To err is human -- to blame it on a computer is even more so."
Hi Ondrej, Apologies on the delay, I'm testing when I have the brain space and my lab is powered up. On 14/10/2025 3:02 am, Ondrej Zajicek wrote:
On Sat, Oct 11, 2025 at 08:46:00AM +1100, William via Bird-users wrote:
Hi Ondrej,
Thanks for the fast reply! Just noticed the assorted typo's in the email.
On 10/10/2025 11:42 pm, Ondrej Zajicek wrote:
On Fri, Oct 10, 2025 at 06:52:28PM +1100, William via Bird-users wrote:
Hi BIRDians, Been tinkering with EPVN in (built from git on Debian 13.1) hooked into an Arista vEOS-LAB network, with an IPv6 underlay. Hi
I am glad to hear someone is playing with it. Do you use the 'evpn' branch? Yes I'm using the evpn branch:
BIRD v2.13.1-161-gc5c9bd81-x ready. You could also try more recent branch 'oz-evpn' (although the following patch related to VNIs is also not included there).
There are some configuration changes in this branch, encapsulation-specific options are in its own subblock:
protocol evpn { eth { table etab2; }; evpn;
encapsulation vxlan { tunnel device "vxlan2"; router address 10.1.1.1; };
rd 1:12; route target (rt, 1, 0);
tag 2; vni 12; }; I wonder if there is something different in the way the IMET routes are put together there? The "encapsulation vxlan" stanza doesn't exist in the main evpn branch which is making me think that might be causing another issue... <snip>
Gotta love standards! So that brings an interesting side case I wouldn't have thought of - the "usable" VNI range is trimmed due to this? No, not really. It is just that EVPN/MPLS (RFC 7432) uses high 20 bits out of 24, while EVPN/VXLAN uses full 24 bits. But the BIRD EVPN BGP does not really care about whether it is MPLS or VXLAN (it could be any encapsulation if it is just an EVPN route reflector).
The attached patch switches BGP code to use full 24 bits. Patched my code and it works nicely! Thank you :) <snip> rd / route distinguisher? It works for me even there:
protocol evpn { ... # rd 1:13; rd 1005001:10040; route target (rt, 1, 0); ... } I've set the RD and RT back and it's working fine, not sure what was going on there
I've got mac-ip and imet routes floating around now, but the next issue is that the wrong next-hop is being set on the imet (and mac-ip) routes. Here's the evpn protocol definition: protocol evpn { eth { table etab; import all; }; evpn { }; rd 1005001:10040; import target (rt, 1004001, 10040); import target (rt, 1004003, 10040); export target (rt, 1005001, 10040); #encapsulation vxlan { tunnel device "vxlan100"; router address 2001:db8:ffff:1ad::501; #}; vni 10040; vid 40; debug all; }; vxlan100 is tied to lo100 with IP 2001:db8:ffff:1ad::501, but the EVPN session is tied to lo10 which has 2001:db8:ffff:1aa::501/128 (distributed via BGP IPv6 sessions to the spines). Here's the vxlan interface under the hood: # ip -d link show vxlan100 13: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br1 state UNKNOWN mode DEFAULT group default qlen 1000 link/ether 42:a5:d1:11:88:4b brd ff:ff:ff:ff:ff:ff promiscuity 1 allmulti 1 minmtu 68 maxmtu 65535 vxlan id 0 local 2001:db8:ffff:1ad::501 srcport 0 0 dstport 4789 ttl auto ageing 300 external nolearning <snip> # bridge vlan show port vlan-id br1 1 PVID Egress Untagged vxlan100 40 vl40 40 PVID Egress Untagged # However when I look at the Arista vEOS Lab nodes, I see this ('sh bgp evpn vni 10040 detail', just the relevant portions pertaining to bird): BGP routing table entry for mac-ip 42a5.d111.884b, Route Distinguisher: 1005001:10040 Paths: 1 available 4201003001 4201005001 2001:db8:ffff:1aa::501 from 2001:db8:ffff:1aa::1 (192.168.1.10) Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, best Extended Community: Route-Target-AS4:1005001:10040 VNI: 10040 ESI: 0000:0000:0000:0000:0000 BGP routing table entry for mac-ip 7e59.8de3.ed71, Route Distinguisher: 1005001:10040 Paths: 1 available 4201003001 4201005001 2001:db8:ffff:1aa::501 from 2001:db8:ffff:1aa::1 (192.168.1.10) Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, best Extended Community: Route-Target-AS4:1005001:10040 VNI: 10040 ESI: 0000:0000:0000:0000:0000 <snip> BGP routing table entry for imet 2001:db8:ffff:1ad::501, Route Distinguisher: 1005001:10040 Paths: 1 available 4201003001 4201005001 2001:db8:ffff:1aa::501 from 2001:db8:ffff:1aa::1 (192.168.1.10) Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, best Extended Community: Route-Target-AS4:1005001:10040 VNI: 10040 PMSI Tunnel: Ingress Replication, MPLS Label: 10040, Leaf Information Required: false, Tunnel ID: 2001:db8:ffff:1ad::501 Looking at other imet and mac-ip routes I can see that the next hop is correct for those nodes. Here's another example for one of the Arista leafs: BGP routing table entry for mac-ip 0050.0000.2c01, Route Distinguisher: 1004003:10040 Paths: 2 available 4201003001 4201004003 2001:db8:ffff:1ad::403 from 2001:db8:ffff:1aa::1 (192.168.1.10) Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, ECMP head, ECMP, best, ECMP contributor Extended Community: Route-Target-AS4:1004003:10040 TunnelEncap:tunnelTypeVxlan VNI: 10040 ESI: 0000:0000:0000:0000:0000 4201003001 4201004003 2001:db8:ffff:1ad::403 from 2001:db8:ffff:1aa::2 (192.168.1.11) Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, ECMP, ECMP contributor Extended Community: Route-Target-AS4:1004003:10040 TunnelEncap:tunnelTypeVxlan VNI: 10040 ESI: 0000:0000:0000:0000:0000 BGP routing table entry for imet 2001:db8:ffff:1ad::403, Route Distinguisher: 1004003:10040 Paths: 2 available 4201003001 4201004003 2001:db8:ffff:1ad::403 from 2001:db8:ffff:1aa::2 (192.168.1.11) Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, ECMP head, ECMP, best, ECMP contributor Extended Community: Route-Target-AS4:1004003:10040 TunnelEncap:tunnelTypeVxlan VNI: 10040 PMSI Tunnel: Ingress Replication, MPLS Label: 10040, Leaf Information Required: false, Tunnel ID: 2001:db8:ffff:1ad::403 4201003001 4201004003 2001:db8:ffff:1ad::403 from 2001:db8:ffff:1aa::1 (192.168.1.10) Origin IGP, metric -, localpref 100, weight 0, tag 0, valid, external, ECMP, ECMP contributor Extended Community: Route-Target-AS4:1004003:10040 TunnelEncap:tunnelTypeVxlan VNI: 10040 PMSI Tunnel: Ingress Replication, MPLS Label: 10040, Leaf Information Required: false, Tunnel ID: 2001:db8:ffff:1ad::403 It shows the proper next-hop IP for remote hosts. That information is happily put into the bridge FDB: 00:50:00:00:12:01 dev vxlan100 vlan 40 extern_learn master br1 00:50:00:00:2c:01 dev vxlan100 vlan 40 extern_learn master br1 42:a5:d1:11:88:4b dev vxlan100 vlan 40 master br1 permanent 42:a5:d1:11:88:4b dev vxlan100 master br1 permanent 00:50:00:00:2c:01 dev vxlan100 dst 2001:db8:ffff:1ad::403 vni 10040 self extern_learn permanent 00:50:00:00:12:01 dev vxlan100 dst 2001:db8:ffff:1ad::401 vni 10040 self extern_learn permanent 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:ffff:1ad::403 vni 10040 self extern_learn permanent 7e:59:8d:e3:ed:71 dev vl40 vlan 40 master br1 permanent 7e:59:8d:e3:ed:71 dev vl40 master br1 permanent 33:33:00:00:00:01 dev vl40 self permanent 01:00:5e:00:00:01 dev vl40 self permanent bird> sh route table etab Table etab: 7e:59:8d:e3:ed:71 vlan 40 unicast [bridge1 18:45:27.117] * L (0) dev vl40 00:00:00:00:00:00 vlan 40 unicast [evpn1 18:58:54.908] * (80) via 2001:db8:ffff:1ad::403 on vxlan100 mpls 10040 42:a5:d1:11:88:4b vlan 40 unicast [bridge1 18:45:27.117] * L (0) dev vxlan100 06:06:4f:2b:a5:e6 vlan 1 unicast [bridge1 18:45:27.117] * L (0) dev br1 00:50:00:00:2c:01 vlan 40 unicast [evpn1 21:41:13.227] * (80) via 2001:db8:ffff:1ad::403 on vxlan100 mpls 10040 00:50:00:00:12:01 vlan 40 unicast [evpn1 21:43:01.278] * (80) via 2001:db8:ffff:1ad::401 on vxlan100 mpls 10040 bird> sh route table evpntab Table evpntab: evpn imet 1005001:10040 0 2001:db8:ffff:1ad::501 [evpn1 18:58:54.908] * (120) evpn imet 1004003:10040 0 2001:db8:ffff:1ad::401 [SPINE1_EVPN 17:52:18.800 from 2001:db8:ffff:1aa::1] * (100) [AS4201004001i] evpn imet 1004003:10040 0 2001:db8:ffff:1ad::403 [SPINE1_EVPN 17:52:18.800 from 2001:db8:ffff:1aa::1] * (100) [AS4201004003i] evpn mac 1005001:10040 0 42:a5:d1:11:88:4b * mpls 10040 [evpn1 18:58:54.908] * (120) evpn mac 1005001:10040 0 7e:59:8d:e3:ed:71 * mpls 10040 [evpn1 18:58:54.908] * (120) evpn mac 1004003:10040 0 00:50:00:00:2c:01 * unicast [SPINE1_EVPN 21:41:13.227 from 2001:db8:ffff:1aa::1] * (100/?) [AS4201004003i] via 2001:db8:ffff:1f3::8 on ens4 mpls 10040 evpn mac 1004003:10040 0 00:50:00:00:12:01 * unicast [SPINE1_EVPN 21:43:01.278 from 2001:db8:ffff:1aa::1] * (100/?) [AS4201004001i] via 2001:db8:ffff:1f3::8 on ens4 mpls 10040 bird> bird> sh route table master6 Table master6: 2001:db8:ffff:1aa::1/128 unicast [SPINE1_BGP 17:52:15.287] * (100) [AS4201003001i] via 2001:db8:ffff:1f3::8 on ens4 2001:db8:ffff:1ad::501/128 unicast [direct1 17:52:13.958] * (240) dev lo100 2001:db8:ffff:1aa::401/128 unicast [SPINE1_BGP 17:52:15.287] * (100) [AS4201004001i] via 2001:db8:ffff:1f3::8 on ens4 2001:db8:ffff:1ad::401/128 unicast [SPINE1_BGP 17:52:15.287] * (100) [AS4201004001i] via 2001:db8:ffff:1f3::8 on ens4 2001:db8:ffff:1aa::501/128 unicast [direct1 17:52:13.958] * (240) dev lo10 2001:db8:ffff:1aa::402/128 unicast [SPINE1_BGP 17:52:15.287] * (100) [AS4201004001i] via 2001:db8:ffff:1f3::8 on ens4 2001:db8:ffff:1ad::403/128 unicast [SPINE1_BGP 17:52:15.287] * (100) [AS4201004003i] via 2001:db8:ffff:1f3::8 on ens4 2001:db8:ffff:1f3::8/127 unicast [direct1 17:52:13.958] * (240) dev ens4 2001:db8:ffff:1aa::2/128 unicast [SPINE1_BGP 17:52:15.331] * (100) [AS4201003001i] via 2001:db8:ffff:1f3::8 on ens4 2001:db8:ffff:1aa::403/128 unicast [SPINE1_BGP 17:52:15.287] * (100) [AS4201004003i] via 2001:db8:ffff:1f3::8 on ens4 bird> I don't know if it's something weird I've done or if my implementation is just whacked out but this seems to be the last hurdle I'm seeing BUM coming through from one of the hosts attached to a leaf: 21:43:47.974972 IP6 2001:db8:ffff:1ad::401.60846 > 2001:db8:ffff:1ad::501.4789: VXLAN, flags [I] (0x08), vni 10040 ARP, Request who-has 100.64.40.254 tell 100.64.40.18, length 46 21:43:47.976451 IP6 2001:db8:ffff:1ad::401.60846 > 2001:db8:ffff:1aa::501.4789: VXLAN, flags [I] (0x08), vni 10040 ARP, Request who-has 100.64.40.254 tell 100.64.40.18, length 46 but it's going to both loopback addresses (the BGP one and the vxlan terminator) For what it's worth, 'bridge vlan add dev vxlan100 vid 40 tunnel_info id 40' errors out. The readme on https://gitlab.nic.cz/labs/bird-tools/-/tree/master/netlab/cf-evpn-bgp has typos in the commands too. Feel free to bug me off-list for more detail, this is getting long-winded... Regards, William -- This email has been checked for viruses by Avast antivirus software. www.avast.com
On Tue, Nov 04, 2025 at 09:48:29PM +1100, William via Bird-users wrote:
Hi Ondrej,
Apologies on the delay, I'm testing when I have the brain space and my lab is powered up.
Hi Thanks for your feedback!
There are some configuration changes in this branch, encapsulation-specific options are in its own subblock:
protocol evpn { eth { table etab2; }; evpn;
encapsulation vxlan { tunnel device "vxlan2"; router address 10.1.1.1; };
rd 1:12; route target (rt, 1, 0);
tag 2; vni 12; }; I wonder if there is something different in the way the IMET routes are put together there? The "encapsulation vxlan" stanza doesn't exist in the main evpn branch which is making me think that might be causing another issue...
The "encapsulation vxlan" stanza is just for encompassing encapsulation-specific options, to future-proof for possible implementations of multiple encapsulations, but that should not matter now. Note that 'oz-evpn' is the main EVPN branch, while 'evpn' is more-less an old snapshot. There are some more changes in 'oz-evpn', like fix for BGP_PMSI_TUNNEL attribute flags (that is used in IMET routes) or announcing encapsulation extended communities.
I've got mac-ip and imet routes floating around now, but the next issue is that the wrong next-hop is being set on the imet (and mac-ip) routes.
Seems like a deviation on our side. EVPN protocol does not attach the bgp_next_hop attribute, and such attribute is attached by BGP when announced to BGP neighbor, based on BGP session endpoint address. We should fix that, meanwhile you can workaround it by setting 'next hop address' BGP channel option (in BGP EVPN channel) to the VNI address: https://bird.nic.cz/doc/bird-2.17.2.html#bgp-next-hop-address
Here's the evpn protocol definition: protocol evpn { eth { table etab; import all; }; evpn { }; rd 1005001:10040; import target (rt, 1004001, 10040); import target (rt, 1004003, 10040); export target (rt, 1005001, 10040);
Note that setting multiple 'import target' options just replace the previous ones with the last one. You need to use a set syntax: import target [(rt, 1004001, 10040), (rt, 1004003, 10040)]; That is likely a bit confusing and at least should generate a warning.
For what it's worth, 'bridge vlan add dev vxlan100 vid 40 tunnel_info id 40' errors out. The readme on https://gitlab.nic.cz/labs/bird-tools/-/tree/master/netlab/cf-evpn-bgp has typos in the commands too.
You mean typo del->add ? instead of: bridge vlan del dev $TUNNEL vid $VID tunnel_info id $VNI should be: bridge vlan add dev $TUNNEL vid $VID tunnel_info id $VNI Note that with the latest 'oz-evpn' branch, BIRD configures VLANs on VXLAN device automatically, but currently only for tagged vlans (as in https://gitlab.nic.cz/labs/bird-tools/-/blob/master/netlab/cf-evpn-bgp-tags/... ) -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) "To err is human -- to blame it on a computer is even more so."
On 5/11/2025 2:57 am, Ondrej Zajicek wrote:
On Tue, Nov 04, 2025 at 09:48:29PM +1100, William via Bird-users wrote:
Hi Ondrej,
Apologies on the delay, I'm testing when I have the brain space and my lab is powered up. Hi
Thanks for your feedback! Always glad to be able to contribute back in some way 🙂 The "encapsulation vxlan" stanza is just for encompassing encapsulation-specific options, to future-proof for possible implementations of multiple encapsulations, but that should not matter now.
Note that 'oz-evpn' is the main EVPN branch, while 'evpn' is more-less an old snapshot.
There are some more changes in 'oz-evpn', like fix for BGP_PMSI_TUNNEL attribute flags (that is used in IMET routes) or announcing encapsulation extended communities. The documentation is missing a chunk about the encapsulation, but I also had to resort to config.Y to find the VLAN definition block.
Adding vni <vni>; to the main protocol config breaks it, but works fine in the vlan <vid> block. Still needed vid <vid> in the main protocol config though. See attached working config for reference.
I've got mac-ip and imet routes floating around now, but the next issue is that the wrong next-hop is being set on the imet (and mac-ip) routes. Seems like a deviation on our side. EVPN protocol does not attach the bgp_next_hop attribute, and such attribute is attached by BGP when announced to BGP neighbor, based on BGP session endpoint address.
We should fix that, meanwhile you can workaround it by setting 'next hop address' BGP channel option (in BGP EVPN channel) to the VNI address:
https://bird.nic.cz/doc/bird-2.17.2.html#bgp-next-hop-address
Did that as part of the evpn stanza in the EVPN BGP chunk and it worked a treat! (See attached config)
Here's the evpn protocol definition: protocol evpn { eth { table etab; import all; }; evpn { }; rd 1005001:10040; import target (rt, 1004001, 10040); import target (rt, 1004003, 10040); export target (rt, 1005001, 10040); Note that setting multiple 'import target' options just replace the previous ones with the last one.
You need to use a set syntax:
import target [(rt, 1004001, 10040), (rt, 1004003, 10040)];
That is likely a bit confusing and at least should generate a warning.
That worked it a lot better :D
For what it's worth, 'bridge vlan add dev vxlan100 vid 40 tunnel_info id 40' errors out. The readme on https://gitlab.nic.cz/labs/bird-tools/-/tree/master/netlab/cf-evpn-bgp has typos in the commands too. You mean typo del->add ?
instead of: bridge vlan del dev $TUNNEL vid $VID tunnel_info id $VNI should be: bridge vlan add dev $TUNNEL vid $VID tunnel_info id $VNI
Note that with the latest 'oz-evpn' branch, BIRD configures VLANs on VXLAN device automatically, but currently only for tagged vlans (as in https://gitlab.nic.cz/labs/bird-tools/-/blob/master/netlab/cf-evpn-bgp-tags/... )
That was correct, del needed to be changed to add. I also found that bridge(8) won't let you set the tunnel info without the VLAN being defined first. While everything *nearly* worked, after a couple of weeks I managed to work out that the bridge FDB entry that was being added was missing the source IP definition. As a result the packet that entered the vxlan interface wasn't being forwarded upstream: root@dc1-acc-1-deb:~# bridge fdb | grep vxlan 00:50:00:00:12:01 dev vxlan100 vlan 40 extern_learn master br1 42:a5:d1:11:88:4b dev vxlan100 vlan 40 master br1 permanent 42:a5:d1:11:88:4b dev vxlan100 master br1 permanent 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:ffff:1ad::401 vni 10040 self extern_learn permanent 00:50:00:00:12:01 dev vxlan100 dst 2001:db8:ffff:1ad::401 vni 10040 self extern_learn permanent root@dc1-acc-1-deb:~# bridge fdb add 00:50:00:00:13:01 dev vxlan100 vni 10040 dst 2001:db8:ffff:1ad::401 via vxlan100 src_vni 10040 root@dc1-acc-1-deb:~# bridge fdb add 00:50:00:00:12:01 dev vxlan100 vni 10040 dst 2001:db8:ffff:1ad::401 via vxlan100 src_vni 10040 root@dc1-acc-1-deb:~# With this in place it worked a treat: root@dc1-acc-1-deb:~# bridge fdb | grep vxlan 00:50:00:00:12:01 dev vxlan100 vlan 40 extern_learn master br1 42:a5:d1:11:88:4b dev vxlan100 vlan 40 master br1 permanent 42:a5:d1:11:88:4b dev vxlan100 master br1 permanent 00:50:00:00:12:01 dev vxlan100 dst 2001:db8:ffff:1ad::401 vni 10040 src_vni 10040 via vxlan100 self permanent 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:ffff:1ad::401 vni 10040 self extern_learn permanent 00:50:00:00:13:01 dev vxlan100 dst 2001:db8:ffff:1ad::401 vni 10040 src_vni 10040 via vxlan100 self permanent 00:50:00:00:12:01 dev vxlan100 dst 2001:db8:ffff:1ad::401 vni 10040 self extern_learn permanent root@dc1-acc-1-deb:~# can happily ping and throw traffic around now. Haven't tried with extra VLANs/VNI's but I think it will be ok. Probably worthy of another email, but bird segfaulted when br1 vanished (e.g. ip link delete br1). I had noticed it previously when playing around, just replicated it for good measure. 2025-12-25T21:17:56.421191+11:00 dc1-acc-1-deb kernel: bird[1353]: segfault at 10 ip 00007f1c94833d6a sp 00007fff73a44ec8 error 4 in libc.so.6[182d6a,7f1c946d9000+165000] likely on CPU 2 (core 0, socket 2) 2025-12-25T21:17:56.421193+11:00 dc1-acc-1-deb kernel: Code: 8b 04 82 29 c8 c3 66 2e 0f 1f 84 00 00 00 00 00 89 f1 89 f8 48 83 e1 3f 48 83 e0 3f 83 f9 30 77 3f 83 f8 30 77 3a f3 0f 6f 0f <f3> 0f 6f 16 66 0f ef c0 66 0f 74 c1 66 0f 74 ca 66 0f f8 c8 66 0f Merry Christmas, more work! :D Regards, William -- This email has been checked for viruses by Avast antivirus software. www.avast.com
participants (2)
-
Ondrej Zajicek -
William