Hello BIRDs! Hello Ondra. In my lab I have an all-active multihomed host to PE1 and PE2, BIRD is there as a PE3. I noticed that when both PE1 and PE2 announce a Type 2 route for the multihomed MAC (b8:69:f4:e0:4e:04) BIRD only uses one and does not fallback to the other when the first is withdrawn. I understand that this is starting to touch unsupported EVPN parts and BIRD doesn't do much with Type 1 and Type 4 routes apart of adding them to the evpn table, but it should at least handle multiple Type 2 routes for the same MAC. With ESI and LAG configured on PE1&PE2 BIRD accepts and shows all relevant routes and both Type 2s are there: ``` bird> sh route table evpntab Table evpntab: evpn mac 10.10.10.1:10010 0 b8:69:f4:e0:4e:04 * unicast [bgp_Arista 22:25:26.352 from 10.10.10.1] * (100/12) [i] via 100.90.109.98 on ens192 mpls 10010 evpn mac 10.10.10.2:10010 0 b8:69:f4:e0:4e:04 * unicast [bgp_Huawei 22:25:45.280 from 10.10.10.2] * (100/1) [?] via 100.90.109.98 on ens192 mpls 10010 evpn ead 10.10.10.1:1 4294967295 00:00:00:00:11:11:22:22:33:33 unicast [bgp_Arista 22:24:58.357 from 10.10.10.1] * (100/12) [i] via 100.90.109.98 on ens192 mpls 0 evpn ead 10.10.10.1:10010 0 00:00:00:00:11:11:22:22:33:33 unicast [bgp_Arista 22:25:08.352 from 10.10.10.1] * (100/12) [i] via 100.90.109.98 on ens192 mpls 10010 evpn ead 10.10.10.2:0 4294967295 00:00:00:00:11:11:22:22:33:33 unicast [bgp_Huawei 22:25:51.723 from 10.10.10.2] * (100/1) [?] via 100.90.109.98 on ens192 mpls 0 evpn ead 10.10.10.2:10010 0 00:00:00:00:11:11:22:22:33:33 unicast [bgp_Huawei 17:29:05.249 from 10.10.10.2] * (100/1) [?] via 100.90.109.98 on ens192 mpls 10010 evpn es 10.10.10.2:0 00:00:00:00:11:11:22:22:33:33 10.10.10.2 [bgp_Huawei 22:26:06.724 from 10.10.10.2] * (100) [?] evpn es 10.10.10.1:1 00:00:00:00:11:11:22:22:33:33 10.10.10.1 [bgp_Arista 22:24:58.357 from 10.10.10.1] * (100) [i] ``` When I inspect the ethernet table inside BIRD only one of these Type 2 routes is added (omitting flood entries to keep it short) and this route is also added to the kernel: ``` bird> sh route table etab10010 all ... b8:69:f4:e0:4e:04 mpls 10010 unicast [evpn10010 22:25:45.280] * (80) via 10.10.10.2 on vxlan10010 mpls 10010 Type: EVPN univ mpls_label: 10010``` Once one link from the LAG goes down and one of PE1 or PE2 withdraws its routes (does not matter which one) BIRD then receives withdraw for Type 2 route with MAC b8:69:f4:e0:4e:04 and removes it from ethernet table and from the bridge fdb without checking if any other route is still available in the evpntab to replace it with. As a result the traffic is flooded. I am somewhat convinced that this is partially caused by the channel being RA_ANY rather than RA_OPTIMAL for the non-BUM entries and because multiple routes are not considered in the BIRD eth table. To at least hack around this and make sure BIRD falls back to another available route I came up with the attached patch. - similarly to IMET routes it uses RD to create unique sources for received Type 2 routes and adds all to the eth table - uses first route from net->routes in kbr_rt_notify to install it to the kernel Using the first route likely is not optimal right now as kbr_rte_better isn't implemented to do BGP-style comparison. With my patch i see both routes in eth table and only one is installed to fdb: ``` bird> sh route table etab10010 all ... b8:69:f4:e0:4e:04 mpls 10010 unicast [evpn10010 11:04:26.321] * (80) via 10.10.10.1 on vxlan10010 mpls 10010 Type: EVPN univ mpls_label: 10010 unicast [evpn10010 11:04:48.048] (80) via 10.10.10.2 on vxlan10010 mpls 10010 Type: EVPN univ mpls_label: 10010 [root@col ~]# bridge fdb show br br10010 b8:69:f4:e0:4e:04 dev vxlan10010 vlan 1 extern_learn master br10010 b8:69:f4:e0:4e:04 dev vxlan10010 extern_learn master br10010 00:00:00:00:00:00 dev vxlan10010 dst 10.10.10.1 self extern_learn permanent 00:00:00:00:00:00 dev vxlan10010 dst 10.10.10.2 self extern_learn permanent b8:69:f4:e0:4e:04 dev vxlan10010 dst 10.10.10.1 self extern_learn permanent ```and when the route from 10.10.10.1 is withdrawn it gets immediately replaced by the route from 10.10.10.2. I'm curious to hear your opinion how you feel about having multiple routes for same MAC in the eth bridge protocol. Would it be cleaner to do this between EVPN table and the eth table rather than eth table and kernel? As in install optimal route from evpn table into eth table, if route is withdrawn put the next best route from evpn table into eth table. And 1:1 sync eth table to kernel. I don't plan to further adjust my "patch" so feel free to treat this as a bug report but I would appreciate to hear what you think if I may steal some of your time. Thanks for the fun, Tomáš