We are setting up a container based proof of concept with IPv6 only using ECMP, ANYCAST, BGP with BFP and Bird2 and I've hit a problem where BFD doesn't recover after a local link goes down and comes back and are seeking some advice as whether this is expected behaviour or a bug. We have installed Bird2 into 2 LXC containers both with BFD and BGP enabled talking via IPv6 using ULA. Each container is on a different subnet (fd00::1/112 & fd01::1/112). Each container is connected to a different bridge (lxdbr0 & lxdbr1) and the host has routes to link them together. This works fine! We are now testing what happens when the link is broken in various ways. 1. Deleting a host route - works fine BFP sessions recovers when route is re-added. 2. Link down on the host side veth that is attached to the bridge - BFD fails to recover when link is brought back up. The problem seems to be "bfd1: Socket error: bind: Cannot assign requested address" If on the container we then do "birdc resart bfd1" the link recovers. "CONFIG" log syslog all; define BGP_ID = 10.9.8.7; define ROUTE_IP = fd00::1; define HOST_ASN = 64512; define STATIC_IP = fd00:b00b:1e00::/64; define NEIGH_IP = fd01::1; define NEIGH_ASN = 64513; define NEIGH_PASS = "test"; router id BGP_ID; protocol device { scan time 10; # Set scanning time. Default is disabled. } protocol direct { disabled; # Enabled by Default. } filter noroutes { if net = STATIC_IP then accept; else reject; # Disabled by Default. } filter allroutes { accept; # Disabled by Default. } protocol kernel { ipv6 { # Connect protocol to IPv6 table by channel. import none; # Import to table. Default is import all. export none; # Export to protocol. Default is export none. }; } protocol static { ipv6; route STATIC_IP via ROUTE_IP; } protocol bgp infra_v6 { # Name of the BGP interface. description "BGP uplink 0"; # Description of the BGP interface. local ROUTE_IP as HOST_ASN; neighbor NEIGH_IP as NEIGH_ASN; hold time 600; # Hold time, time in seconds to wait for the keepalive to initiate. password NEIGH_PASS; # Password, type whatever password you wish. multihop; # Multi-hop enabled for more servers in a cluster. bfd on; # Switch to turn on BFD ipv6 { import filter noroutes; # Importing previous IPv6 filters. export filter allroutes; # Exporting all routed filters shown above (Line 30-33). }; } protocol bfd { multihop { interval 2s; }; neighbor NEIGH_IP; } "END OF CONFIG" Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Lost carrier Jan 27 15:01:17 c2 systemd-networkd[7304]: NDISC: Stopping IPv6 Router Solicitation client Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Removing address fd01::1 Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Removing address fd42:1af0:83cf:4510:216:3eff:fe98:2650 Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Removing route: dst: fd00::1/112, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: static, type: unicast Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Removing route: dst: n/a, src: n/a, gw: fe80::216:3eff:fe41:d39f, prefsrc: n/a, scope: global, table: main, proto: ra, type: unicast Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Removing route: dst: fd42:1af0:83cf:4510::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: ra, type: unicast Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: State is configured, dropping config Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Removing route: dst: fd00::/112, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: static, type: unicast Jan 27 15:01:17 c2 systemd-networkd[7304]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=26 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Forgetting address: fd01::1/112 (valid forever) Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Forgetting route: dst: fd01::1/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local, proto: kernel, type: local Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Forgetting route: dst: fd01::/112, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: kernel, type: unicast Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Forgetting address: fd42:1af0:83cf:4510:216:3eff:fe98:2650/64 (valid for 59min 34s) Jan 27 15:01:17 c2 systemd-networkd[7304]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_310 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=27 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Jan 27 15:01:17 c2 systemd-networkd[7304]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=28 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Forgetting route: dst: fd42:1af0:83cf:4510:216:3eff:fe98:2650/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local, proto: kernel, type: local Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Forgetting route: dst: fd00::/112, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: static, type: unicast Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Forgetting route: dst: n/a, src: n/a, gw: fe80::216:3eff:fe41:d39f, prefsrc: n/a, scope: global, table: main, proto: ra, type: unicast Jan 27 15:01:17 c2 systemd-networkd[7304]: eth0: Forgetting route: dst: fd42:1af0:83cf:4510::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: ra, type: unicast Jan 27 15:01:18 c2 bird[6234]: bfd1: Socket error: Network is unreachable Jan 27 15:01:20 c2 bird[6234]: bfd1: Socket error: Network is unreachable Jan 27 15:01:20 c2 systemd-networkd[7304]: rtnl: received non-static neighbor, ignoring. Jan 27 15:01:21 c2 bird[6234]: bfd1: Socket error: Network is unreachable Jan 27 15:01:23 c2 bird[6234]: bfd1: Socket error: Network is unreachable Jan 27 15:01:25 c2 bird[6234]: bfd1: Socket error: Network is unreachable Jan 27 15:01:26 c2 bird[6234]: bfd1: Socket error: Network is unreachable Jan 27 15:01:26 c2 bird[6234]: bfd1: Socket error: Network is unreachable Jan 27 15:01:26 c2 bird[6234]: bfd1: Socket error: bind: Cannot assign requested address Jan 27 15:01:32 c2 systemd-networkd[7304]: eth0: Flags change: +LOWER_UP +RUNNING Jan 27 15:01:32 c2 systemd-networkd[7304]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_310 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=29 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Jan 27 15:01:32 c2 systemd-networkd[7304]: eth0: Gained carrier Jan 27 15:01:32 c2 systemd-networkd[7304]: eth0: Discovering IPv6 routers Jan 27 15:01:32 c2 systemd-networkd[7304]: NDISC: Started IPv6 Router Solicitation client Jan 27 15:01:32 c2 systemd-networkd[7304]: eth0: State changed: configured -> configuring Jan 27 15:01:32 c2 systemd-networkd[7304]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_310 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=30 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Jan 27 15:01:32 c2 systemd-networkd[7304]: eth0: Setting addresses Jan 27 15:01:32 c2 systemd-networkd[7304]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=31 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Jan 27 15:01:32 c2 systemd-networkd[7304]: eth0: Remembering route: dst: fd01::/112, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: kernel, type: unicast Jan 27 15:01:32 c2 systemd-networkd[7304]: eth0: Remembering updated address: fd01::1/112 (valid forever) Jan 27 15:01:32 c2 systemd-networkd[7304]: eth0: Addresses set Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Remembering updated address: fd01::1/112 (valid forever) Jan 27 15:01:33 c2 systemd-networkd[7304]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_310 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=32 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Configuring route: dst: fd00::1/112, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: static, type: unicast Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Setting routes Jan 27 15:01:33 c2 systemd-networkd[7304]: NDISC: Sent Router Solicitation, next solicitation in 3s Jan 27 15:01:33 c2 systemd-networkd[7304]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=33 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Remembering route: dst: fd01::1/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local, proto: kernel, type: local Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Remembering route: dst: fd00::/112, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: static, type: unicast Jan 27 15:01:33 c2 systemd-networkd[7304]: NDISC: Received Router Advertisement: flags OTHER preference medium lifetime 1800 sec Jan 27 15:01:33 c2 systemd-networkd[7304]: NDISC: Invoking callback for 'router' event. Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Acquiring DHCPv6 lease on NDisc request Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Configuring route: dst: n/a, src: n/a, gw: fe80::216:3eff:fe41:d39f, prefsrc: n/a, scope: global, table: main, proto: ra, type: unicast Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Configuring route: dst: fd42:1af0:83cf:4510::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: ra, type: unicast Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Routes set Jan 27 15:01:33 c2 systemd-networkd[7304]: rtnl: received non-static neighbor, ignoring. Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Received remembered route: dst: n/a, src: n/a, gw: fe80::216:3eff:fe41:d39f, prefsrc: n/a, scope: global, table: main, proto: ra, type: unicast Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Received remembered route: dst: fd42:1af0:83cf:4510::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: ra, type: unicast Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Remembering updated address: fd42:1af0:83cf:4510:216:3eff:fe98:2650/64 (valid for 1h) Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Configuring route: dst: fd00::1/112, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main, proto: static, type: unicast Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Setting routes Jan 27 15:01:33 c2 systemd-networkd[7304]: eth0: Routes set Jan 27 15:01:33 c2 systemd-networkd[7304]: rtnl: received non-static neighbor, ignoring. Jan 27 15:01:34 c2 systemd-networkd[7304]: eth0: Remembering updated address: fd42:1af0:83cf:4510:216:3eff:fe98:2650/64 (valid for 59min 59s) Jan 27 15:01:34 c2 systemd-networkd[7304]: eth0: State changed: configuring -> configured Jan 27 15:01:34 c2 systemd-networkd[7304]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_310 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=34 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a Jan 27 15:01:34 c2 systemd-networkd[7304]: eth0: Remembering route: dst: fd42:1af0:83cf:4510:216:3eff:fe98:2650/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local, proto: kernel, type: local Jan 27 15:01:39 c2 systemd-networkd[7304]: rtnl: received non-static neighbor, ignoring. Jan 27 15:01:39 c2 systemd-networkd[7304]: rtnl: received non-static neighbor, ignoring. Jan 27 15:01:44 c2 systemd-networkd[7304]: NDISC: No RA received before link confirmation timeout Jan 27 15:01:44 c2 systemd-networkd[7304]: NDISC: Invoking callback for 'timeout' event.