Feature request: CARP awareness in BIRD?
Hello :) I am using BIRD on FreeBSD and it is working okay, apart from missing CARP awareness. I am wondering if anyone knows if work has been done in this area already? As you may know, CARP is a patent-free VRRP alternative originally from OpenBSD. A network interface can have one or more VHIDs, each VHID can be either MASTER or BACKUP. The IPs tied to a VHID are only usable when the VHID is MASTER. Currently "birdc show interfaces" shows no difference between two nodes where one is CARP MASTER and the other is CARP BACKUP. The consequence of this is that BIRD thinks it has a working local direct route which it then tries to use, but the OS refuses, since it understands that VHID is CARP BACKUP. Ideally BIRD would understand CARP semantics so when it has a DIRECT protocol it would only consume the routes from CARP MASTER VHIDs, and consider the CARP BACKUP VHID IPs as unusable. Please consider this a feature request. I am interested getting some BIRD experts opinion on 1) is this a feature you would be willing to include in BIRD and 2) how big a job is it? I would be willing to pay to have it developed, although I am just myself so I don't have unlimited funds. Thanks in advance, and thank you for BIRD; even without CARP support it is already awesome. Best regards, Thomas Steen Rasmussen
On Thu, 02 Jun 2022, Thomas Steen Rasmussen wrote:
As you may know, CARP is a patent-free VRRP alternative originally from OpenBSD.
Correct. And to elaborate on history: https://en.wikipedia.org/wiki/Common_Address_Redundancy_Protocol#History Besides from OpenBSD to FreeBSD it is also ported to Linux, like Ubuntu: http://manpages.ubuntu.com/manpages/jammy/man8/ucarp.8.html JFYI -- Met vriendelijke groet, With kind regards, Leo Vandewoestijne <***@dns.company> <www.dns.company>
W dniu 2.06.2022 o 16:16, Thomas Steen Rasmussen pisze:
Hello :)
I am using BIRD on FreeBSD and it is working okay, apart from missing CARP awareness. I am wondering if anyone knows if work has been done in this area already?
As you may know, CARP is a patent-free VRRP alternative originally from OpenBSD. A network interface can have one or more VHIDs, each VHID can be either MASTER or BACKUP. The IPs tied to a VHID are only usable when the VHID is MASTER.
Currently "birdc show interfaces" shows no difference between two nodes where one is CARP MASTER and the other is CARP BACKUP. The consequence of this is that BIRD thinks it has a working local direct route which it then tries to use, but the OS refuses, since it understands that VHID is CARP BACKUP.
Ideally BIRD would understand CARP semantics so when it has a DIRECT protocol it would only consume the routes from CARP MASTER VHIDs, and consider the CARP BACKUP VHID IPs as unusable.
Please consider this a feature request. I am interested getting some BIRD experts opinion on 1) is this a feature you would be willing to include in BIRD and 2) how big a job is it?
I would be willing to pay to have it developed, although I am just myself so I don't have unlimited funds.
Thanks in advance, and thank you for BIRD; even without CARP support it is already awesome.
Yes, it's decent piece of software and runs smooth on FreeBSD. FreeBSD users still believe that at some point it will better utilize new FreeBSD routing stack and will gain ECMP support working with this OS.
Best regards,
Thomas Steen Rasmussen
You can solve this problem by creating two configs for BIRD and deploying devd(8) with simple helper shell script to trigger reconfiguration of BIRD when carp switches from BACKUP to MASTER or from MASTER to BACKUP. I have had some WiP scripts, but abandoned the idea since it would require doubling active BGP sessions for each peer what would be probably waste of resources and not acceptable in some cases. Best regards, -- Marek Zarychta
On 6/2/22 7:04 PM, Marek Zarychta wrote:
W dniu 2.06.2022 o 16:16, Thomas Steen Rasmussen pisze:
Hello :)
I am using BIRD on FreeBSD and it is working okay, apart from missing CARP awareness. I am wondering if anyone knows if work has been done in this area already?
You can solve this problem by creating two configs for BIRD and deploying devd(8) with simple helper shell script to trigger reconfiguration of BIRD when carp switches from BACKUP to MASTER or from MASTER to BACKUP. I have had some WiP scripts, but abandoned the idea since it would require doubling active BGP sessions for each peer what would be probably waste of resources and not acceptable in some cases.
Hello, Thank you for your response! Devd "config mangling" is indeed what I am doing now. But it is far from elegant, and it still has quite some rough edges and corner-cases making it a suboptimal solution. In other words I consider it a hack/workaround I would love to get rid of - hence this thread. What would it require to teach BIRD about CARP states? I can think of two things needed, without knowing anything about BIRD internals: - BIRD has to understand that an IP can be configured on an interface, but still not be in a usable state. A new flag on each IP perhaps? - BIRD has to be informed/learn about the current VHID state on each interface, so it can change the internal state of each IP based on VHID MASTER/BACKUP But I don't know the scope of this job, and I am curious to know if such a change would be accepted into BIRD upstream? Thanks! Best regards, Thomas Steen Rasmussen
Just a thought: maybe use OSPF stub networking to detect the fail-overed address? Cheers, Kees On 02-06-2022 19:35, Thomas Steen Rasmussen wrote:
On 6/2/22 7:04 PM, Marek Zarychta wrote:
W dniu 2.06.2022 o 16:16, Thomas Steen Rasmussen pisze:
Hello :)
I am using BIRD on FreeBSD and it is working okay, apart from missing CARP awareness. I am wondering if anyone knows if work has been done in this area already?
You can solve this problem by creating two configs for BIRD and deploying devd(8) with simple helper shell script to trigger reconfiguration of BIRD when carp switches from BACKUP to MASTER or from MASTER to BACKUP. I have had some WiP scripts, but abandoned the idea since it would require doubling active BGP sessions for each peer what would be probably waste of resources and not acceptable in some cases.
Hello,
Thank you for your response! Devd "config mangling" is indeed what I am doing now. But it is far from elegant, and it still has quite some rough edges and corner-cases making it a suboptimal solution. In other words I consider it a hack/workaround I would love to get rid of - hence this thread.
What would it require to teach BIRD about CARP states? I can think of two things needed, without knowing anything about BIRD internals:
- BIRD has to understand that an IP can be configured on an interface, but still not be in a usable state. A new flag on each IP perhaps? - BIRD has to be informed/learn about the current VHID state on each interface, so it can change the internal state of each IP based on VHID MASTER/BACKUP
But I don't know the scope of this job, and I am curious to know if such a change would be accepted into BIRD upstream?
Thanks!
Best regards,
Thomas Steen Rasmussen
On 02.06.22 16:16, Thomas Steen Rasmussen wrote:
Hello :)
I am using BIRD on FreeBSD and it is working okay, apart from missing CARP awareness. I am wondering if anyone knows if work has been done in this area already?
As you may know, CARP is a patent-free VRRP alternative originally from OpenBSD. A network interface can have one or more VHIDs, each VHID can be either MASTER or BACKUP. The IPs tied to a VHID are only usable when the VHID is MASTER.
Currently "birdc show interfaces" shows no difference between two nodes where one is CARP MASTER and the other is CARP BACKUP. The consequence of this is that BIRD thinks it has a working local direct route which it then tries to use, but the OS refuses, since it understands that VHID is CARP BACKUP.
Ideally BIRD would understand CARP semantics so when it has a DIRECT protocol it would only consume the routes from CARP MASTER VHIDs, and consider the CARP BACKUP VHID IPs as unusable.
Please consider this a feature request. I am interested getting some BIRD experts opinion on 1) is this a feature you would be willing to include in BIRD and 2) how big a job is it?
I would be willing to pay to have it developed, although I am just myself so I don't have unlimited funds.
Thanks in advance, and thank you for BIRD; even without CARP support it is already awesome. Part of the problem is that FreeBSD changed their CARP implementation to make CARP addresses part of the parent interface instead of keeping the CARP pseudointerface OpenBSD uses (which would probably work because it's link state goes up/down depending on the CARP state). You can workaround missing integration of the FreeBSD CARP implementation into bird by using FreeBSD's devd hotplug event daemon because CARP state transitions are made available to userspace as messages on /dev/devctl and consumed by the devd daemon (enabled by default). You can drop a config snipped into /etc/devd/carp.conf to match the events and execute a shell script as root to take the appropriate action e.g. run ifconfig to get the CARP address or send a command to bird. Of course it's not as clean as proper CARP supports, but it's flexible and responds quickly without aggressive polling.
On Thu, Jun 02, 2022 at 04:16:53PM +0200, Thomas Steen Rasmussen wrote:
Hello :)
I am using BIRD on FreeBSD and it is working okay, apart from missing CARP awareness. I am wondering if anyone knows if work has been done in this area already?
Ideally BIRD would understand CARP semantics so when it has a DIRECT protocol it would only consume the routes from CARP MASTER VHIDs, and consider the CARP BACKUP VHID IPs as unusable.
Please consider this a feature request. I am interested getting some BIRD experts opinion on 1) is this a feature you would be willing to include in BIRD and 2) how big a job is it?
Hello That is an interesting idea, and from your description it seems like not hard to support, esp. if that is just some kind of flag on iface/address. But i do not have experience with this, so i have some questions: 1) it is per-iface state or per-address state? 2) Could you send me output of ifconfig for master/backup/non-carp iface for comparison? 3) If someone use that on Linux, could send me output of 'ip address show' for these ifaces? 4) What is a general behavior of backup ifaces/ips? Are they usable, or it is more like iface that is AdminUp/OperDown (in SNMP terminology)?
Thanks in advance, and thank you for BIRD; even without CARP support it is already awesome.
Thanks, glad to read that. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Fri 03 Jun 2022 17:40:23 GMT, Ondrej Zajicek wrote:
3) If someone use that on Linux, could send me output of 'ip address show' for these ifaces?
I’m using vrrp on linux, with the macvlan option to be compatible with cisco. Inactive router: asbr02 ~ # ip a sh enp3s0f1.30 12: enp3s0f1.30@enp3s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:1b:21:48:68:99 brd ff:ff:ff:ff:ff:ff inet 89.234.186.6/27 brd 89.234.186.31 scope global enp3s0f1.30 valid_lft forever preferred_lft forever inet6 2a00:5884::6/64 scope global valid_lft forever preferred_lft forever inet6 fe80::21b:21ff:fe48:6899/64 scope link valid_lft forever preferred_lft forever asbr02 ~ # ip a sh vrrp.1 17: vrrp.1@enp3s0f1.30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:00:5e:00:01:01 brd ff:ff:ff:ff:ff:ff asbr02 ~ # ip a sh vrrp6.1 18: vrrp6.1@enp3s0f1.30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:00:5e:00:02:01 brd ff:ff:ff:ff:ff:ff asbr02 ~ # Active router: asbr03 ~ # ip a sh enp1s0d1.30 6: enp1s0d1.30@enp1s0d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:02:c9:28:92:bf brd ff:ff:ff:ff:ff:ff inet 89.234.186.15/27 brd 89.234.186.31 scope global enp1s0d1.30 valid_lft forever preferred_lft forever inet6 2a00:5884::d/64 scope global valid_lft forever preferred_lft forever inet6 fe80::202:c9ff:fe28:92bf/64 scope link valid_lft forever preferred_lft forever asbr03 ~ # ip a sh vrrp.1 11: vrrp.1@enp1s0d1.30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:00:5e:00:01:01 brd ff:ff:ff:ff:ff:ff inet 89.234.186.1/32 scope global vrrp.1 valid_lft forever preferred_lft forever asbr03 ~ # ip a sh vrrp6.1 12: vrrp6.1@enp1s0d1.30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:00:5e:00:02:01 brd ff:ff:ff:ff:ff:ff inet6 fe80::204:92:100:1/128 scope link nodad deprecated valid_lft forever preferred_lft 0sec asbr03 ~ # Here are my configurations: asbr02 ~ # grep -Pv '^$|^#|^\s+#' /etc/keepalived/keepalived.conf vrrp_instance vrrp_vip_asbr02_ipv4 { interface enp3s0f1.30 virtual_router_id 1 priority 100 advert_int 1 use_vmac vmac_xmit_base notify /usr/local/bin/keepalivednotify.sh virtual_ipaddress { 89.234.186.1/32 } } vrrp_instance vrrp_vip_asbr02_ipv6 { state MASTER interface enp3s0f1.30 virtual_router_id 1 priority 151 advert_int 1 use_vmac vmac_xmit_base notify /usr/local/bin/keepalivednotify.sh virtual_ipaddress { FE80:0:0:0:204:92:100:1/128 } } asbr03 ~ # grep -Pv '^$|^#|^\s+#' /etc/keepalived/keepalived.conf vrrp_instance vrrp_vip_vlan30_ipv4 { state BACKUP interface enp1s0d1.30 virtual_router_id 1 priority 200 advert_int 1 use_vmac vmac_xmit_base notify /usr/local/bin/keepalivednotify.sh virtual_ipaddress { 89.234.186.1/32 } } vrrp_instance vrrp_vip_vlan30_ipv6 { state BACKUP interface enp1s0d1.30 virtual_router_id 1 priority 200 advert_int 1 use_vmac vmac_xmit_base notify /usr/local/bin/keepalivednotify.sh virtual_ipaddress { FE80:0:0:0:204:92:100:1/128 } } -- Alarig
On 6/3/22 5:40 PM, Ondrej Zajicek wrote:
That is an interesting idea, and from your description it seems like not hard to support, esp. if that is just some kind of flag on iface/address.
That is what I was hoping for :)
But i do not have experience with this, so i have some questions:
1) it is per-iface state or per-address state?
It is a per-address state (actually it is per VHID state, where each VHID is a group of 1-N IP addresses, and each interface can have 1-N VHID).
2) Could you send me output of ifconfig for master/backup/non-carp iface for comparison?
Absolutely. This is interface lagg0.2002 which has a single VHID, first node A which is MASTER: ----------------------------- [tykling@dgncr2a ~]$ ifconfig lagg0.2002 lagg0.2002: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: tyktech infrastructure servers vlan 2002 - 85.209.118.0/28 and 2a09:94c4:55d1:7600::/64 options=4600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether 3c:ec:ef:46:e1:58 inet6 fe80::3eec:efff:fe46:e158%lagg0.2002/64 scopeid 0x11 inet6 2a09:94c4:55d1:7600::2a/64 inet6 2a09:94c4:55d1:7600::1/64 vhid 5 inet 85.209.118.1/28 broadcast 85.209.118.15 vhid 5 groups: vlan carp carp: MASTER vhid 5 advbase 1 advskew 100 vlan: 2002 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> [tykling@dgncr2a ~]$ ----------------------------- and then node B which is BACKUP: ----------------------------- [tykling@dgncr2b ~]$ ifconfig lagg0.2002 lagg0.2002: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: tyktech infrastructure servers vlan 2002 - 85.209.118.0/28 and 2a09:94c4:55d1:7600::/64 options=4600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP> ether 3c:ec:ef:46:e0:b8 inet6 fe80::3eec:efff:fe46:e0b8%lagg0.2002/64 scopeid 0xf inet6 2a09:94c4:55d1:7600::2b/64 inet6 2a09:94c4:55d1:7600::1/64 vhid 5 inet 85.209.118.1/28 broadcast 85.209.118.15 vhid 5 groups: vlan carp carp: BACKUP vhid 5 advbase 1 advskew 100 vlan: 2002 vlanproto: 802.1q vlanpcp: 0 parent interface: lagg0 media: Ethernet autoselect status: active nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> [tykling@dgncr2b ~]$ ----------------------------- As you can see CARP is not really an interface, it is more like a property of an IP. Each node has both CARP IPs (marked with a vhid N at the end of the line) and non-CARP IPs (in this example the only non-CARP IP is for ipv6). An interface can have multiple VHIDs which can be MASTER or BACKUP independent of eachother. From birdc the "show interfaces" for these look like this: On the MASTER (A) node: ----------------------------- lagg0.2002 up (index=17) MultiAccess Broadcast Multicast AdminUp LinkUp MTU=1500 85.209.118.1/28 (Preferred, scope univ) fe80::3eec:efff:fe46:e158/64 (Preferred, scope link) 2a09:94c4:55d1:7600::2a/64 (Preferred, scope univ) 2a09:94c4:55d1:7600::1/64 (scope univ) ----------------------------- On the BACKUP (B) node: ----------------------------- lagg0.2002 up (index=15) MultiAccess Broadcast Multicast AdminUp LinkUp MTU=1500 85.209.118.1/28 (Preferred, scope univ) 2a09:94c4:55d1:7600::2b/64 (Preferred, scope univ) fe80::3eec:efff:fe46:e0b8/64 (Preferred, scope link) 2a09:94c4:55d1:7600::1/64 (scope univ) ----------------------------- As you can see they are considered equally valid which causes BIRD to think it has a usable IP for the prefix even though it does not.
4) What is a general behavior of backup ifaces/ips? Are they usable, or it is more like iface that is AdminUp/OperDown (in SNMP terminology)?
The interfaces as such are fully usable, but the OS will not respond to ARP/NDP for virtual IPs in a BACKUP VHID. The OS still permits binding to them (so daemons can bind and be running and ready on BACKUP nodes). Best regards, Thomas Steen Rasmussen
participants (7)
-
Alarig Le Lay -
Jan Bramkamp -
Kees Meijs | Nefos -
Leo Vandewoestijne -
Marek Zarychta -
Ondrej Zajicek -
Thomas Steen Rasmussen