BGP strict bind socket error
Hi all, I have encountered a problem with how BIRD opens the listening socket for strict bind BGP. In particular, BIRD sometimes tries to open the listening socket for the protocol when the configured local IP is not yet present on the interface. It seems that BIRD will try to bind the BGP socket to the configured IP if any address exists on the protocol interface that can be used to reach the neighbor. Let me give an example to clarify. I have a minimal reproducible example with the following BGP configuration: router id 192.168.0.1; protocol device { scan time 10; } protocol bgp bgp_01 { ipv4 { import all; export all; }; debug all; local 192.168.0.1 port 179 as 1; neighbor 192.168.0.2 port 179 as 1; interface "host1"; direct; strict bind yes; } Given this configuration, I can reproduce the problem behaviour by starting the protocol with no addressing on the host1 interface and then adding an address which includes the neighbor IP in its subnet, e.g. 192.168.0.3/24. Once this address/route is added, BIRD logs the following: Jan 07 12:09:31 router03 bird[57497]: bgp_01: Neighbor ready Jan 07 12:09:31 router03 bird[57497]: bgp_01: Socket error: bind: Cannot assign requested address Jan 07 12:09:31 router03 bird[57497]: bgp_01: Cannot open listening socket And the protocol enters an error state: BIRD 2.0.6 ready. Name Proto Table State Since Info device1 Device --- up 11:55:51.049 bgp_01 BGP --- down 12:09:31.264 Error: No listening socket The reason this error is so problematic for me is that the protocol cannot recover from this state without administrative intervention (e.g. configuring bird or restarting the protocol). So, I have two questions: 1. Is this correct behaviour? 2. Can I configure BIRD so that my protocol is able to recover after suffering a socket error like this? I should also mention I can actually observe this socket error occur when adding the 'correct' address to the protocol interface, but only in the context of a larger application and only some of the time. Thanks, Ben
Hi, You could very likely mitigate this with: sysctl -w net.ipv4.ip_nonlocal_bind=1 K. On 7 January 2020 21:45:43 CET, Ben Tremblay <ben@tremblay.dev> wrote:
Hi all,
I have encountered a problem with how BIRD opens the listening socket for strict bind BGP.
In particular, BIRD sometimes tries to open the listening socket for the protocol when the configured local IP is not yet present on the interface. It seems that BIRD will try to bind the BGP socket to the configured IP if any address exists on the protocol interface that can be used to reach the neighbor. Let me give an example to clarify.
I have a minimal reproducible example with the following BGP configuration:
router id 192.168.0.1;
protocol device { scan time 10; }
protocol bgp bgp_01 { ipv4 { import all; export all; }; debug all; local 192.168.0.1 port 179 as 1; neighbor 192.168.0.2 port 179 as 1; interface "host1"; direct; strict bind yes; }
Given this configuration, I can reproduce the problem behaviour by starting the protocol with no addressing on the host1 interface and then adding an address which includes the neighbor IP in its subnet, e.g. 192.168.0.3/24. Once this address/route is added, BIRD logs the following:
Jan 07 12:09:31 router03 bird[57497]: bgp_01: Neighbor ready Jan 07 12:09:31 router03 bird[57497]: bgp_01: Socket error: bind: Cannot assign requested address Jan 07 12:09:31 router03 bird[57497]: bgp_01: Cannot open listening socket
And the protocol enters an error state:
BIRD 2.0.6 ready. Name Proto Table State Since Info device1 Device --- up 11:55:51.049 bgp_01 BGP --- down 12:09:31.264 Error: No listening socket
The reason this error is so problematic for me is that the protocol cannot recover from this state without administrative intervention (e.g. configuring bird or restarting the protocol).
So, I have two questions:
1. Is this correct behaviour? 2. Can I configure BIRD so that my protocol is able to recover after suffering a socket error like this?
I should also mention I can actually observe this socket error occur when adding the 'correct' address to the protocol interface, but only in the context of a larger application and only some of the time.
Thanks, Ben
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Tue, Jan 07, 2020 at 12:45:43PM -0800, Ben Tremblay wrote:
Hi all,
I have encountered a problem with how BIRD opens the listening socket for strict bind BGP.
In particular, BIRD sometimes tries to open the listening socket for the protocol when the configured local IP is not yet present on the interface. It seems that BIRD will try to bind the BGP socket to the configured IP if any address exists on the protocol interface that can be used to reach the neighbor. Let me give an example to clarify.
Hi Yes, there is this issue with strict bind that availability of local IP is not explicitly monitored. Generally, listening socket is started when associated protocol is started/ready. Which means always for multihop BGP, and when peer IP is reachable for direct BGP. But as you noted, it is possible that peer IP is reachable without local IP is reachable, which makes problem with combination with strict bind.
So, I have two questions:
1. Is this correct behaviour?
Well, i would not call that correct but also not a bug. Perhaps rough edge / flaw. We should fix that.
2. Can I configure BIRD so that my protocol is able to recover after suffering a socket error like this?
Probably not. I would suggest the workaround with nonlocal_bind sysctl. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Hi, Ondrej, what do you think of adding some option to bind socket in Linux with IP_FREEBIND or IP_TRANSPARENT setsockopt? On Thu, Jan 9, 2020 at 6:19 PM Ondrej Zajicek <santiago@crfreenet.org> wrote:
On Tue, Jan 07, 2020 at 12:45:43PM -0800, Ben Tremblay wrote:
Hi all,
I have encountered a problem with how BIRD opens the listening socket for strict bind BGP.
In particular, BIRD sometimes tries to open the listening socket for the protocol when the configured local IP is not yet present on the interface. It seems that BIRD will try to bind the BGP socket to the configured IP if any address exists on the protocol interface that can be used to reach the neighbor. Let me give an example to clarify.
Hi
Yes, there is this issue with strict bind that availability of local IP is not explicitly monitored.
Generally, listening socket is started when associated protocol is started/ready. Which means always for multihop BGP, and when peer IP is reachable for direct BGP. But as you noted, it is possible that peer IP is reachable without local IP is reachable, which makes problem with combination with strict bind.
So, I have two questions:
1. Is this correct behaviour?
Well, i would not call that correct but also not a bug. Perhaps rough edge / flaw. We should fix that.
2. Can I configure BIRD so that my protocol is able to recover after suffering a socket error like this?
Probably not. I would suggest the workaround with nonlocal_bind sysctl.
-- Elen sila lumenn' omentielvo
Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Thu, Jan 09, 2020 at 08:15:25PM +0100, Alexander Zubkov wrote:
Hi,
Ondrej, what do you think of adding some option to bind socket in Linux with IP_FREEBIND or IP_TRANSPARENT setsockopt?
Hi Using IP_FREEBIND looks like an interesting way to fix this. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Hi, Check the attached patches. The first adds option to sockets to use nonlocal bind (IP_FREEBIND in Linux) and the second adds bgp option to use such sockets ("nonlocal bind yes|no"). Some additional thoughts: - probably the option could be implemented for any protocol, not only for bgp - I think it is better to check if this option is changed during reconfiguration, so to reload protocol in a hard way - did not check how it works with bfd enabled, maybe it will need to inherit this socket option from bgp somehow - it can be also considered to enable nonlocal bind for all bgp unconditionally, at least I see no obvious problems yet What do you think? On Sat, Jan 11, 2020 at 6:14 PM Ondrej Zajicek <santiago@crfreenet.org> wrote:
On Thu, Jan 09, 2020 at 08:15:25PM +0100, Alexander Zubkov wrote:
Hi,
Ondrej, what do you think of adding some option to bind socket in Linux with IP_FREEBIND or IP_TRANSPARENT setsockopt?
Hi
Using IP_FREEBIND looks like an interesting way to fix this.
-- Elen sila lumenn' omentielvo
Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Wed, Jan 15, 2020 at 01:57:19AM +0100, Alexander Zubkov wrote:
Hi,
Check the attached patches. The first adds option to sockets to use nonlocal bind (IP_FREEBIND in Linux) and the second adds bgp option to use such sockets ("nonlocal bind yes|no"). Some additional thoughts: - probably the option could be implemented for any protocol, not only for bgp ... - it can be also considered to enable nonlocal bind for all bgp unconditionally, at least I see no obvious problems yet
Hi Is there a reason for such option? Is there a downside of using it always? One minor nitpick is that sysdep/unix/io.c should not use non-portable syscalls/sockopts directly, they should be defined as functions in sysdep/X/sysio.h (with implementations for Linux and BSD) and such function called from sysdep/unix/io.c code. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Thu, Jan 23, 2020 at 5:21 PM Ondrej Zajicek <santiago@crfreenet.org> wrote:
On Wed, Jan 15, 2020 at 01:57:19AM +0100, Alexander Zubkov wrote:
Hi,
Check the attached patches. The first adds option to sockets to use nonlocal bind (IP_FREEBIND in Linux) and the second adds bgp option to use such sockets ("nonlocal bind yes|no"). Some additional thoughts: - probably the option could be implemented for any protocol, not only for bgp ... - it can be also considered to enable nonlocal bind for all bgp unconditionally, at least I see no obvious problems yet
Hi
Is there a reason for such option? Is there a downside of using it always?
If I remember right, when address is not available during bind, in strict bind mode bird logs en error and puts protocol down. It is inconvenient when one uses VRRP, for example, with migrating address. Or there could be some problems during applying some configuration changes to the interfaces in the system. We use this socket option with VRRP and also we have interfaces without ip addresses configured yet for some reason. I do not know if using it always-on is a good idea or not. We made it always-on in our setup and are just happy with it. But I'm not sure if it does not break something somewhere. It will make at least bird's behaviour different on different systems with the same config and that may be confusing. On the other hand, the extra option to put in config may be unreasonable payment. I also not tested how it will behave if ip address migrates from one interface to the other in the system. If it ties to the interfaces somehow and because of this option does not mention the change, that could be a problem.
One minor nitpick is that sysdep/unix/io.c should not use non-portable syscalls/sockopts directly, they should be defined as functions in sysdep/X/sysio.h (with implementations for Linux and BSD) and such function called from sysdep/unix/io.c code.
Probably you are right, it might be somewhere there if it would end into the vanilla bird.
-- Elen sila lumenn' omentielvo
Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Hi, I want to bring this question up again. In our company we use it in production with patches, but I think it would be useful in upstream version too. Short version of the story: bird can try to bind socket when IP-address is absent in the system, it will result in a error and the protocol will remain in down state after that. Suggested change is to allow it to bind non-local addresses. First, lets start with just the flag in the socket interface. I adapted the patch, now system-dependend code is in a separate function sk_set_freebind(), which is defined in sysdep/X/sysio.h, as it was suggested. If this variant is OK, than the next step is to choose wether it would be some configuration option or maybe a compile-time flag. On Thu, Jan 23, 2020 at 11:05 PM Alexander Zubkov <green@qrator.net> wrote:
On Thu, Jan 23, 2020 at 5:21 PM Ondrej Zajicek <santiago@crfreenet.org> wrote:
On Wed, Jan 15, 2020 at 01:57:19AM +0100, Alexander Zubkov wrote:
Hi,
Check the attached patches. The first adds option to sockets to use nonlocal bind (IP_FREEBIND in Linux) and the second adds bgp option to use such sockets ("nonlocal bind yes|no"). Some additional thoughts: - probably the option could be implemented for any protocol, not only for bgp ... - it can be also considered to enable nonlocal bind for all bgp unconditionally, at least I see no obvious problems yet
Hi
Is there a reason for such option? Is there a downside of using it always?
If I remember right, when address is not available during bind, in strict bind mode bird logs en error and puts protocol down. It is inconvenient when one uses VRRP, for example, with migrating address. Or there could be some problems during applying some configuration changes to the interfaces in the system. We use this socket option with VRRP and also we have interfaces without ip addresses configured yet for some reason. I do not know if using it always-on is a good idea or not. We made it always-on in our setup and are just happy with it. But I'm not sure if it does not break something somewhere. It will make at least bird's behaviour different on different systems with the same config and that may be confusing. On the other hand, the extra option to put in config may be unreasonable payment. I also not tested how it will behave if ip address migrates from one interface to the other in the system. If it ties to the interfaces somehow and because of this option does not mention the change, that could be a problem.
One minor nitpick is that sysdep/unix/io.c should not use non-portable syscalls/sockopts directly, they should be defined as functions in sysdep/X/sysio.h (with implementations for Linux and BSD) and such function called from sysdep/unix/io.c code.
Probably you are right, it might be somewhere there if it would end into the vanilla bird.
-- Elen sila lumenn' omentielvo
Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Tue, Dec 28, 2021 at 06:34:28PM +0100, Alexander Zubkov wrote:
Hi,
I want to bring this question up again. In our company we use it in production with patches, but I think it would be useful in upstream version too. Short version of the story: bird can try to bind socket when IP-address is absent in the system, it will result in a error and the protocol will remain in down state after that. Suggested change is to allow it to bind non-local addresses.
If this variant is OK, than the next step is to choose wether it would be some configuration option or maybe a compile-time flag.
Hi My main objection is that whether to use IP_FREEBIND should be primarily developer decision, not admin decision. Either the code should work correctly without IP_FREEBIND, or we should use it always or automatically when necessary. I looked for disadvantages of always using IP_FREEBIND, i found nothing except that in case of misconfigured IP address it does not report error. But BIRD (and modern daemons in general) are supposed to wait for IP to appear instead of assuming that all valid IPs are available when daemon starts, so this is not an issue. So it makes sense to use IP_FREEBIND by default if available. So i think that there could be a protocol option for freebind, which should have platform-specific defaults (like rt_default_ecmp is platform-specific default for ECMP option), independently for IPv4 and IPv6. This option is primarily intended for disabling freebind in case of some unexpected case where it is not desirable. Also note that the patch does not handle IPv6 case (there is IPV6_FREEBIND) and BSD case (there is IP_BINDANY, which seems that does the same, but it is less clear and requires some privilege, so perhaps it makes sense to skip it or not make it default). I will try the patch, modify it and merge it. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Yes, probably it is ok to use it by default, at least in our case we use it always-on (as in attached patch). Only in this case it may be better to lower the log level for it then for the cases when it does not work. With default on there may be cases when someone will be surprised to see the bird listening on nonexistent address in netstat. And may be someone has some sort of security concerns with it, then please speak now or forever hold your peace. :) I looked a bit about the FreeBSD, but there are different options for it and it can probably be disabled with sysctl there. So I thought it would be better if some people with more FreeBSD (or other *BSD) experience updated this part later. On Wed, Dec 29, 2021 at 5:41 PM Ondrej Zajicek <santiago@crfreenet.org> wrote:
On Tue, Dec 28, 2021 at 06:34:28PM +0100, Alexander Zubkov wrote:
Hi,
I want to bring this question up again. In our company we use it in production with patches, but I think it would be useful in upstream version too. Short version of the story: bird can try to bind socket when IP-address is absent in the system, it will result in a error and the protocol will remain in down state after that. Suggested change is to allow it to bind non-local addresses.
If this variant is OK, than the next step is to choose wether it would be some configuration option or maybe a compile-time flag.
Hi
My main objection is that whether to use IP_FREEBIND should be primarily developer decision, not admin decision. Either the code should work correctly without IP_FREEBIND, or we should use it always or automatically when necessary.
I looked for disadvantages of always using IP_FREEBIND, i found nothing except that in case of misconfigured IP address it does not report error. But BIRD (and modern daemons in general) are supposed to wait for IP to appear instead of assuming that all valid IPs are available when daemon starts, so this is not an issue. So it makes sense to use IP_FREEBIND by default if available.
So i think that there could be a protocol option for freebind, which should have platform-specific defaults (like rt_default_ecmp is platform-specific default for ECMP option), independently for IPv4 and IPv6. This option is primarily intended for disabling freebind in case of some unexpected case where it is not desirable.
Also note that the patch does not handle IPv6 case (there is IPV6_FREEBIND) and BSD case (there is IP_BINDANY, which seems that does the same, but it is less clear and requires some privilege, so perhaps it makes sense to skip it or not make it default).
I will try the patch, modify it and merge it.
-- Elen sila lumenn' omentielvo
Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Wed, Dec 29, 2021 at 06:11:32PM +0100, Alexander Zubkov wrote:
Yes, probably it is ok to use it by default, at least in our case we use it always-on (as in attached patch). Only in this case it may be better to lower the log level for it then for the cases when it does not work. With default on there may be cases when someone will be surprised to see the bird listening on nonexistent address in netstat. And may be someone has some sort of security concerns with it, then please speak now or forever hold your peace. :)
Thanks, merged (the first patch). After all, i make it 'free bind' BGP option, disabled by default [*], applying SKF_FREEBIND to the listening socket. But your second patch enabled SKF_FREEBIND for both listening and active socket, i see reasons for listening one, but why to do it for the active socket (in bgp_connect())? [*] commit 60e9def9ef7b5d16f868b0fb4ab1192d59fd7541
I looked a bit about the FreeBSD, but there are different options for it and it can probably be disabled with sysctl there. So I thought it would be better if some people with more FreeBSD (or other *BSD) experience updated this part later.
Yes, after some research it seems that it is not an equivalent option.
diff --git a/proto/bgp/bgp.c b/proto/bgp/bgp.c index e4d754b1..5d985e25 100644 --- a/proto/bgp/bgp.c +++ b/proto/bgp/bgp.c @@ -180,7 +180,7 @@ bgp_open(struct bgp_proto *p) sk->sport = port; sk->iface = ifa; sk->vrf = p->p.vrf; - sk->flags = 0; + sk->flags = SKF_FREEBIND; sk->tos = IP_PREC_INTERNET_CONTROL; sk->rbsize = BGP_RX_BUFFER_SIZE; sk->tbsize = BGP_TX_BUFFER_SIZE; @@ -1117,6 +1117,7 @@ bgp_connect(struct bgp_proto *p) /* Enter Connect state and start establishing c s->dport = p->cf->remote_port; s->iface = p->neigh ? p->neigh->iface : NULL; s->vrf = p->p.vrf; + s->flags = SKF_FREEBIND; s->ttl = p->cf->ttl_security ? 255 : hops; s->rbsize = p->cf->enable_extended_messages ? BGP_RX_BUFFER_EXT_SIZE : BGP_RX_BUFFER_SIZE; s->tbsize = p->cf->enable_extended_messages ? BGP_TX_BUFFER_EXT_SIZE : BGP_TX_BUFFER_SIZE;
-- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Sun, Jan 9, 2022 at 3:41 AM Ondrej Zajicek <santiago@crfreenet.org> wrote:
On Wed, Dec 29, 2021 at 06:11:32PM +0100, Alexander Zubkov wrote:
Yes, probably it is ok to use it by default, at least in our case we use it always-on (as in attached patch). Only in this case it may be better to lower the log level for it then for the cases when it does not work. With default on there may be cases when someone will be surprised to see the bird listening on nonexistent address in netstat. And may be someone has some sort of security concerns with it, then please speak now or forever hold your peace. :)
Thanks, merged (the first patch). After all, i make it 'free bind' BGP option, disabled by default [*], applying SKF_FREEBIND to the listening socket. But your second patch enabled SKF_FREEBIND for both listening and active socket, i see reasons for listening one, but why to do it for the active socket (in bgp_connect())?
Cool, thanks! Now, that you asked this question, I think there is really no reason for that. :) It receives an error anyway if it is IP_FREEBIND and not IP_TRANSPARENT, when trying to connect. Maybe I was worried that BIRD could put the protocol down when trying to connect too and added it there just for the case.
[*] commit 60e9def9ef7b5d16f868b0fb4ab1192d59fd7541
I looked a bit about the FreeBSD, but there are different options for it and it can probably be disabled with sysctl there. So I thought it would be better if some people with more FreeBSD (or other *BSD) experience updated this part later.
Yes, after some research it seems that it is not an equivalent option.
diff --git a/proto/bgp/bgp.c b/proto/bgp/bgp.c index e4d754b1..5d985e25 100644 --- a/proto/bgp/bgp.c +++ b/proto/bgp/bgp.c @@ -180,7 +180,7 @@ bgp_open(struct bgp_proto *p) sk->sport = port; sk->iface = ifa; sk->vrf = p->p.vrf; - sk->flags = 0; + sk->flags = SKF_FREEBIND; sk->tos = IP_PREC_INTERNET_CONTROL; sk->rbsize = BGP_RX_BUFFER_SIZE; sk->tbsize = BGP_TX_BUFFER_SIZE; @@ -1117,6 +1117,7 @@ bgp_connect(struct bgp_proto *p) /* Enter Connect state and start establishing c s->dport = p->cf->remote_port; s->iface = p->neigh ? p->neigh->iface : NULL; s->vrf = p->p.vrf; + s->flags = SKF_FREEBIND; s->ttl = p->cf->ttl_security ? 255 : hops; s->rbsize = p->cf->enable_extended_messages ? BGP_RX_BUFFER_EXT_SIZE : BGP_RX_BUFFER_SIZE; s->tbsize = p->cf->enable_extended_messages ? BGP_TX_BUFFER_EXT_SIZE : BGP_TX_BUFFER_SIZE;
-- Elen sila lumenn' omentielvo
Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
participants (4)
-
Alexander Zubkov -
Ben Tremblay -
Kees Meijs -
Ondrej Zajicek