BGP on /32 (/128) interfaces
Hi, A message between bug report and feature request, unless it's a silly oversight on my side. On Linux it's possible to configure interfaces with /32 (or v6 /128) IPs. I detailed our use-case and examples on https://phabricator.wikimedia.org/phame/post/view/312/ganeti_on_modern_netwo... if you're curious. So focusing on v4 only for now I have this configuration on a VM: testvm2006:~$ ip addr inet 10.192.24.4/32 brd 10.192.24.4 scope global ens13 testvm2006:~$ ip route default via 10.192.24.1 dev ens13 onlink And this on the hypervisor (but it could also be 2 physical servers, or a router and a server) : ganeti2033:~$ ip addr inet 10.192.24.1/32 scope link tap0 ganeti2033:~$ ip route 10.192.24.4 dev tap0 proto static scope link Note that 10.192.24.1 is close to 10.192.24.4, but they could be totally different IPs (see the v6 example bellow). Overall connectivity works well, TCP handshake as well: testvm2006:~$ nc -zv 10.192.24.1 179 Connection to 10.192.24.1 179 port [tcp/bgp] succeeded! The issue is that with a basic configuration like: protocol bgp { ipv4 { import none; export none; }; local 10.192.24.4 as 64613; neighbor 10.192.24.1 external; } Bird 2.0.12 doesn't try to established the session, and waits with: 2024-04-12T08:42:29.497756+00:00 testvm2006 bird: bgp1: Waiting for 10.192.24.1 to become my neighbor For IPv4 it's possible to workaround the issue by setting the session as "multihop;" on both side bgp1: Connecting to 10.192.24.1 from local address 10.192.24.4 bgp1: Connected bgp1: Sending OPEN(ver=4,as=64613,hold=240,id=0ac01804) bgp1: Got OPEN(as=64612,hold=240,id=10.192.21.6) bgp1: Sending KEEPALIVE bgp1: Got KEEPALIVE bgp1: BGP session established But for IPv6, it's cleaner to only require the router's link local address: testvm2006:~$ ip -6 addr inet6 2620:0:860:140:10:192:24:4/128 scope global testvm2006:~$ ip -6 route default via fe80::2022:22ff:fe22:2201 dev ens13 metric 1024 pref medium In Bird: neighbor fe80::2022:22ff:fe22:2201%ens13 external; But then the link local address doesn't work with multihop (for obvious reason). bird: /etc/bird/bird.conf:22:1 Multihop BGP cannot be used with link-local addresses The question is then, would it be possible to support that usecase, something along the lines of if the local IP is a /32 (or v6 /128), Treat the neighbor as directly connected, especially if it's a link local IP (unless nultihop is specified). I haven't looked at BFD yet, so not sure if anything would need to change there as well. Don't hesitate to ask if something needs any clarifications. Thank for that great tool, Bird is awesome, -- Arzhel
Hey Arzhel! On 12.04.24 11:57 AM, Arzhel Younsi wrote:
Hi,
A message between bug report and feature request, unless it's a silly oversight on my side.
Hopefully _I_ do not miss a detail here, but: Option A: Set a v4 and v6 addr only on loopback, and using IPv6 LLA with OSPF2/3 to make these addresses available within the network. Then use these addresses for the BGP peering. Option B: Assign the v4 addresses on each interface and emulate "unnumbered" interfaces. In general, I would go with option A and use a Route Reflector. I/we had used ganeti in the past (with a previous employee) too and did some proof of concept design. Each server is also a router and a vxlan endpoint. Each VM got briged to a vxlan-bridge; OSPF makes the v6 address available within the network; everything connects to 2 route reflector, and bgp multi protocol is used to make the v4 addresses available.... Hope this somehow helps you. Best, Bernd
On Linux it's possible to configure interfaces with /32 (or v6 /128) IPs. I detailed our use-case and examples on https://phabricator.wikimedia.org/phame/post/view/312/ganeti_on_modern_netwo... if you're curious.
So focusing on v4 only for now I have this configuration on a VM: testvm2006:~$ ip addr inet 10.192.24.4/32 brd 10.192.24.4 scope global ens13 testvm2006:~$ ip route default via 10.192.24.1 dev ens13 onlink
And this on the hypervisor (but it could also be 2 physical servers, or a router and a server) : ganeti2033:~$ ip addr inet 10.192.24.1/32 scope link tap0 ganeti2033:~$ ip route 10.192.24.4 dev tap0 proto static scope link
Note that 10.192.24.1 is close to 10.192.24.4, but they could be totally different IPs (see the v6 example bellow).
Overall connectivity works well, TCP handshake as well: testvm2006:~$ nc -zv 10.192.24.1 179 Connection to 10.192.24.1 179 port [tcp/bgp] succeeded!
The issue is that with a basic configuration like: protocol bgp { ipv4 { import none; export none; }; local 10.192.24.4 as 64613; neighbor 10.192.24.1 external; } Bird 2.0.12 doesn't try to established the session, and waits with: 2024-04-12T08:42:29.497756+00:00 testvm2006 bird: bgp1: Waiting for 10.192.24.1 to become my neighbor
For IPv4 it's possible to workaround the issue by setting the session as "multihop;" on both side bgp1: Connecting to 10.192.24.1 from local address 10.192.24.4 bgp1: Connected bgp1: Sending OPEN(ver=4,as=64613,hold=240,id=0ac01804) bgp1: Got OPEN(as=64612,hold=240,id=10.192.21.6) bgp1: Sending KEEPALIVE bgp1: Got KEEPALIVE bgp1: BGP session established
But for IPv6, it's cleaner to only require the router's link local address: testvm2006:~$ ip -6 addr inet6 2620:0:860:140:10:192:24:4/128 scope global testvm2006:~$ ip -6 route default via fe80::2022:22ff:fe22:2201 dev ens13 metric 1024 pref medium
In Bird: neighbor fe80::2022:22ff:fe22:2201%ens13 external;
But then the link local address doesn't work with multihop (for obvious reason). bird: /etc/bird/bird.conf:22:1 Multihop BGP cannot be used with link-local addresses
The question is then, would it be possible to support that usecase, something along the lines of if the local IP is a /32 (or v6 /128), Treat the neighbor as directly connected, especially if it's a link local IP (unless nultihop is specified). I haven't looked at BFD yet, so not sure if anything would need to change there as well.
Don't hesitate to ask if something needs any clarifications.
Thank for that great tool, Bird is awesome,
Hi Arzhel, On Fri, Apr 12, 2024 at 11:57:38AM +0200, Arzhel Younsi wrote:
But for IPv6, it's cleaner to only require the router's link local address: testvm2006:~$ ip -6 addr inet6 2620:0:860:140:10:192:24:4/128 scope global testvm2006:~$ ip -6 route default via fe80::2022:22ff:fe22:2201 dev ens13 metric 1024 pref medium
In Bird: neighbor fe80::2022:22ff:fe22:2201%ens13 external;
But then the link local address doesn't work with multihop (for obvious reason). bird: /etc/bird/bird.conf:22:1 Multihop BGP cannot be used with link-local addresses
I use lladdrs for BGP endpoints in my network and that works fine. I think using `direct` instead of `multihop` in the v6-lladdr case would make it work for you. One word of advice: don't use the %scope syntax, use the `interface` directive instead. I don't recall exactly why but I had some subtle problem with that. As for your v4/32 problem, give `multihop 1` a try. That enforces no routers on the path to the peer like direct but allows off-subnet endpoints. Do keep in mind the docs recommend setting the source address explicitly when enabling multihop. multihop [number] Configure multihop BGP session to a neighbor that isn't directly connected. Accurately, this option should be used if the configured neighbor IP address does not match with any local network subnets. Such IP address have to be reachable through system routing table. The alternative is the direct option. For multihop BGP it is recommended to explicitly configure the source address to have it stable. Optional number argument can be used to specify the number of hops (used for TTL). Note that the number of networks (edges) in a path is counted; i.e., if two BGP speakers are separated by one router, the number of hops is 2. Default: enabled for iBGP. The reason why direct isn't working is also clear from the docs: direct Specify that the neighbor is directly connected. The IP address of the neighbor must be from a directly reachable IP range (i.e. associated with one of your router's interfaces), >>>otherwise the BGP session wouldn't start but it would wait for such interface to appear<<<. The alternative is the multihop option. Default: enabled for eBGP. Hope that helps, --Daniel
On Mon, Apr 15, 2024 at 12:10:05PM +0200, Daniel Gröber wrote:
Hi Arzhel,
On Fri, Apr 12, 2024 at 11:57:38AM +0200, Arzhel Younsi wrote:
But for IPv6, it's cleaner to only require the router's link local address: testvm2006:~$ ip -6 addr inet6 2620:0:860:140:10:192:24:4/128 scope global testvm2006:~$ ip -6 route default via fe80::2022:22ff:fe22:2201 dev ens13 metric 1024 pref medium
In Bird: neighbor fe80::2022:22ff:fe22:2201%ens13 external;
But then the link local address doesn't work with multihop (for obvious reason). bird: /etc/bird/bird.conf:22:1 Multihop BGP cannot be used with link-local addresses
I use lladdrs for BGP endpoints in my network and that works fine. I think using `direct` instead of `multihop` in the v6-lladdr case would make it work for you.
One word of advice: don't use the %scope syntax, use the `interface` directive instead. I don't recall exactly why but I had some subtle problem with that.
As for your v4/32 problem, give `multihop 1` a try. That enforces no routers on the path to the peer like direct but allows off-subnet endpoints. Do keep in mind the docs recommend setting the source address explicitly when enabling multihop.
Hi Note that using multihop fixes the issue with waiting for the address range to appear, but there is still an issue with next hop resolution. Multihop routes use recursive next hop resolution and in the case of /32 address ranges, there is no route for resolving neighbor IP announced as next hop. One would need a static route like: route NBR-IP/32 via "IFACE"; So the next hop will be resolved. Thinking about it, it makes sense to have something like direct mode that works with unnumbered interfaces (or ones with /32 address). -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) "To err is human -- to blame it on a computer is even more so."
Thank you all for your replies,
Thinking about it, it makes sense to have something like direct mode that works with unnumbered interfaces (or ones with /32 address).
We also think that's would be very useful, either transparently to the user (depending on next hop resolution, eBGP/iBGP, IP's subnet mask) or through the addition of an explicit "direct" keyword. What would be the process to turn this thread into a feature request ? And would the Bird maintainers be interested in implementing it ? Thanks again ! On Mon, Apr 15, 2024 at 4:25 PM Ondrej Zajicek <santiago@crfreenet.org> wrote:
On Mon, Apr 15, 2024 at 12:10:05PM +0200, Daniel Gröber wrote:
Hi Arzhel,
On Fri, Apr 12, 2024 at 11:57:38AM +0200, Arzhel Younsi wrote:
But for IPv6, it's cleaner to only require the router's link local address: testvm2006:~$ ip -6 addr inet6 2620:0:860:140:10:192:24:4/128 scope global testvm2006:~$ ip -6 route default via fe80::2022:22ff:fe22:2201 dev ens13 metric 1024 pref medium
In Bird: neighbor fe80::2022:22ff:fe22:2201%ens13 external;
But then the link local address doesn't work with multihop (for obvious reason). bird: /etc/bird/bird.conf:22:1 Multihop BGP cannot be used with link-local addresses
I use lladdrs for BGP endpoints in my network and that works fine. I think using `direct` instead of `multihop` in the v6-lladdr case would make it work for you.
One word of advice: don't use the %scope syntax, use the `interface` directive instead. I don't recall exactly why but I had some subtle problem with that.
As for your v4/32 problem, give `multihop 1` a try. That enforces no routers on the path to the peer like direct but allows off-subnet endpoints. Do keep in mind the docs recommend setting the source address explicitly when enabling multihop.
Hi
Note that using multihop fixes the issue with waiting for the address range to appear, but there is still an issue with next hop resolution. Multihop routes use recursive next hop resolution and in the case of /32 address ranges, there is no route for resolving neighbor IP announced as next hop.
One would need a static route like:
route NBR-IP/32 via "IFACE";
So the next hop will be resolved.
Thinking about it, it makes sense to have something like direct mode that works with unnumbered interfaces (or ones with /32 address).
-- Elen sila lumenn' omentielvo
Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) "To err is human -- to blame it on a computer is even more so."
-- Arzhel
Perhaps the presence of the ‘direct’ keyword combined with a manual ‘interface X’ in a protocol block could signal to bypass the usual evaluation and force a neighbor to be considered ‘direct’? Regards, Cathal. From: Bird-users <bird-users-bounces@network.cz> On Behalf Of Arzhel Younsi Sent: Tuesday 14 May 2024 14:58 To: Ondrej Zajicek <santiago@crfreenet.org> Cc: bird-users@network.cz Subject: Re: BGP on /32 (/128) interfaces Thank you all for your replies,
Thinking about it, it makes sense to have something like direct mode that works with unnumbered interfaces (or ones with /32 address).
We also think that's would be very useful, either transparently to the user (depending on next hop resolution, eBGP/iBGP, IP's subnet mask) or through the addition of an explicit "direct" keyword. What would be the process to turn this thread into a feature request ? And would the Bird maintainers be interested in implementing it ? Thanks again ! On Mon, Apr 15, 2024 at 4:25 PM Ondrej Zajicek <santiago@crfreenet.org <mailto:santiago@crfreenet.org> > wrote: On Mon, Apr 15, 2024 at 12:10:05PM +0200, Daniel Gröber wrote:
Hi Arzhel,
On Fri, Apr 12, 2024 at 11:57:38AM +0200, Arzhel Younsi wrote:
But for IPv6, it's cleaner to only require the router's link local address: testvm2006:~$ ip -6 addr inet6 2620:0:860:140:10:192:24:4/128 scope global testvm2006:~$ ip -6 route default via fe80::2022:22ff:fe22:2201 dev ens13 metric 1024 pref medium
In Bird: neighbor fe80::2022:22ff:fe22:2201%ens13 external;
But then the link local address doesn't work with multihop (for obvious reason). bird: /etc/bird/bird.conf:22:1 Multihop BGP cannot be used with link-local addresses
I use lladdrs for BGP endpoints in my network and that works fine. I think using `direct` instead of `multihop` in the v6-lladdr case would make it work for you.
One word of advice: don't use the %scope syntax, use the `interface` directive instead. I don't recall exactly why but I had some subtle problem with that.
As for your v4/32 problem, give `multihop 1` a try. That enforces no routers on the path to the peer like direct but allows off-subnet endpoints. Do keep in mind the docs recommend setting the source address explicitly when enabling multihop.
Hi Note that using multihop fixes the issue with waiting for the address range to appear, but there is still an issue with next hop resolution. Multihop routes use recursive next hop resolution and in the case of /32 address ranges, there is no route for resolving neighbor IP announced as next hop. One would need a static route like: route NBR-IP/32 via "IFACE"; So the next hop will be resolved. Thinking about it, it makes sense to have something like direct mode that works with unnumbered interfaces (or ones with /32 address). -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org <mailto:santiago@crfreenet.org> ) "To err is human -- to blame it on a computer is even more so." -- Arzhel
On Tue, May 14, 2024 at 03:58:06PM +0200, Arzhel Younsi wrote:
Thank you all for your replies,
Thinking about it, it makes sense to have something like direct mode that works with unnumbered interfaces (or ones with /32 address).
We also think that's would be very useful, either transparently to the user (depending on next hop resolution, eBGP/iBGP, IP's subnet mask) or through the addition of an explicit "direct" keyword.
What would be the process to turn this thread into a feature request ? And would the Bird maintainers be interested in implementing it ?
Hi I already made it a feature request and plan to implement it.
Perhaps the presence of the ‘direct’ keyword combined with a manual ‘interface X’ in a protocol block could signal to bypass the usual evaluation and force a neighbor to be considered ‘direct’?
That seems like a reasonable idea, but it has some caveats. Common IP range is used not only to find the interface but also to select the local address. If both 'direct' and 'interface' were changed to not wait for common IP range, then it may start the session immediately after the interface went up but before the IP address is assigned to it, using different IP address, therefore changing the current behavior. One way would be to make specifying local IP mandatory, another one would be to add option like 'onlink' to state explicitly that the neighbor is not expected to be on a shared IP range. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) "To err is human -- to blame it on a computer is even more so."
participants (5)
-
Arzhel Younsi -
Bernd Naumann -
Cathal Mooney -
Daniel Gröber -
Ondrej Zajicek