October 2021 - Bird-users

Re: BIRD trying to reinsert existing kernel routes, netlink issue?
by Zhaofeng Li 18 Feb '22

18 Feb '22

I know this was an old thread, but would really appreciate if anyone is interested in digging in deeper. I can confirm the exact same behavior happens to me on NixOS with the 5.4.69 kernel but not on 4.19.149. BIRD 2.0.7 on 5.4.69 would incessantly attempt to insert the same routes, rendering birdc unusable as it was presumably taking up so much time in the main loop (it was receiving an IPv4 full table over iBGP as well). While it was doing that, it stopped announcing the prefixes of stub interfaces to OSPF neighbors. Thanks, Zhaofeng Li On Sun, Aug 2, 2020, at 17:36 PM, Sasha Romijn wrote: > Hello, > I’m running into a rather complex issue where BIRD tries to reinsert routes into the kernel routing table that are already in there. I’m at the point of suspecting a netlink bug in the 5.x linux kernel. However, I may be entirely off, and would like to hear what people with more BIRD and/or netlink expertise think about it. > My setup is bird 2.0.7 on Ubuntu 20.04. I’ve been able to reproduce this with the default 5.4.0-42-generic kernel as well as the ubuntu mainline 5.7.0-050700-generic kernel. This bird instance has an IBGP peering with another bird 2.0.7 instance, which sends an IPv4 full table. The IBGP peering is set up over a GRE tunnel. There is no other routing software running. I am seeing this issue on another server too, ubuntu 20.04 with 5.4.0-42-generic. A third one with ubuntu 18.04 and a 4.15.0-20-generic kernel, is fine.

3 2

Route Flapping and Bird - RFD and MRAI
by Douglas Fischer 16 Feb '22

16 Feb '22

Hello all. Last weeks I'm noticing the increase the increase of BGP messages on some routers that I have access to. Specially those connected to IXPs with a considerable number of participants. I Checked and the BGP process has increased a bit the use of CPU Also. So, I suspect that Route-Servers are receiving and forwarding Route Flaps from one or more participants... I know that the right place to fix that would be on the source of those flaps... But... Considering IXPs with one ou two thousand participants, there will always be someone flapping some routes. Bird is widely used as the BGP engine on IXP route-servers. And exactly because of that I came here to ask about Route Flap avoidance mechanisms on Bird. The discussion of methods to avoid Route Flapping in BGP is very controversial. - RFD - Route Flapping Dampening comes and goes. - MRAI is not a consensus, but there is a lot of effort on that, including som magic algorithms define some variable interval according to conditions. - I even saw some ultra-rigorous ideas like rate limiting the BGP messages... Before I come here to bore you with this, I searched a bit on mail list and documentation of Bird and I found the links bellow related to that: There is any work running to deal with that route-flapping question? https://bird.network.cz/ - 7.1 Future work <https://bird.network.cz/?get_doc&v=20&f=bird-7.html> Question about BGP advertisement-interval in BIRD <http://trubka.network.cz/pipermail/bird-users/2020-August/014786.html> [BGP] MRAI connection-based implementation review <http://trubka.network.cz/pipermail/bird-users/2020-April/014433.html> Route Flap Dampening <http://trubka.network.cz/pipermail/bird-users/2019-July/013589.html> [BGP] bird and RFD <http://trubka.network.cz/pipermail/bird-users/2020-May/014612.html> Thank you all! -- Douglas Fernando Fischer Engº de Controle e Automação

3 4

BGP strict bind socket error
by Ben Tremblay 09 Jan '22

09 Jan '22

Hi all, I have encountered a problem with how BIRD opens the listening socket for strict bind BGP. In particular, BIRD sometimes tries to open the listening socket for the protocol when the configured local IP is not yet present on the interface. It seems that BIRD will try to bind the BGP socket to the configured IP if any address exists on the protocol interface that can be used to reach the neighbor. Let me give an example to clarify. I have a minimal reproducible example with the following BGP configuration: router id 192.168.0.1; protocol device { scan time 10; } protocol bgp bgp_01 { ipv4 { import all; export all; }; debug all; local 192.168.0.1 port 179 as 1; neighbor 192.168.0.2 port 179 as 1; interface "host1"; direct; strict bind yes; } Given this configuration, I can reproduce the problem behaviour by starting the protocol with no addressing on the host1 interface and then adding an address which includes the neighbor IP in its subnet, e.g. 192.168.0.3/24. Once this address/route is added, BIRD logs the following: Jan 07 12:09:31 router03 bird[57497]: bgp_01: Neighbor ready Jan 07 12:09:31 router03 bird[57497]: bgp_01: Socket error: bind: Cannot assign requested address Jan 07 12:09:31 router03 bird[57497]: bgp_01: Cannot open listening socket And the protocol enters an error state: BIRD 2.0.6 ready. Name Proto Table State Since Info device1 Device --- up 11:55:51.049 bgp_01 BGP --- down 12:09:31.264 Error: No listening socket The reason this error is so problematic for me is that the protocol cannot recover from this state without administrative intervention (e.g. configuring bird or restarting the protocol). So, I have two questions: 1. Is this correct behaviour? 2. Can I configure BIRD so that my protocol is able to recover after suffering a socket error like this? I should also mention I can actually observe this socket error occur when adding the 'correct' address to the protocol interface, but only in the context of a larger application and only some of the time. Thanks, Ben

4 12

[BIRD 2.0.x] Netlink: ignore dead routes
by Vincent Bernat 09 Jan '22

09 Jan '22

With net.ipv4.conf.XXX.ignore_routes_with_linkdown sysctl, a user can ensure the kernel does not use a route whose target interface is down. The route is marked with a "dead"/RTNH_F_DEAD flag. Currently, BIRD still uses and distributes this route. This patch just ignores such a route. This patch could be backported to 1.6.x. Signed-off-by: Vincent Bernat <vincent(a)bernat.ch> --- sysdep/linux/netlink.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sysdep/linux/netlink.c b/sysdep/linux/netlink.c index f85bcf35685b..c28126510e6e 100644 --- a/sysdep/linux/netlink.c +++ b/sysdep/linux/netlink.c @@ -1690,6 +1690,9 @@ nl_parse_route(struct nl_parse_state *s, struct nlmsghdr *h) if (i->rtm_flags & RTNH_F_ONLINK) ra->nh.flags |= RNF_ONLINK; + if (i->rtm_flags & RTNH_F_DEAD) + return; + neighbor *nbr; nbr = neigh_find(&p->p, ra->nh.gw, ra->nh.iface, (ra->nh.flags & RNF_ONLINK) ? NEF_ONLINK : 0); -- 2.28.0

5 15

Possibility to treat /32 and /128 non-gateway routes as onlink on BSD?
by Stefan Haller 27 Dec '21

27 Dec '21

Hi everyone, In a recent change in the FreeBSD Wireguard kernel module the POINTTOPOINT interface flag is dropped for wg interfaces[1]. This behaviour will probably not change, because Wireguard is not peer-to-peer or a broadcast domain, but peer-to-multipoint. Hence my previous configuration of setting peer addresses on the interface is not working anymore: ifconfig wg0 inet 192.168.0.10/32 192.168.0.4 The above syntax is for POINTTOPOINT interfaces. In this case 192.168.0.10 is the local address of the interface and 192.168.0.4 is the address of the remote peer. So if one does not want to add a dedicated subnet to every tunnel device, one has to add static host routes similar to these ones: ifconfig wg0 192.168.0.10/32 route add 192.168.0.4 -iface wg0 Note that this actually models the real world better, because Wireguard is not point-to-point but point-to-multipoint. So multiple peers can be done in the following way: ifconfig wg0 192.168.0.10/32 route add 192.168.0.4 -iface wg0 route add 192.168.0.8 -iface wg0 However, this does not work with bird 2.0.8: bird will not recognize the p2p peer addresses as gateways and it will log on kernel table rescan: KRT: Received route 192.168.42.0/24 with strange next-hop 192.168.0.4 KRT: Error sending route 192.168.42.0/24 to kernel: File exists I did have a look why this is happening: bird discards the route, because the gateway does not belong to any of its interface's addresses (the peer address is configured as a static host route and not on the interface). At the same time it forgets that it was bird itself who added the route (because the route is discarded on kernel table rescan, it thinks it is absent in the kernel table but present in bird's internal routing table -> "add" logic triggers) and tries to add the route again (instead of replacing/keeping the route). This explains the cascade of error messages above. While the Linux netlink code allows to handle onlink routes, this functionality is absent in the BSD code path. Unfortunately, on BSD there is no dedicated flag to mark routes as being directly connected onlink routes. On Linux one could add an onlink route like so: ip route add 192.168.0.4 via 192.168.0.4 dev wg0 onlink The command looks quite odd, but it is working. The via clause is necessary in conjunction with onlink. Afterwards a `birdc dump neighbors` verifies that the peer is recognized: 000055db793035c0 192.168.0.4 wg0 wg0 kernel1 000000000000000000000000 scope univ ONLINK (Also the "strange next-hop" message will go away.) I created a patch that emulates the behaviour on FreeBSD. In this patch the onlink setting is heuristically determined by inspecting the RTF_GATEWAY flag (which must be absent) and checking for a /32 or /128 prefix length. I do not see how to handle cases where the destination is a subnet. The patch was tested on FreeBSD 13.0-RELEASE and 14.0-CURRENT. During rescan of the kernel table the above route is now not dropped because the gateway is recognized to be onlink. While in the Linux route the gateway is still set, on FreeBSD it is unset (~RTF_GATEWAY). So I actually use the destination address of the route, because that's the address of the peer. While I tested the functionality on my machines and also checked the output of `birdc dump neighbors`, the patch might be faulty. Please be kind with me and treat it more as a proof of concept. As I am not experienced with bird's code base I do not fully understand how the neighbor cache is working in detail, so the suggested patch might have unexpected side effects. I would appreciate feedback. In case you think the whole idea is not sensible, I would also love to hear suggestions how to get the above setup working. Kind regards, Stefan [1]: https://git.zx2c4.com/wireguard-freebsd/commit/?id=8801509656e955c27ebf4b9b… sysdep/bsd/krt-sock.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/sysdep/bsd/krt-sock.c b/sysdep/bsd/krt-sock.c index ea0cc4d9..4a0f3a7a 100644 --- a/sysdep/bsd/krt-sock.c +++ b/sysdep/bsd/krt-sock.c @@ -547,6 +547,12 @@ krt_read_route(struct ks_msg *msg, struct krt_proto *p, int scan) net->n.addr, a.nh.gw); return; } + } else if ((!ipv6 && pxlen == 32) || (ipv6 && pxlen == 128)) { + /* Treat non gateway /32 and /128 routes as onlink routes and inject this + information into neighbor cache. */ + if (!neigh_find(&p->p, idst, a.nh.iface, NEF_ONLINK)) + log(L_ERR, "KRT: Failed to inject onlink neighbor %I for interface %s", + idst, a.nh.iface->name); } done: -- 2.31.1

2 5

[patch] Add contextual out-of-bound checks in RTR Prefix PDU handler
by Job Snijders 18 Dec '21

18 Dec '21

Hi, A broken RPKI Cache could contextually underflow or overflow the Max Length value in RTR IPv4 / IPv6 Prefix PDUs. Below is a changeset proposal which adds specific out-of-bounds checks and schedules a reconnect for a later moment. This aligns BIRD's behavior with other COTS BGP implementations. RFC 6810 and RFC 8210 specify that the "Max Length" value MUST NOT be less than the Prefix Length element (underflow). On the other side overflow of the Max Length element also is possible, it being an 8-bit unsigned integer allows for values larger than 32 or 128. When a PDU is received where the Max Length field is corrputed, the RTR client (BIRD) should immediately terminate the session, flush all data learned from that cache, and log an error for the operator. If this condition is hit, it seems better to follow the same approach as for RPKI_CS_ERROR_TRANSPORT, because rpki_force_restart_proto() causes an "as fast as possible" reconnect loop between BIRD and the RPKI Cache server, which increases resource consumption on both sides. Kind regards, Job diff --git proto/rpki/packets.c proto/rpki/packets.c index dd11f997..09e20f38 100644 --- proto/rpki/packets.c +++ proto/rpki/packets.c @@ -737,6 +737,22 @@ rpki_handle_prefix_pdu(struct rpki_cache *cache, const struct pdu_header *pdu) net_addr_union addr = {}; rpki_prefix_pdu_2_net_addr(pdu, &addr); + if (type == IPV4_PREFIX) { + if (addr.roa4.max_pxlen > IP4_MAX_PREFIX_LENGTH || addr.roa4.max_pxlen < addr.roa4.pxlen) { + RPKI_WARN(cache->p, "Received a corrupted packet from cache server"); + rpki_send_error_pdu(cache, CORRUPT_DATA, pdu->len, pdu, "Corrupted PDU"); + rpki_cache_change_state(cache, RPKI_CS_ERROR_FATAL); + return RPKI_ERROR; + } + } else { + if (addr.roa6.max_pxlen > IP6_MAX_PREFIX_LENGTH || addr.roa6.max_pxlen < addr.roa6.pxlen) { + RPKI_WARN(cache->p, "Received a corrupted packet from cache server"); + rpki_send_error_pdu(cache, CORRUPT_DATA, pdu->len, pdu, "Corrupted PDU"); + rpki_cache_change_state(cache, RPKI_CS_ERROR_FATAL); + return RPKI_ERROR; + } + } + if (cf->ignore_max_length) { if (type == IPV4_PREFIX) diff --git proto/rpki/rpki.c proto/rpki/rpki.c index ab0837f3..91b69da0 100644 --- proto/rpki/rpki.c +++ proto/rpki/rpki.c @@ -288,9 +288,6 @@ rpki_cache_change_state(struct rpki_cache *cache, const enum rpki_cache_state ne case RPKI_CS_ERROR_FATAL: /* Fatal protocol error occurred. */ - rpki_force_restart_proto(cache->p); - break; - case RPKI_CS_ERROR_TRANSPORT: /* Error on the transport socket occurred. */ rpki_close_connection(cache);

2 5

Bird 2.0.8 - Automatic RPKI reload behaviour
by Tiago Felipe Goncalves 05 Nov '21

05 Nov '21

Hello, I'm curious to understand how "Automatic channel reloads based on RPKI changes" works, as I've noticed that after I enabled RPKI reload, the daemon is constantly reloading the ingress policies/filters. I have set it up in the laboratory and observed that the ingress policies/filters are constantly re-processed even if the following: - We don't have RPKI RTR sessions flapping, being renegotiated, and/or changing state. - We don't have a refresh requested from/to the clients/neighbours BGP sessions. - We don't have BGP updates coming or going from/to BGP clients/neighbours at the moment that I see the ingress policy/filter being re-processed in bird2. - We don't have RPKI ROA update/changes for the specific prefix during this meantime. >From Bird doc: "BIRD offers crude automatic re-validating of affected routes after RPKI update, see option rpki reload. ... In contrast to to other filter operators, this status for the same route may change as the content of ROA tables changes. When this option is active, BIRD activates automatic reload of affected channels whenever ROA tables are updated (after a short settle time)." I sincerely expected that the automatic reload would be triggered after an event (RPKI RTR session flapping/changing state) or if the specific ROA changes (not any ROA changes). For small deployments, the extra load wouldn't be an issue, but I have deployments with 3k active clients/neighbours, and the extra load concerns me to use this function in production. The lab env is using as minimal configuration as possible, doing RPKI validation (one RTR session with routinator), and 2 clients/neighbours. I just added a "print" to ingress protocol filter to see each time that the daemon process it: Oct 26 07:21:19 bird_lab-01 bird: Reconfigured Oct 26 07:21:43 bird_lab-01 bird: Started Oct 26 07:21:48 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 07:21:48 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) Oct 26 07:26:43 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 07:26:43 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) Oct 26 07:47:21 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 07:47:21 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) Oct 26 08:07:52 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 08:07:52 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) Oct 26 08:20:39 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 08:20:39 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) Oct 26 08:43:35 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 08:43:35 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) Oct 26 09:04:07 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 09:04:07 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) Oct 26 09:26:56 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 09:26:56 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) Oct 26 09:47:28 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 09:47:28 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) Oct 26 10:08:00 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 10:08:00 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) Oct 26 10:28:36 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 10:28:36 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) Oct 26 10:51:29 bird_lab-01 bird: INGRESS FILTER => 192.168.0.0/24 => (path 65002) Oct 26 10:51:29 bird_lab-01 bird: INGRESS FILTER => 192.168.1.0/24 => (path 65003) ... bird> show proto Name Proto Table State Since Info device1 Device --- up 2021-10-26 07:21:43 kernel1 Kernel master4 up 2021-10-26 07:21:43 rpki1 RPKI --- up 2021-10-26 07:21:43 Established TEST BGP --- up 2021-10-26 07:21:47 Established TEST2 BGP --- up 2021-10-26 07:21:47 Established Configuration snippet(not production configuration): ... filter BGP_IN { RPKI_IPV4(65002); print "INGRESS FILTER => ", net, " => ", bgp_path; accept; } protocol bgp TEST from PEERS { neighbor 192.168.242.49 as 65002; ipv4 { import keep filtered on; import table yes; import filter BGP_IN; export all; next hop keep; }; } ... When/if I disable the automatic RPKI reload(comment out import table yes), this behaviour(constantly re-processing ingress filters) is gone. If possible, could you please help me to understand if this is the expected behaviour for the automatic RPKI reload? Thanks in advance! Best regards, Tiago Felipe Gonçalves PGP Fingerprint - A2:82:BD:48:EE:8D:C4:99:C2:4E:81:D4:C4:7B:1C:2E:C7:F3:04:C9

2 2

unstable BGP sessions after debian 10 -> 11 upgrade
by Christoph 02 Nov '21

02 Nov '21

Hello, we upgraded our debian BGP routers from debian buster to bullseye. On debian 10 they used the repo https://bird.network.cz/debian/ now they use the bird2 package directly from the official debian repos, both repos contain BIRD version 2.0.7. After upgrading and rebooting we noticed that iBGP sessions are constantly dying every few minutes. The logs also show multiple of these log entries: <WARN> Kernel dropped some netlink messages, will resync on next scan. <WARN> Event ... took 31415 ms <WARN> I/O loop cycle took 31529 ms for 9 events <WARN> Netlink: File exists Especially the warning "Kernel dropped some netlink messages" was not new but we previously solved them with these sysctl.conf settings: # solves the "Kernel dropped some netlink messages, will resync on next scan." BIRD Warnings # https://bird.network.cz/pipermail/bird-users/2017-September/011542.html # (we did not change the wmem_default since this is apparently not needed) net.core.rmem_max=4194304 net.core.rmem_default=4194304 net.core.wmem_max=4194304 The most obvious change is the linux kernel version: 4.19.208 (buster) 5.10.70 (bullseye) To avoid constantly dying iBGP sessions we increased the keepalive and hold timers. Are you observing similar issues on your debian 11 systems? Does the newer kernel need higher rmem values? best regards, Christoph

3 3

BFD+BGP behavior difference between bird 1.6 and 2.0
by Kseniya Blashchuk 29 Oct '21

29 Oct '21

Hello! After upgrading to 2.0 I have noticed that "bgp on" in BGP peer configuration creates a separate BFD session, however it wasn't like that in bird 1.6, we had to specify neighbors in BFD section. Now for bird 2.0 it looks like specifying neighbors in BFD section is not necessary when "bgp on" is set in BGP configuration. The problem I have faced was "unknown session id" errors between two BGP routers, one running bird 1.6 and another running bird 2.0, same config. It looked like bird 2.0 was creating two BFD sessions comparing to bird 1.6 that created only one. Removing neighbor from BFD section at bird 2.0 fixed the issue. Can anybody tell me if anything has changed in 2.0 or am I missing something? The documentation looks quite similar for both versions in this case.

2 2

How to make bird now its own connectivity (OSPF)
by Lukas Haase 24 Oct '21

24 Oct '21

Hi, Assume the following toy example: R1: R2: 10.1.1.0/24 <--- 10.1.1.1, 192.168.1.1 <---------> 192.168.1.2, 10.1.2.1 ---> 10.1.2.0/24 Net 10.1.1.0/24 has R1 as default route, net 10.1.2.0/24 R2. Since the network connectivity is known, 10.1.2.0/24 could reach 10.1.2.0/24 and vice versa. Of course, R1 and R2 need to exchange that information. It was my understanding that OSPF would do this automatically. But this does not seem to be the case with Bird: Dynamic routes (10.1.1.0/24 and 192.168.1.0/24 for R1 and 192.168.1.0/24 and 10.1.2.0/24 for R2) do not land automatically in bird's routing table. I would like to understand exactly why and what is the "right" way to configure bird. Many protocols have flags to allow this but all say it would only be for "exceptional" purposes. But the above network is pretty much the most common one. Where else would routing information come from if not from the network connectivity? Note that this is not OSPF specific: My Mikrotik router automatically sends me all the routes of its connected networks, as I would expect it. So, I think there are the following options: 1.) The direct protocol. But the doc makes it sound this is not the right path: "[...] Although there are some use cases that use the direct protocol (like abusing eBGP as an IGP routing protocol), in most cases it is not needed to have these device routes in BIRD routing table and to use the direct protocol. [...]" 2.) The kernel protocol. But also here the docs have a strong bias importing dynamic routes (device routes): "[...] Unfortunately, there is one thing that makes the routing table synchronization a bit more complicated. In the kernel routing table there are also device routes for directly connected networks. These routes are usually managed by OS itself (as a part of IP address configuration) and we don't want to touch that. They are completely ignored during the scan of the kernel tables and also the export of device routes from BIRD tables to kernel routing tables is restricted to prevent accidental interference." 3.) The static protocol. I could add something like (for R1): protocol static { export all; route 10.1.1.0/24 via 192.168.1.1; } But then I would manually re-create the already known network connectivity in the config file. Why does no protocol in Bird want to take responsibility for the router's own connectivity? How on earth is this supposed to work? Thanks, Lukas

3 3