- Bird-users - Mailing Lists

Address update causing infinite loop
by James Oakley 01 Feb '22

01 Feb '22

Hi, We've had an occasional issue where we found BIRD stuck using 100% CPU a few times over the past couple of years. Whenever it happened, all protocols stopped running and birdc was unable to communicate with the process. Unfortunately, it was usually in a situation where we could not quite see what was going so we ended up having to kill the process and restart it to get things working again. However, today it finally happened again, and I was able to grab the debug symbols for the build so I could inspect it in gdb. So here is the backtrace: (gdb) bt #0 if_recalc_preferred (i=0x56541cd48de0) at nest/iface.c:518 #1 0x000056541c662d55 in if_end_partial_update (i=i@entry=0x56541cd48de0) at nest/iface.c:359 #2 0x000056541c6b19cb in nl_parse_addr4 (new=1, scan=0, i=0x56541cc20700) at sysdep/linux/netlink.c:1020 #3 nl_parse_addr (h=h@entry=0x56541cc206f0, scan=scan@entry=0) at sysdep/linux/netlink.c:1128 #4 0x000056541c6b3127 in nl_async_msg (h=0x56541cc206f0) at sysdep/linux/netlink.c:1932 #5 nl_async_hook (sk=<optimized out>, size=<optimized out>) at sysdep/linux/netlink.c:1985 #6 0x000056541c6b74a0 in sk_read (s=s@entry=0x56541cc22700, revents=<optimized out>) at sysdep/unix/io.c:1896 #7 0x000056541c6b80ec in io_loop () at sysdep/unix/io.c:2345 #8 0x000056541c633f61 in main (argc=<optimized out>, argv=<optimized out>) at sysdep/unix/main.c:939 If I keep hitting next, it appears to be stuck in a loop: (gdb) n 524 nest/iface.c: No such file or directory. (gdb) n 514 in nest/iface.c (gdb) n 517 in nest/iface.c (gdb) n 518 in nest/iface.c (gdb) n 524 in nest/iface.c (gdb) n 514 in nest/iface.c (gdb) n 517 in nest/iface.c (gdb) n 518 in nest/iface.c (gdb) n 524 in nest/iface.c (gdb) n 514 in nest/iface.c So here is what that code looks like: 510 struct ifa *a; 511 WALK_LIST(a, i->addrs) 512 { 513 /* Secondary address is never selected */ 514 if (a->flags & IA_SECONDARY) 515 continue; 516 517 if (ipa_is_ip4(a->ip)) { 518 if (!a4 || ipa_equal(a->ip, pref_v4)) 519 a4 = a; 520 } else if (!ipa_is_link_local(a->ip)) { 521 if (!a6 || ipa_equal(a->ip, ic->pref_v6)) 522 a6 = a; 523 } else { 524 if (!ll || ipa_equal(a->ip, ic->pref_ll)) 525 ll = a; 526 } 527 } OK, we are iterating over a linked link. Let's look at the list: (gdb) p i->addrs $30 = {{head_node = {next = 0x56541cd02b30, prev = 0x0}, head_padding = 0x56541cd02b30}, {tail_padding = 0x56541cd02b30, tail_node = {next = 0x0, prev = 0x56541cd02b30}}, {head = 0x56541cd02b30, null = 0x0, tail = 0x56541cd02b30}} (gdb) p i->addrs.head $31 = (struct node *) 0x56541cd02b30 (gdb) p *i->addrs.head $32 = {next = 0x56541cd02b30, prev = 0x56541cd02b30} So there is a single item, with next pointing to itself. Looking up the stack, it seems the only place here where this list is modified is when nl_parse_addr4() calls ifa_update() which then calls add_tail(), which looks like this: EXPENSIVE_CHECK(check_list(l, NULL)); ASSUME(n->prev == NULL); ASSUME(n->next == NULL); node *z = l->tail; n->next = &l->tail_node; n->prev = z; z->next = n; l->tail = n; So this is where I am stumped. How could it even be possible for the item next pointer to be set to it's own address, and not that of the static tail node? Any insights? -- James Oakley james(a)ttgi.io

2 2

Two router ha setup questions
by Alessandro Brega 22 Jan '22

22 Jan '22

Hello guys, I have two identically configured routers (bird2, only the local ip is different), connected to one upstream and a few internal vlans. Each router has a BGP session with the upstream router. I have a couple of small local subnets (/27, /28, ..), which are on different vlan interfaces, which I need to announce to upstream. They all belong to one big network (/23). As the upstream router doesn't accept routes smaller than /24, I only announce a single route to the /23 network. I uploaded my bird configuration https://controlc.com/aa226135. For completeness: for first hop redundancy (default gateway of the clients using the small subnets) I use keepalived on my routers, which works fine so far. What's the problem: As both routers announce the same network to upstream, upstream sends some of its traffic to router1 and some to router2 (but doesn't seem balanced in any way). This seems to cause (at least) tcp ordering issues (I suspect, couldn't confirm it in any way), because sometime some connections from local client to clients behind the upstream "hang" for a couple of seconds. If I stop bird on the backup router (so all traffic only goes to the active one) no hangs occur. The question: Is my setup ok or is it (completely) broken? Is it ok to have both routers announce the same subnets at the same time? Possible solution I'm thinking about but don't know how to do it: Both routers should have an active BGP sessions (for fast failover), but only the router which is active (and thus has the gateway ips) should announce the "aggregated" route (the /23). This way no traffic would go over the backup. But I wonder how this could be configured in bird? I could configure keepalived to not only create the gateway ips but also the small subnets (in my current config the subnets are created by bird). Would this make things easier for bird? My last resort would be to have keepalived execute scripts which reconfigure bird on failover. But somehow this feels wrong? Or is this the way to do it? Thank you very much for reading and any help. I'm struggling with this for days now... :-( Alessandro

4 3

[PATCH] RFC: Add netlink KRT dump filters on Linux
by Tomas Hlavacek 18 Jan '22

18 Jan '22

Add netlink KRT dump filter on Linux to avoid PMTU cache records from FNHE table dump along with KRT. Linux Kernel added FNHE table dump to the netlink API in patch https://patchwork.ozlabs.org/project/netdev/patch/8d3b68cd37fb5fddc470904cd… The filter mitigates the risk of receiving unknown and potentially large number of FNHE records that would block BIRD I/O in each sync. There is a known issue caused by the GRE tunnels on Linux that seems to be creating one FNHE record for each destination IP address that is routed through the tunnel, even when the PMTU equals to GRE interface MTU (tested with kernel 5.5 - 5.16-rc7). --- sysdep/linux/netlink.c | 44 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 41 insertions(+), 3 deletions(-) diff --git a/sysdep/linux/netlink.c b/sysdep/linux/netlink.c index f85bcf35..79414122 100644 --- a/sysdep/linux/netlink.c +++ b/sysdep/linux/netlink.c @@ -128,7 +128,7 @@ struct nl_sock uint last_size; }; -#define NL_RX_SIZE 8192 +#define NL_RX_SIZE 32768 #define NL_OP_DELETE 0 #define NL_OP_ADD (NLM_F_CREATE|NLM_F_EXCL) @@ -143,11 +143,18 @@ static struct nl_sock nl_req = {.fd = -1}; /* Netlink socket for requests */ static void nl_open_sock(struct nl_sock *nl) { + int sndbuf = 32768; + int rcvbuf = 1024*1024; + if (nl->fd < 0) { - nl->fd = socket(PF_NETLINK, SOCK_RAW, NETLINK_ROUTE); + nl->fd = socket(PF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, NETLINK_ROUTE); if (nl->fd < 0) die("Unable to open rtnetlink socket: %m"); + + setsockopt(nl->fd, SOL_SOCKET, SO_SNDBUF, &sndbuf, sizeof(sndbuf)); + setsockopt(nl->fd, SOL_SOCKET, SO_RCVBUF, &rcvbuf, sizeof(rcvbuf)); + nl->seq = (u32) (current_time() TO_S); /* Or perhaps random_u32() ? */ nl->rx_buffer = xmalloc(NL_RX_SIZE); nl->last_hdr = NULL; @@ -155,6 +162,12 @@ nl_open_sock(struct nl_sock *nl) } } +static void +nl_set_strict_dump(struct nl_sock *nl, int strict) +{ + setsockopt(nl->fd, SOL_NETLINK, NETLINK_GET_STRICT_CHK, &strict, sizeof(strict)); +} + static void nl_open(void) { @@ -192,6 +205,29 @@ nl_request_dump(int af, int cmd) nl_send(&nl_scan, &req.nh); } +static void +nl_request_dump_rt(int af, int cmd) +{ + struct { + struct nlmsghdr nh; + struct rtmsg rtm; + char buf[128]; + } req = { + .nh.nlmsg_type = cmd, + .nh.nlmsg_len = NLMSG_LENGTH(sizeof(struct rtmsg)), + .nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP, + .nh.nlmsg_seq = ++(nl_scan.seq), + .nh.nlmsg_pid = 0, + .rtm.rtm_protocol = RTPROT_UNSPEC, + .rtm.rtm_family = af + /* .rtm.rtm_flags is defaults to zero, hence RTM_F_CLONED is not set */ + }; + + send(nl_scan.fd, &req, sizeof(req), 0); + nl_scan.last_hdr = NULL; +} + + static struct nlmsghdr * nl_get_reply(struct nl_sock *nl) { @@ -1864,13 +1900,15 @@ krt_do_scan(struct krt_proto *p UNUSED) /* CONFIG_ALL_TABLES_AT_ONCE => p is NUL struct nl_parse_state s; nl_parse_begin(&s, 1); - nl_request_dump(AF_UNSPEC, RTM_GETROUTE); + nl_set_strict_dump(&nl_scan, 1); + nl_request_dump_rt(AF_UNSPEC, RTM_GETROUTE); while (h = nl_get_scan()) if (h->nlmsg_type == RTM_NEWROUTE || h->nlmsg_type == RTM_DELROUTE) nl_parse_route(&s, h); else log(L_DEBUG "nl_scan_fire: Unknown packet received (type=%d)", h->nlmsg_type); nl_parse_end(&s); + nl_set_strict_dump(&nl_scan, 0); } /* -- 2.25.1

2 2

Traffic Engineering (TE) Extensions to OSPF Version 2 RFC3630 Support
by Senthil Kumar Nagappan 13 Jan '22

13 Jan '22

Hi, Checked the documents, Looks like support for "Traffic Engineering (TE) Extensions to OSPF Version 2 RFC3630" is not there yet in bird routing. Can you please confirm? Any plans to support it? Thanks and Regards,Senthil

1 0

Bird as BGP flow-spec announcement filter
by Blažej Krajňák 10 Jan '22

10 Jan '22

Hello, is it possible to do BGP Flowspec announcement filtering with Bird? For ex. I want to provide capability to my customer to annouce to my network BGP Flowspec rules. However, I must check, if announced rule have dst from customers IP range. Bird should act as route reflector only. Thanks, Blažej

4 4

Incomplete bgp connection can close a completed bgp connection
by 生. 10 Jan '22

10 Jan '22

Hello, I have found an issue in Bird 2.0.8. If the tcp connection is not completely, then a new bgp conntection is success. When the hold timer of the first tcp connection is expired, it will close both of the connection.              router1                                        router2            192.168.1.100                      192.168.1.200 first        <----------------------------------------(tcp syn)                 -----------------------------------------> (tcp syn ack)                <---------------------------------------------(tcp ack, open) second   <---------------------------------------------> (tcp )                <---------------------------------------------> (bgp established) first holdtime expire                <---------------------------------------------   (bgp close first)                    <---------------------------------------------   (bgp close second)       The full packet is in the attachment, is this a correct operation ?

1 0

BGP strict bind socket error
by Ben Tremblay 09 Jan '22

09 Jan '22

Hi all, I have encountered a problem with how BIRD opens the listening socket for strict bind BGP. In particular, BIRD sometimes tries to open the listening socket for the protocol when the configured local IP is not yet present on the interface. It seems that BIRD will try to bind the BGP socket to the configured IP if any address exists on the protocol interface that can be used to reach the neighbor. Let me give an example to clarify. I have a minimal reproducible example with the following BGP configuration: router id 192.168.0.1; protocol device { scan time 10; } protocol bgp bgp_01 { ipv4 { import all; export all; }; debug all; local 192.168.0.1 port 179 as 1; neighbor 192.168.0.2 port 179 as 1; interface "host1"; direct; strict bind yes; } Given this configuration, I can reproduce the problem behaviour by starting the protocol with no addressing on the host1 interface and then adding an address which includes the neighbor IP in its subnet, e.g. 192.168.0.3/24. Once this address/route is added, BIRD logs the following: Jan 07 12:09:31 router03 bird[57497]: bgp_01: Neighbor ready Jan 07 12:09:31 router03 bird[57497]: bgp_01: Socket error: bind: Cannot assign requested address Jan 07 12:09:31 router03 bird[57497]: bgp_01: Cannot open listening socket And the protocol enters an error state: BIRD 2.0.6 ready. Name Proto Table State Since Info device1 Device --- up 11:55:51.049 bgp_01 BGP --- down 12:09:31.264 Error: No listening socket The reason this error is so problematic for me is that the protocol cannot recover from this state without administrative intervention (e.g. configuring bird or restarting the protocol). So, I have two questions: 1. Is this correct behaviour? 2. Can I configure BIRD so that my protocol is able to recover after suffering a socket error like this? I should also mention I can actually observe this socket error occur when adding the 'correct' address to the protocol interface, but only in the context of a larger application and only some of the time. Thanks, Ben

4 12

[BIRD 2.0.x] Netlink: ignore dead routes
by Vincent Bernat 09 Jan '22

09 Jan '22

With net.ipv4.conf.XXX.ignore_routes_with_linkdown sysctl, a user can ensure the kernel does not use a route whose target interface is down. The route is marked with a "dead"/RTNH_F_DEAD flag. Currently, BIRD still uses and distributes this route. This patch just ignores such a route. This patch could be backported to 1.6.x. Signed-off-by: Vincent Bernat <vincent(a)bernat.ch> --- sysdep/linux/netlink.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sysdep/linux/netlink.c b/sysdep/linux/netlink.c index f85bcf35685b..c28126510e6e 100644 --- a/sysdep/linux/netlink.c +++ b/sysdep/linux/netlink.c @@ -1690,6 +1690,9 @@ nl_parse_route(struct nl_parse_state *s, struct nlmsghdr *h) if (i->rtm_flags & RTNH_F_ONLINK) ra->nh.flags |= RNF_ONLINK; + if (i->rtm_flags & RTNH_F_DEAD) + return; + neighbor *nbr; nbr = neigh_find(&p->p, ra->nh.gw, ra->nh.iface, (ra->nh.flags & RNF_ONLINK) ? NEF_ONLINK : 0); -- 2.28.0

5 15

netlink filtering to avoid clostly FNHE table dumps on Linux
by Tomas Hlavacek 09 Jan '22

09 Jan '22

Hi! This is not a direct follow-up on the thread "BIRD trying to reinsert existing kernel routes, netlink issue?" ( https://www.mail-archive.com/bird-users@network.cz/msg05429.html ) but I think that the cause of my issue can be the same. My scenario: I have a server in LAB that has a public IP on a GRE tunnel interface and the tunnel goes to a router in the Internet. BIRD receives full BGP feed over the GRE tunnel. The tunnel is terminated in a VRF and BIRD exports the BGP feed to the kernel RT associated with the VRF. I observed pretty much the same set of issues as Sasha and others described in the linked thread two years ago: BIRD starts normally, BGP session comes up but after 3 or 5 minutes it max-out one CPU, mostly in sys (doing the netlink I/O). Then BIRD stops responding normally over birdc - each command takes several minutes to complete and I also saw the log messages like: Jan 2 00:00:35 lab bird: Netlink: File exists Jan 2 00:00:35 lab bird: Netlink: File exists Jan 2 00:00:35 lab bird: ... Jan 2 00:00:35 lab bird: I/O loop cycle took 73127 ms for 11 events I realized that BIRD actually transfers huge amounts of data from kernel over netlink (several gigabytes of messages per minute or so, which translates to hundreds of millions of FIB records) in the beginning of each sync and it ultimately stalls BIRD I/O long enough to miss BGP keepalive deadlines and the sessions start flapping, which makes the situation hard to understand using the conventional profiling tools, tcpdump for netlink dump etc. FYI I am attaching my quick and dirty patch to BIRD that I used to collect stats from netlink interactions to understand the problem and finally to add experimental PoC fix - see nl_request_rt_dump() in the patch. The large table that BIRD pulled from the kernel was a FNHE table where Linux collects PMTU records for *all* destination IPs that are routed to the tunnel (which does not seem to be right and I will discuss it in LKML shortly). These records have (default) 600s expiration time and in my scenario I happen to receive some backscatter traffic that in most cases gets ICMP or TCP reset responses that could ultimately create millions of these records in a few minutes. The reason why this problem occured only in Linux ~5.2+ lies in the patch https://patchwork.ozlabs.org/project/netdev/patch/8d3b68cd37fb5fddc470904cd… that changed the semantics of netlink dump requests. Now the kernel dumps the FIB Next Hop Exceptions table (previously known as route cache) alongside the RT unless the requester sets sockopt NETLINK_GET_STRICT_CHK and clear the flag RTM_F_CLONED in the dump request. BIRD does not apply the filters so the kernel dumps everything. But iproute2 and other programs that use netlink utilize the filters, so no similar performance issue occurs unless I explicitly dump the FNHE table (ip route show cache). I believe that many different types of Linux tunnels create the PMTU records for all packets transmitted over the tunnel as well. And it works like that for a long time - the code that creates the route cache (at that time, now it is FNHE table) records has been introduced in Linux 3.10 (https://elixir.bootlin.com/linux/v3.10/source/net/ipv4/ip_tunnel.c#L591) Regardless of what may or may not happen on the kernel side I think that implementing the netlink filter in BIRD to avoid the described situation makes sense. I am almost certain that my experimental fix breaks other things (most likely OSPF) but I would be glad to help make it right. What do you think? Best regards, Tomas

2 2

fix reduce/reduce conflict for rpki remote
by Alexander Zubkov 03 Jan '22

03 Jan '22

Hi, There is a reduce/reduce conflict during build because in rpki proto it is possible to specify remote both as IP or text hostname. And both in place of both tokens there can be a named constant. I added a new token type that returns a union of IP and string to use in such cases where we need something like hostname.

1 0