January 2022 - Bird-users

Re: comparing open source BGP stacks performance
by Justin Pietsch 04 Mar '22

04 Mar '22

(added bird-users@ back, I didn't mean to remove them before) Sorry that I missed that you'd already added the pull request. I merged it. we do need a way to specify exabgp vs bird, but that's just another thing to add. I agree about containers and versions. Just haven't had time to think it through carefully. The policy support is what bgperf had before I forked it. I don't use it, I don't know how to usefully test with it. One more feature I'd like to see, yet I'm not sure how to do this > properly – to make the measurement more accurate, the target should not > only run on its own machine, but bgperf with all the testers should also > watch their load, issuing warning when the bgperf suite may be the > bottleneck. This will probably help testing some hardware routers in a > somehow comparable way. > I agree. My best quick hack for this is to make sure that there are extra resources on the CPU when running. That's why bgperf now tracks %idle. As you can probably tell, bgperf isn't a great architecture now, and I'm trying to get useful results as quickly as possible. So much could be done here. thanks a lot! Justin On Tue, Aug 31, 2021 at 2:30 AM Maria Matejka <maria.matejka(a)nic.cz> wrote: > Hello! > > On 8/31/21 1:29 AM, Justin Pietsch wrote: > > I meant to ask if you'd compared exabgp to bird as testers? I'll try to > > compare later if I have some time. > Well, I have no exact numbers, I just implemented the bird tester and it > seems to eat less memory and less time. You could probably run the same > batch first with exabgp testers and then with bird testers, comparing > how much this affects the results. > > > On Mon, Aug 30, 2021 at 4:27 PM Justin Pietsch <jpietsch(a)gmail.com > > <mailto:jpietsch@gmail.com>> wrote: > > > > This is fantastic!!!! Can you submit those changes as PRs to my > > bgperf? I'm working on moving it to a more permanent place. Then we > > can track issues! :) > > Well, yes, the tester exchange is already there pending: > > https://github.com/jopietsch/bgperf/pull/1 > > and it seems that there are two more from somebody else. > > The tester exchange is anyway quite stupid and may need some similar > code to MRT testers instead of what is there now. > > > your suggestion for specifying branches used to work in bgperf, but > > I broke it in trying to get everything working. I moved some of the > > targets to just download prebuilt docker images, which makes it > > harder. Also with batch I'm not sure how to have multiple versions > > of the same container. Just more things to figure out, but that all > > makes sense. > > I think it would be better to have separate containers for different > branches, yet I couldn't figure out how to do that, except for brutal > solutions which were too disgusting to implement. > > Anyway, when BIRD is used as tester, it shouldn't imho change with the > target, which is probably the main reason why I prefer separate > containers for branches. > > One more thing – when running update -c, the container should also pull, > not only checkout, imho. (Haven't had time to fix this yet.) > > > * fixed policy configuration in BIRD (no clue whether it works > with > > other stacks) > > This may need thorough examination first to formally define what the > policies should mean. It would be really stupid to compare route > filtering with different policies in each stack. > > One more feature I'd like to see, yet I'm not sure how to do this > properly – to make the measurement more accurate, the target should not > only run on its own machine, but bgperf with all the testers should also > watch their load, issuing warning when the bgperf suite may be the > bottleneck. This will probably help testing some hardware routers in a > somehow comparable way. > > Maria >

3 2

Inserting fulltable into kernel FIB makes bird crazy
by Alarig Le Lay 20 Feb '22

20 Feb '22

Hello, Now that the IPv6 bug is supposed to be resolved since 5.8, I tried to upgrade a router from 4.14 to 5.10 Bird starts, however while inserting routes to FIB, I have long I/O loop cycles and at some point bird is unable to keep up. I already recompiled bird in case of a header change or something like that, and to switch to a pre-compiled kernel, neither have any effect. When bird begins to loose track of itself, I have this kind of messages: Sep 24 08:44:43 edge04-hostzealot bird: Netlink: File exists Sep 24 08:44:43 edge04-hostzealot bird: Netlink: File exists Sep 24 08:44:43 edge04-hostzealot bird: Netlink: File exists Sep 24 08:44:43 edge04-hostzealot bird: Netlink: File exists Sep 24 08:44:43 edge04-hostzealot bird: Netlink: File exists Sep 24 08:44:43 edge04-hostzealot bird: ... Sep 24 08:44:43 edge04-hostzealot bird: I/O loop cycle took 28703 ms for 1 events Sep 24 08:44:43 edge04-hostzealot bird: Kernel dropped some netlink messages, will resync on next scan. Sep 24 08:45:50 edge04-hostzealot bird: Netlink: File exists Sep 24 08:45:50 edge04-hostzealot bird: Netlink: File exists Sep 24 08:45:50 edge04-hostzealot bird: Netlink: File exists Sep 24 08:45:50 edge04-hostzealot bird: Netlink: File exists Sep 24 08:45:50 edge04-hostzealot bird: Netlink: File exists Sep 24 08:45:50 edge04-hostzealot bird: ... Sep 24 08:45:51 edge04-hostzealot bird: I/O loop cycle took 36201 ms for 1 events And then ospf begins to flap and routes are re-calculated based on remaining bgp ones. Sep 24 08:46:54 edge04-hostzealot bird: Next hop address 185.107.95.180 resolvable through recursive route for 185.107.92.0/22 (I have a way more specific route in OSPF) I activated the debug, and I can see that bird is re-scanning the entire kernel table when the “I/O loop” message appears Sep 24 09:07:30 edge04-hostzealot bird: kernel_grt_ipv4: 1.0.0.0/24: seen Sep 24 09:07:30 edge04-hostzealot bird: kernel_grt_ipv4: 1.0.4.0/24: seen And it tries to insert already inserted routes Sep 24 09:08:04 edge04-hostzealot bird: kernel_grt_ipv4: 122.76.248.0/23: installing Sep 24 09:08:04 edge04-hostzealot bird: Netlink: File exists And then OSPF is clearly going down Sep 24 09:08:04 edge04-hostzealot bird: ospf_ipv4: Inactivity timer expired for nbr 45.91.126.248 on gre4 Sep 24 09:08:04 edge04-hostzealot bird: ospf_ipv4: Neighbor 45.91.126.248 on gre4 changed state from Full to Down Sep 24 09:08:04 edge04-hostzealot bird: ospf_ipv4: Neighbor 45.91.126.248 on gre4 removed Here are some more detailed logs: https://paste.swordarmor.fr/raw/HX45 https://paste.swordarmor.fr/raw/oM9s This server isn’t the fastest one on the marked, but stuffed enough to handle full views. And with an older kernel it works very well. I have RRs running on 5.10 kernels, so it’s more likely a kernel issue, but I’m not able to determine if it’s caused by the kernel itself or by the way bird is using netlink. I’m using bird 2.0.8, I didn’t try an older version. Thanks a lot, -- Alarig Le Lay

2 3

Re: BIRD trying to reinsert existing kernel routes, netlink issue?
by Zhaofeng Li 18 Feb '22

18 Feb '22

I know this was an old thread, but would really appreciate if anyone is interested in digging in deeper. I can confirm the exact same behavior happens to me on NixOS with the 5.4.69 kernel but not on 4.19.149. BIRD 2.0.7 on 5.4.69 would incessantly attempt to insert the same routes, rendering birdc unusable as it was presumably taking up so much time in the main loop (it was receiving an IPv4 full table over iBGP as well). While it was doing that, it stopped announcing the prefixes of stub interfaces to OSPF neighbors. Thanks, Zhaofeng Li On Sun, Aug 2, 2020, at 17:36 PM, Sasha Romijn wrote: > Hello, > I’m running into a rather complex issue where BIRD tries to reinsert routes into the kernel routing table that are already in there. I’m at the point of suspecting a netlink bug in the 5.x linux kernel. However, I may be entirely off, and would like to hear what people with more BIRD and/or netlink expertise think about it. > My setup is bird 2.0.7 on Ubuntu 20.04. I’ve been able to reproduce this with the default 5.4.0-42-generic kernel as well as the ubuntu mainline 5.7.0-050700-generic kernel. This bird instance has an IBGP peering with another bird 2.0.7 instance, which sends an IPv4 full table. The IBGP peering is set up over a GRE tunnel. There is no other routing software running. I am seeing this issue on another server too, ubuntu 20.04 with 5.4.0-42-generic. A third one with ubuntu 18.04 and a 4.15.0-20-generic kernel, is fine.

3 2

Route Flapping and Bird - RFD and MRAI
by Douglas Fischer 16 Feb '22

16 Feb '22

Hello all. Last weeks I'm noticing the increase the increase of BGP messages on some routers that I have access to. Specially those connected to IXPs with a considerable number of participants. I Checked and the BGP process has increased a bit the use of CPU Also. So, I suspect that Route-Servers are receiving and forwarding Route Flaps from one or more participants... I know that the right place to fix that would be on the source of those flaps... But... Considering IXPs with one ou two thousand participants, there will always be someone flapping some routes. Bird is widely used as the BGP engine on IXP route-servers. And exactly because of that I came here to ask about Route Flap avoidance mechanisms on Bird. The discussion of methods to avoid Route Flapping in BGP is very controversial. - RFD - Route Flapping Dampening comes and goes. - MRAI is not a consensus, but there is a lot of effort on that, including som magic algorithms define some variable interval according to conditions. - I even saw some ultra-rigorous ideas like rate limiting the BGP messages... Before I come here to bore you with this, I searched a bit on mail list and documentation of Bird and I found the links bellow related to that: There is any work running to deal with that route-flapping question? https://bird.network.cz/ - 7.1 Future work <https://bird.network.cz/?get_doc&v=20&f=bird-7.html> Question about BGP advertisement-interval in BIRD <http://trubka.network.cz/pipermail/bird-users/2020-August/014786.html> [BGP] MRAI connection-based implementation review <http://trubka.network.cz/pipermail/bird-users/2020-April/014433.html> Route Flap Dampening <http://trubka.network.cz/pipermail/bird-users/2019-July/013589.html> [BGP] bird and RFD <http://trubka.network.cz/pipermail/bird-users/2020-May/014612.html> Thank you all! -- Douglas Fernando Fischer Engº de Controle e Automação

3 4

BIRD multithreaded progress
by Ross Tajvar 10 Feb '22

10 Feb '22

Hi BIRD team, I have been periodically checking the cz.nic blog for more posts in the "BIRD multithreading" saga, but the latest one was about 5.5 months ago ( https://en.blog.nic.cz/2021/06/14/bird-journey-to-threads-chapter-2-asynchr…) Has any more progress been made on this? I understand the team is busy and I'm not trying to push, but I enjoy reading these posts and keeping up with the work being done. Best, Ross

4 7

Address update causing infinite loop
by James Oakley 01 Feb '22

01 Feb '22

Hi, We've had an occasional issue where we found BIRD stuck using 100% CPU a few times over the past couple of years. Whenever it happened, all protocols stopped running and birdc was unable to communicate with the process. Unfortunately, it was usually in a situation where we could not quite see what was going so we ended up having to kill the process and restart it to get things working again. However, today it finally happened again, and I was able to grab the debug symbols for the build so I could inspect it in gdb. So here is the backtrace: (gdb) bt #0 if_recalc_preferred (i=0x56541cd48de0) at nest/iface.c:518 #1 0x000056541c662d55 in if_end_partial_update (i=i@entry=0x56541cd48de0) at nest/iface.c:359 #2 0x000056541c6b19cb in nl_parse_addr4 (new=1, scan=0, i=0x56541cc20700) at sysdep/linux/netlink.c:1020 #3 nl_parse_addr (h=h@entry=0x56541cc206f0, scan=scan@entry=0) at sysdep/linux/netlink.c:1128 #4 0x000056541c6b3127 in nl_async_msg (h=0x56541cc206f0) at sysdep/linux/netlink.c:1932 #5 nl_async_hook (sk=<optimized out>, size=<optimized out>) at sysdep/linux/netlink.c:1985 #6 0x000056541c6b74a0 in sk_read (s=s@entry=0x56541cc22700, revents=<optimized out>) at sysdep/unix/io.c:1896 #7 0x000056541c6b80ec in io_loop () at sysdep/unix/io.c:2345 #8 0x000056541c633f61 in main (argc=<optimized out>, argv=<optimized out>) at sysdep/unix/main.c:939 If I keep hitting next, it appears to be stuck in a loop: (gdb) n 524 nest/iface.c: No such file or directory. (gdb) n 514 in nest/iface.c (gdb) n 517 in nest/iface.c (gdb) n 518 in nest/iface.c (gdb) n 524 in nest/iface.c (gdb) n 514 in nest/iface.c (gdb) n 517 in nest/iface.c (gdb) n 518 in nest/iface.c (gdb) n 524 in nest/iface.c (gdb) n 514 in nest/iface.c So here is what that code looks like: 510 struct ifa *a; 511 WALK_LIST(a, i->addrs) 512 { 513 /* Secondary address is never selected */ 514 if (a->flags & IA_SECONDARY) 515 continue; 516 517 if (ipa_is_ip4(a->ip)) { 518 if (!a4 || ipa_equal(a->ip, pref_v4)) 519 a4 = a; 520 } else if (!ipa_is_link_local(a->ip)) { 521 if (!a6 || ipa_equal(a->ip, ic->pref_v6)) 522 a6 = a; 523 } else { 524 if (!ll || ipa_equal(a->ip, ic->pref_ll)) 525 ll = a; 526 } 527 } OK, we are iterating over a linked link. Let's look at the list: (gdb) p i->addrs $30 = {{head_node = {next = 0x56541cd02b30, prev = 0x0}, head_padding = 0x56541cd02b30}, {tail_padding = 0x56541cd02b30, tail_node = {next = 0x0, prev = 0x56541cd02b30}}, {head = 0x56541cd02b30, null = 0x0, tail = 0x56541cd02b30}} (gdb) p i->addrs.head $31 = (struct node *) 0x56541cd02b30 (gdb) p *i->addrs.head $32 = {next = 0x56541cd02b30, prev = 0x56541cd02b30} So there is a single item, with next pointing to itself. Looking up the stack, it seems the only place here where this list is modified is when nl_parse_addr4() calls ifa_update() which then calls add_tail(), which looks like this: EXPENSIVE_CHECK(check_list(l, NULL)); ASSUME(n->prev == NULL); ASSUME(n->next == NULL); node *z = l->tail; n->next = &l->tail_node; n->prev = z; z->next = n; l->tail = n; So this is where I am stumped. How could it even be possible for the item next pointer to be set to it's own address, and not that of the static tail node? Any insights? -- James Oakley james(a)ttgi.io

2 2

Two router ha setup questions
by Alessandro Brega 22 Jan '22

22 Jan '22

Hello guys, I have two identically configured routers (bird2, only the local ip is different), connected to one upstream and a few internal vlans. Each router has a BGP session with the upstream router. I have a couple of small local subnets (/27, /28, ..), which are on different vlan interfaces, which I need to announce to upstream. They all belong to one big network (/23). As the upstream router doesn't accept routes smaller than /24, I only announce a single route to the /23 network. I uploaded my bird configuration https://controlc.com/aa226135. For completeness: for first hop redundancy (default gateway of the clients using the small subnets) I use keepalived on my routers, which works fine so far. What's the problem: As both routers announce the same network to upstream, upstream sends some of its traffic to router1 and some to router2 (but doesn't seem balanced in any way). This seems to cause (at least) tcp ordering issues (I suspect, couldn't confirm it in any way), because sometime some connections from local client to clients behind the upstream "hang" for a couple of seconds. If I stop bird on the backup router (so all traffic only goes to the active one) no hangs occur. The question: Is my setup ok or is it (completely) broken? Is it ok to have both routers announce the same subnets at the same time? Possible solution I'm thinking about but don't know how to do it: Both routers should have an active BGP sessions (for fast failover), but only the router which is active (and thus has the gateway ips) should announce the "aggregated" route (the /23). This way no traffic would go over the backup. But I wonder how this could be configured in bird? I could configure keepalived to not only create the gateway ips but also the small subnets (in my current config the subnets are created by bird). Would this make things easier for bird? My last resort would be to have keepalived execute scripts which reconfigure bird on failover. But somehow this feels wrong? Or is this the way to do it? Thank you very much for reading and any help. I'm struggling with this for days now... :-( Alessandro

4 3

[PATCH] RFC: Add netlink KRT dump filters on Linux
by Tomas Hlavacek 18 Jan '22

18 Jan '22

Add netlink KRT dump filter on Linux to avoid PMTU cache records from FNHE table dump along with KRT. Linux Kernel added FNHE table dump to the netlink API in patch https://patchwork.ozlabs.org/project/netdev/patch/8d3b68cd37fb5fddc470904cd… The filter mitigates the risk of receiving unknown and potentially large number of FNHE records that would block BIRD I/O in each sync. There is a known issue caused by the GRE tunnels on Linux that seems to be creating one FNHE record for each destination IP address that is routed through the tunnel, even when the PMTU equals to GRE interface MTU (tested with kernel 5.5 - 5.16-rc7). --- sysdep/linux/netlink.c | 44 +++++++++++++++++++++++++++++++++++++++--- 1 file changed, 41 insertions(+), 3 deletions(-) diff --git a/sysdep/linux/netlink.c b/sysdep/linux/netlink.c index f85bcf35..79414122 100644 --- a/sysdep/linux/netlink.c +++ b/sysdep/linux/netlink.c @@ -128,7 +128,7 @@ struct nl_sock uint last_size; }; -#define NL_RX_SIZE 8192 +#define NL_RX_SIZE 32768 #define NL_OP_DELETE 0 #define NL_OP_ADD (NLM_F_CREATE|NLM_F_EXCL) @@ -143,11 +143,18 @@ static struct nl_sock nl_req = {.fd = -1}; /* Netlink socket for requests */ static void nl_open_sock(struct nl_sock *nl) { + int sndbuf = 32768; + int rcvbuf = 1024*1024; + if (nl->fd < 0) { - nl->fd = socket(PF_NETLINK, SOCK_RAW, NETLINK_ROUTE); + nl->fd = socket(PF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, NETLINK_ROUTE); if (nl->fd < 0) die("Unable to open rtnetlink socket: %m"); + + setsockopt(nl->fd, SOL_SOCKET, SO_SNDBUF, &sndbuf, sizeof(sndbuf)); + setsockopt(nl->fd, SOL_SOCKET, SO_RCVBUF, &rcvbuf, sizeof(rcvbuf)); + nl->seq = (u32) (current_time() TO_S); /* Or perhaps random_u32() ? */ nl->rx_buffer = xmalloc(NL_RX_SIZE); nl->last_hdr = NULL; @@ -155,6 +162,12 @@ nl_open_sock(struct nl_sock *nl) } } +static void +nl_set_strict_dump(struct nl_sock *nl, int strict) +{ + setsockopt(nl->fd, SOL_NETLINK, NETLINK_GET_STRICT_CHK, &strict, sizeof(strict)); +} + static void nl_open(void) { @@ -192,6 +205,29 @@ nl_request_dump(int af, int cmd) nl_send(&nl_scan, &req.nh); } +static void +nl_request_dump_rt(int af, int cmd) +{ + struct { + struct nlmsghdr nh; + struct rtmsg rtm; + char buf[128]; + } req = { + .nh.nlmsg_type = cmd, + .nh.nlmsg_len = NLMSG_LENGTH(sizeof(struct rtmsg)), + .nh.nlmsg_flags = NLM_F_REQUEST | NLM_F_DUMP, + .nh.nlmsg_seq = ++(nl_scan.seq), + .nh.nlmsg_pid = 0, + .rtm.rtm_protocol = RTPROT_UNSPEC, + .rtm.rtm_family = af + /* .rtm.rtm_flags is defaults to zero, hence RTM_F_CLONED is not set */ + }; + + send(nl_scan.fd, &req, sizeof(req), 0); + nl_scan.last_hdr = NULL; +} + + static struct nlmsghdr * nl_get_reply(struct nl_sock *nl) { @@ -1864,13 +1900,15 @@ krt_do_scan(struct krt_proto *p UNUSED) /* CONFIG_ALL_TABLES_AT_ONCE => p is NUL struct nl_parse_state s; nl_parse_begin(&s, 1); - nl_request_dump(AF_UNSPEC, RTM_GETROUTE); + nl_set_strict_dump(&nl_scan, 1); + nl_request_dump_rt(AF_UNSPEC, RTM_GETROUTE); while (h = nl_get_scan()) if (h->nlmsg_type == RTM_NEWROUTE || h->nlmsg_type == RTM_DELROUTE) nl_parse_route(&s, h); else log(L_DEBUG "nl_scan_fire: Unknown packet received (type=%d)", h->nlmsg_type); nl_parse_end(&s); + nl_set_strict_dump(&nl_scan, 0); } /* -- 2.25.1

2 2

Traffic Engineering (TE) Extensions to OSPF Version 2 RFC3630 Support
by Senthil Kumar Nagappan 13 Jan '22

13 Jan '22

Hi, Checked the documents, Looks like support for "Traffic Engineering (TE) Extensions to OSPF Version 2 RFC3630" is not there yet in bird routing. Can you please confirm? Any plans to support it? Thanks and Regards,Senthil

1 0

Bird as BGP flow-spec announcement filter
by Blažej Krajňák 10 Jan '22

10 Jan '22

Hello, is it possible to do BGP Flowspec announcement filtering with Bird? For ex. I want to provide capability to my customer to annouce to my network BGP Flowspec rules. However, I must check, if announced rule have dst from customers IP range. Bird should act as route reflector only. Thanks, Blažej

4 4