Hi, # BIRD 3.3.2 — `Assertion '!new || rte_is_valid(new)' failed at nest/rt-table.c:1562` Apparent race between `rt_next_hop_update_net()` and `channel_notify_optimal_req()`. Addresses, ASNs and hostnames below are replaced with documentation ranges (RFC 5398, RFC 3849). Pointer values, offsets and line numbers are verbatim. ## Summary BIRD 3.3.2 aborts on an internal assertion roughly every 20–40 seconds on two independent route servers once recursive next-hop resolution is active and the table is under churn. ``` Assertion '!new || rte_is_valid(new)' failed at nest/rt-table.c:1562 ``` **84 and 81 aborts** in about 40 minutes. The two machines are separate VMs on separate hypervisors, same configuration shape, same peers. They began aborting **three seconds apart**, which pointed at a data/timing trigger arriving over BGP rather than anything host-local. The backtrace shows the assertion firing in the main loop while a worker thread is suspended mid-update inside `rt_next_hop_update_net()`. ## Environment ``` BIRD 3.3.2 (Debian package bird3 3.3.2-cznic.1~trixie, amd64) OS Debian GNU/Linux 13 (trixie) Kernel 6.12.107+deb13-cloud-amd64 Role IX-style route server ``` Ruled out: 16 GB RAM, ~1 GB in use, BIRD RSS 538–564 MB at abort. No kernel OOM records, `dmesg` clean. Configuration parses (`birdc configure check` → OK). ## Backtrace Symbolised with `bird3-dbgsym` 3.3.2-cznic.1~trixie. **Thread 1 — main loop, the abort:** ``` #3 bug (msg=msg@entry=0x... "Assertion '%s' failed at %s:%d") at sysdep/unix/log.c:446 #4 channel_notify_optimal_req (c=c@entry=0x562028c3a000, req=req@entry=0x562028c3a128) at nest/rt-table.c:1562 new = <optimized out> old = <optimized out> trte = <optimized out> u = 0x7f8308f4f2b0 #5 channel_notify_optimal (_channel=0x562028c3a000) at nest/rt-table.c:1591 c = 0x562028c3a000 #6 ev_run_list_limited (l=0x... <global_work_list>, limit=7, limit@entry=10) at lib/event.c:336 #7 io_loop () at sysdep/unix/io.c:2642 #8 main (argc=<optimized out>, argv=<optimized out>) at sysdep/unix/main.c:1114 ``` **Thread 2 — next-hop update worker, suspended mid-update:** ``` #5 birdloop_yield () at sysdep/unix/io-loop.c:2372 #6 synchronize_rcu () at ./lib/rcu.h:96 #7 rt_next_hop_update_net (tab=<optimized out>, ni=<optimized out>, n=<optimized out>) at nest/rt-table.c:4747 #8 rt_next_hop_update (_tab=<optimized out>) at nest/rt-table.c:4951 #9 ev_run_list_limited (l=l@entry=0x..., limit=4294967294, limit@entry=4294967295) at lib/event.c:336 #10 birdloop_run (_loop=0x...) at sysdep/unix/io-loop.c:2114 #11 ev_run_list_limited (...) at lib/event.c:336 #12 bird_thread_main (arg=0x...) at sysdep/unix/io-loop.c:1034 ``` **Thread 3 — idle in `poll()`**, `bird_thread_main` at `sysdep/unix/io-loop.c:1094`. ## What this looks like Thread 2 is inside `rt_next_hop_update_net()` and has yielded in `synchronize_rcu()` → `birdloop_yield()`, so the network it is updating is part-way through modification. Thread 1, in the main loop, runs `channel_notify_optimal()` for a channel and `channel_notify_optimal_req()` asserts that the route it has been handed is valid. We have not read the source to confirm the exact window, and the relevant locals are optimised out, so this is offered as where to look rather than as a diagnosis. ## Reproduction Consistent and quick once the conditions are present: 1. Route server with ~1.07M IPv4 and ~250k IPv6 in `master4`/`master6`, learned over two multihop eBGP sessions from edge routers. 2. Eight multihop eBGP customer sessions sourced from a loopback, whose next hops require **recursive resolution** against the local table. 3. A default route installed such that those next hops actually resolve. 4. Churn — a session flap, or `systemctl restart bird` to reload the full table. The abort then occurs within seconds to a couple of minutes, repeatedly. Log immediately before one abort, showing recursive resolution in bulk: ``` Next hop address 2001:db8:0:2::211 resolvable through recursive route for 2001:db8::/32 Next hop address 2001:db8:1000:700::11 resolvable through recursive route for 2001:db8::/32 Next hop address 2001:db8:1::1 resolvable through recursive route for 2001:db8:1::/48 Next hop address 2001:db8:1::1 resolvable through recursive route for 2001:db8:1::/48 Next hop address 2001:db8:1::2 resolvable through recursive route for 2001:db8:1::/48 Next hop address 2001:db8:1::2 resolvable through recursive route for 2001:db8:1::/48 Assertion '!new || rte_is_valid(new)' failed at nest/rt-table.c:1562 ``` Before another: ``` edge_a.ipv4: table prune after refresh end: rr 4096 set 5 valid 5 pruning 5 pruned 5 fabric_b: Connecting to 2001:db8:100:b00b::3 from local address 2001:db8:1000:701::1 fabric_b: Connection lost (Connection refused) fabric_b: Connect delayed by 5 seconds device1: Scanning interfaces Assertion '!new || rte_is_valid(new)' failed at nest/rt-table.c:1562 ``` ## The workaround, which also isolates the trigger Changing **one line** — the static default used as the recursive resolution target — is the difference between a stable process and one aborting every 20–40 seconds. Same box, same table, same peers. ``` # aborts, 84 times in ~40 minutes protocol static resolver6 { ipv6; route ::/0 via fe80::1%eth0; } protocol static resolver4 { ipv4; route 0.0.0.0/0 via "eth0"; } # stable, 30+ minutes and counting protocol static resolver6 { ipv6; route ::/0 blackhole; } protocol static resolver4 { ipv4; route 0.0.0.0/0 blackhole; } ``` With `blackhole` the default still exists in the table and is still exported to peers, but nothing resolves recursively through it — so `rt_next_hop_update_*` has no hostentries to process and Thread 2's code path is never entered. This was verified as a controlled test rather than assumed: the change was applied to **one** of the two route servers while the other was left alone. The changed box stopped aborting immediately (restart counter frozen, uptime growing past 30 minutes); the untouched box continued aborting every 20–40 seconds until the same change was applied to it. Re-applying the gateway form to the second box reproduced the abort on demand and produced the core above. ## Configuration shape ``` Protocols 12 BGP, 5 Static, 2 Pipe, 2 Kernel, 1 RPKI, 1 BFD, 1 Device Tables master4, master6, t_cust4, t_cust6, rpki4, rpki6, aspa1 Pipes master4 <=> t_cust4, master6 <=> t_cust6 Kernel export none, learn off (BIRD installs nothing in the FIB) RPKI RTR session with roa4, roa6 and aspa channels ``` BGP: two multihop eBGP to edge routers carrying a full table, two single-hop eBGP to switches, eight multihop eBGP customer sessions sourced from a loopback with recursive next-hop resolution. Import filters call `roa_check()` and `aspa_check_downstream()` / `aspa_check_upstream()`. ## Impact Both route servers of a redundant pair abort on the same input at the same time, and `Restart=on-abort` restarts each in a loop. Every restart drops all BGP sessions; the edge routers then re-push a full table, which does not complete before the next abort. Customer sessions cannot stay established. The redundancy provides no protection because both machines fail identically. ## Core dumps etc The compressed core (57–105 MB per dump) https://www.filemail.com/d/bpysuryqkeusczh Best regards Lars Strandos