Good day, While setting up (using Claude’ assistance) an internal PrivateASN “client” setup, we hit some snags on kernel protocols, but also some crashes. See the report from Claude about the crashes. Any better formats to report the analysis on this group? ## Summary of report. A kernel protocol whose krt_sys_start() fails ("Kernel table N already registered by X") stays in PS_START with p->scan_timer == NULL. Any later shutdown of that protocol (reconfiguration that restarts or removes it, `disable`, SIGTERM, graceful restart) runs krt_shutdown() -> krt_scan_timer_stop() -> tm_stop(NULL) and the daemon segfaults. Reproducible unprivileged with a 12-line config on 3.3.3 and 3.4.0. It took down two production routers on 2026-10-14 (once on `configure`, once on SIGTERM, after the SIGTERM path had already withdrawn all routes). ## Versions - bird3 3.3.3-cznic.1~trixie and 3.4.0-cznic.1~trixie (pkg.labs.nic.cz), Linux 7.0, Devuan. - Code involved is identical in v3.3.3, v3.4.0 and thread-next dcf094d3f (2026-10-12). - master (2.20.0): krt_shutdown() has the same order (krt_scan_timer_stop() before the PS_START check) and krt_start() the same early return; tm_stop() in lib/timer.c also reads t->expires first. Not tested there. ## Reproducer a.conf: log stderr all; router id 192.0.2.1; ipv4 table t2; protocol device { } protocol kernel k1 { kernel table 254; ipv4 { table master4; import none; export none; }; } protocol kernel k2 { kernel table 254; metric 10; ipv4 { table t2; import none; export all; }; } b.conf = a.conf with k2 changed to `kernel table 101;` (no metric). cp a.conf run.conf unshare -rn sleep 60 & NSP=$! nsenter -t $NSP -U -n --preserve-credentials --wd=$PWD ip link set lo up nsenter -t $NSP -U -n --preserve-credentials --wd=$PWD bird -f -c run.conf -s ctl -P pid & nsenter -t $NSP -U -n --preserve-credentials --wd=$PWD birdc -s ctl show protocols cp b.conf run.conf nsenter -t $NSP -U -n --preserve-credentials --wd=$PWD birdc -s ctl configure `bird -p` accepts a.conf. At start: <ERR> k2: Kernel table 254 already registered by k1 `show protocols` shows k2 in state `start`. The `configure` logs <INFO> Reconfiguring <INFO> Restarting protocol k2 and birdc reports "Connection closed by server"; bird is gone (SIGSEGV). `birdc -s ctl disable k2` instead of the configure gives the same result. | version | configure (table 254 -> 101) | disable k2 | |---|---|---| | 3.3.3-cznic.1 | SIGSEGV | SIGSEGV | | 3.4.0-cznic.1 | SIGSEGV | SIGSEGV | ## Observed in production (3.3.3, two routers, 2026-10-14) Router A, `birdc configure` moving the stuck protocol to its own kernel table: 22:52:13 bird: Reconfiguring 22:52:13 bird: Restarting protocol kuls4 22:52:13 kernel: bird[1814]: segfault at 30 ip 0000555b82c5d994 sp 00007ffdf1b62828 error 4 in bird[72994,555b82c00000+116000] Router B, SIGTERM (init script restart) with the same stuck protocol still present: 23:37:49 bird: Shutting down 23:38:14 kernel: bird[1648]: segfault at 30 ip 0000559ad06c7994 sp 00007ffd3ea32008 error 4 in bird[72994,559ad066a000+116000] Same text offset 0x72994 both times. With bird3-dbgsym 3.3.3-cznic.1, addr2line resolves it to tm_stop() lib/timer.c:133 (`if (!t->expires)`). Consistent with the fault address: struct timer = resource (32 bytes) + hook (8) + data (8), so `expires` is at offset 0x30 of a NULL timer. Both faults are on the main-thread stack (sp 0x7ffd...). On router B the 25 s between "Shutting down" and the fault were the regular shutdown withdrawing ~1M kernel routes and ~100 BGP sessions on the main loop; the crash came when the stuck protocol's turn came. ## Analysis (3.3.3 line numbers) 1. sysdep/linux/netlink.c:2798 krt_sys_start(): HASH_FIND(nl_table_map, af, table id) finds the other kernel protocol, logs "Kernel table %u already registered by %s" and returns 0. `metric` is not part of the key. 2. sysdep/unix/krt.c:946 krt_start(): line 969 `if (!krt_sys_start(p)) { rem_node(&p->krt_node); return PS_START; }` runs before line 975 krt_scan_timer_start(p). p->scan_timer is only ever created inside krt_scan_timer_start() (tm_new_init), so it stays NULL. The protocol remains in PS_START; nothing else happens, it ran 34 min like that. 3. sysdep/unix/krt.c:984 krt_shutdown(): line 988 krt_scan_timer_stop(p) is the first statement. krt.c:690 krt_scan_timer_stop() calls tm_stop(p->scan_timer) with NULL; lib/timer.c:133 reads t->expires. The guard `if (p->p.proto_state == PS_START) return PS_FLUSH;` is at line 990, after the call. 4. On hosts where strict netlink dumps are unavailable (krt_use_shared_scan(), krt_scan_all_tables = 1) the same statement takes the other branch: krt_scan_all_timer_stop() with ASSERT(krt_scan_all_count > 0) and a decrement that this protocol never matched with an increment. Not tested. Every shutdown path reaches krt_shutdown(): protocol restart or removal on reconfigure, `disable`, SIGTERM, and `graceful restart` (PDC_CMD_GR_DOWN is only consulted after the timer stop). Once a kernel protocol is in this state there is no way to end it without the fault. ## Tested change Moving the PS_START check above the timer stop (krt-shutdown.patch, against the 3.3.3 release tarball): --- sysdep/unix/krt.c.orig +++ sysdep/unix/krt.c @@ -985,11 +985,11 @@ { struct krt_proto *p = (struct krt_proto *) P; - krt_scan_timer_stop(p); - if (p->p.proto_state == PS_START) return PS_FLUSH; + krt_scan_timer_stop(p); + /* FIXME we should flush routes even when persist during reconfiguration */ if ((p->sync_state != KPS_INIT) && !KRT_CF->persist && (P->down_code != PDC_CMD_GR_DOWN)) { With it, the reproducer survives both cases: after `configure` k2 comes up on table 101; after `disable` k2 is down and the daemon keeps running. Two remarks on how we got there: - There is no config-time rejection of two kernel protocols on the same (af, kernel table), unlike "Kernel syncer (%s) already attached to table %s" for BIRD tables. `bird -p` passes such a config. - doc/bird.sgml (kernel `metric`): "routes with different metrics do not clash with each other, therefore using dedicated metric value is a reliable way to avoid overwriting routes from other sources". We read that as two kernel protocols on one kernel table with different metrics being supported. ## Second, separate crash on router A (not reproduced) After crash 1, router A was started fresh (22:53:54) with the conflict removed (kuls4/6 on kernel table 101, all four kernel protocols `up`). 78 s later, with no CLI activity, it segfaulted again: 22:55:12 kernel: bird[13330]: segfault at 30 ip 0000562a8033ea23 sp 00007fc85c2462c0 error 4 in bird[6ea23,562a802e5000+116000] Text offset 0x6ea23 -> mb_alloc() lib/resource.c:452, `ASSERT_DIE(DG_IS_LOCKED(p->domain))`. struct pool = resource (32 bytes) + TLIST_LIST(resource) inside (16 bytes), so `domain` is at offset 0x30: the pool pointer passed to mb_alloc() was NULL. (A NULL p->domain would fault at 0x28, the `order` field behind the 40-byte mutex in struct domain_generic.) Unlike crash 1, sp is a thread stack (0x7fc8...), i.e. a protocol or table loop, not the main loop where the kernel protocols run. State at the time: - Graceful restart recovery in progress (`graceful restart wait 120`, 112 channels waiting). Two configured BGP peers never connect (Active), so the recovery lock holds until the 120 s timer; the kernel protocols were still in KPS_LEARNING with exports not started. - ~100 BGP sessions, two with full tables still being received (last log line 1 s before the fault was an AS_SET withdrawal from one of them). - 4 RPKI sessions, Established 0-3 s after start, with `import table yes` and ROA autoreload on all BGP channels, so channel_roa_changed() was running on the BGP loops during recovery (that function allocates from c->proto->pool). - 4 kernel protocols on kernel tables 254, 100, 101; 3 pipes; 1 BMP. I could not find what called mb_alloc() with a NULL pool: no core dump (the daemon is not dumpable on that host), no backtrace. nest/proto.c sets p->pool only in proto_start() (line 1690) and clears it after PS_DOWN_XX (line 2884). The June 2026 ROA-autoreload / reload-during-GR fixes (24194edaa, 7a7ebdcc0, 0f5e8e988, b18793719 on stable-v3.3) are in 3.3.3. The only correlation I have: this configuration (the extra kernel protocol on table 101, its BIRD table and pipe) was running for the first time. The same configuration without that kernel protocol (table, pipe and BGP sessions kept) has been stable on both routers since, through several restarts and reconfigurations. The 12-line reproducer above does not cover this case (no BGP, RPKI or graceful restart). I can arrange a core dump for a future occurrence if useful. --- Hendrik Visage Director/Owner HeViS.Co Systems t/a Envisage Cloud Solutions hvisage@hevis.co.za GSM/SMS/Signal: +27-84-612-5345 InstantMessenger: https://t.me/hvisage