<html aria-label="message body"><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div>Hello all,</div><div><br></div><div><br></div><div>I'm hitting crash in BIRD 3.3.0 on several routers.</div><div><br></div><div>The machines are running</div><div>- NixOS x86_64-linux 7.0.10 </div><div>- 2-5 BGP IPv4/IPv6 peers per affected host</div><div>- prometheus-bird-exporter 1.4.5, scraping every 30s</div><div><br></div><div>prometheus-bird-exporter runs `show protocols all` over</div><div>/run/bird/bird.ctl every ~30s, and on hosts with active BGP churn that</div><div>crashes bird inside bgp_show_proto_info(). systemd restarts it, the next</div><div>scrape kills it again, and NRestarts climbs ~120/hr...</div><div><br></div><div>toompea and timah took 14 and 13 cores in their last ~20 minutes of</div><div>uptime, highline took 4 over a slower span. The other seven routers</div><div>(butte, cradle, baldy, kongo, lantau, roraima, rysy) run the same build</div><div>and exporter with no cores (most likely because these machines are only</div><div>getting default routes). The crash needs concurrent BGP state changes.</div><div><br></div><div>For example:</div><div><br></div><div>```console</div><div>$ colmena exec --verbose --on @router -- 'sudo coredumpctl list || true'</div><div> timah |</div><div>highline |</div><div> kongo |</div><div> toompea |</div><div> butte |</div><div> cradle |</div><div> baldy |</div><div> rysy |</div><div> lantau |</div><div> roraima |</div><div> rysy | No coredumps found.</div><div> rysy | Succeeded</div><div> toompea | TIME PID UID GID SIG COREFILE EXE SIZE</div><div> toompea | Thu 2026-05-28 15:41:25 UTC 1185401 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 98.9M</div><div> toompea | Thu 2026-05-28 15:41:53 UTC 1265107 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.3M</div><div> toompea | Thu 2026-05-28 15:42:22 UTC 1265317 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.9M</div><div> toompea | Thu 2026-05-28 15:42:54 UTC 1265529 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 106.1M</div><div> toompea | Thu 2026-05-28 15:43:23 UTC 1265740 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 101.8M</div><div> toompea | Thu 2026-05-28 15:43:54 UTC 1265952 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 105.9M</div><div> toompea | Thu 2026-05-28 15:44:22 UTC 1266165 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 90.2M</div><div> toompea | Thu 2026-05-28 15:44:53 UTC 1266380 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.6M</div><div> toompea | Thu 2026-05-28 19:45:24 UTC 1266590 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 102.9M</div><div> toompea | Thu 2026-05-28 19:45:53 UTC 1355754 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 88.6M</div><div> toompea | Thu 2026-05-28 19:46:22 UTC 1355959 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 87.4M</div><div> toompea | Thu 2026-05-28 19:46:53 UTC 1356167 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.8M</div><div> toompea | Thu 2026-05-28 19:47:22 UTC 1356371 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 88.8M</div><div> toompea | Thu 2026-05-28 19:47:54 UTC 1356389 993 991 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 104.8M</div><div> toompea | Succeeded</div><div> butte | No coredumps found.</div><div> butte | Succeeded</div><div> timah | TIME PID UID GID SIG COREFILE EXE SIZE</div><div> timah | Thu 2026-05-28 15:36:51 UTC 1268745 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 96.9M</div><div> timah | Thu 2026-05-28 15:37:17 UTC 1268960 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 98.6M</div><div> timah | Thu 2026-05-28 15:37:52 UTC 1268974 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 95.9M</div><div> timah | Thu 2026-05-28 15:38:18 UTC 1269382 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 89.2M</div><div> timah | Thu 2026-05-28 15:38:53 UTC 1269397 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 95.4M</div><div> timah | Thu 2026-05-28 19:39:27 UTC 1269828 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 131.6M</div><div> timah | Thu 2026-05-28 19:39:48 UTC 1359330 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 76.2M</div><div> timah | Thu 2026-05-28 19:40:20 UTC 1359346 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 99.8M</div><div> timah | Thu 2026-05-28 19:40:49 UTC 1359574 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 94.1M</div><div> timah | Thu 2026-05-28 19:41:20 UTC 1359832 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 98.9M</div><div> timah | Thu 2026-05-28 19:41:51 UTC 1360097 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 93.9M</div><div> timah | Thu 2026-05-28 19:42:22 UTC 1360320 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 88.9M</div><div> timah | Thu 2026-05-28 19:42:48 UTC 1360526 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 78.8M</div><div> timah | Succeeded</div><div>highline | TIME PID UID GID SIG COREFILE EXE SIZE</div><div>highline | Thu 2026-05-28 11:34:31 UTC 1203740 999 999 SIGSEGV present /nix/store/afhrc47kar31rwmvbj0qrymm6xcpglvk-bird-3.3.0/bin/bird 99.4M</div><div>highline | Thu 2026-05-28 11:37:02 UTC 1204447 999 999 SIGSEGV present /nix/store/afhrc47kar31rwmvbj0qrymm6xcpglvk-bird-3.3.0/bin/bird 81.9M</div><div>highline | Thu 2026-05-28 11:40:00 UTC 1205349 999 999 SIGSEGV present /nix/store/afhrc47kar31rwmvbj0qrymm6xcpglvk-bird-3.3.0/bin/bird 108.9M</div><div>highline | Thu 2026-05-28 12:15:56 UTC 1220240 999 999 SIGSEGV present /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 80.3M</div><div>highline | Succeeded</div><div> baldy | No coredumps found.</div><div> baldy | Succeeded</div><div> kongo | No coredumps found.</div><div> kongo | Succeeded</div><div> roraima | No coredumps found.</div><div> roraima | Succeeded</div><div> lantau | No coredumps found.</div><div> lantau | Succeeded</div><div> cradle | No coredumps found.</div><div> cradle | Succeeded</div><div> | All done!</div><div>```</div><div><br></div><div>```console</div><div>$ sudo coredumpctl info</div><div> PID: 1356389 (bird)</div><div> UID: 993 (bird)</div><div> GID: 991 (bird)</div><div> Signal: 11 (SEGV)</div><div> Timestamp: Thu 2026-05-28 19:47:42 UTC (1h 37min ago)</div><div> Command Line: /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird -c /etc/bird/bird.conf</div><div> Executable: /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird</div><div> Control Group: /system.slice/bird.service</div><div> Unit: bird.service</div><div> Slice: system.slice</div><div> Boot ID: fd4c001e44dc44e1a05cbc5d60440ecb</div><div> Machine ID: 77293224076f4ff7845c2358cb35a4c0</div><div> Hostname: toompea</div><div> Storage: /var/lib/systemd/coredump/core.bird.993.fd4c001e44dc44e1a05cbc5d60440ecb.1356389.1779997662000000.zst (present)</div><div> Size on Disk: 104.8M</div><div> Message: Process 1356389 (bird) of user 993 dumped core.</div><div><br></div><div> Module /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird without build-id.</div><div> Module libz.so.1 without build-id.</div><div> Module libssh.so.4 without build-id.</div><div> Stack trace of thread 1356389:</div><div> #0 0x000059d538758d08 bgp_show_proto_info.lto_priv.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0xdcd08)</div><div> #1 0x000059d538706fb2 proto_cmd_show (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x8afb2)</div><div> #2 0x000059d5387077c1 proto_apply_cmd.isra.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x8b7c1)</div><div> #3 0x000059d5386a2de8 cf_parse.isra.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x26de8)</div><div> #4 0x000059d5386ac462 cli_parse (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x30462)</div><div> #5 0x000059d5386f96e5 cli_command (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x7d6e5)</div><div> #6 0x000059d5386f9980 cli_event.lto_priv.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x7d980)</div><div> #7 0x000059d5386dff35 ev_run_list_limited (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x63f35)</div><div> #8 0x000059d538693c8c main (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x17c8c)</div><div> #9 0x00007ae37602b285 __libc_start_call_main (libc.so.6 + 0x2b285)</div><div> #10 0x00007ae37602b338 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2b338)</div><div> #11 0x000059d538694885 _start (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x18885)</div><div><br></div><div> Stack trace of thread 1356391:</div><div> #0 0x00007ae3760a6922 __syscall_cancel_arch (libc.so.6 + 0xa6922)</div><div> #1 0x00007ae37609a00c __internal_syscall_cancel (libc.so.6 + 0x9a00c)</div><div> #2 0x00007ae37609a084 __syscall_cancel (libc.so.6 + 0x9a084)</div><div> #3 0x00007ae37611778e __poll (libc.so.6 + 0x11778e)</div><div> #4 0x000059d5387b3bfe bird_thread_main.lto_priv.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x137bfe)</div><div> #5 0x00007ae37609dd53 start_thread (libc.so.6 + 0x9dd53)</div><div> #6 0x00007ae37612563c __clone3 (libc.so.6 + 0x12563c)</div><div> ELF object binary architecture: AMD x86-64</div><div>```</div><div><br></div><div>I temporarily worked around this by removing the prefix count showed</div><div>when invoking `birdc show proto all`. But I’m not sure if this type of patches</div><div>are acceptable for upstreaming (please see reasoning in the commit</div><div>message in the patch):</div><div><br></div><div><a href="https://raw.githubusercontent.com/stepbrobd/inc/refs/heads/master/pkgs/bird3/fix.patch">https://raw.githubusercontent.com/stepbrobd/inc/refs/heads/master/pkgs/bird3/fix.patch</a></div><div><br></div><div><br></div><div>Best,</div><div>Yifei</div></body></html>