Races in birdc causing daemon to crash on low spec routers

2 Jun 2026

      Hello all,

I'm hitting crash in BIRD 3.3.0 on several routers.

The machines are running
- NixOS x86_64-linux 7.0.10 
- 2-5 BGP IPv4/IPv6 peers per affected host
- prometheus-bird-exporter 1.4.5, scraping every 30s

prometheus-bird-exporter runs `show protocols all` over
/run/bird/bird.ctl every ~30s, and on hosts with active BGP churn that
crashes bird inside bgp_show_proto_info(). systemd restarts it, the next
scrape kills it again, and NRestarts climbs ~120/hr...

toompea and timah took 14 and 13 cores in their last ~20 minutes of
uptime, highline took 4 over a slower span. The other seven routers
(butte, cradle, baldy, kongo, lantau, roraima, rysy) run the same build
and exporter with no cores (most likely because these machines are only
getting default routes). The crash needs concurrent BGP state changes.

For example:

```console
$ colmena exec --verbose --on @router -- 'sudo coredumpctl list || true'
   timah |
highline |
   kongo |
 toompea |
   butte |
  cradle |
   baldy |
    rysy |
  lantau |
 roraima |
    rysy | No coredumps found.
    rysy | Succeeded
 toompea | TIME                            PID UID GID SIG     COREFILE EXE                                                               SIZE
 toompea | Thu 2026-05-28 15:41:25 UTC 1185401 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  98.9M
 toompea | Thu 2026-05-28 15:41:53 UTC 1265107 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.3M
 toompea | Thu 2026-05-28 15:42:22 UTC 1265317 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.9M
 toompea | Thu 2026-05-28 15:42:54 UTC 1265529 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 106.1M
 toompea | Thu 2026-05-28 15:43:23 UTC 1265740 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 101.8M
 toompea | Thu 2026-05-28 15:43:54 UTC 1265952 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 105.9M
 toompea | Thu 2026-05-28 15:44:22 UTC 1266165 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  90.2M
 toompea | Thu 2026-05-28 15:44:53 UTC 1266380 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.6M
 toompea | Thu 2026-05-28 19:45:24 UTC 1266590 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 102.9M
 toompea | Thu 2026-05-28 19:45:53 UTC 1355754 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  88.6M
 toompea | Thu 2026-05-28 19:46:22 UTC 1355959 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  87.4M
 toompea | Thu 2026-05-28 19:46:53 UTC 1356167 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 100.8M
 toompea | Thu 2026-05-28 19:47:22 UTC 1356371 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  88.8M
 toompea | Thu 2026-05-28 19:47:54 UTC 1356389 993 991 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 104.8M
 toompea | Succeeded
   butte | No coredumps found.
   butte | Succeeded
   timah | TIME                            PID UID GID SIG     COREFILE EXE                                                               SIZE
   timah | Thu 2026-05-28 15:36:51 UTC 1268745 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  96.9M
   timah | Thu 2026-05-28 15:37:17 UTC 1268960 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  98.6M
   timah | Thu 2026-05-28 15:37:52 UTC 1268974 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  95.9M
   timah | Thu 2026-05-28 15:38:18 UTC 1269382 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  89.2M
   timah | Thu 2026-05-28 15:38:53 UTC 1269397 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  95.4M
   timah | Thu 2026-05-28 19:39:27 UTC 1269828 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird 131.6M
   timah | Thu 2026-05-28 19:39:48 UTC 1359330 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  76.2M
   timah | Thu 2026-05-28 19:40:20 UTC 1359346 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  99.8M
   timah | Thu 2026-05-28 19:40:49 UTC 1359574 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  94.1M
   timah | Thu 2026-05-28 19:41:20 UTC 1359832 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  98.9M
   timah | Thu 2026-05-28 19:41:51 UTC 1360097 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  93.9M
   timah | Thu 2026-05-28 19:42:22 UTC 1360320 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  88.9M
   timah | Thu 2026-05-28 19:42:48 UTC 1360526 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  78.8M
   timah | Succeeded
highline | TIME                            PID UID GID SIG     COREFILE EXE                                                               SIZE
highline | Thu 2026-05-28 11:34:31 UTC 1203740 999 999 SIGSEGV present  /nix/store/afhrc47kar31rwmvbj0qrymm6xcpglvk-bird-3.3.0/bin/bird  99.4M
highline | Thu 2026-05-28 11:37:02 UTC 1204447 999 999 SIGSEGV present  /nix/store/afhrc47kar31rwmvbj0qrymm6xcpglvk-bird-3.3.0/bin/bird  81.9M
highline | Thu 2026-05-28 11:40:00 UTC 1205349 999 999 SIGSEGV present  /nix/store/afhrc47kar31rwmvbj0qrymm6xcpglvk-bird-3.3.0/bin/bird 108.9M
highline | Thu 2026-05-28 12:15:56 UTC 1220240 999 999 SIGSEGV present  /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird  80.3M
highline | Succeeded
   baldy | No coredumps found.
   baldy | Succeeded
   kongo | No coredumps found.
   kongo | Succeeded
 roraima | No coredumps found.
 roraima | Succeeded
  lantau | No coredumps found.
  lantau | Succeeded
  cradle | No coredumps found.
  cradle | Succeeded
         | All done!
```

```console
$ sudo coredumpctl info
           PID: 1356389 (bird)
           UID: 993 (bird)
           GID: 991 (bird)
        Signal: 11 (SEGV)
     Timestamp: Thu 2026-05-28 19:47:42 UTC (1h 37min ago)
  Command Line: /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird -c /etc/bird/bird.conf
    Executable: /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird
 Control Group: /system.slice/bird.service
          Unit: bird.service
         Slice: system.slice
       Boot ID: fd4c001e44dc44e1a05cbc5d60440ecb
    Machine ID: 77293224076f4ff7845c2358cb35a4c0
      Hostname: toompea
       Storage: /var/lib/systemd/coredump/core.bird.993.fd4c001e44dc44e1a05cbc5d60440ecb.1356389.1779997662000000.zst (present)
  Size on Disk: 104.8M
       Message: Process 1356389 (bird) of user 993 dumped core.

                Module /nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird without build-id.
                Module libz.so.1 without build-id.
                Module libssh.so.4 without build-id.
                Stack trace of thread 1356389:
                #0  0x000059d538758d08 bgp_show_proto_info.lto_priv.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0xdcd08)
                #1  0x000059d538706fb2 proto_cmd_show (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x8afb2)
                #2  0x000059d5387077c1 proto_apply_cmd.isra.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x8b7c1)
                #3  0x000059d5386a2de8 cf_parse.isra.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x26de8)
                #4  0x000059d5386ac462 cli_parse (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x30462)
                #5  0x000059d5386f96e5 cli_command (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x7d6e5)
                #6  0x000059d5386f9980 cli_event.lto_priv.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x7d980)
                #7  0x000059d5386dff35 ev_run_list_limited (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x63f35)
                #8  0x000059d538693c8c main (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x17c8c)
                #9  0x00007ae37602b285 __libc_start_call_main (libc.so.6 + 0x2b285)
                #10 0x00007ae37602b338 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x2b338)
                #11 0x000059d538694885 _start (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x18885)

                Stack trace of thread 1356391:
                #0  0x00007ae3760a6922 __syscall_cancel_arch (libc.so.6 + 0xa6922)
                #1  0x00007ae37609a00c __internal_syscall_cancel (libc.so.6 + 0x9a00c)
                #2  0x00007ae37609a084 __syscall_cancel (libc.so.6 + 0x9a084)
                #3  0x00007ae37611778e __poll (libc.so.6 + 0x11778e)
                #4  0x000059d5387b3bfe bird_thread_main.lto_priv.0 (/nix/store/230z9nyndgbn265mqlbyvf20z5wdciwy-bird-3.3.0/bin/bird + 0x137bfe)
                #5  0x00007ae37609dd53 start_thread (libc.so.6 + 0x9dd53)
                #6  0x00007ae37612563c __clone3 (libc.so.6 + 0x12563c)
                ELF object binary architecture: AMD x86-64
```

I temporarily worked around this by removing the prefix count showed
when invoking `birdc show proto all`. But I’m not sure if this type of patches
are acceptable for upstreaming (please see reasoning in the commit
message in the patch):

https://raw.githubusercontent.com/stepbrobd/inc/refs/heads/master/pkgs/bird3...

Best,
Yifei

Races in birdc causing daemon to crash on low spec routers

Yifei Sun