Having recently installed using 3.3.x (soon to upgrade to 3.4) I also noticed these stalls, typically close after restarts/big-reconfigures, ~100 and ~110 BGP protocol sessions in my instances.

---

Hendrik Visage
Director/Owner
HeViS.Co Systems t/a Envisage Cloud Solutions
hvisage@hevis.co.za
GSM/SMS/Signal: +27-84-612-5345
InstantMessenger: https://t.me/hvisage

On 14 Oct 2026, at 14:22, Pavel Trefný via Bird-users wrote:

Hi BIRD developers and users,

We would like to share a performance problem we encountered with BIRD 3.4.0
on Linux, our investigation, and an experimental patch series that substantially
improved behaviour on our test server. We can provide the patches for review.

The investigation, patches and test runs were carried out with assistance
from OpenAI Codex, an AI coding agent, with us directing the experiments on
our server. The measurements below come from the actual test runs.

Symptoms and environment

After moving from 3.2.4 to 3.4.0, we observed multi-second delays in basic
birdc commands and sustained utilisation of approximately one full CPU core
by the main thread. With 3.2.4, the CLI was much more responsive after
convergence. This was an operational comparison, not a controlled benchmark
of the two unmodified releases; startup kernel scans could also delay 3.2.4.

We had also observed similar performance symptoms with the 3.3 series.
The profiles, patches and quantitative measurements below are for builds
based on 3.4.0; we have not profiled or tested these patches on 3.3.

The server runs Linux on x86_64 with 16 logical CPUs and 12 BIRD worker threads.
It has 158 BGP protocols, of which 143 were established in our baseline,
and seven Kernel protocols. The Kernel channels reported approximately
3.79 million exports in aggregate. These exports span multiple tables and
must not be interpreted as 3.79 million distinct prefixes.

CPU affinity alone did not solve the CLI delays: the CLI still depends on the
main loop, including its synchronous work. Stack sampling showed time spent
in msl_cleanup() and synchronous Linux kernel scans through
nl_get_reply() / nl_get_scan() / krt_do_scan(). We therefore addressed
both allocator work and main-loop latency.

Experimental changes

The series applies to clean upstream 3.4.0, in this order:

1. 0001-bounded-allocator-cleanup.patch

   Make mslab cleanup resumable, with a budget of at most 256 page heads per
   callback across full and partial cleanup. Partition the private full-page
   list into 4096 buckets and use atomic dirty bits to select buckets affected
   by publication or frees. This avoids repeatedly walking the entire
   unchanged full-page population. Collect the published partial list only
   when a possible empty-page transition requires it.

   The implementation retains the producer/consumer model and RCU grace
   periods before republishing partial pages. It handles frees behind a
   retained cursor and destruction with unfinished cleanup. Page headers and
   the public API remain unchanged; the extra metadata is approximately
   33 KiB per mslab. The patch includes regression tests.

2. 0002-main-loop-quota.patch

   Reduce the existing cooperative main-loop operation quota from 2048 to
   128, giving other main-loop work more frequent opportunities to run during
   route feed/export activity.

3. 0003-async-kernel-and-cli.patch

   Use a dedicated nonblocking netlink socket for Linux per-table route dumps.
   Collect and validate a complete dump before beginning refresh/prune, so an
   interrupted, truncated or otherwise invalid dump is discarded and retried
   instead of pruning against an incomplete snapshot. Receive at most eight
   datagrams per callback with a 10 ms time budget; apply/replay at most
   64 messages per callback with the same time budget. Individual operations
   can still exceed the time budget.

   Serialize scan jobs, defer initial export until the scan completes, buffer
   route notifications and replay them in FIFO order, and cancel jobs during
   protocol shutdown. Snapshot and notification buffers are capped at
   256 MiB and 8 MiB respectively, with abort/resync handling.

   Also remove an early continue in the main-loop wakeup-pipe path that
   skipped other ready sockets. Repeatedly rescheduled work could otherwise
   keep the CLI from being serviced despite its socket being ready.

The CLI remains in the main loop; we did not move it to a separate thread.

What profiling changed in our approach

Our first local batching prototype introduced excessive partial-list
recollection. We corrected that in an intermediate version, which we call
V3. That amplification was a defect in our prototype, not a demonstrated
defect in unmodified upstream 3.4.0. V3 nevertheless still consumed almost
one main-thread CPU core: profiling then identified repeated full-page list
walks as the dominant remaining cost. The bucket/dirty-bit approach above
is our final experimental version, V4.

Measurements on this server

The quantitative baseline below is our already modified V3, immediately
before deploying V4, not stock 3.4.0. Both versions use the asynchronous scan
and quota changes. Routing policy, scan intervals and CPU affinity were
unchanged between them. This is a comparison of successive live runs on one
server, not a replay of identical traffic.

Version and observation window	Main-thread CPU
V3, 60 seconds after convergence	96.60%
V4, first 60-second window after convergence	23.40%
V4, second 60-second window after convergence	23.92%
V4, longer window from 360 seconds after startup	27.56%

Here, 100% means one logical CPU. CPU usage comes from the main thread's
/proc/PID/task/PID/stat, rather than percentages inferred from profiling.
Full-page cleanup accounted for 64.46% of userspace instruction-pointer
samples in the immediate V3 baseline and 8.51% in the first V4 profile.
These sample shares are separate from the CPU utilisation figures.

During a 600-second V4 observation, all 120 CLI probes returned, with no
3-second timeouts. For the 73 probes from 360 seconds after startup, the
median was 4.09 ms, p95 6.06 ms and maximum 1269.45 ms. In that later window,
the whole process averaged 50.16% CPU and RSS was approximately 4.04–4.06 GiB.
Export counters continued to advance. All named BGP states matched the V3
baseline and all seven Kernel protocols were up with exports READY at the
final check.

Validation and remaining limitations

We built with default ./configure; make -j2 bird check passed. New allocator
tests cover sparse frees among 60,000 objects, bounded cleanup work, slot
reuse, payload/accounting, frees behind a cursor and unfinished destruction.
The sparse-work regression fails on V3 and passes on V4. A four-thread test
with cross-thread frees passed, including 20 repeated runs on V4.

Linux network-namespace tests passed for route ordering, cancellation,
interface failure, interrupted dumps and a large case with 400,000 IPv4
plus 100,000 IPv6 routes. We checked route counts, export changes, preservation
of existing routes on interrupted dumps, retry behaviour and clean shutdown.
Applying the series to the official 3.4.0 archive reproduced all eight changed
build-source files byte for byte.

This remains experimental and the final V4 observation is short. Occasional
CLI delays around 1.27 seconds remain. Netlink write ACKs, interface/FDB scans,
legacy scans, shutdown and lock waits can still block. We have not established
long-term stability or verified every live forwarding route end to end.
An earlier prototype trial also logged two "Export flag inconsistency"
messages in Kernel channels; we have not established their cause or shown
that these patches resolve them.

We would appreciate feedback on the allocator dirty-bucket design and RCU
handling, and on the snapshot/notification ordering and cancellation approach
for kernel scans. Is there an existing upstream approach we should follow?
We can share the three patches, test fixtures and measurement details, or
split the changes differently if that would make review easier.

Best regards,
Pavel Trefny

PS (for Czech-speaking readers): My mu nerozumime, ale my mu verime :-)