Hi Pavel, thanks for letting us know about the performance issue and your related findings. You can send us the patch files so we can take a look, however just from the patch descriptions alone, we would probably much rather fix the problems in a different way that is less fragile and more easily maintainable long term. We were already planning to move garbage collection to its own worker thread, and migrate main loop to new IO loop implementation. Both of these changes are in our view better fixes than `patch01` and `patch02` to your problem regarding CLI response times after restart. We will look into these changes and hopefully squeeze them in before the next LTS release. Instead of the fragile changes in the `patch03` regarding kernel protocol, we would rather recommend changing your configuration, if possible, to have less then 7 kernel protocols. We do not know what is the use for so many kernel protocols, but we sense that proper BIRD API might help reduce the number. API is still in the works right now, but it is going to be released soon. API route manipulation is expected to be available to public sometime in the spring of next year. Also we would like to know your configuration setup, so we can learn about the case where the performance is affected this way. Thanks again and happy routing, David David Petera (he/him) | BIRD Tech Support | CZ.NIC, z.s.p.o. On 10/14/26 14:22, Pavel Trefný via Bird-users wrote:
Hi BIRD developers and users,
We would like to share a performance problem we encountered with BIRD 3.4.0 on Linux, our investigation, and an experimental patch series that substantially improved behaviour on our test server. We can provide the patches for review.
The investigation, patches and test runs were carried out with assistance from OpenAI Codex, an AI coding agent, with us directing the experiments on our server. The measurements below come from the actual test runs.
### Symptoms and environment
After moving from 3.2.4 to 3.4.0, we observed multi-second delays in basic `birdc` commands and sustained utilisation of approximately one full CPU core by the main thread. With 3.2.4, the CLI was much more responsive after convergence. This was an operational comparison, not a controlled benchmark of the two unmodified releases; startup kernel scans could also delay 3.2.4.
We had also observed similar performance symptoms with the 3.3 series. The profiles, patches and quantitative measurements below are for builds based on 3.4.0; we have not profiled or tested these patches on 3.3.
The server runs Linux on x86_64 with 16 logical CPUs and 12 BIRD worker threads. It has 158 BGP protocols, of which 143 were established in our baseline, and seven Kernel protocols. The Kernel channels reported approximately 3.79 million exports in aggregate. These exports span multiple tables and must not be interpreted as 3.79 million distinct prefixes.
CPU affinity alone did not solve the CLI delays: the CLI still depends on the main loop, including its synchronous work. Stack sampling showed time spent in `msl_cleanup()` and synchronous Linux kernel scans through `nl_get_reply()` / `nl_get_scan()` / `krt_do_scan()`. We therefore addressed both allocator work and main-loop latency.
### Experimental changes
The series applies to clean upstream 3.4.0, in this order:
1. **`0001-bounded-allocator-cleanup.patch`**
Make mslab cleanup resumable, with a budget of at most 256 page heads per callback across full and partial cleanup. Partition the private full-page list into 4096 buckets and use atomic dirty bits to select buckets affected by publication or frees. This avoids repeatedly walking the entire unchanged full-page population. Collect the published partial list only when a possible empty-page transition requires it.
The implementation retains the producer/consumer model and RCU grace periods before republishing partial pages. It handles frees behind a retained cursor and destruction with unfinished cleanup. Page headers and the public API remain unchanged; the extra metadata is approximately 33 KiB per mslab. The patch includes regression tests.
2. **`0002-main-loop-quota.patch`**
Reduce the existing cooperative main-loop operation quota from 2048 to 128, giving other main-loop work more frequent opportunities to run during route feed/export activity.
3. **`0003-async-kernel-and-cli.patch`**
Use a dedicated nonblocking netlink socket for Linux per-table route dumps. Collect and validate a complete dump before beginning refresh/prune, so an interrupted, truncated or otherwise invalid dump is discarded and retried instead of pruning against an incomplete snapshot. Receive at most eight datagrams per callback with a 10 ms time budget; apply/replay at most 64 messages per callback with the same time budget. Individual operations can still exceed the time budget.
Serialize scan jobs, defer initial export until the scan completes, buffer route notifications and replay them in FIFO order, and cancel jobs during protocol shutdown. Snapshot and notification buffers are capped at 256 MiB and 8 MiB respectively, with abort/resync handling.
Also remove an early `continue` in the main-loop wakeup-pipe path that skipped other ready sockets. Repeatedly rescheduled work could otherwise keep the CLI from being serviced despite its socket being ready.
The CLI remains in the main loop; we did not move it to a separate thread.
### What profiling changed in our approach
Our first local batching prototype introduced excessive partial-list recollection. We corrected that in an intermediate version, which we call V3. That amplification was a defect in our prototype, not a demonstrated defect in unmodified upstream 3.4.0. V3 nevertheless still consumed almost one main-thread CPU core: profiling then identified repeated full-page list walks as the dominant remaining cost. The bucket/dirty-bit approach above is our final experimental version, V4.
### Measurements on this server
The quantitative baseline below is **our already modified V3**, immediately before deploying V4, not stock 3.4.0. Both versions use the asynchronous scan and quota changes. Routing policy, scan intervals and CPU affinity were unchanged between them. This is a comparison of successive live runs on one server, not a replay of identical traffic.
| Version and observation window | Main-thread CPU | | --- | ---: | | V3, 60 seconds after convergence | 96.60% | | V4, first 60-second window after convergence | 23.40% | | V4, second 60-second window after convergence | 23.92% | | V4, longer window from 360 seconds after startup | 27.56% |
Here, 100% means one logical CPU. CPU usage comes from the main thread's `/proc/PID/task/PID/stat`, rather than percentages inferred from profiling. Full-page cleanup accounted for 64.46% of userspace instruction-pointer samples in the immediate V3 baseline and 8.51% in the first V4 profile. These sample shares are separate from the CPU utilisation figures.
During a 600-second V4 observation, all 120 CLI probes returned, with no 3-second timeouts. For the 73 probes from 360 seconds after startup, the median was 4.09 ms, p95 6.06 ms and maximum 1269.45 ms. In that later window, the whole process averaged 50.16% CPU and RSS was approximately 4.04–4.06 GiB. Export counters continued to advance. All named BGP states matched the V3 baseline and all seven Kernel protocols were up with exports READY at the final check.
### Validation and remaining limitations
We built with default `./configure`; `make -j2 bird check` passed. New allocator tests cover sparse frees among 60,000 objects, bounded cleanup work, slot reuse, payload/accounting, frees behind a cursor and unfinished destruction. The sparse-work regression fails on V3 and passes on V4. A four-thread test with cross-thread frees passed, including 20 repeated runs on V4.
Linux network-namespace tests passed for route ordering, cancellation, interface failure, interrupted dumps and a large case with 400,000 IPv4 plus 100,000 IPv6 routes. We checked route counts, export changes, preservation of existing routes on interrupted dumps, retry behaviour and clean shutdown. Applying the series to the official 3.4.0 archive reproduced all eight changed build-source files byte for byte.
This remains experimental and the final V4 observation is short. Occasional CLI delays around 1.27 seconds remain. Netlink write ACKs, interface/FDB scans, legacy scans, shutdown and lock waits can still block. We have not established long-term stability or verified every live forwarding route end to end. An earlier prototype trial also logged two "Export flag inconsistency" messages in Kernel channels; we have not established their cause or shown that these patches resolve them.
We would appreciate feedback on the allocator dirty-bucket design and RCU handling, and on the snapshot/notification ordering and cancellation approach for kernel scans. Is there an existing upstream approach we should follow? We can share the three patches, test fixtures and measurement details, or split the changes differently if that would make review easier.
Best regards, Pavel Trefny
PS (for Czech-speaking readers): My mu nerozumime, ale my mu verime :-)