Hello Maria, Thank you, that is very helpful. Good to know an improvement is planned for 3.5.0 and that 3.1.x is not affected. Given what you wrote about other patches and race conditions, I won't send ours. Just two observations from a lab where I let Claude (LLM) experiment, in case they are of any use for your approach: - putting msl_free() inside an RCU critical section made the synchronize_rcu() callers sleep in birdloop_yield() noticeably more often, and throughput dropped; - freeing empty heads right away made the page allocator hand them back to the kernel; convergence then paid about 4x more minor page faults. We would be glad to test 3.5.0 pre-release builds on our route server when they are available. Best regards, Marcelo Balbinot Em qui., 12 de nov. de 2026 às 05:41, Maria Matejka <maria.matejka@nic.cz> escreveu:
Hello, we know about this problem and we are discussing in the core team what to do with that as it is not easy to fix correctly in a way which would stay long-term maintainable.
For now, my expectation is that an improvement will land in 3.5.0 and probably won't be backported. The 3.1.x line should be unaffected, the msl code has been added in 3.3.0.
We have seen some patches already trying to fix that but all of them have some drawbacks which we hope to overcome by a different approach. (There are some rather crazy race conditions which we prefer to avoid.)
Sales/marketing note. It would be actually very helpful if the networks running BIRD with lots of BGP sessions had a support contract. Performance analysis and multithreaded development is a laborous and hardware-intensive process, and this is exactly where your money would go – to those people who are familiar with the whole codebase and can fix things. Check https://bird.nic.cz/commercial-services/ for more info.
I hope this helps. Happy routing! Maria
On November 12, 2026 6:26:18 AM GMT+01:00, Marcelo Balbinot < balbinot.marcelo@gmail.com> wrote:
Hello,
On a BIRD 3 route reflector the main thread runs at 100% CPU all the time, and almost all of it is in msl_cleanup().
Setup: - ~2,700 BGP sessions (iBGP rr clients and eBGP multihop) - threads 20 on a 32 vCPU VM; no IGP, only BGP and static reject routes - worker threads at 5-12% each
Measurements: - BIRD 3.3.2, process up 2h19m: total CPU 15h26m. perf on the main thread: 94% at a single address in msl_cleanup_full_heads() (lib/mslab.c:723, the walk over full_heads). - BIRD 3.4.0, about 10 minutes after a restart, perf on the main thread: 98.7% in msl_cleanup (57k samples in 15 s). - In 8 of our 12 crash cores, the main thread was inside msl_cleanup(). Two cores are aborts on asserts in that code: lib/mslab.c:731 (ASSERT_DIE(num_full < s->objs_per_slab)) and lib/event.c:332.
How we read the code: msl_free() sends the cleanup event whenever a page leaves the full state (or its last object is freed), and every msl_cleanup_full_heads() run walks the whole full_heads list. With a large attribute cache and constant churn from thousands of sessions, the cleanup is rescheduled all the time and each run is O(number of full pages).
We reduced our own churn (empty "igp table" on all BGP channels, since this RR never resolves next hops), which removed most next-hop re-evaluation. The main thread is still saturated in msl_cleanup.
Is this a known limitation of the mslab cleanup? Is there a way to tune it, or a fix we could test? We can run debug builds or collect more perf data.
Disclosure: I used an LLM (Claude) to help analyse the perf and core data. The numbers above are from our production system.
Best regards, Marcelo Balbinot
-- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.