[3.3.1] How to debug excessive memory usage
Hi, I've been observing some memory leak (slowly gets to ~30GB across 18 hours then dies to oomkiller, ~500kB/s growth), how can I debug it furter ? # birdc BIRD 3.3.1 ready. bird> show memory BIRD memory usage Effective Overhead Routing tables: 6059.3 MB 564.4 MB Route attributes: 9510.1 MB 565.8 MB Protocols: 22.3 MB 82.9 kB Current config: 60.8 kB 1232.0 B Standby memory: 0.0 B 32.7 MB Total: 15.2 GB 1163.0 MB Active pages: 16.2 GB Kept free pages: 32.7 MB Cold free pages: 194.2 MB Hot page cache depleted while in RCU: 66929 bird> show route count 2098594 of 2098594 routes for 1051498 networks in table master4 0 of 0 routes for 0 networks in table master6 it's relatively simple config with 2 peers like this: protocol bgp ispX { local as 12345; graceful restart; bfd; allow as sets; neighbor 1.2.3.4 as 65432; ipv4 { import table; export table; export filter bgp_ext_out; import filter bgp_ext_in; import keep filtered on; }; # kernel and OSPF }; protocol kernel kcore { kernel table 1002; learn all; ipv4 { export filter core_to_kernel; import filter kernel_to_core; }; }; protocol ospf core { vrf "core"; ecmp; tick 2; ipv4 { import keep filtered on; import filter ospf_core_in; export filter ospf_core_out; }; area 0 { interface "b-swcore.3000" { type pointopoint; }; interface "b-swcore.3011" { type pointopoint; }; interface "b-swcore.3012" { type pointopoint; }; interface "core" { stub; }; }; }; version from current Debian Testing It only started happening once router was enabled (before it was peered but did not push the routes to kernel table), tho it coincided with upgrade from 3.1 to 3.3.1 as I hit different bug there . I have also tried to disable "export/import table", "import keep filtered on" and "allow as sets" just to narrow it down but appears there is no change Regards, Mariusz -- Mariusz Gronczewski (XANi) <xani+bird@devrandom.pl> GnuPG: 0xEA8ACE64 https://devrandom.eu
Hi Mariusz, On Fri, Jul 10, 2026 at 03:19:03PM +0200, Mariusz Gronczewski wrote:
I've been observing some memory leak (slowly gets to ~30GB across 18 hours then dies to oomkiller, ~500kB/s growth), how can I debug it furter ?
That is not an easy task, and I'm a little worried about your actual memory footprint. This is what I get with one uplink: BIRD 3.3.1 ready. bird> show route count 1047039 of 1047039 routes for 1047039 networks in table master4 245208 of 245208 routes for 245208 networks in table master6 Total: 1292247 of 1292247 routes for 1292247 networks in 2 tables bird> show mem BIRD memory usage Effective Overhead Routing tables: 191.7 MB 12.8 MB Route attributes: 115.3 MB 70.2 MB Protocols: 371.2 kB 13.6 kB Current config: 17.0 kB 880.0 B Standby memory: 0.0 B 33.2 MB Total: 307.5 MB 116.3 MB Active pages: 308.2 MB Kept free pages: 33.2 MB Cold free pages: 129.0 MB Hot page cache depleted while in RCU: 2482
# birdc BIRD 3.3.1 ready. bird> show memory BIRD memory usage Effective Overhead Routing tables: 6059.3 MB 564.4 MB Route attributes: 9510.1 MB 565.8 MB Protocols: 22.3 MB 82.9 kB Current config: 60.8 kB 1232.0 B Standby memory: 0.0 B 32.7 MB Total: 15.2 GB 1163.0 MB
Active pages: 16.2 GB Kept free pages: 32.7 MB Cold free pages: 194.2 MB Hot page cache depleted while in RCU: 66929
bird> show route count 2098594 of 2098594 routes for 1051498 networks in table master4 0 of 0 routes for 0 networks in table master6
This consumption is in itself very fishy. My first look would be into your filters because 10G in route attributes is probably terribly wrong with just 2M routes, and there must be _some_ reason for that, probably impossible deduplication. Another look would be what happens when you slightly change your import filter (e.g. by adding yet another community) and see whether the reported memory footprint raises significantly. Also, it may be handy to set debug protocols { states }; debug channels { routes, states }; debug tables { routes, states }; and see what is happening over time, but as it is slow, it may yield nothing.
[...] learn all;
This may play some role but I don't believe that as the implementation of the kernel protocol almost didn't change between 3.1 and 3.3.
It only started happening once router was enabled (before it was peered but did not push the routes to kernel table), tho it coincided with upgrade from 3.1 to 3.3.1 as I hit different bug there .
I have also tried to disable "export/import table", "import keep filtered on" and "allow as sets" just to narrow it down but appears there is no change
I would expect that with export/import table off, the memory consumption would go away. By any chance, how much are you calling `birdc`? We got reported some memory leaks regarding that. Also, I expect that your OSPF is not producing way too many routes, but if you could elaborate on that number, it may be a clue. Also, `import keep filtered` in OSPF is not needed, there is no benefit from setting it. Thanks for the report! -- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
On Fri, 10 Jul 2026 16:10:09 +0200 Maria Matejka <maria.matejka@nic.cz> wrote:
Hi Mariusz,
On Fri, Jul 10, 2026 at 03:19:03PM +0200, Mariusz Gronczewski wrote:
I've been observing some memory leak (slowly gets to ~30GB across 18 hours then dies to oomkiller, ~500kB/s growth), how can I debug it furter ?
That is not an easy task, and I'm a little worried about your actual memory footprint. This is what I get with one uplink:
BIRD 3.3.1 ready. bird> show route count 1047039 of 1047039 routes for 1047039 networks in table master4 245208 of 245208 routes for 245208 networks in table master6 Total: 1292247 of 1292247 routes for 1292247 networks in 2 tables bird> show mem BIRD memory usage Effective Overhead Routing tables: 191.7 MB 12.8 MB Route attributes: 115.3 MB 70.2 MB Protocols: 371.2 kB 13.6 kB Current config: 17.0 kB 880.0 B Standby memory: 0.0 B 33.2 MB Total: 307.5 MB 116.3 MB
Active pages: 308.2 MB Kept free pages: 33.2 MB Cold free pages: 129.0 MB Hot page cache depleted while in RCU: 2482
# birdc BIRD 3.3.1 ready. bird> show memory BIRD memory usage Effective Overhead Routing tables: 6059.3 MB 564.4 MB Route attributes: 9510.1 MB 565.8 MB Protocols: 22.3 MB 82.9 kB Current config: 60.8 kB 1232.0 B Standby memory: 0.0 B 32.7 MB Total: 15.2 GB 1163.0 MB
Active pages: 16.2 GB Kept free pages: 32.7 MB Cold free pages: 194.2 MB Hot page cache depleted while in RCU: 66929
bird> show route count 2098594 of 2098594 routes for 1051498 networks in table master4 0 of 0 routes for 0 networks in table master6
This consumption is in itself very fishy. My first look would be into your filters because 10G in route attributes is probably terribly wrong with just 2M routes, and there must be _some_ reason for that, probably impossible deduplication.
Another look would be what happens when you slightly change your import filter (e.g. by adding yet another community) and see whether the reported memory footprint raises significantly. Also, it may be handy to set
debug protocols { states }; debug channels { routes, states }; debug tables { routes, states };
and see what is happening over time, but as it is slow, it may yield nothing.
oh no it's very fast, I think I have possibly found the problem (attached log) I noticed there is many changes in 0.0.0.0/0 route but I: * don't import it from any peer * export it only from OSPF * is generated only as static route : protocol static generated { ipv4 { table generated_t; }; check link; igp table master4; route 193.200.227.0/24 blackhole; route 194.50.132.0/22 blackhole; route 0.0.0.0/0 recursive 8.8.8.8 { preference=500; }; route 0.0.0.0/0 recursive 1.1.1.1 { preference=290; }; route 0.0.0.0/0 recursive 13.96.0.1 { preference=280; }; route 0.0.0.0/0 recursive 142.250.0.1 { preference=270; }; route 0.0.0.0/0 recursive 18.244.0.1 { preference=260; }; route 0.0.0.0/0 unreachable { preference=1; }; }; This was poor attempt (I really wanted to avoid external scripting) at generating a route preference based on whether a given router "sees" wider internet (vs just sending traffic to peer that's). But I don't understand why it causes so many updates
[...] learn all;
This may play some role but I don't believe that as the implementation of the kernel protocol almost didn't change between 3.1 and 3.3.
It only started happening once router was enabled (before it was peered but did not push the routes to kernel table), tho it coincided with upgrade from 3.1 to 3.3.1 as I hit different bug there .
I have also tried to disable "export/import table", "import keep filtered on" and "allow as sets" just to narrow it down but appears there is no change
I would expect that with export/import table off, the memory consumption would go away.
nope
By any chance, how much are you calling `birdc`? We got reported some memory leaks regarding that.
Yes, I saw that report, after stopping all of the tooling that does it doesn't really change so I don't think it is related
Also, I expect that your OSPF is not producing way too many routes, but if you could elaborate on that number, it may be a clue.
few hundred at most
Also, `import keep filtered` in OSPF is not needed, there is no benefit from setting it.
Thanks for the report!
-- Mariusz Gronczewski (XANi) <xani+bird@devrandom.pl> GnuPG: 0xEA8ACE64 https://devrandom.eu
On Fri, 10 Jul 2026 16:10:09 +0200 Maria Matejka <maria.matejka@nic.cz> wrote:
Hi Mariusz,
On Fri, Jul 10, 2026 at 03:19:03PM +0200, Mariusz Gronczewski wrote:
I've been observing some memory leak (slowly gets to ~30GB across 18 hours then dies to oomkiller, ~500kB/s growth), how can I debug it furter ?
That is not an easy task, and I'm a little worried about your actual memory footprint. This is what I get with one uplink:
BIRD 3.3.1 ready. bird> show route count 1047039 of 1047039 routes for 1047039 networks in table master4 245208 of 245208 routes for 245208 networks in table master6 Total: 1292247 of 1292247 routes for 1292247 networks in 2 tables bird> show mem BIRD memory usage Effective Overhead Routing tables: 191.7 MB 12.8 MB Route attributes: 115.3 MB 70.2 MB Protocols: 371.2 kB 13.6 kB Current config: 17.0 kB 880.0 B Standby memory: 0.0 B 33.2 MB Total: 307.5 MB 116.3 MB
Active pages: 308.2 MB Kept free pages: 33.2 MB Cold free pages: 129.0 MB Hot page cache depleted while in RCU: 2482
we just have 2 peers with full tables so yeah, route count doesn't really match actually used memory
# birdc BIRD 3.3.1 ready. bird> show memory BIRD memory usage Effective Overhead Routing tables: 6059.3 MB 564.4 MB Route attributes: 9510.1 MB 565.8 MB Protocols: 22.3 MB 82.9 kB Current config: 60.8 kB 1232.0 B Standby memory: 0.0 B 32.7 MB Total: 15.2 GB 1163.0 MB
Active pages: 16.2 GB Kept free pages: 32.7 MB Cold free pages: 194.2 MB Hot page cache depleted while in RCU: 66929
bird> show route count 2098594 of 2098594 routes for 1051498 networks in table master4 0 of 0 routes for 0 networks in table master6
This consumption is in itself very fishy. My first look would be into your filters because 10G in route attributes is probably terribly wrong with just 2M routes, and there must be _some_ reason for that, probably impossible deduplication. Another look would be what happens when you slightly change your import filter (e.g. by adding yet another community) and see whether the reported memory footprint raises significantly. Also, it may be handy to set
debug protocols { states }; debug channels { routes, states }; debug tables { routes, states };
and see what is happening over time, but as it is slow, it may yield nothing.
Oh it isn't slow at all, in my case for some reason there is ton of 0.0.0.0/0 route updates like, kernel get good 2 orders of magnitude more updates than any of the BGP peers produces Now 0.0.0.0/0 isnt imported from any peer (dropped on filters), and only generated in static and then distributed to OSPF: protocol static generated { ipv4 { table generated_t; }; check link; igp table master4; route 193.200.227.0/24 blackhole; route 194.50.132.0/22 blackhole; route 0.0.0.0/0 recursive 8.8.8.8 { preference=500; }; route 0.0.0.0/0 recursive 1.1.1.1 { preference=290; }; route 0.0.0.0/0 recursive 13.96.0.1 { preference=280; }; route 0.0.0.0/0 recursive 142.250.0.1 { preference=270; }; route 0.0.0.0/0 recursive 18.244.0.1 { preference=260; }; route 0.0.0.0/0 unreachable { preference=1; }; }; That was my poor attempt at trying to generate one only on router that actually sees internet (and not just have peer up), I wanted to steer traffic to router that have "working" ISP, but I'm not sure why it would generate such flapping (and either way shouldnt really leak memory) I've attached log and full config
[...] learn all;
This may play some role but I don't believe that as the implementation of the kernel protocol almost didn't change between 3.1 and 3.3.
It only started happening once router was enabled (before it was peered but did not push the routes to kernel table), tho it coincided with upgrade from 3.1 to 3.3.1 as I hit different bug there .
I have also tried to disable "export/import table", "import keep filtered on" and "allow as sets" just to narrow it down but appears there is no change
I would expect that with export/import table off, the memory consumption would go away.
No change after disabling, still grows
By any chance, how much are you calling `birdc`? We got reported some memory leaks regarding that.
saw that thread, I've tried disabling montioring (we do have prometheus exporter exporting it quite often) but no change
Also, I expect that your OSPF is not producing way too many routes, but if you could elaborate on that number, it may be a clue.
around 100 or so so probably not
Also, `import keep filtered` in OSPF is not needed, there is no benefit from setting it.
good to know, thanks
Thanks for the report!
-- Mariusz Gronczewski (XANi) <xani+bird@devrandom.pl> GnuPG: 0xEA8ACE64 https://devrandom.eu
On Fri, 10 Jul 2026 17:23:54 +0200 Mariusz Gronczewski <xani+bird@devrandom.pl> wrote:
Now 0.0.0.0/0 isnt imported from any peer (dropped on filters), and only generated in static and then distributed to OSPF:
That was incorrect. I had kernel import filter set to all so it imported 0.0.0.0/0 route and kernel export filter also allowed 0.0.0.0/0 route however fixing that (no 0.0.0.0/0 exported to kernel any more) didn't fix the problem removing all but one generated route protocol static generated { ipv4 { table generated_t; }; check link; igp table master4; route 193.200.227.0/24 blackhole; route 194.50.132.0/22 blackhole; route 0.0.0.0/0 recursive 8.8.8.8 { preference=500; }; }; seems to fix it however I don't know why... the previous config just showed 0.0.0.0/0 in bird with multiple hops based on preference. -- Mariusz Gronczewski (XANi) <xani+bird@devrandom.pl> GnuPG: 0xEA8ACE64 https://devrandom.eu
This smells of a runaway hostentry resolution cycle or something like that. Do you have some "minimal reproducer" then, which causes this godzilla to emerge? It is probably not easy to fix but we'd like to try. Thanks! Maria On Fri, Jul 10, 2026 at 05:58:43PM +0200, Mariusz Gronczewski wrote:
On Fri, 10 Jul 2026 17:23:54 +0200 Mariusz Gronczewski <xani+bird@devrandom.pl> wrote:
Now 0.0.0.0/0 isnt imported from any peer (dropped on filters), and only generated in static and then distributed to OSPF:
That was incorrect. I had kernel import filter set to all so it imported 0.0.0.0/0 route
and kernel export filter also allowed 0.0.0.0/0 route
however fixing that (no 0.0.0.0/0 exported to kernel any more) didn't fix the problem
removing all but one generated route
protocol static generated { ipv4 { table generated_t;
}; check link; igp table master4; route 193.200.227.0/24 blackhole; route 194.50.132.0/22 blackhole; route 0.0.0.0/0 recursive 8.8.8.8 { preference=500;
};
};
seems to fix it however I don't know why... the previous config just showed 0.0.0.0/0 in bird with multiple hops based on preference.
-- Mariusz Gronczewski (XANi) <xani+bird@devrandom.pl> GnuPG: 0xEA8ACE64 https://devrandom.eu
-- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
Hi Maria, I observed severe memory consumption with bird-3.2.0-1.el8.x86_64 and recently updated to bird-3.3.1-2.el8.x86_64 to mitigate the issue. While the update may have resolved the behavior, I am still interested in investigating the root cause. On several servers, there was consistent, intensive memory growth with RSS reaching up to 100GB. At that stage, birdc became unresponsive, the process showed significant CPU utilization across multiple cores, and BGP neighbors closed their sessions due to timeouts. I had configured debug protocols all; and enabled log rotation, but the process stopped writing to the logs once the memory threshold was reached. I have successfully captured a gcore dump exceeding 100GB. Could you please provide guidance on analyzing this core file to identify the source of the leak? Any advice on how to troubleshoot this effectively would be greatly appreciated. Best regards, Alexander Shevchenko On Fri, Jul 10, 2026 at 5:16 PM Maria Matejka via Bird-users < bird-users@network.cz> wrote:
Hi Mariusz,
On Fri, Jul 10, 2026 at 03:19:03PM +0200, Mariusz Gronczewski wrote:
I’ve been observing some memory leak (slowly gets to ~30GB across 18 hours then dies to oomkiller, ~500kB/s growth), how can I debug it furter ?
That is not an easy task, and I’m a little worried about your actual memory footprint. This is what I get with one uplink:
BIRD 3.3.1 ready. bird> show route count 1047039 of 1047039 routes for 1047039 networks in table master4 245208 of 245208 routes for 245208 networks in table master6 Total: 1292247 of 1292247 routes for 1292247 networks in 2 tables bird> show mem BIRD memory usage Effective Overhead Routing tables: 191.7 MB 12.8 MB Route attributes: 115.3 MB 70.2 MB Protocols: 371.2 kB 13.6 kB Current config: 17.0 kB 880.0 B Standby memory: 0.0 B 33.2 MB Total: 307.5 MB 116.3 MB
Active pages: 308.2 MB Kept free pages: 33.2 MB Cold free pages: 129.0 MB Hot page cache depleted while in RCU: 2482
birdc
BIRD 3.3.1 ready. bird> show memory BIRD memory usage Effective Overhead Routing tables: 6059.3 MB 564.4 MB Route attributes: 9510.1 MB 565.8 MB Protocols: 22.3 MB 82.9 kB Current config: 60.8 kB 1232.0 B Standby memory: 0.0 B 32.7 MB Total: 15.2 GB 1163.0 MB
Active pages: 16.2 GB Kept free pages: 32.7 MB Cold free pages: 194.2 MB Hot page cache depleted while in RCU: 66929
bird> show route count 2098594 of 2098594 routes for 1051498 networks in table master4 0 of 0 routes for 0 networks in table master6
This consumption is in itself very fishy. My first look would be into your filters because 10G in route attributes is probably terribly wrong with just 2M routes, and there must be *some* reason for that, probably impossible deduplication.
Another look would be what happens when you slightly change your import filter (e.g. by adding yet another community) and see whether the reported memory footprint raises significantly. Also, it may be handy to set
debug protocols { states }; debug channels { routes, states }; debug tables { routes, states };
and see what is happening over time, but as it is slow, it may yield nothing.
[…] learn all;
This may play some role but I don’t believe that as the implementation of the kernel protocol almost didn’t change between 3.1 and 3.3.
It only started happening once router was enabled (before it was peered but did not push the routes to kernel table), tho it coincided with upgrade from 3.1 to 3.3.1 as I hit different bug there .
I have also tried to disable “export/import table”, “import keep filtered on” and “allow as sets” just to narrow it down but appears there is no change
I would expect that with export/import table off, the memory consumption would go away.
By any chance, how much are you calling birdc? We got reported some memory leaks regarding that.
Also, I expect that your OSPF is not producing way too many routes, but if you could elaborate on that number, it may be a clue.
Also, import keep filtered in OSPF is not needed, there is no benefit from setting it.
Thanks for the report!
– Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
On Fri, 10 Jul 2026 18:31:42 +0200 Maria Matejka <maria.matejka@nic.cz> wrote:
This smells of a runaway hostentry resolution cycle or something like that. Do you have some "minimal reproducer" then, which causes this godzilla to emerge? It is probably not easy to fix but we'd like to try.
Thanks!
Try the attached config. Thankfully it appears to be reproducible with no dependencies. It appears to be Bird2 -> 3 regression I have tested on 3.3.1 and on 3.1.7 vs 2.19.1 (both from Debian) It's a static protocol with few routes with different preferences attached to a separate table and the routes are moved off this table to the master one via pipe protocol. Now, if static is attached directly to table it works fine. But doing it thru pipe starts the loop: bird> show memory BIRD memory usage Effective Overhead Routing tables: 11.8 MB 1145.0 kB Route attributes: 12.8 MB 252.9 kB Protocols: 39.3 kB 3520.0 B Current config: 8351.0 B 504.0 B Standby memory: 0.0 B 188.0 kB Total: 24.8 MB 1628.1 kB Active pages: 25.6 MB Kept free pages: 188.0 kB Cold free pages: 0.0 B Hot page cache depleted while in RCU: 0 bird> show memory BIRD memory usage Effective Overhead Routing tables: 12.5 MB 1213.2 kB Route attributes: 13.5 MB 263.5 kB Protocols: 39.3 kB 3520.0 B Current config: 8351.0 B 504.0 B Standby memory: 0.0 B 200.0 kB Total: 26.3 MB 1718.9 kB Active pages: 27.1 MB Kept free pages: 200.0 kB Cold free pages: 0.0 B Hot page cache depleted while in RCU: 0 bird> show route count 5 of 5 routes for 1 networks in table master4 0 of 0 routes for 0 networks in table master6 Total: 5 of 5 routes for 1 networks in 2 tables and disabling the pipe stops it bird> disable pipe_generated pipe_generated: disabled bird> show memory BIRD memory usage Effective Overhead Routing tables: 295.1 kB 29.3 kB Route attributes: 28.4 kB 59.5 kB Protocols: 7217.0 B 2048.0 B Current config: 8351.0 B 504.0 B Standby memory: 0.0 B 55.7 MB Total: 483.1 kB 55.8 MB Active pages: 204.0 kB Kept free pages: 55.7 MB Cold free pages: 0.0 B Hot page cache depleted while in RCU: 0 kinda feels like putting pipe in-between fucks with some logic but that doesn't explain the memory growth, just the route churn, and it all appears to be in the pipe protocol: bird> show protocols all Name Proto Table State Since Info pipe_generated Pipe --- up 2026-07-13 11:08:28 master4 <=> generated_t Created: 2026-07-13 11:08:28 Channel main Table: master4 Peer table: generated_t Import state: READY Export state: READY Import filter: ACCEPT Export filter: REJECT Routes: 5 imported, 0 exported Route change stats: received rejected filtered ignored accepted Import updates: 409490 0 0 5 204745 Import withdraws: 0 0 --- 0 0 Export updates: 409480 204740 0 0 0 Export withdraws: 0 0 --- 0 0 device1 Device --- up 2026-07-13 11:08:28 Created: 2026-07-13 11:08:28 generated Static generated_t up 2026-07-13 11:08:28 Created: 2026-07-13 11:08:28 Channel ipv4 State: UP Import state: UP Export state: DOWN Table: generated_t Preference: 200 Input filter: ACCEPT Output filter: REJECT Routes: 5 imported, 0 exported, 1 preferred Route change stats: received rejected filtered ignored RX limit limit accepted Import updates: 5 0 0 0 0 0 5 Import withdraws: 0 0 --- 0 --- --- 0 Export updates: 0 0 0 0 --- 0 0 Export withdraws: 0 --- --- 0 --- --- 0 for comparison, bird2 : ☠ birdc BIRD 2.19.1 ready. bird> show protocols all Name Proto Table State Since Info device1 Device --- up 2026-07-13 11:14:05 generated Static generated_t up 2026-07-13 11:14:05 Channel ipv4 State: UP Table: generated_t Preference: 200 Input filter: ACCEPT Output filter: REJECT Routes: 5 imported, 0 exported, 1 preferred Route change stats: received rejected filtered ignored accepted Import updates: 5 0 0 0 5 Import withdraws: 0 0 --- 0 0 Export updates: 0 0 0 --- 0 Export withdraws: 0 --- --- --- 0 pipe_generated Pipe --- up 2026-07-13 11:14:05 master4 <=> generated_t Channel main Table: master4 Peer table: generated_t Import state: up Export state: up Import filter: ACCEPT Export filter: REJECT Routes: 5 imported, 0 exported Route change stats: received rejected filtered ignored accepted Import updates: 10 0 0 0 10 Import withdraws: 0 0 --- 0 0 Export updates: 10 10 0 0 0 Export withdraws: 0 0 --- 0 0
Maria
On Fri, Jul 10, 2026 at 05:58:43PM +0200, Mariusz Gronczewski wrote:
On Fri, 10 Jul 2026 17:23:54 +0200 Mariusz Gronczewski <xani+bird@devrandom.pl> wrote:
Now 0.0.0.0/0 isnt imported from any peer (dropped on filters), and only generated in static and then distributed to OSPF:
That was incorrect. I had kernel import filter set to all so it imported 0.0.0.0/0 route
and kernel export filter also allowed 0.0.0.0/0 route
however fixing that (no 0.0.0.0/0 exported to kernel any more) didn't fix the problem
removing all but one generated route
protocol static generated { ipv4 { table generated_t;
}; check link; igp table master4; route 193.200.227.0/24 blackhole; route 194.50.132.0/22 blackhole; route 0.0.0.0/0 recursive 8.8.8.8 { preference=500;
};
};
seems to fix it however I don't know why... the previous config just showed 0.0.0.0/0 in bird with multiple hops based on preference.
-- Mariusz Gronczewski (XANi) <xani+bird@devrandom.pl> GnuPG: 0xEA8ACE64 https://devrandom.eu
-- Mariusz Gronczewski (XANi) <xani+bird@devrandom.pl> GnuPG: 0xEA8ACE64 https://devrandom.eu
bisecting it got me the bug being introduced in 182a97f5e6aaeb5fac2859bb30889d88275898c8 On Mon, 13 Jul 2026 11:30:27 +0200 Mariusz Gronczewski <xani+bird@devrandom.pl> wrote:
On Fri, 10 Jul 2026 18:31:42 +0200 Maria Matejka <maria.matejka@nic.cz> wrote:
This smells of a runaway hostentry resolution cycle or something like that. Do you have some "minimal reproducer" then, which causes this godzilla to emerge? It is probably not easy to fix but we'd like to try.
Thanks!
Try the attached config. Thankfully it appears to be reproducible with no dependencies. It appears to be Bird2 -> 3 regression I have tested on 3.3.1 and on 3.1.7 vs 2.19.1 (both from Debian)
It's a static protocol with few routes with different preferences attached to a separate table and the routes are moved off this table to the master one via pipe protocol.
Now, if static is attached directly to table it works fine. But doing it thru pipe starts the loop:
bird> show memory BIRD memory usage Effective Overhead Routing tables: 11.8 MB 1145.0 kB Route attributes: 12.8 MB 252.9 kB Protocols: 39.3 kB 3520.0 B Current config: 8351.0 B 504.0 B Standby memory: 0.0 B 188.0 kB Total: 24.8 MB 1628.1 kB
Active pages: 25.6 MB Kept free pages: 188.0 kB Cold free pages: 0.0 B Hot page cache depleted while in RCU: 0 bird> show memory BIRD memory usage Effective Overhead Routing tables: 12.5 MB 1213.2 kB Route attributes: 13.5 MB 263.5 kB Protocols: 39.3 kB 3520.0 B Current config: 8351.0 B 504.0 B Standby memory: 0.0 B 200.0 kB Total: 26.3 MB 1718.9 kB
Active pages: 27.1 MB Kept free pages: 200.0 kB Cold free pages: 0.0 B Hot page cache depleted while in RCU: 0 bird> show route count 5 of 5 routes for 1 networks in table master4 0 of 0 routes for 0 networks in table master6 Total: 5 of 5 routes for 1 networks in 2 tables
and disabling the pipe stops it
bird> disable pipe_generated pipe_generated: disabled bird> show memory BIRD memory usage Effective Overhead Routing tables: 295.1 kB 29.3 kB Route attributes: 28.4 kB 59.5 kB Protocols: 7217.0 B 2048.0 B Current config: 8351.0 B 504.0 B Standby memory: 0.0 B 55.7 MB Total: 483.1 kB 55.8 MB
Active pages: 204.0 kB Kept free pages: 55.7 MB Cold free pages: 0.0 B Hot page cache depleted while in RCU: 0
kinda feels like putting pipe in-between fucks with some logic but that doesn't explain the memory growth, just the route churn, and it all appears to be in the pipe protocol:
bird> show protocols all Name Proto Table State Since Info pipe_generated Pipe --- up 2026-07-13 11:08:28 master4 <=> generated_t Created: 2026-07-13 11:08:28 Channel main Table: master4 Peer table: generated_t Import state: READY Export state: READY Import filter: ACCEPT Export filter: REJECT Routes: 5 imported, 0 exported Route change stats: received rejected filtered ignored accepted Import updates: 409490 0 0 5 204745 Import withdraws: 0 0 --- 0 0 Export updates: 409480 204740 0 0 0 Export withdraws: 0 0 --- 0 0
device1 Device --- up 2026-07-13 11:08:28 Created: 2026-07-13 11:08:28
generated Static generated_t up 2026-07-13 11:08:28 Created: 2026-07-13 11:08:28 Channel ipv4 State: UP Import state: UP Export state: DOWN Table: generated_t Preference: 200 Input filter: ACCEPT Output filter: REJECT Routes: 5 imported, 0 exported, 1 preferred Route change stats: received rejected filtered ignored RX limit limit accepted Import updates: 5 0 0 0 0 0 5 Import withdraws: 0 0 --- 0 --- --- 0 Export updates: 0 0 0 0 --- 0 0 Export withdraws: 0 --- --- 0 --- --- 0
for comparison, bird2 :
☠ birdc BIRD 2.19.1 ready. bird> show protocols all Name Proto Table State Since Info device1 Device --- up 2026-07-13 11:14:05
generated Static generated_t up 2026-07-13 11:14:05 Channel ipv4 State: UP Table: generated_t Preference: 200 Input filter: ACCEPT Output filter: REJECT Routes: 5 imported, 0 exported, 1 preferred Route change stats: received rejected filtered ignored accepted Import updates: 5 0 0 0 5 Import withdraws: 0 0 --- 0 0 Export updates: 0 0 0 --- 0 Export withdraws: 0 --- --- --- 0
pipe_generated Pipe --- up 2026-07-13 11:14:05 master4 <=> generated_t Channel main Table: master4 Peer table: generated_t Import state: up Export state: up Import filter: ACCEPT Export filter: REJECT Routes: 5 imported, 0 exported Route change stats: received rejected filtered ignored accepted Import updates: 10 0 0 0 10 Import withdraws: 0 0 --- 0 0 Export updates: 10 10 0 0 0 Export withdraws: 0 0 --- 0 0
Maria
On Fri, Jul 10, 2026 at 05:58:43PM +0200, Mariusz Gronczewski wrote:
On Fri, 10 Jul 2026 17:23:54 +0200 Mariusz Gronczewski <xani+bird@devrandom.pl> wrote:
Now 0.0.0.0/0 isnt imported from any peer (dropped on filters), and only generated in static and then distributed to OSPF:
That was incorrect. I had kernel import filter set to all so it imported 0.0.0.0/0 route
and kernel export filter also allowed 0.0.0.0/0 route
however fixing that (no 0.0.0.0/0 exported to kernel any more) didn't fix the problem
removing all but one generated route
protocol static generated { ipv4 { table generated_t;
}; check link; igp table master4; route 193.200.227.0/24 blackhole; route 194.50.132.0/22 blackhole; route 0.0.0.0/0 recursive 8.8.8.8 { preference=500;
};
};
seems to fix it however I don't know why... the previous config just showed 0.0.0.0/0 in bird with multiple hops based on preference.
-- Mariusz Gronczewski (XANi) <xani+bird@devrandom.pl> GnuPG: 0xEA8ACE64 https://devrandom.eu
-- Mariusz Gronczewski (XANi) <xani+bird@devrandom.pl> GnuPG: 0xEA8ACE64 https://devrandom.eu
On 10.07.2026 at 18:31, Maria Matejka via Bird-users wrote:
This smells of a runaway hostentry resolution cycle or something like that. Do you have some “minimal reproducer” then, which causes this godzilla to emerge? It is probably not easy to fix but we’d like to try.
Thanks!
Maria
I was also skeptical about the memory usage of BIRD 3.x compared to BIRD 2.x. In my case, memory consumption roughly doubled. I raised this several times on this mailing list as well as in private messages, but the issue was either ignored or attributed to FreeBSD. Regarding possible memory leaks, after about one month of uptime, BIRD 3.3.1 with four (2 x dual-stack) BGP full-view sessions and one OSPFv3 session currently shows: |top PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 4397 root 2 7 0 1987M 1877M select 2 120.1H 7.81% bird| |bird3-3.3.1 Dynamic multithreaded IP routing daemon BIRD 3.3.1 ready. bird> show memory BIRD memory usage Effective Overhead Routing tables: 311.6 MB 27.4 MB Route attributes: 232.5 MB 17.5 MB Protocols: 4847.1 kB 72.7 kB Current config: 73.5 kB 1400.0 B Standby memory: 0.0 B 35.2 MB Total: 549.1 MB 80.3 MB Active pages: 450.1 MB Kept free pages: 35.2 MB Cold free pages: 1173.5 MB Hot page cache depleted while in RCU: 1266139| Could someone explain what the continuously increasing "Cold free pages" value represents, and why there is such a large difference (around 710 MB after one month) in the memory reported by |show memory|and the |SIZE|/|RES|values reported by |top(1)|on FreeBSD as well? Is this expected behavior, or could it indicate that memory is being retained but not released back to the operating system? Cheers Marek Zarychta
I sent the liebot to do some iteration on the problem and it figured out problem was caused by change in route comparison logic in 182a97f5e, comparing by version caused same routes to be re-evaluated over and over again and after adding extra comparison (attached patch, I don't know C nowhere near enough to vouch for it but it at least shows the problem clearly) I also attach the test script used for bisect Cheers, Mariusz On Fri, 10 Jul 2026 18:31:42 +0200 Maria Matejka <maria.matejka@nic.cz> wrote:
This smells of a runaway hostentry resolution cycle or something like that. Do you have some "minimal reproducer" then, which causes this godzilla to emerge? It is probably not easy to fix but we'd like to try.
Thanks!
Maria
On Fri, Jul 10, 2026 at 05:58:43PM +0200, Mariusz Gronczewski wrote:
On Fri, 10 Jul 2026 17:23:54 +0200 Mariusz Gronczewski <xani+bird@devrandom.pl> wrote:
Now 0.0.0.0/0 isnt imported from any peer (dropped on filters), and only generated in static and then distributed to OSPF:
That was incorrect. I had kernel import filter set to all so it imported 0.0.0.0/0 route
and kernel export filter also allowed 0.0.0.0/0 route
however fixing that (no 0.0.0.0/0 exported to kernel any more) didn't fix the problem
removing all but one generated route
protocol static generated { ipv4 { table generated_t;
}; check link; igp table master4; route 193.200.227.0/24 blackhole; route 194.50.132.0/22 blackhole; route 0.0.0.0/0 recursive 8.8.8.8 { preference=500;
};
};
seems to fix it however I don't know why... the previous config just showed 0.0.0.0/0 in bird with multiple hops based on preference.
-- Mariusz Gronczewski (XANi) <xani+bird@devrandom.pl> GnuPG: 0xEA8ACE64 https://devrandom.eu
-- Mariusz Gronczewski (XANi) <xani+bird@devrandom.pl> GnuPG: 0xEA8ACE64 https://devrandom.eu
Hi Marek, On Mon, Jul 13, 2026 at 03:23:26PM +0200, Marek Zarychta via Bird-users wrote:
I was also skeptical about the memory usage of BIRD 3.x compared to BIRD 2.x. In my case, memory consumption roughly doubled. I raised this several times on this mailing list as well as in private messages, but the issue was either ignored or attributed to FreeBSD.
Regarding possible memory leaks, after about one month of uptime, BIRD 3.3.1 with four (2 x dual-stack) BGP full-view sessions and one OSPFv3 session currently shows:
[...]
Cold free pages: 1173.5 MB Hot page cache depleted while in RCU: 1266139
These two lines tell me that there have been over a million times where BIRD had to map a new memory page in situations where part of the allocator was unavailable due to deadlock prevention. That is way more than I expected when adding this check, and now I have an empirical reason to touch the allocator again.
Could someone explain what the continuously increasing "Cold free pages" value represents, and why there is such a large difference (around 710 MB after one month) in the memory reported by |show memory|and the |SIZE|/|RES|values reported by |top(1)|on FreeBSD as well?
There is a problem of address space fragmentation when we return memory pages randomly, leading to nasty errors. Instead of that, we decided to keep the pages and only mark them via madvise() as free so that kernel can actually unmap them whenever needed.
Is this expected behavior, or could it indicate that memory is being retained but not released back to the operating system?
All of that indicate that: - maybe at FreeBSD we don't use the right madvise() arguments, or the kernel behaves differently - we should look again into the hot page cache depletion problem, and implement some smarter scheduling and memory management algorithms, because this slow virtual memory creep will sooner or later become a problem even at linux machines where we are sure that the pages are freed correctly - there may be some other memory leak (e.g. that one we already know about in CLI) causing another part of the consumed memory growth Thank you for the heads-up. We'll look at it. -- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
participants (4)
-
Alexander Shevchenko -
Marek Zarychta -
Maria Matejka -
Mariusz Gronczewski