freebsd mbuf cluster leak with bird 2.14

[Thomas Steen Rasmussen]

On 2/7/24 15:55, Jan Bramkamp wrote:
>
> On 11.01.24 21:34, Thomas Steen Rasmussen via Bird-users wrote:
>>
>> I have no idea if this is a bird or a freebsd problem but I have to 
>> start somewhere :) I can to some extent test stuff, but the routers 
>> are in production (BGP with 1 ebgp and 1 ibgp peer and no full table) 
>> so nothing too wild.
> Can you please also open a FreeBSD PR for this. It looks like bird is 
> only the reproducer for a kernel bug. Can you periodically record the 
> `vmstat -m` and `netstat -m` output as the resources are leaked?

Hi,

Yes, sorry, I forgot all about it. I am not crazy about re-triggering 
the issue, but I did make a rudimentary prometheus exporter for the 
output of `netstat -m` when I was struggling with this so I have grafana 
graphs of it.

I have attached a screenshot and it case it doesn't go through to the 
list I have also uploaded it here https://imgur.com/a/NAZYoDO and here 
https://hushfile.it/07ae3d87cc4c4#W0tXW2ZVtWkjDt9BuKrz9QVbcMfhd4BGriKBYfwb

What the screenshot shows is the per-second rate of increase of the 
"total" number in the "mbuf clusters in use" line of the `netstat -m` 
output for both routers. The green line is the active and the yellow 
line is the passive router.

The drop in the green line and the following spike towards the end 
(2000-2100ish) is me filtering the blackhole routes from the bird kernel 
export, removing the filter to confirm, and re-adding it.

I have opened https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276870 
in the FreeBSD end.

/Thomas

ps. The very rudimentary netstat -m exporter is here, needs jq and 
sponge installed:

[tykling at dgncr2a ~]$ cat /etc/cron.d/netstat_mbuf_exporter
# Run netstat_mbuf_exporter.sh every minute and put the output in 
prometheus textfile collector directory
* * * * * root /usr/local/bin/netstat_mbuf_exporter.sh | 
/usr/local/bin/sponge /var/tmp/node_exporter/netstat-mbuf.prom

[tykling at dgncr2a ~]$ cat /usr/local/bin/netstat_mbuf_exporter.sh
#!/bin/sh
/usr/bin/netstat -m --libxo json | /usr/local/bin/jq -r 
'."mbuf-statistics" | keys_unsorted[] as $k | "\($k) \(.[$k])"' | 
/usr/bin/tr "-" "_" | /usr/bin/sed "s/^/freebsd_netstat_mbuf_/g"

[tykling at dgncr2a ~]$ head -5 /var/tmp/node_exporter/netstat-mbuf.prom
freebsd_netstat_mbuf_mbuf_current 1495568
freebsd_netstat_mbuf_mbuf_cache 3547
freebsd_netstat_mbuf_mbuf_total 1499115
freebsd_netstat_mbuf_cluster_current 749044
freebsd_netstat_mbuf_cluster_cache 3558
[tykling at dgncr2a ~]$
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Firefox_Screenshot_2024-02-07T15-06-31.373Z.png
Type: image/png
Size: 286724 bytes
Desc: not available
URL: <http://trubka.network.cz/pipermail/bird-users/attachments/20240207/79148ac5/attachment.png>