On 2/7/24 15:55, Jan Bramkamp wrote:
On 11.01.24 21:34, Thomas Steen Rasmussen via Bird-users wrote:
I have no idea if this is a bird or a freebsd problem but I have to start somewhere :) I can to some extent test stuff, but the routers are in production (BGP with 1 ebgp and 1 ibgp peer and no full table) so nothing too wild.
Can you please also open a FreeBSD PR for this. It looks like bird is only the reproducer for a kernel bug. Can you periodically record the `vmstat -m` and `netstat -m` output as the resources are leaked?
Hi, Yes, sorry, I forgot all about it. I am not crazy about re-triggering the issue, but I did make a rudimentary prometheus exporter for the output of `netstat -m` when I was struggling with this so I have grafana graphs of it. I have attached a screenshot and it case it doesn't go through to the list I have also uploaded it here https://imgur.com/a/NAZYoDO and here https://hushfile.it/07ae3d87cc4c4#W0tXW2ZVtWkjDt9BuKrz9QVbcMfhd4BGriKBYfwb What the screenshot shows is the per-second rate of increase of the "total" number in the "mbuf clusters in use" line of the `netstat -m` output for both routers. The green line is the active and the yellow line is the passive router. The drop in the green line and the following spike towards the end (2000-2100ish) is me filtering the blackhole routes from the bird kernel export, removing the filter to confirm, and re-adding it. I have opened https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276870 in the FreeBSD end. /Thomas ps. The very rudimentary netstat -m exporter is here, needs jq and sponge installed: [tykling@dgncr2a ~]$ cat /etc/cron.d/netstat_mbuf_exporter # Run netstat_mbuf_exporter.sh every minute and put the output in prometheus textfile collector directory * * * * * root /usr/local/bin/netstat_mbuf_exporter.sh | /usr/local/bin/sponge /var/tmp/node_exporter/netstat-mbuf.prom [tykling@dgncr2a ~]$ cat /usr/local/bin/netstat_mbuf_exporter.sh #!/bin/sh /usr/bin/netstat -m --libxo json | /usr/local/bin/jq -r '."mbuf-statistics" | keys_unsorted[] as $k | "\($k) \(.[$k])"' | /usr/bin/tr "-" "_" | /usr/bin/sed "s/^/freebsd_netstat_mbuf_/g" [tykling@dgncr2a ~]$ head -5 /var/tmp/node_exporter/netstat-mbuf.prom freebsd_netstat_mbuf_mbuf_current 1495568 freebsd_netstat_mbuf_mbuf_cache 3547 freebsd_netstat_mbuf_mbuf_total 1499115 freebsd_netstat_mbuf_cluster_current 749044 freebsd_netstat_mbuf_cluster_cache 3558 [tykling@dgncr2a ~]$
