Hello, I installed different version of bird on Ubuntu 16.04 (namely, 1.5.0, 1.6.3, 1.6.4). After leaving the daemon running for ~1 hour, the memory used by the host is over 8Gib. However, when running `birdc show memory`, I get the following: BIRD memory usage Routing tables: 21 kB Route attributes: 11 kB ROA tables: 192 B Protocols: 22 kB Total: 102 kB My config is pretty basic (edited, see attached) as we have 2 BGP protocols using 2 ASN (one public, one private), each exporting the routes learnt from the other. The leak seems to be related to the traffic going through the server, as the memory of the host remains still when there is no traffic going on. It is triggered by bird though, because I don't have the leak when the routes are configured statically and bird is shut down. The weird thing about this is that the memory is not used by the bird process itself. It's shown as used memory by the host, but no process is actually using this memory. I know that it's leaking because of bird though because it's the only process installed and running on this host. Has anyone ever encountered such a behavior, or can see an issue in my configuration? Thanks in advance for your help -- Willy
debian stretch, bird v 1.6.3-2 is rock solid FWIW i saw zero leaks or issues on debian wheezy, jessie, devuan jessie either. on multiple rrs taking several million routes I am at: birdc BIRD 1.6.3 ready. bird> show mem BIRD memory usage Routing tables: 126 MB Route attributes: 93 MB ROA tables: 112 B Protocols: 823 kB Total: 219 MB alpine linux also works very well, no issues taking multiple full tables on core/edge as well: (host: core1) (user: root) (time= 11:23:53)
~ # birdc BIRD 1.6.0 ready. bird> show mem BIRD memory usage Routing tables: 182 MB Route attributes: 156 MB ROA tables: 192 B Protocols: 686 kB Total: 339 MB bird>
On Fri, 13 Apr 2018, Willy Goiffon wrote:
Hello,
I installed different version of bird on Ubuntu 16.04 (namely, 1.5.0, 1.6.3, 1.6.4). After leaving the daemon running for ~1 hour, the memory used by the host is over 8Gib. However, when running `birdc show memory`, I get the following:
BIRD memory usage Routing tables: 21 kB Route attributes: 11 kB ROA tables: 192 B Protocols: 22 kB Total: 102 kB
My config is pretty basic (edited, see attached) as we have 2 BGP protocols using 2 ASN (one public, one private), each exporting the routes learnt from the other.
The leak seems to be related to the traffic going through the server, as the memory of the host remains still when there is no traffic going on. It is triggered by bird though, because I don't have the leak when the routes are configured statically and bird is shut down.
The weird thing about this is that the memory is not used by the bird process itself. It's shown as used memory by the host, but no process is actually using this memory. I know that it's leaking because of bird though because it's the only process installed and running on this host.
Has anyone ever encountered such a behavior, or can see an issue in my configuration?
Thanks in advance for your help -- Willy
debian stretch, bird v 1.6.3-2 is rock solid
FWIW i saw zero leaks or issues on debian wheezy, jessie, devuan jessie either.
on multiple rrs taking several million routes I am at:
birdc BIRD 1.6.3 ready. bird> show mem BIRD memory usage Routing tables: 126 MB Route attributes: 93 MB ROA tables: 112 B Protocols: 823 kB Total: 219 MB
As I said, the mem leak does not apppear in the `birdc show memory` command. It is shown at the OS level, and is shown via the `free` command. The leak was also present with v1.6.3 on ubuntu though. Note that I don't have that much routes (~20), so the leak is not related to that IMO.
alpine linux also works very well, no issues taking multiple full tables on core/edge as well:
(host: core1) (user: root) (time= 11:23:53)
~ # birdc BIRD 1.6.0 ready. bird> show mem BIRD memory usage Routing tables: 182 MB Route attributes: 156 MB ROA tables: 192 B Protocols: 686 kB Total: 339 MB bird>
Unfortunately, switching to alpine or debian is not an option in my case. Thank you for the input!
On 2018-04-17 13:31, willy@mailoo.org wrote:
As I said, the mem leak does not apppear in the `birdc show memory` command. It is shown at the OS level, and is shown via the `free` command.
What do you mean by "OS level"? Which exactly level? What is the actual process size as reported by ps ("ps axuw|grep bird")? If ps shows size similar to what you see in birdc "show mem", then the leak is definitely not in bird (though it may be related) - it is in kernel or some other process. -- Best regards, Alexander.
Le 2018-04-17 14:27, Alexander Demenshin a écrit :
On 2018-04-17 13:31, willy@mailoo.org wrote:
As I said, the mem leak does not apppear in the `birdc show memory` command. It is shown at the OS level, and is shown via the `free` command.
What do you mean by "OS level"? Which exactly level? What is the actual process size as reported by ps ("ps axuw|grep bird")?
By "OS Level" I mean the memory used as reported by the kernel from /proc/meminfo. The process size is rather standard: bird 1737 0.0 0.0 15540 2216 ? Ss Apr17 0:01 /usr/sbin/bird -f -u bird -g bird If I sum up the size of all processes, I don't even reach 1Gib. But the memory used reported by the kernel increases gradually until it reaches OOM.
If ps shows size similar to what you see in birdc "show mem", then the leak is definitely not in bird (though it may be related) - it is in kernel or some other process.
It might not be in bird, but it is definitely triggered by it. Thanks -- Willy
On Wed, Apr 18, 2018 at 09:57:16AM +0200, Willy Goiffon wrote:
Le 2018-04-17 14:27, Alexander Demenshin a écrit :
On 2018-04-17 13:31, willy@mailoo.org wrote:
As I said, the mem leak does not apppear in the `birdc show memory` command. It is shown at the OS level, and is shown via the `free` command.
What do you mean by "OS level"? Which exactly level? What is the actual process size as reported by ps ("ps axuw|grep bird")?
By "OS Level" I mean the memory used as reported by the kernel from /proc/meminfo.
Hi Just curios, which field in /proc/meminfo (or /proc/slabinfo) is increasing?
If ps shows size similar to what you see in birdc "show mem", then the leak is definitely not in bird (though it may be related) - it is in kernel or some other process.
It might not be in bird, but it is definitely triggered by it.
That is interesting, one could imagine that difference between running BIRD and just using static routes would be that periodic scans could trigger some kernel bug, but that would cause linear increase independent of network traffic. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Hello, if it occurs also on 1.6.4, could you please add also "birdcl show route count" output and some logs with debug all switched on? How does it get leaked, do you reconfigure many times? Does the memory consumption increase gradually or by larger amounts? Thanks Maria Quidquid latine dictum sit, altum videtur. -----Original Message----- From: Willy Goiffon <willy@mailoo.org> To: bird-users@network.cz Sent: Fri, 13 Apr 2018 6:21 PM Subject: Bird 1.5 and 1.6 memory leak Hello, I installed different version of bird on Ubuntu 16.04 (namely, 1.5.0, 1.6.3, 1.6.4). After leaving the daemon running for ~1 hour, the memory used by the host is over 8Gib. However, when running `birdc show memory`, I get the following: BIRD memory usage Routing tables: 21 kB Route attributes: 11 kB ROA tables: 192 B Protocols: 22 kB Total: 102 kB My config is pretty basic (edited, see attached) as we have 2 BGP protocols using 2 ASN (one public, one private), each exporting the routes learnt from the other. The leak seems to be related to the traffic going through the server, as the memory of the host remains still when there is no traffic going on. It is triggered by bird though, because I don't have the leak when the routes are configured statically and bird is shut down. The weird thing about this is that the memory is not used by the bird process itself. It's shown as used memory by the host, but no process is actually using this memory. I know that it's leaking because of bird though because it's the only process installed and running on this host. Has anyone ever encountered such a behavior, or can see an issue in my configuration? Thanks in advance for your help -- Willy
if it occurs also on 1.6.4, could you please add also "birdcl show route count" output and some logs with debug all switched on?
I had to reinstall version 1.5.0, to remain in the ubuntu default PPA. The route count is ridiculously low though: $ birdc show route count BIRD 1.5.0 ready. 11 of 11 routes for 11 networks
How does it get leaked, do you reconfigure many times? Does the memory consumption increase gradually or by larger amounts?
It seems to leak at the kernel level, as the process memory doesn't increase. I'm not doing any reconfiguration. It increases gradually and is closely tied to the traffic crossing the router. If I stop any traffic from passing (eg, using iptables), the memleak stops. As I said, I also don't have this leak if I leave bird down and configure the routes statically.
participants (6)
-
Alexander Demenshin -
C. Jon Larsen -
Jan Maria Matějka -
Ondrej Zajicek -
Willy Goiffon -
willy＠mailoo.org