Hello list, we are running BIRD on our IXP peering platform with no problems since quite a while now. We need to activate a new peer which is expected to announce a considerable amount of prefixes (~100k), and the first try resulted in having nearly 100% cpu load, ~60-70% for BIRD and the remaining for syslog. The machine was hardly manageable, so after a few minutes we had to disable the peer and restart the deamon. The current settings actually make BIRD log everything: log "/var/log/bird.log" all; log syslog all; so I guess we could start by modifying it: any advice on this? Would you suggest something more, in order to avoid having again a high cpu load? Thanks in advance, Simone.
I had same kind problem some years a go. Problem was that I accidentaly miscofigure bgp session so that one of peers was connected to ip what was announced via other hop. That caused route flapping and full internet table flapping at that router -> 100% cpu load and 10-100Mbps bgp traffic. 8.4.2015, 13:18, Simone Morandini kirjoitti:
Hello list,
we are running BIRD on our IXP peering platform with no problems since quite a while now. We need to activate a new peer which is expected to announce a considerable amount of prefixes (~100k), and the first try resulted in having nearly 100% cpu load, ~60-70% for BIRD and the remaining for syslog. The machine was hardly manageable, so after a few minutes we had to disable the peer and restart the deamon.
The current settings actually make BIRD log everything: log "/var/log/bird.log" all; log syslog all;
so I guess we could start by modifying it: any advice on this? Would you suggest something more, in order to avoid having again a high cpu load?
Thanks in advance, Simone.
-- Tapio Haapala F-Solutions Oy mobile +358400998371 irc Burner@ircnet skype burner-
Hi Simone, I would guess that syslog has better buffering and load handling; so I would try logging only to syslog--this should halve the load caused by verbose logging. Speaking of verbose, I've looked at BIRD logs, and I see lines like this: Route A received by peer Y Route A announced to peer X Route A announced to peer Y So, you multiply the number of roues by the number of peers: (100 000 x 152) = 15 200 000 But you're logging it twice (bird and syslog), so it's now 30 million lines of logs, to be written, just to bring up the peer. At 80 characters per line, would be 2.3 GB data to disk. This could cause serious load on a system with a modest HDD. It's also possible that each line causes a logging process which takes CPU time--do it 30 million times and the CPU time needed becomes large. What kind of CPU or disk do you have? Can you share? Maybe the solution is to buy more CPU and disk performance. On Wed, Apr 8, 2015 at 5:18 AM, Simone Morandini < simone.morandini@mix-it.net> wrote:
Hello list,
we are running BIRD on our IXP peering platform with no problems since quite a while now. We need to activate a new peer which is expected to announce a considerable amount of prefixes (~100k), and the first try resulted in having nearly 100% cpu load, ~60-70% for BIRD and the remaining for syslog. The machine was hardly manageable, so after a few minutes we had to disable the peer and restart the deamon.
The current settings actually make BIRD log everything: log "/var/log/bird.log" all; log syslog all;
so I guess we could start by modifying it: any advice on this? Would you suggest something more, in order to avoid having again a high cpu load?
Thanks in advance, Simone.
-- Jonathan
On Wed, Apr 08, 2015 at 12:18:14PM +0200, Simone Morandini wrote:
Hello list,
we are running BIRD on our IXP peering platform with no problems since quite a while now. We need to activate a new peer which is expected to announce a considerable amount of prefixes (~100k), and the first try resulted in having nearly 100% cpu load, ~60-70% for BIRD and the remaining for syslog. The machine was hardly manageable, so after a few minutes we had to disable the peer and restart the deamon.
The current settings actually make BIRD log everything:
Hi As already mentioned by others, it is most likely excessive logging. You could keep 'all' in 'log' options: log "/var/log/bird.log" all; log syslog all; (although such duplicit logging to both file and syslog is IMHO useless). But you should restrict 'debug' or 'debug protocols' options to perhaps '{ events }'. Assuming that it is these debug messages that are fulling your logs, You should obviously check these to see if it is not some error or warning. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
participants (4)
-
Jonathan Stewart -
Ondrej Zajicek -
Simone Morandini -
Tapio Haapala