System freezes with Bird running on FreeBSD 11.1
Hello Guys, Let me do a question about bird 1.6.3 to see if you can help me on this, we are trying to running it in a server with freebsd 11.1, but the system freeze every 20 seconds, looking for the logs we found some I/O loop cycle messages like this: Jan 2 14:03:15 tau bird: I/O loop cycle took 12509 ms for 8 events We think that the problem can be have a relationship to the Kernel protocol, when the scan run every 20 seconds to the search for changes and insert in the table, we know that this time can be manipulated by the flag scan time, but it will just to delay the problem and not solves that. Enabling or not the learn flag also is not making difference to solve the problem. Any help will be really appreciated. My best Regards, Clistenes Viana Network Engineer Vianet Telecomunicacoes e Internet Ltda
On Wed, Jan 03, 2018 at 11:30:48AM -0200, Clistenes Viana wrote:
Hello Guys,
Let me do a question about bird 1.6.3 to see if you can help me on this, we are trying to running it in a server with freebsd 11.1, but the system freeze every 20 seconds, looking for the logs we found some I/O loop cycle messages like this:
Jan 2 14:03:15 tau bird: I/O loop cycle took 12509 ms for 8 events
Hello How many routes do you have in kernel routing table? How much time it takes to run something like netstat -nr > /dev/null ? -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Following the outputs; # netstat -nr4 | wc -l 688431 # time netstat -nr > /dev/null 3.474u 1.063s 0:04.57 99.1% 183+183k 0+0io 6pf+0w 3.519u 1.196s 0:04.71 99.7% 183+183k 0+0io 0pf+0w 3.468u 0.841s 0:04.31 99.7% 183+183k 0+0io 0pf+0w 3.492u 0.841s 0:04.33 100.0% 183+183k 0+0io 0pf+0w More info about bird; # birdc show protocols all BIRD 1.6.3 ready. name proto table state since info pDirect Direct master up 16:56:18 Routes: 2 imported, 0 exported, 2 preferred pDevice Device master up 16:56:18 Routes: 0 imported, 0 exported, 0 preferred pKernel Kernel master up 16:56:18 Routes: 696 imported, 688426 exported, 6 preferred Route change stats: received rejected filtered ignored accepted Import updates: 701 0 0 0 701 Import withdraws: 5 0 --- 0 5 Export updates: 2231443 739 126 --- 2230578 Export withdraws: 22269 --- --- --- 22303 pStaticOSPF Static master up 16:56:18 Routes: 1 imported, 0 exported, 1 preferred pOspf OSPF master up 16:56:18 Running Routes: 701 imported, 4 exported, 700 preferred Route change stats: received rejected filtered ignored accepted Import updates: 831 0 72 0 759 Import withdraws: 51 0 --- 84 39 Export updates: 2231442 757 2230681 --- 4 Export withdraws: 22269 --- --- --- 0 pStaticBGP Static master up 16:56:18 Routes: 62 imported, 0 exported, 62 preferred pWithDRAXXvia2 BGP master up 16:56:27 Established Routes: 687726 imported, 62 exported, 687711 preferred Route change stats: received rejected filtered ignored accepted Import updates: 1528118 0 0 237156 1290962 Import withdraws: 22146 0 --- 0 22146 Export updates: 2231378 2229821 1495 --- 62 Export withdraws: 22269 --- --- --- 0 pWithDRAXXvia3 BGP master up 16:56:26 Established Routes: 687726 imported, 62 exported, 15 preferred Route change stats: received rejected filtered ignored accepted Import updates: 1524020 0 0 239253 1284767 Import withdraws: 22113 0 --- 0 22113 Export updates: 2231378 2229821 1495 --- 62 Export withdraws: 22269 --- --- --- 0 -------- Mensagem original -------- Data: Wed, 03 Jan 2018 14:43:35 +0100 Assunto: Re: System freezes with Bird running on FreeBSD 11.1 Cc: bird-users@network.cz <bird-users@network.cz> Para: Clistenes Viana <cviana@vianet.com.br> De: Ondrej Zajicek <santiago@crfreenet.org> On Wed, Jan 03, 2018 at 11:30:48AM -0200, Clistenes Viana wrote:
Hello Guys,
Let me do a question about bird 1.6.3 to see if you can help me on this, we are trying to running it in a server with freebsd 11.1, but the system freeze every 20 seconds, looking for the logs we found some I/O loop cycle messages like this:
Jan 2 14:03:15 tau bird: I/O loop cycle took 12509 ms for 8 events
Hello How many routes do you have in kernel routing table? How much time it takes to run something like netstat -nr > /dev/null ? --
On Wed, Jan 03, 2018 at 12:02:24PM -0200, Clistenes Viana wrote:
Following the outputs;
# netstat -nr4 | wc -l 688431
# time netstat -nr > /dev/null 3.474u 1.063s 0:04.57 99.1% 183+183k 0+0io 6pf+0w 3.519u 1.196s 0:04.71 99.7% 183+183k 0+0io 0pf+0w 3.468u 0.841s 0:04.31 99.7% 183+183k 0+0io 0pf+0w 3.492u 0.841s 0:04.33 100.0% 183+183k 0+0io 0pf+0w More info about bird;
Hi I tested that on Linux an FreeBSD 10, and for full BGP table (680k routes) the scan took about 1.3 s on Linux and about 2.4 s on FreeBSD 10. Unfortunately i don't have FreeBSD 11 here now to test it. Also it is 'system freeze' or just BIRD freeze? There are two possible issues - logging and memory: Enabling 'debug {routes}' or 'debug all' for Kernel protocol could slow down the scan significantly. Kernel table scan on BSD require a buffer for all routes, so it may require some nontrivial additional memory during routing table scans. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
The System is freezing, it's fast 10 seconds I guess, but it's freezing when scan run. . Also, the debug (routes or all) is disabled in our config. About that you mencioned with relation the nontrivial additional memory, have some place that can I check or provide some usefully information from my system? My Best Regards, Clistenes Viana Network Engineer Vianet Telecomunicacoes e Internet Ltda -------- Mensagem original -------- Data: Wed, 03 Jan 2018 17:31:11 +0100 Assunto: Re: System freezes with Bird running on FreeBSD 11.1 Cc: bird-users@network.cz <bird-users@network.cz> Para: Clistenes Viana <cviana@vianet.com.br> De: Ondrej Zajicek <santiago@crfreenet.org> On Wed, Jan 03, 2018 at 12:02:24PM -0200, Clistenes Viana wrote:
Following the outputs;
# netstat -nr4 | wc -l 688431
# time netstat -nr > /dev/null 3.474u 1.063s 0:04.57 99.1% 183+183k 0+0io 6pf+0w 3.519u 1.196s 0:04.71 99.7% 183+183k 0+0io 0pf+0w 3.468u 0.841s 0:04.31 99.7% 183+183k 0+0io 0pf+0w 3.492u 0.841s 0:04.33 100.0% 183+183k 0+0io 0pf+0w More info about bird;
Hi I tested that on Linux an FreeBSD 10, and for full BGP table (680k routes) the scan took about 1.3 s on Linux and about 2.4 s on FreeBSD 10. Unfortunately i don't have FreeBSD 11 here now to test it. Also it is 'system freeze' or just BIRD freeze? There are two possible issues - logging and memory: Enabling 'debug {routes}' or 'debug all' for Kernel protocol could slow down the scan significantly. Kernel table scan on BSD require a buffer for all routes, so it may require some nontrivial additional memory during routing table scans. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so." --
On Wed, Jan 03, 2018 at 11:30:48AM -0200, Clistenes Viana wrote:
Hello Guys,
Let me do a question about bird 1.6.3 to see if you can help me on this, we are trying to running it in a server with freebsd 11.1, but the system freeze every 20 seconds, looking for the logs we found some I/O loop cycle messages like this:
Jan 2 14:03:15 tau bird: I/O loop cycle took 12509 ms for 8 events
We think that the problem can be have a relationship to the Kernel protocol, when the scan run every 20 seconds to the search for changes and insert in the table, we know that this time can be manipulated by the flag scan time, but it will just to delay the problem and not solves that.
Enabling or not the learn flag also is not making difference to solve the problem.
Hello, We had the similar problem when server had run out of RAM. -- Alexander Shikov Technical Staff, Digital Telecom IX Tel.: +380 44 201 14 07 http://dtel-ix.net/
understood, my server is running with 4GB of ram memory and the bird is consuming something like 400 MB. -------- Mensagem original -------- Data: Mon, 08 Jan 2018 15:04:39 +0200 Assunto: Re: System freezes with Bird running on FreeBSD 11.1 Cc: bird-users@network.cz <bird-users@network.cz> Para: Clistenes Viana <cviana@vianet.com.br> De: Alexander Shikov <a.shikov@dtel-ix.net> On Wed, Jan 03, 2018 at 11:30:48AM -0200, Clistenes Viana wrote:
Hello Guys,
Let me do a question about bird 1.6.3 to see if you can help me on this, we are trying to running it in a server with freebsd 11.1, but the system freeze every 20 seconds, looking for the logs we found some I/O loop cycle messages like this:
Jan 2 14:03:15 tau bird: I/O loop cycle took 12509 ms for 8 events
We think that the problem can be have a relationship to the Kernel protocol, when the scan run every 20 seconds to the search for changes and insert in the table, we know that this time can be manipulated by the flag scan time, but it will just to delay the problem and not solves that.
Enabling or not the learn flag also is not making difference to solve the problem.
Hello, We had the similar problem when server had run out of RAM. -- Alexander Shikov Technical Staff, Digital Telecom IX Tel.: +380 44 201 14 07 http://dtel-ix.net/ --
participants (3)
-
Alexander Shikov -
Clistenes Viana -
Ondrej Zajicek