Bird 1.6.3 cycle-restarting with SegFault during load-testing
Dear all, good afternoon New user, excuses in advance if the question has been answered elsewhere - despite not having found anything on the mailing list history. For my Master thesis, I am investigating the impact of configuration changes resulting from IRRDB updates on route server infrastructure. During my experiments I got a segfault from Bird, and wanted to ask a few questions about it. The details of the problem are below. Please let me know if you miss anything, and thank you in advance, David ** Problem description: For load testing, I am having my Bird route server instance peer with other 1100 peers (containers, basic bgp config). CPU utilization and Memory are normal (~5%/30% respectively) and I see BGP session establishments go above 1000 sessions without problems, but at 1013 consistently Bird raises seg-fault and restarts. Output shows: 2019-06-27 12:37:49 <ERR> BGP: Error on listening socket: Too many open files Which seem to be related to dynamic buffers running out due to BGP open sessions. Bird version: 1.6.3 CPU details: Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, 4 cores). ** Actions taken: - Resource dump available, also: repeated tests to verify the restart always happen at 1013 bgp sessions. ** Questions: - Is the issue I just described known? - Are there (known) hard-coded limits on Bird? What is the expected behaviour under these conditions? - In general, what are other dimensioning variables to be taken into account with Bird? ** Logs: Zip file containing: resource dump, count of bgp sessions, further analysis. https://we.tl/t-mRmgIgC38a
Hello! Please check your filedescriptor limit by ulimit -n and check how many files are open by lsof -p <BIRD's pid> BIRD has no such limit, it is capable of handling any reasonable (fitting in memory) number of file descriptors. If you wanted any other help, feel free to ask more. BIRD should have no hard limits on scalability, yet some parts of it may be unexpectedly slow in extreme conditions. Maria On June 30, 2019 6:31:53 PM GMT+02:00, David.Garay@os3.nl wrote:
Dear all, good afternoon
New user, excuses in advance if the question has been answered elsewhere - despite not having found anything on the mailing list history.
For my Master thesis, I am investigating the impact of configuration changes resulting from IRRDB updates on route server infrastructure. During my experiments I got a segfault from Bird, and wanted to ask a few questions about it.
The details of the problem are below. Please let me know if you miss anything, and thank you in advance,
David
** Problem description: For load testing, I am having my Bird route server instance peer with other 1100 peers (containers, basic bgp config).
CPU utilization and Memory are normal (~5%/30% respectively) and I see BGP session establishments go above 1000 sessions without problems, but
at 1013 consistently Bird raises seg-fault and restarts.
Output shows:
2019-06-27 12:37:49 <ERR> BGP: Error on listening socket: Too many open
files
Which seem to be related to dynamic buffers running out due to BGP open
sessions.
Bird version: 1.6.3 CPU details: Intel(R) Xeon(R) CPU E3-1220L V2 @ 2.30GHz, 4 cores).
** Actions taken: - Resource dump available, also: repeated tests to verify the restart always happen at 1013 bgp sessions.
** Questions: - Is the issue I just described known? - Are there (known) hard-coded limits on Bird? What is the expected behaviour under these conditions? - In general, what are other dimensioning variables to be taken into account with Bird?
** Logs: Zip file containing: resource dump, count of bgp sessions, further analysis. https://we.tl/t-mRmgIgC38a
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
participants (2)
-
David.Garay＠os3.nl -
Maria Matějka