segfault on running "show mem" command with bird-3.0alpha2
Hi All, I am seeing this segfault with bird-3.0alpha2 on trying to run the "show mem" command. Not sure if it is a known issue. Just thought of reporting it. By the way, bird-3.0 though in alpha, looks really stable. Was able to scale upto 1K BGP Sessions without any major issues. Eagerly looking for a 3.0 stable release soon. Regards Prem Thread 1 "bird" received signal SIGSEGV, Segmentation fault. 0x00005555555841c3 in rmemsize (res=<optimized out>) at lib/resource.c:271 271 if (!r->class->memsize) (gdb) bt #0 0x00005555555841c3 in rmemsize (res=<optimized out>) at lib/resource.c:271 #1 cmd_show_memory () at nest/cmds.c:122 #2 0x000055555560c03e in cf_parse.isra.0 () at nest/config.Y:640 #3 0x000055555556a523 in cli_parse (c=0x7fffffffd920) at conf/conf.c:179 #4 0x00005555555837fd in cli_command (c=c@entry=0x5555556bb338) at nest/cli.c:273 #5 0x0000555555583b3c in cli_event (data=0x5555556bb338) at nest/cli.c:302 #6 0x00005555555786cb in ev_run_list_limited (l=0x555555681000 <global_event_list>, limit=4294967294) at lib/event.c:331 #7 0x0000555555566e14 in io_loop () at sysdep/unix/io.c:2216 #8 main (argc=<optimized out>, argv=<optimized out>) at sysdep/unix/main.c:972 (gdb) f 1 #1 cmd_show_memory () at nest/cmds.c:122 122 struct resmem total = rmemsize(&root_pool); (gdb) print root_pool $3 = {r = {n = {next = 0x0, prev = 0x0, list = 0x0}, class = 0x0}, inside = {first = 0x555555685360, last = 0x5555556bb630}, domain = 0x555555680e40 <the_bird_domain_gen.lto_priv>, name = 0x555555656735 "Root"} (gdb) print root_pool->r->class $5 = (const struct resclass *) 0x0 (gdb)
Hello Prem, Could you please try out the current thread-nextbranch, i.e. this one? https://gitlab.nic.cz/labs/bird/-/tree/thread-next/ This is the development branch which will become 3.0alpha3 soon and if your report is still relevant for this branch, I'll try to fix it before releasing. It's really a pleasure to read that v3 is stable with 1K connections without major issues outside our testing setup. BTW in our testing setup, we managed to run it with 5K connections as well and I suppose we could handle at least 10K … just if I was't so lazy to setup larger tests. It's actually a good question how many connections we can handle at once. With v2, the architectural limitation occurs somewhere around 1.5K BGP connections, but with v3 and e.g. 32 threads, I would suppose that 40K connections should still be OK-ish. We should do such testing in future, just for fun, to find out how far we can stretch BIRD with the new internal architecture. Also, and especially if you are our happy user on such a large scale, please consider getting our BIRD Support Package to get exclusive services (config optimization, training, custom development …) and also to ensure continuous BIRD development in the future. See https://bird.nic.cz/en/commercial-services/and/or send me an off-list e-mail to get a customized quote. Thank you for reporting. Have a nice day! Maria On 2024-02-26 16:45, Prem Anand wrote:
Hi All, I am seeing this segfault with bird-3.0alpha2 on trying to run the "show mem" command. Not sure if it is a known issue. Just thought of reporting it.
By the way, bird-3.0 though in alpha, looks really stable. Was able to scale upto 1K BGP Sessions without any major issues. Eagerly looking for a 3.0 stable release soon.
Regards Prem
Thread 1 "bird" received signal SIGSEGV, Segmentation fault. 0x00005555555841c3 in rmemsize (res=<optimized out>) at lib/resource.c:271 271 if (!r->class->memsize) (gdb) bt #0 0x00005555555841c3 in rmemsize (res=<optimized out>) at lib/resource.c:271 #1 cmd_show_memory () at nest/cmds.c:122 #2 0x000055555560c03e in cf_parse.isra.0 () at nest/config.Y:640 #3 0x000055555556a523 in cli_parse (c=0x7fffffffd920) at conf/conf.c:179 #4 0x00005555555837fd in cli_command (c=c@entry=0x5555556bb338) at nest/cli.c:273 #5 0x0000555555583b3c in cli_event (data=0x5555556bb338) at nest/cli.c:302 #6 0x00005555555786cb in ev_run_list_limited (l=0x555555681000 <global_event_list>, limit=4294967294) at lib/event.c:331 #7 0x0000555555566e14 in io_loop () at sysdep/unix/io.c:2216 #8 main (argc=<optimized out>, argv=<optimized out>) at sysdep/unix/main.c:972 (gdb) f 1 #1 cmd_show_memory () at nest/cmds.c:122 122 struct resmem total = rmemsize(&root_pool); (gdb) print root_pool $3 = {r = {n = {next = 0x0, prev = 0x0, list = 0x0}, class = 0x0}, inside = {first = 0x555555685360, last = 0x5555556bb630}, domain = 0x555555680e40 <the_bird_domain_gen.lto_priv>, name = 0x555555656735 "Root"} (gdb) print root_pool->r->class $5 = (const struct resclass *) 0x0 (gdb)
-- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
Begin forwarded message:
From: Maria Matejka <maria.matejka@nic.cz> Subject: Re: segfault on running "show mem" command with bird-3.0alpha2 Date: 26 February 2024 at 16:43:03 GMT To: Prem Anand <h.prem.anand@gmail.com> Cc: bird-users@network.cz
Hello Prem, Could you please try out the current thread-next branch, i.e. this one? https://gitlab.nic.cz/labs/bird/-/tree/thread-next/ This is the development branch which will become 3.0alpha3 soon and if your report is still relevant for this branch, I'll try to fix it before releasing.
Hi Maria, I don't see this segfault issue with thread-next branch. So I assume this got fixed somewhere along the way :) $ birdc -s ./bird.sock.ctl BIRD v2.13.1-164-ge2728c80 ready. bird> show mem BIRD memory usage Effective Overhead Routing tables: 19.2 kB 1360.0 B Route attributes: 816.0 B 5288.0 B Protocols: 416.0 B 312.0 B Current config: 8768.0 B 448.0 B Standby memory: 0.0 B 60.0 kB Total: 104.8 kB 71.6 kB bird>
It's really a pleasure to read that v3 is stable with 1K connections without major issues outside our testing setup. BTW in our testing setup, we managed to run it with 5K connections as well and I suppose we could handle at least 10K … just if I was't so lazy to setup larger tests. It's actually a good question how many connections we can handle at once. With v2, the architectural limitation occurs somewhere around 1.5K BGP connections, but with v3 and e.g. 32 threads, I would suppose that 40K connections should still be OK-ish. We should do such testing in future, just for fun, to find out how far we can stretch BIRD with the new internal architecture.
Good to know that we can got all the way upto 40K connections. That's really impressive.
Also, and especially if you are our happy user on such a large scale, please consider getting our BIRD Support Package to get exclusive services (config optimization, training, custom development …) and also to ensure continuous BIRD development in the future. See https://bird.nic.cz/en/commercial-services/ and/or send me an off-list e-mail to get a customized quote.
I am currently just evaluating TRex + Bird integration to fit into our workflow. Once we decide to switch to use this in production, I would convince my company to go up with a commercial service. Regards Prem
Thank you for reporting. Have a nice day! Maria On 2024-02-26 16:45, Prem Anand wrote:
Hi All, I am seeing this segfault with bird-3.0alpha2 on trying to run the "show mem" command. Not sure if it is a known issue. Just thought of reporting it.
By the way, bird-3.0 though in alpha, looks really stable. Was able to scale upto 1K BGP Sessions without any major issues. Eagerly looking for a 3.0 stable release soon.
Regards Prem
Thread 1 "bird" received signal SIGSEGV, Segmentation fault. 0x00005555555841c3 in rmemsize (res=<optimized out>) at lib/resource.c:271 271 if (!r->class->memsize) (gdb) bt #0 0x00005555555841c3 in rmemsize (res=<optimized out>) at lib/resource.c:271 #1 cmd_show_memory () at nest/cmds.c:122 #2 0x000055555560c03e in cf_parse.isra.0 () at nest/config.Y:640 #3 0x000055555556a523 in cli_parse (c=0x7fffffffd920) at conf/conf.c:179 #4 0x00005555555837fd in cli_command (c=c@entry=0x5555556bb338) at nest/cli.c:273 #5 0x0000555555583b3c in cli_event (data=0x5555556bb338) at nest/cli.c:302 #6 0x00005555555786cb in ev_run_list_limited (l=0x555555681000 <global_event_list>, limit=4294967294) at lib/event.c:331 #7 0x0000555555566e14 in io_loop () at sysdep/unix/io.c:2216 #8 main (argc=<optimized out>, argv=<optimized out>) at sysdep/unix/main.c:972 (gdb) f 1 #1 cmd_show_memory () at nest/cmds.c:122 122 struct resmem total = rmemsize(&root_pool); (gdb) print root_pool $3 = {r = {n = {next = 0x0, prev = 0x0, list = 0x0}, class = 0x0}, inside = {first = 0x555555685360, last = 0x5555556bb630}, domain = 0x555555680e40 <the_bird_domain_gen.lto_priv>, name = 0x555555656735 "Root"} (gdb) print root_pool->r->class $5 = (const struct resclass *) 0x0 (gdb)
-- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
participants (2)
-
Maria Matejka -
Prem Anand