Hello all - I just compiled and installed BIRD 2.0.0 on a x86_64 Ubuntu 16.04.3 LTS (with ZFS root) test-machine and (most) things look pretty dandy so far. Though ... birdc crashes inside lib-readline when I attempt to go back in the command history (^P or ↑), but only if there already is something to go back to. It also fails to create the .birdc_history file in the first place. Thankfully the 1.6.3 birdc still works fine, so I can continue playing for now. Has anyone else found this as well already? Otherwise I'd dig deeper (would have to install debug tools on the target machine, etc.) Thanks, Clemens PS: Great Work, Team!!! I really appreciate your efforts!
On Tue, Dec 12, 2017 at 04:00:23PM +0100, Clemens Schrimpe wrote:
Hello all -
I just compiled and installed BIRD 2.0.0 on a x86_64 Ubuntu 16.04.3 LTS (with ZFS root) test-machine and (most) things look pretty dandy so far.
Though ... birdc crashes inside lib-readline when I attempt to go back in the command history (^P or ↑), but only if there already is something to go back to. It also fails to create the .birdc_history file in the first place. Thankfully the 1.6.3 birdc still works fine, so I can continue playing for now.
Has anyone else found this as well already? Otherwise I'd dig deeper (would have to install debug tools on the target machine, etc.)
Hi Thanks for the testing and report. It works for us on several platforms, from Debian to NetBSD. Note that .birdc_history is created during exit from birdc, not during start of birdc. Do you get .birdc_history if you use new birdc, do some commands and exit from it without crashing it? What version of readline and history libraries do you have? -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Note that .birdc_history is created during exit from birdc, not during start of birdc. Do you get .birdc_history if you use new birdc, do some commands and exit from it without crashing it?
Yes, works fine. And it also appears to be read when I restart birdc, because now I can crash it with ^P or ↑ without having typed a command before. (strace confirmed: .birdc_history is read successfully)
What version of readline and history libraries do you have?
~/builds/bird-2.0.0> ldd birdc linux-vdso.so.1 => (0x00007fff445fa000) libhistory.so.6 => /lib/x86_64-linux-gnu/libhistory.so.6 (0x00007f41f2bfd000) libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007f41f29b7000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f41f278d000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f41f2570000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f41f21a6000) /lib64/ld-linux-x86-64.so.2 (0x000055d339f07000) /var/log/syslog has: Dec 12 16:45:37 kernel: birdc[6756]: segfault at 10 ip 00007f0243394a0b sp 00007ffec5ef7df8 error 4 in libreadline.so.6.3[7f0243366000+3d000] Groetjes, Clemens
On Tue, Dec 12, 2017 at 04:52:42PM +0100, Clemens Schrimpe wrote:
Note that .birdc_history is created during exit from birdc, not during start of birdc. Do you get .birdc_history if you use new birdc, do some commands and exit from it without crashing it?
Yes, works fine. And it also appears to be read when I restart birdc, because now I can crash it with ^P or ↑ without having typed a command before. (strace confirmed: .birdc_history is read successfully)
You can try attached patch (disables reading of history). You can also try to build BIRD 1.6.3 in the same way like 2.0.0 and see if it by chance does not express the same problem. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Yes, works fine. And it also appears to be read when I restart birdc, because now I can crash it with ^P or ↑ without having typed a command before. (strace confirmed: .birdc_history is read successfully)
You can try attached patch (disables reading of history).
... to no avail ... still crashes the same way.
You can also try to build BIRD 1.6.3 in the same way like 2.0.0 and see if it by chance does not express the same problem.
No, it doesn't (I built the 2.0.0 the same way as the 1.6.3 - and all the others). Running the 2.0.0 client inside gdb yields: (I installed libreadline6-dbg before testing) bird> Program received signal SIGSEGV, Segmentation fault. 0x00007ffff79b6a0b in previous_history () at /build/readline6-RKA9OI/readline6-6.3/history.c:185 185 /build/readline6-RKA9OI/readline6-6.3/history.c: No such file or directory. (gdb) bt #0 0x00007ffff79b6a0b in previous_history () at /build/readline6-RKA9OI/readline6-6.3/history.c:185 #1 0x00007ffff79b65e5 in rl_get_previous_history (count=<optimized out>, key=<optimized out>) at /build/readline6-RKA9OI/readline6-6.3/misc.c:609 #2 0x00007ffff799b990 in _rl_dispatch_subseq (key=65, map=<optimized out>, got_subseq=0) at /build/readline6-RKA9OI/readline6-6.3/readline.c:832 #3 0x00007ffff799c202 in _rl_dispatch_callback (cxt=0x61cb10) at /build/readline6-RKA9OI/readline6-6.3/readline.c:736 #4 0x00007ffff79b27ff in rl_callback_read_char () at /build/readline6-RKA9OI/readline6-6.3/callback.c:188 #5 0x0000000000403885 in input_read () at client/birdc.c:219 #6 0x00000000004020d3 in select_loop () at client/client.c:375 #7 main (argc=<optimized out>, argv=<optimized out>) at client/client.c:447 Maybe this provides a hint?! And here's another interesting datapoint: Since I use BIRD to replace the dreadful Quagga-derivate Ubiquiti delivers with their EdgeRouters I compiled BIRD-2.0.0 for the MIPS platform and there it does not exhibit the observed behavior! Same libreadline version, though: csch# ldd ./birdc-2.0.0 librt.so.1 => /lib/mips-linux-gnu/librt.so.1 (0x77118000) libhistory.so.6 => /lib/mips-linux-gnu/libhistory.so.6 (0x770ff000) libreadline.so.6 => /lib/mips-linux-gnu/libreadline.so.6 (0x770b5000) libtinfo.so.5 => /lib/mips-linux-gnu/libtinfo.so.5 (0x77087000) libpthread.so.0 => /lib/mips-linux-gnu/libpthread.so.0 (0x7705c000) libc.so.6 => /lib/mips-linux-gnu/libc.so.6 (0x76ee1000) /lib/ld.so.1 (0x77136000) Maybe I should download the src for libreadline6 to look at history.c, line 185 ;-) Hälsningar - Clemens
❦ 13 décembre 2017 22:56 +0100, Clemens Schrimpe <clemens.schrimpe@gmail.com> :
bird> Program received signal SIGSEGV, Segmentation fault. 0x00007ffff79b6a0b in previous_history () at /build/readline6-RKA9OI/readline6-6.3/history.c:185 185 /build/readline6-RKA9OI/readline6-6.3/history.c: No such file or directory. (gdb) bt #0 0x00007ffff79b6a0b in previous_history () at /build/readline6-RKA9OI/readline6-6.3/history.c:185 #1 0x00007ffff79b65e5 in rl_get_previous_history (count=<optimized out>, key=<optimized out>) at /build/readline6-RKA9OI/readline6-6.3/misc.c:609 #2 0x00007ffff799b990 in _rl_dispatch_subseq (key=65, map=<optimized out>, got_subseq=0) at /build/readline6-RKA9OI/readline6-6.3/readline.c:832 #3 0x00007ffff799c202 in _rl_dispatch_callback (cxt=0x61cb10) at /build/readline6-RKA9OI/readline6-6.3/readline.c:736 #4 0x00007ffff79b27ff in rl_callback_read_char () at /build/readline6-RKA9OI/readline6-6.3/callback.c:188 #5 0x0000000000403885 in input_read () at client/birdc.c:219 #6 0x00000000004020d3 in select_loop () at client/client.c:375 #7 main (argc=<optimized out>, argv=<optimized out>) at client/client.c:447
Maybe this provides a hint?!
Use "bt full". What's "p the_history" and "p the_history[0]"? -- Having nothing, nothing can he lose. -- William Shakespeare, "Henry VI"
Use "bt full".
What’s "p the_history" and "p the_history[0]"?
Looks legit: Program received signal SIGSEGV, Segmentation fault. 0x00007ffff79b6a0b in previous_history () at /build/readline6-RKA9OI/readline6-6.3/history.c:185 185 /build/readline6-RKA9OI/readline6-6.3/history.c: No such file or directory. (gdb) bt full #0 0x00007ffff79b6a0b in previous_history () at /build/readline6-RKA9OI/readline6-6.3/history.c:185 No locals. #1 0x00007ffff79b65e5 in rl_get_previous_history (count=<optimized out>, key=<optimized out>) at /build/readline6-RKA9OI/readline6-6.3/misc.c:609 old_temp = 0x0 temp = 0x0 key = <optimized out> count = 1 #2 0x00007ffff799b990 in _rl_dispatch_subseq (key=65, map=<optimized out>, got_subseq=0) at /build/readline6-RKA9OI/readline6-6.3/readline.c:832 r = 0 newkey = <optimized out> func = <optimized out> cxt = <optimized out> #3 0x00007ffff799c202 in _rl_dispatch_callback (cxt=0x61cb10) at /build/readline6-RKA9OI/readline6-6.3/readline.c:736 nkey = <optimized out> r = <optimized out> #4 0x00007ffff79b27ff in rl_callback_read_char () at /build/readline6-RKA9OI/readline6-6.3/callback.c:188 line = <optimized out> eof = <optimized out> jcode = <optimized out> olevel = {{__jmpbuf = {6319136, 7997659007791416985, -1, 0, 4212971, 6319559, 7997659007814485657, 7997676843090331289}, __mask_was_saved = 0, __saved_mask = {__val = {0 <repeats 16 times>}}}} #5 0x0000000000403885 in input_read () at client/birdc.c:219 No locals. #6 0x00000000004020d3 in select_loop () at client/client.c:375 select_fds = {__fds_bits = {1, 0 <repeats 15 times>}} rv = <optimized out> #7 main (argc=<optimized out>, argv=<optimized out>) at client/client.c:447 No locals. (gdb) p the_history $1 = (HIST_ENTRY **) 0x609470 (gdb) p the_history[0] $2 = (HIST_ENTRY *) 0x609630 (gdb) p *(the_history[0]) $3 = {line = 0x609650 „show protocols „, timestamp = 0x609610 "", data = 0x0} Double-plus-weird! I keep digging … Clemens
❦ 14 décembre 2017 13:13 +0100, Clemens Schrimpe <clemens.schrimpe@gmail.com> :
(gdb) p the_history $1 = (HIST_ENTRY **) 0x609470 (gdb) p the_history[0] $2 = (HIST_ENTRY *) 0x609630 (gdb) p *(the_history[0]) $3 = {line = 0x609650 „show protocols „, timestamp = 0x609610 "", data = 0x0}
Double-plus-weird!
And "p history_offset"? Should be 0. -- Replace repetitive expressions by calls to a common function. - The Elements of Programming Style (Kernighan & Plauger)
And "p history_offset"? Should be 0.
Nope - it is 2: Program received signal SIGSEGV, Segmentation fault. 0x00007ffff79b6a0b in previous_history () at /build/readline6-RKA9OI/readline6-6.3/history.c:185 185 /build/readline6-RKA9OI/readline6-6.3/history.c: No such file or directory. (gdb) p history_offset $1 = 2 (gdb) p *(the_history[2]) Cannot access memory at address 0x10 We are up to something! :-) -c PS: I did: Invoke birdc, „show proto\n“, either ^P or ↑ → 💣
❦ 14 décembre 2017 16:48 +0100, Clemens Schrimpe <clemens.schrimpe@gmail.com> :
Nope - it is 2:
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff79b6a0b in previous_history () at /build/readline6-RKA9OI/readline6-6.3/history.c:185 185 /build/readline6-RKA9OI/readline6-6.3/history.c: No such file or directory. (gdb) p history_offset $1 = 2 (gdb) p *(the_history[2]) Cannot access memory at address 0x10
We are up to something! :-)
-c
You could start birdc from gdb: gbd --args birdc Then, add a watchpoint on history_offset: watch history_offset Then "run". Each time gdb stops because history_offset changed, grab a "bt". Maybe it's not initialized to 0 properly? It should be increased by add_history calls. I don't see anything suspicious in bird about all that (it's all handled by libreadline itself). -- Make input easy to proofread. - The Elements of Programming Style (Kernighan & Plauger)
On Thu, Dec 14, 2017 at 04:48:37PM +0100, Clemens Schrimpe wrote:
And "p history_offset"? Should be 0.
Nope - it is 2:
Program received signal SIGSEGV, Segmentation fault. 0x00007ffff79b6a0b in previous_history () at /build/readline6-RKA9OI/readline6-6.3/history.c:185 185 /build/readline6-RKA9OI/readline6-6.3/history.c: No such file or directory. (gdb) p history_offset $1 = 2 (gdb) p *(the_history[2]) Cannot access memory at address 0x10
We are up to something! :-)
Hi Perhaps it is just missing initialization? Please try attached patch. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Wed, Dec 13, 2017 at 10:56:49PM +0100, Clemens Schrimpe wrote:
Yes, works fine. And it also appears to be read when I restart birdc, because now I can crash it with ^P or ↑ without having typed a command before. (strace confirmed: .birdc_history is read successfully)
You can try attached patch (disables reading of history).
... to no avail ... still crashes the same way.
You can also try to build BIRD 1.6.3 in the same way like 2.0.0 and see if it by chance does not express the same problem.
No, it doesn't (I built the 2.0.0 the same way as the 1.6.3 - and all the others).
Hi What is output of ldd birdc for 1.6.3? I guess that the problem is not related to BIRD code itself but to the change in libraries, as we changed the order of detected libraries. Could you try to build and test GIT master branch (which is 1.6.x, but also with these changes as in 2.0)? Perhaps there is some problem with combination of libhistory and libtinfo on your setup, while libhistory and libncurses works fine. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On 14.12.2017, at 14:18, Ondrej Zajicek <santiago@crfreenet.org> wrote:
What is output of ldd birdc for 1.6.3?
# for 2.0.0 newnoc:~/builds/bird-2.0.0# ldd birdc linux-vdso.so.1 => (0x00007ffeb07dd000) libhistory.so.6 => /lib/x86_64-linux-gnu/libhistory.so.6 (0x00007fcab708d000) libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007fcab6e47000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fcab6c1d000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fcab6a00000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcab6636000) /lib64/ld-linux-x86-64.so.2 (0x000055a00f20b000) # for 1.6.3 newnoc:~/builds/bird-2.0.0# ldd /usr/local/sbin/birdc linux-vdso.so.1 => (0x00007ffcbb5d2000) libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007f8e4bd8c000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f8e4bb63000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8e4b945000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8e4b57b000) /lib64/ld-linux-x86-64.so.2 (0x0000557f92c50000) Salutations - Clemens
On Thu, Dec 14, 2017 at 04:50:20PM +0100, Clemens Schrimpe wrote:
On 14.12.2017, at 14:18, Ondrej Zajicek <santiago@crfreenet.org> wrote:
What is output of ldd birdc for 1.6.3?
# for 2.0.0 newnoc:~/builds/bird-2.0.0# ldd birdc linux-vdso.so.1 => (0x00007ffeb07dd000) libhistory.so.6 => /lib/x86_64-linux-gnu/libhistory.so.6 (0x00007fcab708d000) libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007fcab6e47000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007fcab6c1d000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fcab6a00000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcab6636000) /lib64/ld-linux-x86-64.so.2 (0x000055a00f20b000)
# for 1.6.3 newnoc:~/builds/bird-2.0.0# ldd /usr/local/sbin/birdc linux-vdso.so.1 => (0x00007ffcbb5d2000) libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007f8e4bd8c000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f8e4bb63000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8e4b945000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8e4b57b000) /lib64/ld-linux-x86-64.so.2 (0x0000557f92c50000)
Hi It seems like history functions are both in independent libhistory and as a part of libreadline. We link birdc to both libraries, which possibly caused some confusion (readline internally called its history functions, while birdc uses functions from libhistory). The same problem is described here: https://lists.gnu.org/archive/html/bug-readline/2015-10/msg00000.html Could you try to build bird/birdc (without any patches i sent before) using: make CLIENT_LIBS="-lreadline -ltinfo" ? -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Could you try to build bird/birdc (without any patches i sent before) using:
make CLIENT_LIBS="-lreadline -ltinfo"
?
And indeed, this fixes it and persistent history still works! \o/ Clemens PS: for your records (again) - this machine run Ubuntu 16.04.3 LTS in a (mostly) vanilla fashion. I just installed libreadline6-dbg to help with this case. So the same problem/solution *should* apply on other machines as well!?
On Fri, Dec 15, 2017 at 04:16:31PM +0100, Clemens Schrimpe wrote:
Could you try to build bird/birdc (without any patches i sent before) using:
make CLIENT_LIBS="-lreadline -ltinfo"
?
And indeed, this fixes it and persistent history still works!
PS: for your records (again) - this machine run Ubuntu 16.04.3 LTS in a (mostly) vanilla fashion. I just installed libreadline6-dbg to help with this case. So the same problem/solution *should* apply on other machines as well!?
Yes, the bad linking is on most other systems, it just does not mainfest on them. Thanks to all for debugging the issue. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
participants (3)
-
Clemens Schrimpe -
Ondrej Zajicek -
Vincent Bernat