Hi,
We've had an occasional issue where we found BIRD stuck using 100% CPU a few
times over the past couple of years. Whenever it happened, all protocols
stopped running and birdc was unable to communicate with the process.
Unfortunately, it was usually in a situation where we could not quite see what
was going so we ended up having to kill the process and restart it to get
things working again.
However, today it finally happened again, and I was able to grab the debug
symbols for the build so I could inspect it in gdb.
So here is the backtrace:
(gdb) bt
#0 if_recalc_preferred (i=0x56541cd48de0) at nest/iface.c:518
#1 0x000056541c662d55 in if_end_partial_update (i=i@entry=0x56541cd48de0) at nest/iface.c:359
#2 0x000056541c6b19cb in nl_parse_addr4 (new=1, scan=0, i=0x56541cc20700) at sysdep/linux/netlink.c:1020
#3 nl_parse_addr (h=h@entry=0x56541cc206f0, scan=scan@entry=0) at sysdep/linux/netlink.c:1128
#4 0x000056541c6b3127 in nl_async_msg (h=0x56541cc206f0) at sysdep/linux/netlink.c:1932
#5 nl_async_hook (sk=<optimized out>, size=<optimized out>) at sysdep/linux/netlink.c:1985
#6 0x000056541c6b74a0 in sk_read (s=s@entry=0x56541cc22700, revents=<optimized out>) at sysdep/unix/io.c:1896
#7 0x000056541c6b80ec in io_loop () at sysdep/unix/io.c:2345
#8 0x000056541c633f61 in main (argc=<optimized out>, argv=<optimized out>) at sysdep/unix/main.c:939
If I keep hitting next, it appears to be stuck in a loop:
(gdb) n
524 nest/iface.c: No such file or directory.
(gdb) n
514 in nest/iface.c
(gdb) n
517 in nest/iface.c
(gdb) n
518 in nest/iface.c
(gdb) n
524 in nest/iface.c
(gdb) n
514 in nest/iface.c
(gdb) n
517 in nest/iface.c
(gdb) n
518 in nest/iface.c
(gdb) n
524 in nest/iface.c
(gdb) n
514 in nest/iface.c
So here is what that code looks like:
510 struct ifa *a;
511 WALK_LIST(a, i->addrs)
512 {
513 /* Secondary address is never selected */
514 if (a->flags & IA_SECONDARY)
515 continue;
516
517 if (ipa_is_ip4(a->ip)) {
518 if (!a4 || ipa_equal(a->ip, pref_v4))
519 a4 = a;
520 } else if (!ipa_is_link_local(a->ip)) {
521 if (!a6 || ipa_equal(a->ip, ic->pref_v6))
522 a6 = a;
523 } else {
524 if (!ll || ipa_equal(a->ip, ic->pref_ll))
525 ll = a;
526 }
527 }
OK, we are iterating over a linked link. Let's look at the list:
(gdb) p i->addrs
$30 = {{head_node = {next = 0x56541cd02b30, prev = 0x0}, head_padding = 0x56541cd02b30}, {tail_padding = 0x56541cd02b30, tail_node = {next = 0x0, prev = 0x56541cd02b30}}, {head = 0x56541cd02b30, null = 0x0,
tail = 0x56541cd02b30}}
(gdb) p i->addrs.head
$31 = (struct node *) 0x56541cd02b30
(gdb) p *i->addrs.head
$32 = {next = 0x56541cd02b30, prev = 0x56541cd02b30}
So there is a single item, with next pointing to itself.
Looking up the stack, it seems the only place here where this list is modified
is when nl_parse_addr4() calls ifa_update() which then calls add_tail(), which
looks like this:
EXPENSIVE_CHECK(check_list(l, NULL));
ASSUME(n->prev == NULL);
ASSUME(n->next == NULL);
node *z = l->tail;
n->next = &l->tail_node;
n->prev = z;
z->next = n;
l->tail = n;
So this is where I am stumped. How could it even be possible for the item
next pointer to be set to it's own address, and not that of the static tail
node?
Any insights?
--
James Oakley
james(a)ttgi.io
