Hello,
we hit a crash in BIRD 3.3.3 on a production edge router with full
BGP tables.
Version: 3.3.3-cznic.1~trixie (Debian trixie, packages from
pkg.labs.nic.cz)
Setup: ~1.08M IPv4 routes in master4, several eBGP full feeds +
iBGP,
exported to the kernel via the Kernel protocol.
Kernel protocol config:
protocol kernel KernelRIB {
learn;
persist;
scan time 20;
netlink rx buffer 67108864;
ipv4 {
import all;
export filter { krt_prefsrc = <loopback IP>;
accept; };
};
}
Symptoms:
1) Repeated warnings, always on the KernelRIB.ipv4 channel,
aligned with
the 20 s scan interval (every 20-60 min):
KernelRIB.ipv4: Export flag inconsistency at
nest/rt-table.c:1452,
route id 307495. If frequent, please tell the developers.
2) Eventually BIRD aborted:
Assertion 'export_in_map' failed at nest/rt-table.c:3059
3) After restart the warnings continue at the same rate, so we
expect
the crash to recur.
Backtrace (with bird3-dbgsym 3.3.3), main thread:
#2 abort ()
#3 bug (msg="Assertion '%s' failed at %s:%d") at
sysdep/unix/log.c:446
#4 rt_feed_net_best (... seen=0x5620b0f02b60, f=0x5620b0f02af8,
_first=0x0)
at nest/rt-table.c:3059
#5 rt_export_get_next_feed (...) at nest/rt-export.c:273
#6 rt_export_next_feed (...) at nest/rt-export.c:317
#7 rt_export_get (r=...) at nest/rt-export.c:103 es =
TES_PARTIAL
#8 channel_notify_optimal_req (c=..., req=...) at
nest/rt-table.c:1478
#9 channel_notify_optimal (_channel=...) at
nest/rt-table.c:1591
c->name = "ipv4", c->proto->name = "KernelRIB"
Observations:
- The KernelRIB channel's "Export updates" counter grows by
~54k/s,
i.e. the whole table (~1.08M routes) every 20 s scan. Meanwhile
"ip monitor route" shows only ~17 changes/s to the kernel, and
kernel
routes match BIRD's best routes. So the per-scan full
re-evaluation
appears to be internal, and it exercises the partial-feed path
(TES_PARTIAL) where the assertion fires.
- The fix "Nest: Check that there is a best route in
rt_net_feed_index()"
(828f9e8) is already included in 3.3.3, so this looks like a
different
issue. We found nothing related in the 3.4.0 changelog.
Full backtrace (thread apply all bt full) is attached. We still
have the
core dump and can run gdb commands on request.
Is this a known issue? Is there a workaround other than increasing
the
scan time or going back to 2.x?
Thank you,
Przemyslaw Swiderski