Bird is not adding routes into kernel anymore

26 Dec 2023

      Good morning,

yesterday we upgrade in a lab from 2.13.1 to 2.14 and (great idea) at
the same time modified some configuration bits. Since then bird does not
seem to write routes into the kernel anymore, even though relevant bits
seem to be in the configuration.

The config (in parts) looks like this:

--------------------------------------------------------------------------------
protocol device { }

protocol kernel kernel_v4 {
        ipv4 { export all; };
}

protocol kernel kernel_v6 {
        ipv6 { export all; };
}

...
protocol static {
    ipv6;
    route 2a0a:5480::/48 unreachable;   # LAN
    route 2a0a:5480:1::/48 unreachable; # k8s
    route 2a0a:5480:3::/48 unreachable; # router specific
    route 2a0a:5480:0:c001::/96 via 2a0a:5480::20d:b9ff:fe5a:816c; # jool
}

protocol static {
    ipv4;
    route  185.203.113.0/24 unreachable;
}

protocol bfd {
  # openvpn
  interface "tun*" {
    interval 500 ms;
    idle tx interval 2 s;
  };
  # transfer
  interface "eth0" {
    interval 500 ms;
    idle tx interval 2 s;
  };
}

--------------------------------------------------------------------------------

Plus various BGP instances at the end.

If I check birdc, things seems to be good:

--------------------------------------------------------------------------------
bird> show route
Table master4:
185.203.114.0/23     unicast [server121_v6_v4 22:24:45.009] * (100) [AS213081i]
        via 2a0a:e5c0:32:4::1 on tun1
185.203.113.2/32     unicast [r2ge_v6_v4 22:35:40.181 from 2a0a:5480::2] * (100/10) [i]
        via fe80::20d:b9ff:fe46:3bd4 on eth0
185.203.113.0/24     unreachable [static2 22:12:35.949] * (200)
                     unicast [r2ge_v6_v4 22:35:40.181 from 2a0a:5480::2] (100/10) [i]
        via fe80::20d:b9ff:fe46:3bd4 on eth0
148.222.248.1/32     unicast [server121_v6_v4 09:39:56.279] * (100) [AS209898i]
        via 2a0a:e5c0:32:4::1 on tun1
                     unicast [r2ge_v6_v4 09:39:57.048 from 2a0a:5480::2] (100/20) [AS209898i]
        via fe80::20d:b9ff:fe46:3bd4 on eth0
...
bird> show protocols
Name       Proto      Table      State  Since         Info
device1    Device     ---        up     22:12:35.949
kernel_v4  Kernel     master4    up     22:12:35.949
kernel_v6  Kernel     master6    up     22:12:35.949
static1    Static     master6    up     22:12:35.949
static2    Static     master4    up     22:12:35.949
bfd1       BFD        ---        up     22:12:35.949
r2ge_v6_v4 BGP        ---        up     22:35:40.175  Established
genauso_v6 BGP        ---        start  22:12:35.949  Passive
server121_v6_v4 BGP        ---        up     22:24:15.183  Established
radv1      RAdv       master6    up     22:12:35.949
ospf6      OSPF       master6    up     22:12:35.949  Running
fiberstream Static     master4    up     22:12:35.949
direct1    Direct     ---        up     22:12:35.949
genauso_1  BGP        ---        up     22:12:36.974  Established
genauso_2  BGP        ---        up     22:12:36.979  Established
genauso_3  BGP        ---        up     22:12:36.985  Established
genauso_4  BGP        ---        start  22:14:01.848  Idle
bird>
bird> show protocols all kernel_v4
Name       Proto      Table      State  Since         Info
kernel_v4  Kernel     master4    up     22:12:35.949
  Channel ipv4
    State:          UP
    Table:          master4
    Preference:     10
    Input filter:   ACCEPT
    Output filter:  ACCEPT
    Routes:         0 imported, 23 exported, 0 preferred
    Route change stats:     received   rejected   filtered    ignored   accepted
      Import updates:              0          0          0          0          0
      Import withdraws:            0          0        ---          0          0
      Export updates:            265          0          0        ---        265
      Export withdraws:          113        ---        ---        ---        113

bird> show protocols all kernel_v6
Name       Proto      Table      State  Since         Info
kernel_v6  Kernel     master6    up     22:12:35.949
  Channel ipv6
    State:          UP
    Table:          master6
    Preference:     10
    Input filter:   ACCEPT
    Output filter:  ACCEPT
    Routes:         0 imported, 83 exported, 0 preferred
    Route change stats:     received   rejected   filtered    ignored   accepted
      Import updates:              0          0          0          0          0
      Import withdraws:            0          0        ---          0          0
      Export updates:           2797          0          0        ---       2797
      Export withdraws:          431        ---        ---        ---        431

bird>

--------------------------------------------------------------------------------

However checking the route table in the kernel, there is not a single
route injected by bird:

--------------------------------------------------------------------------------
 # ip r
default via 192.168.1.254 dev eth1 proto dhcp src 192.168.1.159 metric 1003
192.168.1.0/24 dev eth1 proto dhcp scope link src 192.168.1.159 metric 1003
194.5.220.0/24 via 192.168.1.254 dev eth1
/ # ip -6 r
2a0a:5480::/64 dev eth0 proto kernel metric 256 pref medium
2a0a:5480:0:a::b dev lo proto kernel metric 256 pref medium
2a0a:5480:3:1::/64 dev k8sbridge proto kernel metric 256 pref medium
2a0a:5480:3:3::3 dev lo proto kernel metric 256 pref medium
2a0a:5480:3:4::/64 dev tun0 proto kernel metric 256 pref medium
2a0a:e5c0:32:4::/64 dev tun1 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
fe80::/64 dev k8sbridge proto kernel metric 256 pref medium
fe80::/64 dev veth99f0c958 proto kernel metric 256 pref medium
fe80::/64 dev vethe20cee78 proto kernel metric 256 pref medium
fe80::/64 dev veth8f504fac proto kernel metric 256 pref medium
fe80::/64 dev vethc03bfec4 proto kernel metric 256 pref medium
fe80::/64 dev veth046f7cf0 proto kernel metric 256 pref medium
fe80::/64 dev veth434f4ff5 proto kernel metric 256 pref medium
fe80::/64 dev vethd021bdfa proto kernel metric 256 pref medium
fe80::/64 dev veth935a701b proto kernel metric 256 pref medium
fe80::/64 dev veth473f9ed3 proto kernel metric 256 pref medium
fe80::/64 dev veth931ba96a proto kernel metric 256 pref medium
fe80::/64 dev vethdbaa705e proto kernel metric 256 pref medium
fe80::/64 dev tun1 proto kernel metric 256 pref medium
fe80::/64 dev tun0 proto kernel metric 256 pref medium
/ #
--------------------------------------------------------------------------------

I think I saw some routes shortly being installed and then vanish within
a second again, but I cannot even take a screenshot of it - so it might
be that bird writes the routes and that they instantly disappear.

I can reproduce the missing routes on 2 lab machines and I was even able
to downgrade to 2.13.1 and reproduce it there, too.

I have attached the log output of bird below.

Does anyone have an idea on what might be going wrong?

Best regards,

Nico

--------------------------------------------------------------------------------
2023-12-25 22:12:35.949 <INFO> Started
2023-12-25 22:12:36.210 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for ::/0
2023-12-25 22:12:36.216 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:12:36.216 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:12:36.218 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:12:36.218 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:12:36.909 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:12:36.909 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:12:36.909 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:12:36.909 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:12:38.723 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:12:38.723 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:12:38.723 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:12:38.723 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:16:45.534 <WARN> Netlink: No such device
2023-12-25 22:20:13.582 <WARN> Netlink: No such device
2023-12-25 22:23:11.536 <RMT> server121_v6_v4: Received: Administrative shutdown
2023-12-25 22:28:13.570 <WARN> Netlink: No such device
2023-12-25 22:29:40.051 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:29:40.051 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:29:40.051 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:29:40.051 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:29:40.059 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:29:40.059 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:29:40.059 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:29:40.059 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:29:40.091 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:29:40.091 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:29:40.091 <WARN> Next hop address 2a0a:5480:2:4::2 resolvable through recursive route for 2a0a:5480:2:4::/64
2023-12-25 22:29:40.091 <WARN> Next hop address 2a0a:e5c0:31:4::1 resolvable through recursive route for 2a0a:e5c0:31:4::/64
2023-12-25 22:30:02.584 <RMT> r2ge_v6_v4: Received: Administrative shutdown
2023-12-25 22:34:33.277 <WARN> Netlink: No such device
2023-12-25 22:49:44.311 <WARN> Netlink: No such device
2023-12-25 22:50:35.660 <WARN> Netlink: No such device
2023-12-25 22:50:43.548 <WARN> Netlink: No such device
2023-12-25 23:08:38.133 <WARN> Netlink: No such device
2023-12-25 23:28:39.044 <WARN> Netlink: No such device
2023-12-25 23:53:43.584 <WARN> Netlink: No such device
2023-12-26 00:02:43.576 <WARN> Netlink: No such device
2023-12-26 00:09:43.593 <WARN> Netlink: No such device
2023-12-26 00:17:44.840 <WARN> Netlink: No such device
2023-12-26 00:19:44.792 <WARN> Netlink: No such device
2023-12-26 00:24:48.079 <WARN> Netlink: No such device
2023-12-26 00:52:52.080 <WARN> Netlink: No such device
2023-12-26 01:04:44.532 <WARN> Netlink: No such device
2023-12-26 01:32:43.574 <WARN> Netlink: No such device
2023-12-26 02:17:43.583 <WARN> Netlink: No such device
2023-12-26 02:43:44.123 <WARN> Netlink: No such device
2023-12-26 03:07:06.367 <WARN> Netlink: No such device
2023-12-26 03:08:53.050 <WARN> Netlink: No such device
2023-12-26 03:39:10.264 <WARN> Netlink: No such device
2023-12-26 04:03:13.540 <WARN> Netlink: No such device
2023-12-26 04:03:43.591 <WARN> Netlink: No such device
2023-12-26 04:07:13.660 <WARN> Netlink: No such device
2023-12-26 04:22:44.732 <WARN> Netlink: No such device
2023-12-26 04:27:15.276 <WARN> Netlink: No such device
2023-12-26 04:39:43.578 <WARN> Netlink: No such device
2023-12-26 04:41:43.598 <WARN> Netlink: No such device
2023-12-26 05:09:18.230 <WARN> Netlink: No such device
2023-12-26 06:10:45.058 <WARN> Netlink: No such device
2023-12-26 06:15:27.087 <WARN> Netlink: No such device
2023-12-26 06:45:32.092 <WARN> Netlink: No such device
2023-12-26 06:55:33.326 <WARN> Netlink: No such device
2023-12-26 07:05:43.542 <WARN> Netlink: No such device
2023-12-26 07:17:35.427 <WARN> Netlink: No such device
2023-12-26 07:29:35.980 <WARN> Netlink: No such device
2023-12-26 08:01:39.077 <WARN> Netlink: No such device
2023-12-26 09:01:44.615 <WARN> Netlink: No such device
2023-12-26 09:02:43.566 <WARN> Netlink: No such device
2023-12-26 09:17:47.696 <WARN> Netlink: No such device
2023-12-26 09:23:48.668 <WARN> Netlink: No such device
2023-12-26 09:27:49.466 <WARN> Netlink: No such device
2023-12-26 09:29:43.564 <WARN> Netlink: No such device
2023-12-26 09:39:52.004 <WARN> Netlink: No such device
2023-12-26 09:44:58.208 <WARN> Netlink: No such device
2023-12-26 09:45:43.545 <WARN> Netlink: No such device
2023-12-26 09:55:53.271 <WARN> Netlink: No such device
2023-12-26 10:09:53.189 <WARN> Netlink: No such device
--------------------------------------------------------------------------------

--
Sustainable and modern Infrastructures by ungleich.ch

Bird is not adding routes into kernel anymore

Nico Schottelius