Different OSPF behaviour between Static and Kernel/Pipe

7 Apr 2020

      Hi all!

I've got a small mesh deployment using BIRD 2.0.7 and OSPFv2 which has been
working well for a while now distributing a mix of device and static routes
going into the kernel table export-only.

I've now run into a need to have a number of "dynamic" static routes
managed by an external system and was looking into using a 2nd import-only
kernel routing table for that managed via netlink rather than constantly
reloading BIRD - but I've run into an issue where things aren't behaving as
I'd expect when multiple routers advertise the same route sourced from the
kernel protocol. What I'm seeing is that when the route gets added on
the first router, it gets exported via BIRD to the kernel table as directly
connected (as specified) and on the other router as a next-hop of the first
router (as you'd expect). But when the same route is added on the other
router (while still on the first one), the kernel routing table of the 2nd
router does not change to be direct, it keeps the next-hop of the 1st
router. However, if I specify the exact same routes in the static protocol
and reload BIRD, everything works correctly and both routers end up with a
direct route in the kernel table! (and other routers in the area get
multi-path routes via the ones that have it added.)

I think I've tracked the behaviour difference down to how they're being
added / appearing in OSPF, but I'm not sure why they might be behaving
differently (it's as if the static routes get merged with what is already
in the table, but the pipe isn't doing that) - I've included the config and
some command output:

BIRD config (the same apart from id/prefsrc on both I'm testing with,
trimmed out of the "production" config which also includes PtMP neighbours
on remote routers):

router id 172.19.4.2;

debug protocols { states, routes, filters, interfaces, events };

ipv4 table clustr;

protocol device {
}

protocol direct {
  ipv4;
  interface "ens224";
}

protocol kernel {
  ipv4 {
    import none;
    export filter {
      krt_prefsrc = 172.19.4.2;
      accept;
    };
  };
  merge paths yes;
}

protocol kernel {
  ipv4 {
    table clustr;
    import all;
    export none;
  };
  scan time 2;
  learn yes;
  kernel table 100;
}

protocol pipe {
  table master4;
  peer table clustr;
  import all;
  export none;
}

protocol ospf v2 {
  ipv4 {
    import all;
    export where dest != RTD_UNREACHABLE;
  };
  area 0 {
    interface "ens224" {
      bfd yes;
    };
  };
  ecmp yes;
  merge external yes;
}

protocol bfd {
  interface "ens*" {
    min rx interval 100 ms;
    min tx interval 200 ms;
    idle tx interval 1000 ms;
  };
}

protocol static {
  ipv4;
  route 172.27.0.0/19 via "ens224";
  route 10.0.0.0/8 unreachable;
  route 100.64.0.0/10 unreachable;
  route 169.254.0.0/16 unreachable;
  route 172.16.0.0/12 unreachable;
  route 192.168.0.0/16 unreachable;
}

Initial OSPF state:

[root@ukdev-bastion-003 ~]# birdc show ospf state all
BIRD 2.0.7 ready.

area 0.0.0.0

        router 172.19.4.2
                distance 0
                network 172.19.0.0/19 metric 10
                stubnet 172.19.2.2/32 metric 0
                external 172.19.2.2/32 metric2 10000
                external 172.19.0.0/19 metric2 10000
                external 172.27.0.0/19 metric2 10000

        router 172.19.4.3
                distance 10
                network 172.19.0.0/19 metric 10
                stubnet 172.19.2.3/32 metric 0
                external 172.19.2.3/32 metric2 10000
                external 172.19.0.0/19 metric2 10000
                external 172.27.0.0/19 metric2 10000

        network 172.19.0.0/19
                dr 172.19.4.2
                distance 10
                router 172.19.4.2
                router 172.19.4.3

Adding a test route on the first router:

[root@ukdev-bastion-003 ~]# ip route add 192.168.0.100/32 table 100 dev
ens224 proto static metric 32

To avoid overwhelming everyone with lots of repeated birdc show ospf state
all output (but do ask if anything would be useful!) I'll describe what
happens then:

   1. The route shows up in the router section it was added for as an
   external (as I'd expect).
   2. When the same route is added on the 2nd router, there is no change to
   the OSPF state.
   3. When the route is removed from the 1st router (while still on the
   2nd) the external is removed from the first router and appears on the 2nd
   one!
   4. Finally, removing it from the 2nd router causes the external to
   disappear from the OSPF state.

The kernel routing table behaviour (described previously) behaves exactly
as I'd expect given what OSPF is doing here. But that isn't what I'm trying
to achieve, at step 2 I would expect the external network to be registered
by both routers simultaneously - and this is how it behaves if the route is
configured using the static protocol (as you can see with the 172.27.0.0/19
route).

If anyone has any insight as to what I might be doing wrong here (or other
ways of accomplishing the same goal) it would be greatly appreciated! I'm
no expert and mostly just muddling my way through :)

Many thanks,
Asher

P.S. I think a API-based routing protocol as discussed on the list a few
years back would be really cool to see for this sort of stuff.

~~~~~
"Their heads are green, and their hands are blue,
      And they went to sea in a Sieve." - Edward Lear

Different OSPF behaviour between Static and Kernel/Pipe

Asher Baker