FreeBSD, multiple routing tables, "KRT: Error sending route 192.222.249.232/29 to kernel: File exists"
Hey there, We've got a fairly simple setup where we have two fibs. Interface routes are in both fibs the way BSD does it. (Table 1 is the second fib, if it matters). log syslog all; protocol kernel { scan time 20; ipv4 { import all; export all; table master4; }; kernel table 1; } protocol kernel { ipv6 { import all; export all; table master6; }; kernel table 1; } protocol device { scan time 10; } setfib 1 netstat -nr: Routing tables (fib: 1) Internet: Destination Gateway Flags Netif Expire default 192.222.249.233 UGS bge0 127.0.0.1 link#5 UH lo0 127.0.0.2 link#6 UH lo1 192.222.249.232/29 link#1 U bge0 fib 0 would be the same. An old thread seemed to imply this was because the "direct" protocol was missing. Adding the "direct" protocol doesn't stop this error every 20 seconds in syslog (which seems to be related to the scan time). Mar 12 15:24:11 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:24:31 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:24:51 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:25:11 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:25:31 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Is it possible bird isn't differentiating routes learned from each fib and is learning from both, separately, and not properly deduplicating those routes? -Dan
W dniu 12.03.2022 o 16:48, Dan Mahoney pisze:
Hey there,
We've got a fairly simple setup where we have two fibs. Interface routes are in both fibs the way BSD does it. (Table 1 is the second fib, if it matters).
No, by default interface route is added only in interface fib for FreeBSD 13.0 and later. Setting "net.add_addr_allfibs=1" brings back old behavior.
trying log syslog all;
protocol kernel { scan time 20; ipv4 { import all; export all; table master4; }; kernel table 1; }
protocol kernel { ipv6 { import all; export all; table master6; }; kernel table 1; }
protocol device { scan time 10; }
setfib 1 netstat -nr:
Routing tables (fib: 1)
Internet: Destination Gateway Flags Netif Expire default 192.222.249.233 UGS bge0 127.0.0.1 link#5 UH lo0 127.0.0.2 link#6 UH lo1 192.222.249.232/29 link#1 U bge0
fib 0 would be the same.
An old thread seemed to imply this was because the "direct" protocol was missing.
Adding the "direct" protocol doesn't stop this error every 20 seconds in syslog (which seems to be related to the scan time).
Mar 12 15:24:11 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:24:31 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:24:51 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:25:11 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:25:31 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists
Is it possible bird isn't differentiating routes learned from each fib and is learning from both, separately, and not properly deduplicating those routes?
-Dan
What FreeBSD version are you running? FreeBSD routing stack has been significantly rewritten for 13.0-RELEASE, moreover this rewritten routing stack in stable/13 was polished a bit, so if you are encountering this on 13.0-RELEASE, then please upgrade to 13.1-BETA. You can also probably either: 1. Filter this route to prevent it from being added. 2. Disable aforementioned "net.add_addr_allfibs". -- Marek Zarychta
On Mar 12, 2022, at 9:39 PM, Marek Zarychta <zarychtam@plan-b.pwste.edu.pl> wrote:
W dniu 12.03.2022 o 16:48, Dan Mahoney pisze:
Hey there,
We've got a fairly simple setup where we have two fibs. Interface routes are in both fibs the way BSD does it. (Table 1 is the second fib, if it matters).
No, by default interface route is added only in interface fib for FreeBSD 13.0 and later. Setting "net.add_addr_allfibs=1" brings back old behavior.
trying log syslog all;
protocol kernel { scan time 20; ipv4 { import all; export all; table master4; }; kernel table 1; }
protocol kernel { ipv6 { import all; export all; table master6; }; kernel table 1; }
protocol device { scan time 10; }
setfib 1 netstat -nr:
Routing tables (fib: 1)
Internet: Destination Gateway Flags Netif Expire default 192.222.249.233 UGS bge0 127.0.0.1 link#5 UH lo0 127.0.0.2 link#6 UH lo1 192.222.249.232/29 link#1 U bge0
fib 0 would be the same.
An old thread seemed to imply this was because the "direct" protocol was missing.
Adding the "direct" protocol doesn't stop this error every 20 seconds in syslog (which seems to be related to the scan time).
Mar 12 15:24:11 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:24:31 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:24:51 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:25:11 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists Mar 12 15:25:31 box1 bird[34463]: KRT: Error sending route 192.222.249.232/29 to kernel: File exists
Is it possible bird isn't differentiating routes learned from each fib and is learning from both, separately, and not properly deduplicating those routes?
-Dan
What FreeBSD version are you running? FreeBSD routing stack has been significantly rewritten for 13.0-RELEASE, moreover this rewritten routing stack in stable/13 was polished a bit, so if you are encountering this on 13.0-RELEASE, then please upgrade to 13.1-BETA.
We're on 12.3. We're not ready to run a .0 OS release in production.
You can also probably either:
1. Filter this route to prevent it from being added.
That's an option, yes. We'd have to write some automation to exclude this specific route, but still allow the export of all the rest of our BGP routes. I'd still like to understand why bird *feels the need* to send this route to the kernel. Is it learning it from fib 0 and attempting to update fib 1 or something? Is there a log I can enable that would confirm that's happening?
2. Disable aforementioned "net.add_addr_allfibs".
Thanks, these look promising, even if we do have to wait for the roll to 13.0 or 13.1. I'd still ideally like to find a solution that works on 12.x, which is a production release for a while now. -Dan
participants (2)
-
Dan Mahoney -
Marek Zarychta