Re: bird patches review
On Mon, Oct 03, 2011 at 10:03:30PM +0400, Alexander V. Chernikov wrote:
In most hardware routers (and even in Quagga) it is possibly to see if RIB entry is installed to FIB (e.g. * ). Maybe kernel can set some flag at least when it sees installed prefix gets removed from kernel table? (or adds some flags on route installation and remove it on route removal)
There is already such flag (because kernel sync is a bit tricky and abuses routing table to rememer its state), perhaps we could show that flag in some way.
Should I [as usual] submit patch for that? :)
I found that the flag signalizes whether route is expected to be installed (but the installation might failed). I added the flag signalizing that the last attempt to install the route to the kernel table failed, and showed that flag in 'show route': bird> show route 10.200.0.0/16 via xx.xx.xx.xx on eth0 [static1 16:13] ! (200) 192.168.0.0/16 via xx.xx.xx.xx on eth0 [static1 16:13] * (200) 10.220.0.0/16 via xx.xx.xx.xx on eth0 [static1 16:13] * (200) Showing just whether the route is (supposed to be) installed seems a bit pointless - it is usually 'export all' or 'export none' for kernel protocol. If someone removes the route from FIB (kernel) and BIRD notices that, it reinstalls the route, so unless there is a problem in route installation, it remains synced. -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On 06.10.2011 14:43, Ondrej Zajicek wrote:
On Mon, Oct 03, 2011 at 10:03:30PM +0400, Alexander V. Chernikov wrote:
In most hardware routers (and even in Quagga) it is possibly to see if RIB entry is installed to FIB (e.g. * ). Maybe kernel can set some flag at least when it sees installed prefix gets removed from kernel table? (or adds some flags on route installation and remove it on route removal)
There is already such flag (because kernel sync is a bit tricky and abuses routing table to rememer its state), perhaps we could show that flag in some way.
Should I [as usual] submit patch for that? :) Sorry, got -ETIME error :( I found that the flag signalizes whether route is expected to be installed (but the installation might failed). I added the flag signalizing that the last attempt to install the route to the kernel table failed, and showed that flag in 'show route':
bird> show route 10.200.0.0/16 via xx.xx.xx.xx on eth0 [static1 16:13] ! (200) 192.168.0.0/16 via xx.xx.xx.xx on eth0 [static1 16:13] * (200) 10.220.0.0/16 via xx.xx.xx.xx on eth0 [static1 16:13] * (200)
Showing just whether the route is (supposed to be) installed seems a bit pointless - it is usually 'export all' or 'export none' for kernel protocol. If someone removes the route from FIB (kernel) and BIRD notices that, it reinstalls the route, so unless there is a problem in route installation, it remains synced. We're talking about immediate or delayed reinstallaion ? If route gets removed accidentally, immediate reinstallation is great. If, for some reasons, we have to remove route manually (for example if we're trying to do ifconfig some_iface a.b.c.d/z where a.b.c.0/z already exists from IGP), it is much better not to restore route immediately giving user chance to do what needs to be done.
Quagga (I can be wrong here) marks route inactive on route removal event and tries to restore it on next table scan. -- Alexander V. Chernikov Yandex NOC
On Thu, Oct 06, 2011 at 02:58:25PM +0400, Alexander V. Chernikov wrote:
I found that the flag signalizes whether route is expected to be installed (but the installation might failed). I added the flag signalizing that the last attempt to install the route to the kernel table failed, and showed that flag in 'show route':
bird> show route 10.200.0.0/16 via xx.xx.xx.xx on eth0 [static1 16:13] ! (200) 192.168.0.0/16 via xx.xx.xx.xx on eth0 [static1 16:13] * (200) 10.220.0.0/16 via xx.xx.xx.xx on eth0 [static1 16:13] * (200)
Showing just whether the route is (supposed to be) installed seems a bit pointless - it is usually 'export all' or 'export none' for kernel protocol. If someone removes the route from FIB (kernel) and BIRD notices that, it reinstalls the route, so unless there is a problem in route installation, it remains synced. We're talking about immediate or delayed reinstallaion ? If route gets removed accidentally, immediate reinstallation is great. If, for some reasons, we have to remove route manually (for example if we're trying to do ifconfig some_iface a.b.c.d/z where a.b.c.0/z already exists from IGP), it is much better not to restore route immediately giving user chance to do what needs to be done.
When BIRD notices that during periodic scan, it reinstalls that immediately. Asynchronous (immediate) notifications about removed BIRD routes are currently not processed (so it will be reinstalled during the next periodic scan). -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
If someone removes the route from FIB (kernel) and BIRD notices that, it reinstalls the route, so unless there is a problem in route installation, it remains synced.
Oh, you mean like this: 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2620:x:10a0::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2001:x:ce0::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2607:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2001:x:ff03::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2607:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2606:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2a00:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2606:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2400:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2001:x:2d4::/0 to kernel This is for IP6 on FreeBSD. I've also got the same for IP4: 2011-10-06 14:00:04 <ERR> KRT: Error sending route 199.189.xx.0/24 to kernel 2011-10-06 14:00:04 <ERR> KRT: Error sending route 212.104.xx.0/24 to kernel 2011-10-06 14:00:04 <ERR> KRT: Error sending route 188.171.xx.0/24 to kernel 2011-10-06 14:00:04 <ERR> KRT: Error sending route 175.103.xx.0/24 to kernel 2011-10-06 14:00:08 <ERR> KRT: Error sending route 199.189.xx.0/24 to kernel 2011-10-06 14:00:08 <ERR> KRT: Error sending route 212.104.xx.0/24 to kernel 2011-10-06 14:00:08 <ERR> KRT: Error sending route 188.171.xx.0/24 to kernel 2011-10-06 14:00:08 <ERR> KRT: Error sending route 175.103.xx.0/24 to kernel I've got a couple of gigs of these messages in my log... The only one whose can insert and delete this routes is bird. And it seems it only happens if there is a update to a route. 2011-10-04 16:05:49 <TRACE> b6_sekkaa_1_cr2 < added 2620:x:10a0::/48 via x on igb4 2011-10-04 16:05:49 <TRACE> b6_sekkaa_1_cr2 > added [best] 2620:x:10a0::/48 via x on igb3 2011-10-04 16:05:49 <TRACE> kernel1 < replaced 2620:x:10a0::/48 via x on igb3 2011-10-04 16:05:49 <TRACE> b6_sekkaa_1_er1 < replaced 2620:x:10a0::/48 via x on igb3 2011-10-04 16:05:49 <TRACE> b6_sekkaa_1_er2 < replaced 2620:x:10a0::/48 via x on igb3 2011-10-04 16:05:49 <INFO> (export_allowed) community 52011, 970 block announce of '2620:x:10a0::/48' to ASx 2011-10-04 16:05:49 <TRACE> b6_sekkaa_1_cr2 < removed 2620:x:10a0::/48 via x on igb4 2011-10-04 16:05:51 <ERR> KRT: Error sending route 2620:x:10a0::/0 to kernel .... 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2620:x:10a0::/0 to kernel -- //fredan
On Thu, Oct 06, 2011 at 02:10:49PM +0200, fredrik danerklint wrote:
If someone removes the route from FIB (kernel) and BIRD notices that, it reinstalls the route, so unless there is a problem in route installation, it remains synced.
Oh, you mean like this:
2011-10-06 13:57:57 <ERR> KRT: Error sending route 2620:x:10a0::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2001:x:ce0::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2607:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2001:x:ff03::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2607:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2606:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2a00:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2606:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2400:x::/0 to kernel 2011-10-06 13:57:57 <ERR> KRT: Error sending route 2001:x:2d4::/0 to kernel
Really x:x::/0 (prefix length 0) ?
This is for IP6 on FreeBSD. I've also got the same for IP4:
2011-10-06 14:00:04 <ERR> KRT: Error sending route 199.189.xx.0/24 to kernel 2011-10-06 14:00:04 <ERR> KRT: Error sending route 212.104.xx.0/24 to kernel 2011-10-06 14:00:04 <ERR> KRT: Error sending route 188.171.xx.0/24 to kernel 2011-10-06 14:00:04 <ERR> KRT: Error sending route 175.103.xx.0/24 to kernel 2011-10-06 14:00:08 <ERR> KRT: Error sending route 199.189.xx.0/24 to kernel 2011-10-06 14:00:08 <ERR> KRT: Error sending route 212.104.xx.0/24 to kernel 2011-10-06 14:00:08 <ERR> KRT: Error sending route 188.171.xx.0/24 to kernel 2011-10-06 14:00:08 <ERR> KRT: Error sending route 175.103.xx.0/24 to kernel
I've got a couple of gigs of these messages in my log...
Generally, there should not be such problems. Could you send me your config file and 'netstat -nr' output? -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
route 2001:x:2d4::/0 to kernel
Really x:x::/0 (prefix length 0) ?
Oh, yes! # netstat -nr -f inet6 | grep ::/0 2001:218::/0 fe80::21b:21xx:xx:xx%igb2 UG1 igb2 2001:2b8:c2::/0 fe80::21b:21xx:xx:xx%igb2 UG1 igb2 2001:400::/0 fe80::21b:21xx:xx:xx%igb1 UG1 igb1 2001:428:b20::/0 fe80::21b:21xx:xx:xx%igb1 UG1 igb1 On this router I've got 135 of them. # netstat -nr -f inet6 | grep ::/0 | wc 135
Could you send me your config file and 'netstat -nr' output?
Please see private e-mail. -- //fredan
On Fri, Oct 07, 2011 at 01:04:43AM +0200, fredrik danerklint wrote:
route 2001:x:2d4::/0 to kernel
Really x:x::/0 (prefix length 0) ?
Oh, yes!
Do you also have these strange routes in BIRD table? My theory is that BSD kernel created the strange route in the kernel table, BIRD noticed that route, tried to remove it (as it is marked as its route but BIRD does not know about it) and the removal failed. I don't know why the BSD would do that, but i do not really track BSD. Alexander, don't you know something about that? The netstat output looks like: 2001:xxx::/32 fe80::xxxx%igb1 UG1 igb1 => 2001:xxx::/0 fe80::xxxx%igb1 UG1 igb1 -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Do you also have these strange routes in BIRD table?
No, I don't think so. 2011-10-10 15:23:20 <ERR> KRT: Error sending route 2a03:xx00::/0 to kernel # birdc6 BIRD 1.3.4 ready. bird> show route for 2a03:xx00::1 2a03:xx00::/32 via fe80::21b:21ff:fe80:xxx on igb1 [b6_sekkaa_1_cr2 2011-10-11 02:33:17 from 2a03:b000:xxxx::xxxx:f02] * (100/6) [AS42038i] via fe80::21b:21ff:fe80:xxxx on igb2 [b6_sekkaa_1_cr1 2011-10-11 02:33:17 from 2a03:b000:xxxx::xxxx:f01] (100/5) [AS42038i] bird> However. When I rebooted my routes to use the new version 1.3.4 I did see the following. ER1 is connected to both CR1 and CR2. My primary traffic is coming from CR2. CR1 is used as a backup. Internally I have a lower local_pref from CR1 to achieve this. I started first with ER1 then with CR1 and finally with CR2. In ER1 when bird noticed this, it replaced the route from CR2 with CR1, since the route from CR2 was the active one. It is now the problem seems to occur. When there is an change of the route for a prefix, it seems that it might be something with the kernel that doesn't seems right. On the ER1 i've got: #netstat -rn -a inet6 | grep ::/0 | wc 252 directly after the change from CR2 to CR1. Now I've also got this in the log file: 2011-10-11 03:45:35 <ERR> KRT: Error sending route 2407:xxxx:2::/0 to kernel: No such process Is it just me that have these strange this on FreeBSD? -- //fredan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 11.10.2011 04:05, Ondrej Zajicek wrote:
On Fri, Oct 07, 2011 at 01:04:43AM +0200, fredrik danerklint wrote:
route 2001:x:2d4::/0 to kernel
Really x:x::/0 (prefix length 0) ?
Oh, yes!
Do you also have these strange routes in BIRD table?
My theory is that BSD kernel created the strange route in the kernel Kernel itself usually operate interface routes only. Addtitionally, it doesn't install routes with flag 1. Either some other dynamic routing software is running or something strange is happening with bird6. table, BIRD noticed that route, tried to remove it (as it is marked as its route but BIRD does not know about it) and the removal failed.
I don't know why the BSD would do that, but i do not really track BSD. Alexander, don't you know something about that? The netstat output looks like:
2001:xxx::/32 fe80::xxxx%igb1 UG1 igb1 => 2001:xxx::/0 fe80::xxxx%igb1 UG1 igb1 Unfortunately I dont't have any bird6 in production because having bird/ospf3 running is the best and easiest way to cause all quagga ospf3 processes in she same subnet to crash (till 0.99.18 ?)
fredrik, can you try the following: issue "route -n monitor" on the server with bird6, run bird6 with kernel protocol debug, get to the problem situation (you wrote you've got a more or less easy way to trigger this) and show us monitor output and bird logs ? -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk6T0xYACgkQwcJ4iSZ1q2m5jQCgkSa2KWF8J6DbzVf4EqhlMkha d7MAnR5pYRvd59MBLQ4r5a8ir+M9mKUc =99zf -----END PGP SIGNATURE-----
participants (3)
-
Alexander V. Chernikov -
fredrik danerklint -
Ondrej Zajicek