BIRD 1.3.11 (ports) on FreeBSD 9.1-p7
Good morning! My name is Markus and I have an issue with the BIRD 1.3.11. Since I had switched from Quagga 0.9x.x to BIRD 1.3.11 I have more and more trouble in my network with OSPFv2. After some minutes or hours the ARP resolution doesn't work and tons of messages are logged by BIRD and the "kernel". All FreeBSD 9.1-p7 servers sending messages to their syslog (see below). Virtual servers are no longer reachable. What's the problem? The same messages are generated by OSPFv3 processes. --- Some Logs --- arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 [..] arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.197 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 [..] arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 ### The server with the NFS service is running BIRD 1.3.11 (FreeBSD ports). ### Over 1000 messages will be logged. $ dmesg | grep arpresolve | wc -l 1189 ### After a while some servers are not longer available because ARP resolution failed. Oct 9 00:55:39 clarion kernel: newnfs server trinitron:/archiv/clarion: not responding ### When I restart BIRD the ARP resolution will work again. Custom Kernel: 9.1-RELEASE-p7 FreeBSD 9.1-RELEASE-p7 #4: Tue Sep 10 14:31:40 CEST 2013 amd64 ### All OSPF neighbors sending tons of warnings! :-( Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0005, Id: 10.82.206.193, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.1.33, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.210, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.216, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.217, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0002, Id: 8X.2X.1.33, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0002, Id: 8X.2X.32.210, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0005, Id: 10.82.206.193, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.1.33, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.210, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.216, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.217, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0002, Id: 82.2X.1.33, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0002, Id: 8X.2X.32.210, Rt: 8X.2X.32.210) ### It's a little bit too much ;-) grep "LSA disappeared" /var/log/messages|wc -l 3585 ### /var/log/messages on a other machine :-( [..] Oct 9 05:02:19 clarion bird: OSPF: LSA disappeared (Type: 0005, Id: 0.0.0.0, Rt: 82.2X.32.193) Oct 9 05:02:24 clarion bird: OSPF: LSA disappeared (Type: 0002, Id: 8X.2X.32.193, Rt: 8X.2X.32.193) Oct 9 05:02:24 clarion bird: OSPF: LSA disappeared (Type: 0005, Id: 0.0.0.0, Rt: 82.2X.32.193) Oct 9 05:02:28 clarion bird: OSPF: LSA disappeared (Type: 0002, Id: 8X.2X.32.193, Rt: 8X.2X.32.193) Oct 9 05:02:59 clarion last message repeated 6 times Oct 9 05:05:04 clarion last message repeated 25 times Oct 9 05:15:08 clarion last message repeated 121 times Oct 9 05:25:09 clarion last message repeated 120 times Oct 9 05:35:08 clarion last message repeated 120 times Oct 9 05:45:10 clarion last message repeated 120 times Oct 9 05:55:11 clarion last message repeated 120 times Oct 9 06:05:10 clarion last message repeated 120 times Oct 9 06:15:10 clarion last message repeated 120 times Oct 9 06:25:10 clarion last message repeated 120 times Oct 9 06:35:10 clarion last message repeated 120 times Oct 9 06:45:11 clarion last message repeated 120 times Oct 9 06:55:12 clarion last message repeated 120 times Oct 9 07:05:12 clarion last message repeated 120 times Oct 9 07:15:13 clarion last message repeated 120 times Oct 9 07:25:14 clarion last message repeated 120 times Oct 9 07:35:15 clarion last message repeated 120 times Oct 9 07:45:14 clarion last message repeated 120 times Oct 9 07:55:15 clarion last message repeated 120 times Oct 9 08:05:13 clarion last message repeated 119 times [..]
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09.10.2013 11:07, Markus Grundmann wrote:
Good morning!
My name is Markus and I have an issue with the BIRD 1.3.11. Since I had switched from Quagga 0.9x.x to BIRD 1.3.11 I have more and more trouble in my network with OSPFv2. After some minutes or hours the ARP resolution doesn't work and tons of messages are logged by BIRD and the "kernel". All FreeBSD 9.1-p7 servers sending messages to their syslog Did this happen with quagga? (see below).
Virtual servers are no longer reachable. What's the problem? The same messages are generated by OSPFv3 processes. Em, "arpresolve" messages are generated by kernel. Are you talking about "LSA disappeared" ones?
--- Some Logs --- arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 [..] arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.197 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 [..] arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211 arpresolve: can't allocate llinfo for 8X.2X.32.211
Can you provide bird config (at least direct/ospf protocol parts) and describe network topology? E.g. how the server can reach "8X.2X.32.211" or other addresses in dmesg. What does output of "route get 8X.2X.32.211" show (when working "normal" and when problem starts) ?
### The server with the NFS service is running BIRD 1.3.11 (FreeBSD ports). ### Over 1000 messages will be logged.
$ dmesg | grep arpresolve | wc -l 1189
### After a while some servers are not longer available because ARP resolution failed.
Oct 9 00:55:39 clarion kernel: newnfs server trinitron:/archiv/clarion: not responding
### When I restart BIRD the ARP resolution will work again.
Custom Kernel: 9.1-RELEASE-p7 FreeBSD 9.1-RELEASE-p7 #4: Tue Sep 10 14:31:40 CEST 2013 amd64
### All OSPF neighbors sending tons of warnings! :-(
Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0005, Id: 10.82.206.193, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.1.33, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.210, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.216, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.217, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0002, Id: 8X.2X.1.33, Rt: 8X.2X.32.210) Oct 9 08:30:35 j bird: OSPF: LSA disappeared (Type: 0002, Id: 8X.2X.32.210, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0005, Id: 10.82.206.193, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.1.33, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.210, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.216, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0005, Id: 8X.2X.32.217, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0002, Id: 82.2X.1.33, Rt: 8X.2X.32.210) Oct 9 08:30:40 j bird: OSPF: LSA disappeared (Type: 0002, Id: 8X.2X.32.210, Rt: 8X.2X.32.210)
This is annoying, but harmless.
### It's a little bit too much ;-)
grep "LSA disappeared" /var/log/messages|wc -l 3585
### /var/log/messages on a other machine :-(
[..] Oct 9 05:02:19 clarion bird: OSPF: LSA disappeared (Type: 0005, Id: 0.0.0.0, Rt: 82.2X.32.193) Oct 9 05:02:24 clarion bird: OSPF: LSA disappeared (Type: 0002, Id: 8X.2X.32.193, Rt: 8X.2X.32.193) Oct 9 05:02:24 clarion bird: OSPF: LSA disappeared (Type: 0005, Id: 0.0.0.0, Rt: 82.2X.32.193) Oct 9 05:02:28 clarion bird: OSPF: LSA disappeared (Type: 0002, Id: 8X.2X.32.193, Rt: 8X.2X.32.193) Oct 9 05:02:59 clarion last message repeated 6 times Oct 9 05:05:04 clarion last message repeated 25 times Oct 9 05:15:08 clarion last message repeated 121 times Oct 9 05:25:09 clarion last message repeated 120 times Oct 9 05:35:08 clarion last message repeated 120 times Oct 9 05:45:10 clarion last message repeated 120 times Oct 9 05:55:11 clarion last message repeated 120 times Oct 9 06:05:10 clarion last message repeated 120 times Oct 9 06:15:10 clarion last message repeated 120 times Oct 9 06:25:10 clarion last message repeated 120 times Oct 9 06:35:10 clarion last message repeated 120 times Oct 9 06:45:11 clarion last message repeated 120 times Oct 9 06:55:12 clarion last message repeated 120 times Oct 9 07:05:12 clarion last message repeated 120 times Oct 9 07:15:13 clarion last message repeated 120 times Oct 9 07:25:14 clarion last message repeated 120 times Oct 9 07:35:15 clarion last message repeated 120 times Oct 9 07:45:14 clarion last message repeated 120 times Oct 9 07:55:15 clarion last message repeated 120 times Oct 9 08:05:13 clarion last message repeated 119 times [..]
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJVBO0ACgkQwcJ4iSZ1q2kHHgCfdhCedWyXnIMWpzy1cMFVmP6w RG0AoID+kJ/9RIHQLyq3pEJWTgkJsBVJ =oIIX -----END PGP SIGNATURE-----
Hi Alexander! Here some additional information do you requested. On a other server (same OS) I running a BIRD 1.3.11 with OSPFv2, OSPFv3 and a BGP Peering with Full-Prefix-Table. That works fine. --- BIRD configuration --- #log syslog { debug, trace, info, remote, warning, error, auth, fatal, bug }; router id 8X.2X.32.210; #protocol direct { // Commented out for testing purposes # interface "vlan*"; #} protocol kernel { learn; persist; scan time 20; import all; export all; } protocol device { scan time 10; } protocol ospf activezone { rfc1583compat yes; area 0.0.0.0 { interface "vlan3" { cost 10; type broadcast; }; interface "vlan2" { cost 10; type broadcast; passwords { password "totaly secret ;-)" { id 1; }; }; authentication cryptographic; }; }; export all; import all; } Am 09.10.2013 09:25, schrieb Alexander V. Chernikov:
Did this happen with quagga?
No! Quagga works fine but I think BIRD is more powerful (Filtering).
Virtual servers are no longer reachable. What's the problem? The same messages are generated by OSPFv3 processes. Em, "arpresolve" messages are generated by kernel. Are you talking about "LSA disappeared" ones?
Yes! The "arpresolv" messages are an result of starting BIRD daemon.
Can you provide bird config (at least direct/ospf protocol parts) and describe network topology?
See above.
E.g. how the server can reach "8X.2X.32.211" or other addresses in dmesg. What does output of "route get 8X.2X.32.211" show (when working "normal" and when problem starts) ?
route get 8X.2X.32.211 route to: <hostname> destination: 8X.2X.32.192 mask: 255.255.255.224 interface: vlan2 flags: <UP,DONE> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 Thanks + Regards, Markus
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 09.10.2013 11:36, Markus Grundmann wrote:
Hi Alexander!
Here some additional information do you requested. On a other server (same OS) I running a BIRD 1.3.11 with OSPFv2, OSPFv3 and a BGP Peering with Full-Prefix-Table. That works fine.
--- BIRD configuration ---
#log syslog { debug, trace, info, remote, warning, error, auth, fatal, bug }; router id 8X.2X.32.210;
#protocol direct { // Commented out for testing purposes # interface "vlan*"; #} It is significant. Did you have this uncommented initially?
protocol kernel { learn; persist; scan time 20; import all; export all; }
protocol device { scan time 10; }
protocol ospf activezone { rfc1583compat yes; Do you really need this?
area 0.0.0.0 { interface "vlan3" { cost 10; type broadcast; };
interface "vlan2" { cost 10; type broadcast; passwords { password "totaly secret ;-)" { id 1; }; };
authentication cryptographic; };
};
export all; import all; }
Am 09.10.2013 09:25, schrieb Alexander V. Chernikov:
Did this happen with quagga?
No! Quagga works fine but I think BIRD is more powerful (Filtering).
Virtual servers are no longer reachable. What's the problem? The same messages are generated by OSPFv3 processes. Em, "arpresolve" messages are generated by kernel. Are you talking about "LSA disappeared" ones?
Yes! The "arpresolv" messages are an result of starting BIRD daemon.
Can you provide bird config (at least direct/ospf protocol parts) and describe network topology?
See above.
E.g. how the server can reach "8X.2X.32.211" or other addresses in dmesg. What does output of "route get 8X.2X.32.211" show (when working "normal" and when problem starts) ?
route get 8X.2X.32.211 route to: <hostname> destination: 8X.2X.32.192 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Should it be directly reachable or?
mask: 255.255.255.224 interface: vlan2 flags: <UP,DONE> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0
There are 2 FreeBSD cases which probably can play here: 1) Due to absence of Direct routes (or Direct protocol route reannounce) bird can change direct interface route to OSPF-one (this is probably what we see here). This is fixed (for IPv4 case) in 9.2 - there are special RTM_PINNED flag for non-interface routes. I hope to fix the rest (IPv6 and tunnels stuff) soon. 2) Given that you have several addresses on interface Direct route reannounce can happen on primary (first) address change. So, probably the best way is to update to 9.2 (or at least r248895). If your interface addresses are stable, adding "Direct" protocol back should mitigate the problem
Thanks + Regards, Markus
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJVC58ACgkQwcJ4iSZ1q2kDMgCeK4QqOsQMrCRLDBSCoHJ4dsf6 JH0AoJ+1qeOl1q4fJ2e872VJ21fQ3bOx =ZlaC -----END PGP SIGNATURE-----
Alexander thank you for your reply. On 10/09/2013 09:54 AM, Alexander V. Chernikov wrote:
It is significant. Did you have this uncommented initially?
OK. I have probed some configuration options to find out where my problem possibly exist
protocol ospf activezone { rfc1583compat yes; Do you really need this?
No!
route get 8X.2X.32.211 route to: <hostname> destination: 8X.2X.32.192 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Should it be directly reachable or?
Yes Alexander! .211 is in same broadcast domain.
There are 2 FreeBSD cases which probably can play here:
1) Due to absence of Direct routes (or Direct protocol route reannounce) bird can change direct interface route to OSPF-one (this is probably what we see here). This is fixed (for IPv4 case) in 9.2 - there are special RTM_PINNED flag for non-interface routes. I hope to fix the rest (IPv6 and tunnels stuff) soon.
OK. Yesterday I have downloaded Release 9.2 but I'm a bit cautious to do an upgrade on this maschine. I hope in the next Patchlevel for R 9.1 this problem was also solved.
2) Given that you have several addresses on interface Direct route reannounce can happen on primary (first) address change.
So, probably the best way is to update to 9.2 (or at least r248895). If your interface addresses are stable, adding "Direct" protocol back should mitigate the problem
Thank you Alexander! This post is very helpful for me. -Markus
Now I have the problem again. I have start remotly the BIRD (config modified / quagga was stopped). After 20 minutes that the "switch" was done the server is now unreachable. The virtual machine with the IP .195 works but can't login into the host. All other servers can't connect to IP .210 and the system is isolated (offline) :-( Regards, Markus
On 09.10.2013 14:45, Markus Grundmann wrote:
Now I have the problem again. I have start remotly the BIRD (config modified / quagga was stopped). After 20 minutes that the "switch" was done the server is now unreachable. The virtual machine with the IP .195 works but can't login into the host. All other servers can't connect to IP .210 and the system is isolated (offline) :-( You can try link-local IPv6 address to access host.
Regards, Markus
On 09.10.2013 14:10, Markus Grundmann wrote:
Alexander thank you for your reply.
On 10/09/2013 09:54 AM, Alexander V. Chernikov wrote:
It is significant. Did you have this uncommented initially? OK. I have probed some configuration options to find out where my problem possibly exist
protocol ospf activezone { rfc1583compat yes; Do you really need this? No!
route get 8X.2X.32.211 route to: <hostname> destination: 8X.2X.32.192 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Should it be directly reachable or? Yes Alexander! .211 is in same broadcast domain.
There are 2 FreeBSD cases which probably can play here:
1) Due to absence of Direct routes (or Direct protocol route reannounce) bird can change direct interface route to OSPF-one (this is probably what we see here). This is fixed (for IPv4 case) in 9.2 - there are special RTM_PINNED flag for non-interface routes. I hope to fix the rest (IPv6 and tunnels stuff) soon. OK. Yesterday I have downloaded Release 9.2 but I'm a bit cautious to do an upgrade on this maschine. I hope in the next Patchlevel for R 9.1 this problem was also solved. Usually patchlevel incorporates security fixes only.
However, you can try to write specific filter in kernel protocol which explicitly forbids to install routes which reside on your vlan2 interface: like if net ~ [8X.2X.32.216/28, ...] then reject; accept;
2) Given that you have several addresses on interface Direct route reannounce can happen on primary (first) address change.
So, probably the best way is to update to 9.2 (or at least r248895). If your interface addresses are stable, adding "Direct" protocol back should mitigate the problem
Thank you Alexander! This post is very helpful for me.
-Markus
Soo! I have modified the BIRD configuration.This seems better than before. In my last post you can see that 8X.2X.32.211 was routed via 8X.2X.32.192 (= error; Network address) and now ... # route -n get 8X.2X.32.211 route to: 8X.2X.32.211 destination: 8X.2X.32.211 gateway: 8X.2x.32.211 interface: vlan2 flags: <UP,GATEWAY,HOST,DONE,PROTO1> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 The new configuration see below. [OK! OK! The label of the filter is "bull..it" but it works ;-)] A little bit coriuos is that all services are available from outside but I can't login from host in the broadcast doamin. --SNIP-- filter nodirect_kernel { if net ~ [8X.2X.32.192/27] then reject; accept; } protocol direct { interface "vlan*"; } protocol kernel { learn; persist; scan time 20; import filter nodirect_kernel; export all; } --SNAP-- On 10/09/2013 12:54 PM, Alexander V. Chernikov wrote:
if net ~ [8X.2X.32.216/28, ...] then reject; accept;
Thank you very much Alexander!!! -- Best regards, Markus
participants (3)
-
Alexander V. Chernikov -
Alexander V. Chernikov -
Markus Grundmann