Hi, we're running redundant pairs of routers using keepalived for VRRP and BIRD for OSPF. (bird-1.3.3 through bird-1.3.6 actually) Static routes are configured in BIRD so when keepalived transitions to master state it adds ip addresses to a bunch of vlan interfaces and the routes' gateways become directly reachable and are being announced. Now when I failover from one primary router to a backup router by stopping keepalived the interfaces' addresses are being deleted on the former master and added to the backup router. Immediately after such a transition all BIRD routers fill their log files periodically with this kind of messages: Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170) proto/ospf/lsupd.c says this should never happen: /* Probably flushed LSA, this should not happen */ log(L_WARN "OSPF: LSA disappeared (Type: %04x, Id: %R, Rt: %R)", lsr->lsh.type, lsr->lsh.id, lsr->lsh.rt); Is this some bug in BIRD or caused by the way we use it? Regards Bjoern -- Bjoern Engels OpenIT GmbH engels@openit.de In der Steele 33a-41 PGP keyID 1024D/895F13C3 D-40599 Duesseldorf ________________________________________________________________________ HRB 38815 Amtsgericht Duesseldorf USt-Id DE 812951861 Geschaeftsfuehrer: Oliver Haakert, Maurice Kemmann
I see this too, different setup. I had to roll back to 1.3.1. Sorry can not provide more info, other than, seems to be an issue for me too ;-) Setup: freebsd 8.2 bird 1.3.3 and 1.3.4 On 2012.01.25, at 06:14, Bjoern Engels wrote:
Hi,
we're running redundant pairs of routers using keepalived for VRRP and BIRD for OSPF. (bird-1.3.3 through bird-1.3.6 actually)
Static routes are configured in BIRD so when keepalived transitions to master state it adds ip addresses to a bunch of vlan interfaces and the routes' gateways become directly reachable and are being announced.
Now when I failover from one primary router to a backup router by stopping keepalived the interfaces' addresses are being deleted on the former master and added to the backup router.
Immediately after such a transition all BIRD routers fill their log files periodically with this kind of messages:
Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170)
proto/ospf/lsupd.c says this should never happen:
/* Probably flushed LSA, this should not happen */ log(L_WARN "OSPF: LSA disappeared (Type: %04x, Id: %R, Rt: %R)", lsr->lsh.type, lsr->lsh.id, lsr->lsh.rt);
Is this some bug in BIRD or caused by the way we use it?
Regards Bjoern -- Bjoern Engels OpenIT GmbH engels@openit.de In der Steele 33a-41 PGP keyID 1024D/895F13C3 D-40599 Duesseldorf ________________________________________________________________________ HRB 38815 Amtsgericht Duesseldorf USt-Id DE 812951861 Geschaeftsfuehrer: Oliver Haakert, Maurice Kemmann
I can also see these messages in logs. Our setup is several Linux boxes with bird 1.3.3 and one Cisco 7206. Linux servers are used as PPPoE access concentrators. Amount of these messages differs between servers and days. For today, there are just 50 messages on 4 of them, ~100k messages on other and ~1.5M on last one. One of the machines that has only 50 messages in logs for today, yesterday had 8M. On Wed, Jan 25, 2012 at 8:41 PM, harold barker <hvb@dsms.com> wrote:
I see this too, different setup. I had to roll back to 1.3.1. Sorry can not provide more info, other than, seems to be an issue for me too ;-)
Setup: freebsd 8.2 bird 1.3.3 and 1.3.4
On 2012.01.25, at 06:14, Bjoern Engels wrote:
Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) Jan 25 14:47:11 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 198.51.100.160, Rt: 10.10.10.170)
-- Timur Irmatov, xmpp:irmatov@jabber.ru
On Wed, Jan 25, 2012 at 03:14:10PM +0100, Bjoern Engels wrote:
Hi,
we're running redundant pairs of routers using keepalived for VRRP and BIRD for OSPF. (bird-1.3.3 through bird-1.3.6 actually)
Static routes are configured in BIRD so when keepalived transitions to master state it adds ip addresses to a bunch of vlan interfaces and the routes' gateways become directly reachable and are being announced.
Now when I failover from one primary router to a backup router by stopping keepalived the interfaces' addresses are being deleted on the former master and added to the backup router.
So the failover would cause removal of these static routes?
Immediately after such a transition all BIRD routers fill their log files periodically with this kind of messages:
Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170)
I guess these LSAs represents old static routes (Rt: 10.10.10.170 is the router that removed the static routes), is this true? Are there any problems with this other than log filling? You could check using 'show ospf state' and 'show ospf lsadb' whether the old LSAs/routes are properly removed and the new are properly installed during failover (and during the failover back to the primary).
proto/ospf/lsupd.c says this should never happen:
/* Probably flushed LSA, this should not happen */ log(L_WARN "OSPF: LSA disappeared (Type: %04x, Id: %R, Rt: %R)", lsr->lsh.type, lsr->lsh.id, lsr->lsh.rt);
Is this some bug in BIRD or caused by the way we use it?
This is probably some minor bug in LSA flushing, i will check that. One problem with log filling is that the message is repeated during each update. -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
2012/1/30 Ondrej Zajicek <santiago@crfreenet.org>:
Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170) I guess these LSAs represents old static routes (Rt: 10.10.10.170 is the router that removed the static routes), is this true?
My guess is that happens to routes/destinations which are present in both, the kernel pool and OSPF pool. I was getting this kind of messages since 1.3.0 or even earlier: Jan 30 11:06:53 nat2 bird: OSPF: LSA disappeared (Type: 0002, Id: x.100.x.248, Rt: x.121.x.248) Jan 30 11:06:55 nat2 bird: OSPF: LSA disappeared (Type: 0002, Id: x.100.x.248, Rt: x.121.x.248) And both, x.100.x.248 and x.121.x.248 interfaces are present in OSPF topology and are "seen" by kernel, since the 248's belong to the backup router. -- rafamiga
On Mon, Jan 30, 2012 at 11:19:26AM +0100, Ondrej Zajicek wrote:
On Wed, Jan 25, 2012 at 03:14:10PM +0100, Bjoern Engels wrote:
we're running redundant pairs of routers using keepalived for VRRP and BIRD for OSPF. (bird-1.3.3 through bird-1.3.6 actually)
Static routes are configured in BIRD so when keepalived transitions to master state it adds ip addresses to a bunch of vlan interfaces and the routes' gateways become directly reachable and are being announced.
Now when I failover from one primary router to a backup router by stopping keepalived the interfaces' addresses are being deleted on the former master and added to the backup router.
So the failover would cause removal of these static routes?
Yes. When all addresses have been deleted on an interface it changes to DOWN so the routes' next hops become unreachable and the they are being withdrawn.
Immediately after such a transition all BIRD routers fill their log files periodically with this kind of messages:
Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.0, Rt: 10.10.10.170) Jan 25 14:47:06 cr4b bird: OSPF: LSA disappeared (Type: 0005, Id: 192.0.2.128, Rt: 10.10.10.170)
I guess these LSAs represents old static routes (Rt: 10.10.10.170 is the router that removed the static routes), is this true?
Correct.
Are there any problems with this other than log filling? You could check using 'show ospf state' and 'show ospf lsadb' whether the old LSAs/routes are properly removed and the new are properly installed during failover (and during the failover back to the primary).
Log filling (>> 100.000 entries per day*router) seems to be the only problem, "show ospf lsadb" shows LSAs that have been advertised by the active (master state) routers and LSAs of inactive/backup routers have been deleted.
proto/ospf/lsupd.c says this should never happen:
/* Probably flushed LSA, this should not happen */ log(L_WARN "OSPF: LSA disappeared (Type: %04x, Id: %R, Rt: %R)", lsr->lsh.type, lsr->lsh.id, lsr->lsh.rt);
Is this some bug in BIRD or caused by the way we use it?
This is probably some minor bug in LSA flushing, i will check that.
Alright, thank you! Regards Bjoern -- Bjoern Engels OpenIT GmbH engels@openit.de In der Steele 33a-41 PGP keyID 1024D/895F13C3 D-40599 Duesseldorf ________________________________________________________________________ HRB 38815 Amtsgericht Duesseldorf USt-Id DE 812951861 Geschaeftsfuehrer: Oliver Haakert, Maurice Kemmann
participants (5)
-
Bjoern Engels -
harold barker -
Ondrej Zajicek -
rafamiga -
Timur Irmatov