upgrade 1.2.4 - 1.3.1 reverted
Hello, upgraded from bird 1.2.4 to bird 1.3.1 last weekend, but needed to revert due to complaints from high bgp traffic to bgpmon.net and 100% cpu load on a 2x bird bgp routers setup connected with iBGP. Did some research, and noticed a high bgp traffic situation on the iBGP sessions (about 10.000 bgp packets in 1.7 second) Based on graph statistics, i noticed a irregular traffic on the trans-ip's as well (noticed after rollback in statistical graphs) In the logging (mostly on the first bgp router) , i noticed quite some messages (33414 in ~12h) like: 29-05-2011 10:57:49 <WARN> Next hop address X.X.X.242 resolvable through recursive route for X.X.X.0/24 29-05-2011 10:57:49 <WARN> Next hop address Y.Y.35.194 resolvable through recursive route for Y.Y.32.0/19 29-05-2011 10:57:52 <WARN> Next hop address X.X.X.241 resolvable through recursive route for X.X.X.0/24 29-05-2011 10:57:52 <WARN> Next hop address X.X.X.242 resolvable through recursive route for X.X.X.0/24 29-05-2011 10:57:55 <WARN> Next hop address Y.Y.35.194 resolvable through recursive route for Y.Y.32.0/19 29-05-2011 10:57:55 <WARN> Next hop address X.X.X.241 resolvable through recursive route for X.X.X.0/24 29-05-2011 10:57:55 <WARN> Next hop address X.X.X.242 resolvable through recursive route for X.X.X.0/24 These are the subnets of the upstream trans-ip links of it's neigbor router On the second (neighbor) bgp router, noticed only such 15 recursive messages I had those upstream trans-ip interface IP's manually added once to the bird.conf with a "protocol static" definition with a subnet of /32. I thought this might be the problem, and wel, disabled those static trans-ip interface routes, except i forgot one. Noticed a lower CPU afterwards, but problem persisted, and i reverted to 1.2.4. Later i noticed i missed one static route to disable, but no time left to upgrade and test again. I think i should have looked at the recursive routes in bird, but i didn't. Does one know what was going on here, and what to do to prevent it when executing another upgrade? I created a 10.000 packet tcpdump capture (full size) of an iBGP session, in case somone would like to look at it. Regards, Arjan Filius -- Arjan Filius mailto:iafilius@xs4all.nl
Hello Arjan, On Monday 30 May 2011 at 09:23 (CET), Arjan Filius wrote:
upgraded from bird 1.2.4 to bird 1.3.1 last weekend, but needed to revert due to complaints from high bgp traffic to bgpmon.net and 100% cpu load on a 2x bird bgp routers setup connected with iBGP.
Did you add "gateway direct;" to your iBGP session configuration? Bird 1.3.0 had some drastic changes regarding gateways and iBGP. Regards, Ruben Laban
Hello, Ruben, Ondrey no i didn't. next planned upgrade that's what i will do. regards, On Mon, 30 May 2011, Ruben Laban wrote:
Hello Arjan,
On Monday 30 May 2011 at 09:23 (CET), Arjan Filius wrote:
upgraded from bird 1.2.4 to bird 1.3.1 last weekend, but needed to revert due to complaints from high bgp traffic to bgpmon.net and 100% cpu load on a 2x bird bgp routers setup connected with iBGP.
Did you add "gateway direct;" to your iBGP session configuration? Bird 1.3.0 had some drastic changes regarding gateways and iBGP.
Regards, Ruben Laban
-- Arjan Filius mailto:iafilius@xs4all.nl
On Mon, May 30, 2011 at 09:23:12AM +0200, Arjan Filius wrote:
Hello,
upgraded from bird 1.2.4 to bird 1.3.1 last weekend, but needed to revert due to complaints from high bgp traffic to bgpmon.net and 100% cpu load on a 2x bird bgp routers setup connected with iBGP.
Did some research, and noticed a high bgp traffic situation on the iBGP sessions (about 10.000 bgp packets in 1.7 second)
Based on graph statistics, i noticed a irregular traffic on the trans-ip's as well (noticed after rollback in statistical graphs)
In the logging (mostly on the first bgp router) , i noticed quite some messages (33414 in ~12h) like: 29-05-2011 10:57:49 <WARN> Next hop address X.X.X.242 resolvable through recursive route for X.X.X.0/24 29-05-2011 10:57:49 <WARN> Next hop address Y.Y.35.194 resolvable through recursive route for Y.Y.32.0/19 29-05-2011 10:57:52 <WARN> Next hop address X.X.X.241 resolvable through recursive route for X.X.X.0/24 29-05-2011 10:57:52 <WARN> Next hop address X.X.X.242 resolvable through recursive route for X.X.X.0/24 29-05-2011 10:57:55 <WARN> Next hop address Y.Y.35.194 resolvable through recursive route for Y.Y.32.0/19 29-05-2011 10:57:55 <WARN> Next hop address X.X.X.241 resolvable through recursive route for X.X.X.0/24 29-05-2011 10:57:55 <WARN> Next hop address X.X.X.242 resolvable through recursive route for X.X.X.0/24
These are the subnets of the upstream trans-ip links of it's neigbor router
On the second (neighbor) bgp router, noticed only such 15 recursive messages
I had those upstream trans-ip interface IP's manually added once to the bird.conf with a "protocol static" definition with a subnet of /32. I thought this might be the problem, and wel, disabled those static trans-ip interface routes, except i forgot one. Noticed a lower CPU afterwards, but problem persisted, and i reverted to 1.2.4. Later i noticed i missed one static route to disable, but no time left to upgrade and test again.
Generally, the IP address from the BGP NEXT_HOP attribute (i.e. the ones mentioned in 'Next hop address X.X.X.242 resolvable ..' messages should be resolvable through a route which is not from BGP. So, if you do not use OSPF, adding static /32 routes for these IPs should do the trick. I don't know why that caused problems in your setting (Perhaps the routes have different NEXT_HOP attribute?). Other posibilities (like using 'gateway direct' to switch to the old behavior) were discussed here: http://marc.info/?l=bird-users&m=130173865608861&w=2
I created a 10.000 packet tcpdump capture (full size) of an iBGP session, in case somone would like to look at it.
For such kind of problem tcpdump is not really useful, more useful would be BIRD log with 'debug all all'; -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Hello Ondrej Zajicek, 2011-05-30, 17:04:16 you wrote:
Generally, the IP address from the BGP NEXT_HOP attribute (i.e. the ones mentioned in 'Next hop address X.X.X.242 resolvable ..' messages should be resolvable through a route which is not from BGP.
I just stumbled over that when trying to remove "next hop self" after 1.3 has recursive lookup. What's the problem with using iBGP routes for recursive lookups (when using iBGP as IGP)? The message even says that the route is resolvable. Setup (simplified, RR removed): [Router A] <-- VLAN 1 --> [Router B] <-- VLAN 2 --> [Router C] <-- VLAN 3 A and B have an iBGP session, B announces the net on VLAN 2 via itself and the net on VLAN 3 via Router C. Greetings Timo -- Timo Weingärtner Abteilung Technische InfraStruktur Karlsruher Institut für Technologie (KIT)
On Mon, Oct 22, 2012 at 07:52:48PM +0200, Timo Weingärtner wrote:
Hello Ondrej Zajicek,
2011-05-30, 17:04:16 you wrote:
Generally, the IP address from the BGP NEXT_HOP attribute (i.e. the ones mentioned in 'Next hop address X.X.X.242 resolvable ..' messages should be resolvable through a route which is not from BGP.
I just stumbled over that when trying to remove "next hop self" after 1.3 has recursive lookup.
What's the problem with using iBGP routes for recursive lookups (when using iBGP as IGP)? The message even says that the route is resolvable.
BIRD does not allow recursive routes to depend on another recursive route. If your iBGP is configured to generate non-recursive routes (using 'gateway direct' option), then recursive routes (from another BGP) could depend on that routes, but i am not sure if such setting could be useful, esp. it does not work if you want to use the same iBGP session for both recursive and non-recursive routes. -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Hallo Ondrej Zajicek, 2012-10-24 um 13:54:37 schriebst Du:
On Mon, Oct 22, 2012 at 07:52:48PM +0200, Timo Weingärtner wrote:
2011-05-30, 17:04:16 you wrote:
Generally, the IP address from the BGP NEXT_HOP attribute (i.e. the ones mentioned in 'Next hop address X.X.X.242 resolvable ..' messages should be resolvable through a route which is not from BGP.
I just stumbled over that when trying to remove "next hop self" after 1.3 has recursive lookup.
What's the problem with using iBGP routes for recursive lookups (when using iBGP as IGP)? The message even says that the route is resolvable.
BIRD does not allow recursive routes to depend on another recursive route. If your iBGP is configured to generate non-recursive routes (using 'gateway direct' option), then recursive routes (from another BGP) could depend on that routes, but i am not sure if such setting could be useful, esp. it does not work if you want to use the same iBGP session for both recursive and non-recursive routes.
Let me rephrase my question: Why does BIRD not allow recursive routes to depend on another recursive route. What could go wrong if it was allowed? Grüße Timo -- Timo Weingärtner Abteilung Technische InfraStruktur Karlsruher Institut für Technologie (KIT)
On Wed, Oct 24, 2012 at 03:50:24PM +0200, Timo Weingärtner wrote:
What's the problem with using iBGP routes for recursive lookups (when using iBGP as IGP)? The message even says that the route is resolvable.
BIRD does not allow recursive routes to depend on another recursive route. If your iBGP is configured to generate non-recursive routes (using 'gateway direct' option), then recursive routes (from another BGP) could depend on that routes, but i am not sure if such setting could be useful, esp. it does not work if you want to use the same iBGP session for both recursive and non-recursive routes.
Let me rephrase my question: Why does BIRD not allow recursive routes to depend on another recursive route. What could go wrong if it was allowed?
It is mainly for simplicity of some implementation issues and preventing problems like mutually recursive routes. -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
participants (4)
-
Arjan Filius -
Ondrej Zajicek -
Ruben Laban -
Timo Weingärtner