1 out of 2 BGP sessions down, all routes become unreachable
Hello, First message here, although a long-time bird user (not an expert in any way). We're using bird as a bgp speaker for all our VMs that either need to announce routes or import routes from VRFs etc. The setup is pretty straight forward I'd guess. The VM has 2 IPv6 BGP sessions with 2 L3 switches (one per switch). The VM announces one /128 route and imports all routes learned from the BGP sessions. The same routes are learned from both sessions, so whenever I do a 'ip -6 r' I see 2 entries for each route (with different next hops) like: 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::a dev eth2 proto bird metric 32 pref medium 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::b dev eth2 proto bird metric 32 pref medium At some point, both BGP sessions went down and one of them did not recover for an hour. Normally, this should not be a problem but in this specific case, I could no longer see 2 routes in the kernel table, one reachable and another unreachable, just an unreachable one, so even though the anycast route was being announced and traffic was reaching the VM, it couldn't send the packets to their destination as all routes were unreachable in the kernel table: unreachable 2a02:xxxx:d00d::/45 dev lo proto bird metric 32 error -113 pref medium . I'm running 2.0.7 (upgrading soon) on centos7 and here's the relevant part of my config: protocol device { } protocol direct direct_v6_lo { interface "lo*"; # announce loopback ipv6; } protocol direct direct_v6_eth { interface "eth*"; ipv6; } protocol bgp bgp_v6_a { local 2a02:xxxx:yyyy:53::37 as 66666 neighbor 2a02:xxxx:yyyy:53::a as 66666; # switch A ipv6 { export where proto = "direct_v6_lo"; }; } protocol bgp bgp_v6_b { local 2a02:1388:4000:53::37 as 66666; neighbor 2a02:1388:4000:53::b as 66666; # switch B ipv6 { export where proto = "direct_v6_lo"; }; } protocol kernel { ipv6 { export filter { if proto = "direct_v6_eth" then reject; accept; }; }; merge paths limit 2; } any ideas? thanks in advance and regards, Yannis
Hi, And what do you see in your bird's console? "show route" etc. On Fri, Apr 9, 2021 at 7:10 PM Dez C <dez@otenet.gr> wrote:
Hello,
First message here, although a long-time bird user (not an expert in any way). We're using bird as a bgp speaker for all our VMs that either need to announce routes or import routes from VRFs etc.
The setup is pretty straight forward I'd guess. The VM has 2 IPv6 BGP sessions with 2 L3 switches (one per switch). The VM announces one /128 route and imports all routes learned from the BGP sessions. The same routes are learned from both sessions, so whenever I do a 'ip -6 r' I see 2 entries for each route (with different next hops) like: 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::a dev eth2 proto bird metric 32 pref medium 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::b dev eth2 proto bird metric 32 pref medium
At some point, both BGP sessions went down and one of them did not recover for an hour. Normally, this should not be a problem but in this specific case, I could no longer see 2 routes in the kernel table, one reachable and another unreachable, just an unreachable one, so even though the anycast route was being announced and traffic was reaching the VM, it couldn't send the packets to their destination as all routes were unreachable in the kernel table: unreachable 2a02:xxxx:d00d::/45 dev lo proto bird metric 32 error -113 pref medium .
I'm running 2.0.7 (upgrading soon) on centos7 and here's the relevant part of my config:
protocol device { } protocol direct direct_v6_lo { interface "lo*"; # announce loopback ipv6; } protocol direct direct_v6_eth { interface "eth*"; ipv6; } protocol bgp bgp_v6_a { local 2a02:xxxx:yyyy:53::37 as 66666 neighbor 2a02:xxxx:yyyy:53::a as 66666; # switch A ipv6 { export where proto = "direct_v6_lo"; }; } protocol bgp bgp_v6_b { local 2a02:1388:4000:53::37 as 66666; neighbor 2a02:1388:4000:53::b as 66666; # switch B ipv6 { export where proto = "direct_v6_lo"; }; } protocol kernel { ipv6 { export filter { if proto = "direct_v6_eth" then reject; accept; }; }; merge paths limit 2; }
any ideas? thanks in advance and regards, Yannis
Hi, all external routes were unreachable (apart from the local loopback which was imported via direct protocol). master6 consisted solely of entries like: 2a02:xxxx:d00d::/45 unreachable [bgp_v6_a 19:38:23.222 from 2a02:xxxx:yyyy:53::a] * (100) [ASxxxxx1i] No entries whatsoever from the other bgp session (bgp_v6_b), like in normal operation: 2a02:xxxx:d00d::/45 unicast [bgp_v6_a 2021-04-07] * (100) [ASxxxxx1i] via 2a02:xxxx:yyyy:53::a on eth2 unicast [bgp_v6_b 2021-04-07] (100) [ASxxxxx1i] via 2a02:xxxx:yyyy:53::b on eth2 regards, Yannis On 4/9/21 11:22 PM, Alexander Zubkov wrote:
Hi,
And what do you see in your bird's console? "show route" etc.
On Fri, Apr 9, 2021 at 7:10 PM Dez C <dez@otenet.gr> wrote:
Hello,
First message here, although a long-time bird user (not an expert in any way). We're using bird as a bgp speaker for all our VMs that either need to announce routes or import routes from VRFs etc.
The setup is pretty straight forward I'd guess. The VM has 2 IPv6 BGP sessions with 2 L3 switches (one per switch). The VM announces one /128 route and imports all routes learned from the BGP sessions. The same routes are learned from both sessions, so whenever I do a 'ip -6 r' I see 2 entries for each route (with different next hops) like: 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::a dev eth2 proto bird metric 32 pref medium 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::b dev eth2 proto bird metric 32 pref medium
At some point, both BGP sessions went down and one of them did not recover for an hour. Normally, this should not be a problem but in this specific case, I could no longer see 2 routes in the kernel table, one reachable and another unreachable, just an unreachable one, so even though the anycast route was being announced and traffic was reaching the VM, it couldn't send the packets to their destination as all routes were unreachable in the kernel table: unreachable 2a02:xxxx:d00d::/45 dev lo proto bird metric 32 error -113 pref medium .
I'm running 2.0.7 (upgrading soon) on centos7 and here's the relevant part of my config:
protocol device { } protocol direct direct_v6_lo { interface "lo*"; # announce loopback ipv6; } protocol direct direct_v6_eth { interface "eth*"; ipv6; } protocol bgp bgp_v6_a { local 2a02:xxxx:yyyy:53::37 as 66666 neighbor 2a02:xxxx:yyyy:53::a as 66666; # switch A ipv6 { export where proto = "direct_v6_lo"; }; } protocol bgp bgp_v6_b { local 2a02:1388:4000:53::37 as 66666; neighbor 2a02:1388:4000:53::b as 66666; # switch B ipv6 { export where proto = "direct_v6_lo"; }; } protocol kernel { ipv6 { export filter { if proto = "direct_v6_eth" then reject; accept; }; }; merge paths limit 2; }
any ideas? thanks in advance and regards, Yannis
Hi, Please show also: show interface show route all So when you have both bgp_v6_a and bgp_v6_b established - you have normal routes. And when you have only bgp_v6_a established - you have that unreachable route only? Your bgp sessions are internal (iBGP), so they may probably have bgp_nex_hop different from the router's address. And also because of recursive resolution they may depend on other routes you have in your bird. On Sat, Apr 10, 2021 at 8:02 PM Dez C <dez@otenet.gr> wrote:
Hi,
all external routes were unreachable (apart from the local loopback which was imported via direct protocol). master6 consisted solely of entries like:
2a02:xxxx:d00d::/45 unreachable [bgp_v6_a 19:38:23.222 from 2a02:xxxx:yyyy:53::a] * (100) [ASxxxxx1i]
No entries whatsoever from the other bgp session (bgp_v6_b), like in normal operation:
2a02:xxxx:d00d::/45 unicast [bgp_v6_a 2021-04-07] * (100) [ASxxxxx1i] via 2a02:xxxx:yyyy:53::a on eth2 unicast [bgp_v6_b 2021-04-07] (100) [ASxxxxx1i] via 2a02:xxxx:yyyy:53::b on eth2
regards, Yannis
On 4/9/21 11:22 PM, Alexander Zubkov wrote:
Hi,
And what do you see in your bird's console? "show route" etc.
On Fri, Apr 9, 2021 at 7:10 PM Dez C <dez@otenet.gr> wrote:
Hello,
First message here, although a long-time bird user (not an expert in any way). We're using bird as a bgp speaker for all our VMs that either need to announce routes or import routes from VRFs etc.
The setup is pretty straight forward I'd guess. The VM has 2 IPv6 BGP sessions with 2 L3 switches (one per switch). The VM announces one /128 route and imports all routes learned from the BGP sessions. The same routes are learned from both sessions, so whenever I do a 'ip -6 r' I see 2 entries for each route (with different next hops) like: 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::a dev eth2 proto bird metric 32 pref medium 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::b dev eth2 proto bird metric 32 pref medium
At some point, both BGP sessions went down and one of them did not recover for an hour. Normally, this should not be a problem but in this specific case, I could no longer see 2 routes in the kernel table, one reachable and another unreachable, just an unreachable one, so even though the anycast route was being announced and traffic was reaching the VM, it couldn't send the packets to their destination as all routes were unreachable in the kernel table: unreachable 2a02:xxxx:d00d::/45 dev lo proto bird metric 32 error -113 pref medium .
I'm running 2.0.7 (upgrading soon) on centos7 and here's the relevant part of my config:
protocol device { } protocol direct direct_v6_lo { interface "lo*"; # announce loopback ipv6; } protocol direct direct_v6_eth { interface "eth*"; ipv6; } protocol bgp bgp_v6_a { local 2a02:xxxx:yyyy:53::37 as 66666 neighbor 2a02:xxxx:yyyy:53::a as 66666; # switch A ipv6 { export where proto = "direct_v6_lo"; }; } protocol bgp bgp_v6_b { local 2a02:1388:4000:53::37 as 66666; neighbor 2a02:1388:4000:53::b as 66666; # switch B ipv6 { export where proto = "direct_v6_lo"; }; } protocol kernel { ipv6 { export filter { if proto = "direct_v6_eth" then reject; accept; }; }; merge paths limit 2; }
any ideas? thanks in advance and regards, Yannis
Hello again, On 4/10/21 10:00 PM, Alexander Zubkov wrote:
Hi,
Please show also: show interface lo up (index=1) MultiAccess AdminUp LinkUp Loopback Ignored MTU=65536 127.0.0.1/8 (Preferred, scope host) a.b.c.1/32 (scope univ) a.b.c.2/32 (scope univ) 2a02:x:2::53/128 (Preferred, scope univ) 2a02:x:1:2::53/128 (scope univ) ::1/128 (scope host) eth0 up (index=2) MultiAccess Broadcast Multicast AdminUp LinkUp MTU=8950 192.168.0.6/24 (Preferred, scope site) fe80::x:x:x:x/64 (Preferred, scope link) eth1 up (index=3) MultiAccess Broadcast Multicast AdminUp LinkUp MTU=9000 a.b.71.70/27 (Preferred, scope univ) 2a02:x:y:33::70/64 (Preferred, scope univ) fe80::y:y:y:y/64 (Preferred, scope link) eth2 up (index=4) MultiAccess Broadcast Multicast AdminUp LinkUp MTU=9000 10.20.18.69/28 (Preferred, scope site) 2a02:x:x:53::69/64 (Preferred, scope univ) fe80::z:z:z:z/64 (Preferred, scope link)
show route all
unfortunately, I didn't take this output during the timeframe the problem occured
So when you have both bgp_v6_a and bgp_v6_b established - you have normal routes. And when you have only bgp_v6_a established - you have that unreachable route only?
correct on both counts
Your bgp sessions are internal (iBGP), so they may probably have bgp_nex_hop different from the router's address.
correct
And also because of recursive resolution they may depend on other routes you have in your bird.
that, I don't follow regards, Yannis
On Sat, Apr 10, 2021 at 8:02 PM Dez C <dez@otenet.gr> wrote:
Hi,
all external routes were unreachable (apart from the local loopback which was imported via direct protocol). master6 consisted solely of entries like:
2a02:xxxx:d00d::/45 unreachable [bgp_v6_a 19:38:23.222 from 2a02:xxxx:yyyy:53::a] * (100) [ASxxxxx1i]
No entries whatsoever from the other bgp session (bgp_v6_b), like in normal operation:
2a02:xxxx:d00d::/45 unicast [bgp_v6_a 2021-04-07] * (100) [ASxxxxx1i] via 2a02:xxxx:yyyy:53::a on eth2 unicast [bgp_v6_b 2021-04-07] (100) [ASxxxxx1i] via 2a02:xxxx:yyyy:53::b on eth2
regards, Yannis
On 4/9/21 11:22 PM, Alexander Zubkov wrote:
Hi,
And what do you see in your bird's console? "show route" etc.
On Fri, Apr 9, 2021 at 7:10 PM Dez C <dez@otenet.gr> wrote:
Hello,
First message here, although a long-time bird user (not an expert in any way). We're using bird as a bgp speaker for all our VMs that either need to announce routes or import routes from VRFs etc.
The setup is pretty straight forward I'd guess. The VM has 2 IPv6 BGP sessions with 2 L3 switches (one per switch). The VM announces one /128 route and imports all routes learned from the BGP sessions. The same routes are learned from both sessions, so whenever I do a 'ip -6 r' I see 2 entries for each route (with different next hops) like: 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::a dev eth2 proto bird metric 32 pref medium 2a02:xxxx:d00d::/45 via 2a02:xxxx:yyyy:zz::b dev eth2 proto bird metric 32 pref medium
At some point, both BGP sessions went down and one of them did not recover for an hour. Normally, this should not be a problem but in this specific case, I could no longer see 2 routes in the kernel table, one reachable and another unreachable, just an unreachable one, so even though the anycast route was being announced and traffic was reaching the VM, it couldn't send the packets to their destination as all routes were unreachable in the kernel table: unreachable 2a02:xxxx:d00d::/45 dev lo proto bird metric 32 error -113 pref medium .
I'm running 2.0.7 (upgrading soon) on centos7 and here's the relevant part of my config:
protocol device { } protocol direct direct_v6_lo { interface "lo*"; # announce loopback ipv6; } protocol direct direct_v6_eth { interface "eth*"; ipv6; } protocol bgp bgp_v6_a { local 2a02:xxxx:yyyy:53::37 as 66666 neighbor 2a02:xxxx:yyyy:53::a as 66666; # switch A ipv6 { export where proto = "direct_v6_lo"; }; } protocol bgp bgp_v6_b { local 2a02:1388:4000:53::37 as 66666; neighbor 2a02:1388:4000:53::b as 66666; # switch B ipv6 { export where proto = "direct_v6_lo"; }; } protocol kernel { ipv6 { export filter { if proto = "direct_v6_eth" then reject; accept; }; }; merge paths limit 2; }
any ideas? thanks in advance and regards, Yannis
participants (3)
-
Alexander Zubkov -
Dez C -
Yannis Nikolopoulos