Ondrej Zajicek <santiago@crfreenet.org> writes:
Yes, this is kind of confusing error message (as i noted in response to Simon Ruderich).
If there is one thing I'd suggest to improve first: print out the specific route and the next_hop - this would already help so much to debug the issue.
In detail:
router1, router2 are peered to apu-router1,apu-router2 via OSPF + BGP.
apu-router1,apu-router2 are peered to a set of kubernetes hosts.
The goal is to have router1 + router2 import the routes sent by the kubernetes hosts:
router1 router2---------| | \ | | | \ | | | \ | | apu-router1 apu-router2 | . | . | |-------------------------- . . [ kubernetes cluster via apu-routers ]
The problem: router1+router reject the routes with:
Dec 14 20:33:51 router1 daemon.err bird: apu_router1_place5_ungleich_ch_v6: Invalid NEXT_HOP attribute
The setup:
router1, router2, apu-router1, apu-router2 = ASN209898 kubernetes hosts = ASN65533 kubernetes peers with apu-routers only.
So the BGP link between kubernetes and APU-ROUTER is EBGP, while between APU-ROUTER and ROUTER is IBGP?
Correct.
I expect it is in multihop / gateway-recursive mode, as it is default for IBGP?
It's in direct mode for all links, as the APU-ROUTERS have physical links inside the kubernetes cluster as well as physical connection to the ROUTER.
The routes: Kubernetes announces parts of 2a0a:e5c0:0:12::/64 and 2a0a:e5c0:0:13::/64, for instance the route 2a0a:e5c0:0:12:b01a:5ae3:1bd4:1e00/122.
Kubernetes nodes live in 2a0a:e5c0::/64.
apu-routers have a leg in 2a0a:e5c0::/64, via eth1.2. They reach the cluster directly. They have the routes.
routers1+2 receive the route for 2a0a:e5c0::/64 via ospf:
bird> show route 2a0a:e5c0::/64 Table master6: 2a0a:e5c0::/64 unicast [ospf6 17:08:18.515] * I (150/20) [0.0.0.47] via fe80::20d:b9ff:fe57:2f91 on bond0.8 The apu-routers: - They import the route [0]
Could you show 'show route all' on apu-router for failed routes to see their BGP_NEXT_HOP attribute? And also ideally tcpdump output to see BGP_NEXT_HOP as sent from apu-router to router?
bird> show route all 2a0a:e5c0:0:12:b01a:5ae3:1bd4:1e00/122 Table master6: 2a0a:e5c0:0:12:b01a:5ae3:1bd4:1e00/122 unicast [k8s_p5_1_6 2021-12-14 from 2a0a:e5c0::225:90ff:fe1e:3e74] * (100) [AS65533i] via 2a0a:e5c0::225:90ff:fe1e:3e62 on eth1.2 Type: BGP univ BGP.origin: IGP BGP.as_path: 65533 BGP.next_hop: 2a0a:e5c0::225:90ff:fe1e:3e62 BGP.local_pref: 100 unicast [k8s_p5_1_5 2021-12-14 from 2a0a:e5c0::225:90ff:fe1a:d680] (100) [AS65533i] via 2a0a:e5c0::225:90ff:fe1e:3e62 on eth1.2 Type: BGP univ BGP.origin: IGP BGP.as_path: 65533 BGP.next_hop: 2a0a:e5c0::225:90ff:fe1e:3e62 BGP.local_pref: 100 unicast [k8s_p5_1_2 2021-12-14] (100) [AS65533i] via 2a0a:e5c0::225:90ff:fe1e:3e62 on eth1.2 Type: BGP univ BGP.origin: IGP BGP.as_path: 65533 BGP.next_hop: 2a0a:e5c0::225:90ff:fe1e:3e62 BGP.local_pref: 100 unicast [k8s_p5_1_1 2021-12-14 from 2a0a:e5c0::225:90ff:fe1a:d682] (100) [AS65533i] via 2a0a:e5c0::225:90ff:fe1e:3e62 on eth1.2 Type: BGP univ BGP.origin: IGP BGP.as_path: 65533 BGP.next_hop: 2a0a:e5c0::225:90ff:fe1e:3e62 BGP.local_pref: 100 unicast [k8s_p5_1_3 2021-12-14 from 2a0a:e5c0::225:90ff:fe1e:3e64] (100) [AS65533i] via 2a0a:e5c0::225:90ff:fe1e:3e62 on eth1.2 Type: BGP univ BGP.origin: IGP BGP.as_path: 65533 BGP.next_hop: 2a0a:e5c0::225:90ff:fe1e:3e62 BGP.local_pref: 100 unicast [k8s_p5_1_4 2021-12-14 from 2a0a:e5c0::225:90ff:fe1e:62d6] (100) [AS65533i] via 2a0a:e5c0::225:90ff:fe1e:3e62 on eth1.2 Type: BGP univ BGP.origin: IGP BGP.as_path: 65533 BGP.next_hop: 2a0a:e5c0::225:90ff:fe1e:3e62 BGP.local_pref: 100 Pcap is attached as well.
- They export the route to the routers [1]
Could you also show 'show protocol all' for the session from apu-router to the kubernetes hosts?
bird> show protocols ... k8s_p5_1_1 BGP --- up 2021-12-14 Established k8s_p5_1_2 BGP --- up 2021-12-14 Established k8s_p5_1_3 BGP --- up 2021-12-14 Established k8s_p5_1_4 BGP --- up 2021-12-14 Established k8s_p5_1_5 BGP --- up 2021-12-14 Established k8s_p5_1_6 BGP --- up 2021-12-14 Established ... bird> show protocols all k8s_p5_1_6 Name Proto Table State Since Info k8s_p5_1_6 BGP --- up 2021-12-14 Established BGP state: Established Neighbor address: 2a0a:e5c0::225:90ff:fe1e:3e74 Neighbor AS: 65533 Local AS: 209898 Neighbor ID: 77.29.68.245 Local capabilities Multiprotocol AF announced: ipv6 Route refresh Graceful restart 4-octet AS numbers Enhanced refresh Long-lived graceful restart Neighbor capabilities Multiprotocol AF announced: ipv6 Route refresh Graceful restart Restart time: 120 AF supported: ipv6 AF preserved: 4-octet AS numbers ADD-PATH RX: ipv6 TX: ipv6 Enhanced refresh Long-lived graceful restart Session: external AS4 Source address: 2a0a:e5c0::46 Hold timer: 158.633/240 Keepalive timer: 53.250/80 Channel ipv6 State: UP Table: master6 Preference: 100 Input filter: ungleich_networks_no_igp Output filter: REJECT Routes: 4 imported, 0 exported, 4 preferred Route change stats: received rejected filtered ignored accepted Import updates: 4 0 0 0 4 Import withdraws: 0 0 --- 0 0 Export updates: 43 6 37 --- 0 Export withdraws: 0 --- --- --- 0 BGP Next hop: 2a0a:e5c0::46 fe80::20d:b9ff:fe57:2f91
The routers: - print 4x the Invalid NEXT_HOP attribute, once per exported kubernetes network - They ignore the 4 routes [2]
Question: why does bird on the routers not accept the routes? Or is there a different problem I am not seeing? Aside from that, shouldn't bird on the apu-routers set itself as nexthop for the kubernetes routes?
Not, because sending it to routerx over IBGP link, where BGP_NEXT_HOP is kept unmodified by default (unless 'next hop self' option is used).
Correct, sorry for my confusion. Actually setting the next hop self does to some degree fix the problem, with the usual disadvantage of rewriting the attribute. Curious to what we are running into here. For reference the bgp configuration sections:
From the routers:
-------------------------------------------------------------------------------- protocol bgp apu_router1_place5_ungleich_ch_v6 { local as 209898; neighbor 2a0a:e5c0:1:8::46 as 209898; direct; ipv6 { # What we accept from this protocol -> others send us import filter ungleich_networks; # What we export into this protocol -> what we send export filter ungleich_networks_no_igp; }; # Highest preference on internal traffic default bgp_local_pref pref_normal; } --------------------------------------------------------------------------------
From the apu-routers:
protocol bgp router1_place5_ungleich_ch_v6 { local as 209898; neighbor 2a0a:e5c0:1:8::3 as 209898; direct; ipv6 { # What we accept from this protocol -> others send us import none; # What we export into this protocol -> what we send export filter ungleich_networks_no_igp; }; # Highest preference on internal traffic default bgp_local_pref pref_normal; } protocol bgp k8s_p5_1_v6 { local as 209898; neighbor range 2a0a:e5c0:0::/64 as 65533; dynamic name "k8s_p5_1_"; ipv6 { # What we accept from this protocol -> others send us import filter ungleich_networks_no_igp; # What we export into this protocol -> what we send export none; }; # Highest preference on internal traffic default bgp_local_pref pref_normal; } -------------------------------------------------------------------------------- I actually just realised that the "k8s_p5_1_v6" protocol did not have the direct statement, however adding it does not change the situation (probably because bird detects it as a direct link anyway). Best regards from the snowy mountains, Nico -- Sustainable and modern Infrastructures by ungleich.ch