handling of unreachable routes with gateway direct
Hello, We have some routes that are passed via transit bird bgp-daemons without installing them to the kernel. And for some of these routes we need to keep their next hop attribute, because it is meaningful on the destination host, but may be unreachable on those transit hosts. If we have "gateway recursive" sessions with these transit hosts, then such routes are accepted, just marked unreachable, but we can work with them. And if we make the sessions "gateway direct", then such routes are ignored by the bird with "Invalid NEXT_HOP attribute" messages in log. Looks like it is caused by this piece of code in proto/bgp/packets.c in function bgp_apply_next_hop(): if (!nbr || (nbr->scope == SCOPE_HOST)) WITHDRAW(BAD_NEXT_HOP); It seems reasonable, of course, but it breaks things for our case. So I want to suggest some changes to make our setup possible. At the moment, I consider the possibility of adding something like "gateway <direct_with_recursive_fallback>" so that it would behave like "gateway direct", but in the case of a non-reachable neighbour, it would switch to recursive logic. What do you think? Whether this is a good idea or maybe you can suggest some better variants? We have a couple of reasons why we want "gateway direct": We want to avoid double recursion, which is not allowed - we might have some other recursive routes that use direct routes received in this session. We could make separate sessions for direct and transit routes, but I'm not sure that the whole thing would be simpler. And there is an issue with next-hop resolution when we have the same ips on different interfaces. As I understand, bird daemon currently does not support that case.
I tried to add a goto to the recursive section, but got segfaults, so that way may require more complex changes. I also tried it another way - just set the dest to unreachable. Looks like that it works as expected for me. Do you think that it is a reasonable "solution" and could we make an option for it? In attachment is a patch with proof of concept and my test setup (creates a couple of namespaces, links them and runs birds) to show what I am talking about. With vanilla bird I get these routes on bird b: 10.0.11.0/24 unicast [static1 18:56:55.705] * (200) via 192.168.0.1 on eth0 Type: static univ 10.0.12.0/24 unreachable [static1 18:56:55.705] * (200) Type: static univ 10.0.1.0/24 unicast [bgp1 18:57:00.085] * (100) [i] via 192.168.0.1 on eth0 Type: BGP univ BGP.origin: IGP BGP.as_path: BGP.next_hop: 192.168.0.1 BGP.local_pref: 100 Vanilla bird again, but I enabled "gateway recursive" in the channel. Route 10.0.2.0/24 appeared, but route 10.0.11.0/24 became unreachable because of double recursion. 10.0.11.0/24 unreachable [static1 18:59:44.735] * (200) Type: static univ 10.0.12.0/24 unreachable [static1 18:59:44.735] * (200) Type: static univ 10.0.1.0/24 unicast [bgp1 18:59:48.587] * (100) [i] via 192.168.0.1 on eth0 Type: BGP univ BGP.origin: IGP BGP.as_path: BGP.next_hop: 192.168.0.1 BGP.local_pref: 100 10.0.2.0/24 unreachable [bgp1 18:59:48.587 from 192.168.0.1] * (100) [i] Type: BGP univ BGP.origin: IGP BGP.as_path: BGP.next_hop: 192.168.1.1 BGP.local_pref: 100 Patched bird with gateway direct. Route 10.0.11.0/24 is reachable again and 10.0.2.0/24 is still here even with a "bad" next_hop. 10.0.11.0/24 unicast [static1 18:58:03.050] * (200) via 192.168.0.1 on eth0 Type: static univ 10.0.12.0/24 unreachable [static1 18:58:03.050] * (200) Type: static univ 10.0.1.0/24 unicast [bgp1 18:58:07.524] * (100) [i] via 192.168.0.1 on eth0 Type: BGP univ BGP.origin: IGP BGP.as_path: BGP.next_hop: 192.168.0.1 BGP.local_pref: 100 10.0.2.0/24 unreachable [bgp1 18:58:07.524 from 192.168.0.1] * (100) [i] Type: BGP univ BGP.origin: IGP BGP.as_path: BGP.next_hop: 192.168.1.1 BGP.local_pref: 100 On Thu, Oct 1, 2020 at 11:19 AM Alexander Zubkov <green@qrator.net> wrote:
Hello,
We have some routes that are passed via transit bird bgp-daemons without installing them to the kernel. And for some of these routes we need to keep their next hop attribute, because it is meaningful on the destination host, but may be unreachable on those transit hosts. If we have "gateway recursive" sessions with these transit hosts, then such routes are accepted, just marked unreachable, but we can work with them. And if we make the sessions "gateway direct", then such routes are ignored by the bird with "Invalid NEXT_HOP attribute" messages in log. Looks like it is caused by this piece of code in proto/bgp/packets.c in function bgp_apply_next_hop():
if (!nbr || (nbr->scope == SCOPE_HOST)) WITHDRAW(BAD_NEXT_HOP);
It seems reasonable, of course, but it breaks things for our case. So I want to suggest some changes to make our setup possible. At the moment, I consider the possibility of adding something like "gateway <direct_with_recursive_fallback>" so that it would behave like "gateway direct", but in the case of a non-reachable neighbour, it would switch to recursive logic. What do you think? Whether this is a good idea or maybe you can suggest some better variants?
We have a couple of reasons why we want "gateway direct": We want to avoid double recursion, which is not allowed - we might have some other recursive routes that use direct routes received in this session. We could make separate sessions for direct and transit routes, but I'm not sure that the whole thing would be simpler. And there is an issue with next-hop resolution when we have the same ips on different interfaces. As I understand, bird daemon currently does not support that case.
Hello again, Waiting for some feedback, please. On Thu, Oct 1, 2020 at 7:03 PM Alexander Zubkov <green@qrator.net> wrote:
I tried to add a goto to the recursive section, but got segfaults, so that way may require more complex changes. I also tried it another way - just set the dest to unreachable. Looks like that it works as expected for me. Do you think that it is a reasonable "solution" and could we make an option for it? In attachment is a patch with proof of concept and my test setup (creates a couple of namespaces, links them and runs birds) to show what I am talking about.
With vanilla bird I get these routes on bird b:
10.0.11.0/24 unicast [static1 18:56:55.705] * (200) via 192.168.0.1 on eth0 Type: static univ 10.0.12.0/24 unreachable [static1 18:56:55.705] * (200) Type: static univ 10.0.1.0/24 unicast [bgp1 18:57:00.085] * (100) [i] via 192.168.0.1 on eth0 Type: BGP univ BGP.origin: IGP BGP.as_path: BGP.next_hop: 192.168.0.1 BGP.local_pref: 100
Vanilla bird again, but I enabled "gateway recursive" in the channel. Route 10.0.2.0/24 appeared, but route 10.0.11.0/24 became unreachable because of double recursion.
10.0.11.0/24 unreachable [static1 18:59:44.735] * (200) Type: static univ 10.0.12.0/24 unreachable [static1 18:59:44.735] * (200) Type: static univ 10.0.1.0/24 unicast [bgp1 18:59:48.587] * (100) [i] via 192.168.0.1 on eth0 Type: BGP univ BGP.origin: IGP BGP.as_path: BGP.next_hop: 192.168.0.1 BGP.local_pref: 100 10.0.2.0/24 unreachable [bgp1 18:59:48.587 from 192.168.0.1] * (100) [i] Type: BGP univ BGP.origin: IGP BGP.as_path: BGP.next_hop: 192.168.1.1 BGP.local_pref: 100
Patched bird with gateway direct. Route 10.0.11.0/24 is reachable again and 10.0.2.0/24 is still here even with a "bad" next_hop.
10.0.11.0/24 unicast [static1 18:58:03.050] * (200) via 192.168.0.1 on eth0 Type: static univ 10.0.12.0/24 unreachable [static1 18:58:03.050] * (200) Type: static univ 10.0.1.0/24 unicast [bgp1 18:58:07.524] * (100) [i] via 192.168.0.1 on eth0 Type: BGP univ BGP.origin: IGP BGP.as_path: BGP.next_hop: 192.168.0.1 BGP.local_pref: 100 10.0.2.0/24 unreachable [bgp1 18:58:07.524 from 192.168.0.1] * (100) [i] Type: BGP univ BGP.origin: IGP BGP.as_path: BGP.next_hop: 192.168.1.1 BGP.local_pref: 100
On Thu, Oct 1, 2020 at 11:19 AM Alexander Zubkov <green@qrator.net> wrote:
Hello,
We have some routes that are passed via transit bird bgp-daemons without installing them to the kernel. And for some of these routes we need to keep their next hop attribute, because it is meaningful on the destination host, but may be unreachable on those transit hosts. If we have "gateway recursive" sessions with these transit hosts, then such routes are accepted, just marked unreachable, but we can work with them. And if we make the sessions "gateway direct", then such routes are ignored by the bird with "Invalid NEXT_HOP attribute" messages in log. Looks like it is caused by this piece of code in proto/bgp/packets.c in function bgp_apply_next_hop():
if (!nbr || (nbr->scope == SCOPE_HOST)) WITHDRAW(BAD_NEXT_HOP);
It seems reasonable, of course, but it breaks things for our case. So I want to suggest some changes to make our setup possible. At the moment, I consider the possibility of adding something like "gateway <direct_with_recursive_fallback>" so that it would behave like "gateway direct", but in the case of a non-reachable neighbour, it would switch to recursive logic. What do you think? Whether this is a good idea or maybe you can suggest some better variants?
We have a couple of reasons why we want "gateway direct": We want to avoid double recursion, which is not allowed - we might have some other recursive routes that use direct routes received in this session. We could make separate sessions for direct and transit routes, but I'm not sure that the whole thing would be simpler. And there is an issue with next-hop resolution when we have the same ips on different interfaces. As I understand, bird daemon currently does not support that case.
On Thu, Oct 01, 2020 at 11:19:24AM +0200, Alexander Zubkov wrote:
Hello,
It seems reasonable, of course, but it breaks things for our case. So I want to suggest some changes to make our setup possible. At the moment, I consider the possibility of adding something like "gateway <direct_with_recursive_fallback>" so that it would behave like "gateway direct", but in the case of a non-reachable neighbour, it would switch to recursive logic. What do you think? Whether this is a good idea or maybe you can suggest some better variants?
Hello Sorry for late answer. Perhaps it would make sense to have such hybrid mode (i would imagine it like 'if bgp_next_hop is from same network as BGP neighbor, use gateway direct, otherwise use gateway recursive'), but it seems like just another hack to avoid limited recursive handling (not supporting double recursion, which is a valid configuration). Also, it is something that could be done in import filter. Setting 'gw' attribyte removes recursive behavior for a route, so if you set BGP to 'gateway recursive', you can use: if bgp_next_hop = from then gw = bgp_next_hop; To accomplish what you need.
We have a couple of reasons why we want "gateway direct": We want to avoid double recursion, which is not allowed - we might have some other recursive routes that use direct routes received in this session. We could make separate sessions for direct and transit routes, but I'm not sure that the whole thing would be simpler. And there is an issue with next-hop resolution when we have the same ips on different interfaces. As I understand, bird daemon currently does not support that case.
-- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Hello, Thanks for the reply. Yes, I also thought that setting gw will drop the "recursive bit" and your solution is quite good. But anyway, I also had another problem in that setup. I configured the same link-local ipv6 on different interfaces and when I use "gateway direct" it correctly sets the interface. But with "gateway recursive", it performs a lookup, which may assign another interface with the same ip to the route. Maybe it would also make sense in recursive mode to make lookup considering the interface, the route was received from. I also would like to know if the patch I attached looks good in the sense that it does not break some bird's internal assumptions, etc. Because, as I said, when I tried to add recursive resolution, I got segfaults. I probably miss some understanding of the code. So I am afraid that such a patch could make some things bad too. I did some basic tests and it works, but I have not tried it in production yet. On Sun, Nov 22, 2020 at 3:17 PM Ondrej Zajicek <santiago@crfreenet.org> wrote:
On Thu, Oct 01, 2020 at 11:19:24AM +0200, Alexander Zubkov wrote:
Hello,
It seems reasonable, of course, but it breaks things for our case. So I want to suggest some changes to make our setup possible. At the moment, I consider the possibility of adding something like "gateway <direct_with_recursive_fallback>" so that it would behave like "gateway direct", but in the case of a non-reachable neighbour, it would switch to recursive logic. What do you think? Whether this is a good idea or maybe you can suggest some better variants?
Hello
Sorry for late answer. Perhaps it would make sense to have such hybrid mode (i would imagine it like 'if bgp_next_hop is from same network as BGP neighbor, use gateway direct, otherwise use gateway recursive'), but it seems like just another hack to avoid limited recursive handling (not supporting double recursion, which is a valid configuration).
Also, it is something that could be done in import filter. Setting 'gw' attribyte removes recursive behavior for a route, so if you set BGP to 'gateway recursive', you can use:
if bgp_next_hop = from then gw = bgp_next_hop;
To accomplish what you need.
We have a couple of reasons why we want "gateway direct": We want to avoid double recursion, which is not allowed - we might have some other recursive routes that use direct routes received in this session. We could make separate sessions for direct and transit routes, but I'm not sure that the whole thing would be simpler. And there is an issue with next-hop resolution when we have the same ips on different interfaces. As I understand, bird daemon currently does not support that case.
-- Elen sila lumenn' omentielvo
Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
participants (2)
-
Alexander Zubkov -
Ondrej Zajicek