Doc suggestion - clarifying behaviour when routes are moving between protocols
Hi folks Thank you for Bird, I have enjoyed working with it over the holidays for a personal project routing between offices. Along the way I got stuck on an issue that might catch other people as well, so I thought to describe it here in the hope that docs are updated to make the behaviour more clear. I have a metropolitan area network between a few sites. Each of those sites has its own internet connection, but the MAN is much faster, so my goal was to route traffic to whichever exit point had the best bandwidth, using the different sites as backup internet gateways. At each site, I was testing the local connectivity (looking beyond simple link up to whether I could reach common sites) and then, if the link was good, adding a static route out from that location. I wanted to use Bird to export and distribute those static routes to other sites, with custom metrics to prioritise the gateways that had the best bandwidth. The expected behaviour for me was that a local, statically defined default route would be imported and have the custom metric set based on its bandwidth, and that the 'best' route would then win and be distributed by OSPF to the other sites. The problem I ran into was that the implicit protocol preference comparison prevented the local route from even being considered at all. So once a route had propagated to a site with Bird, it could not be displaced by a 'better' static route, because of the relative preferences. I ended up with a config like this, where the explicit preference solved the issue: protocol kernel { learn; # Learn alien routes from the kernel ipv4 { # Connect protocol to IPv4 table by channel import filter { if net = 0.0.0.0/0 then { # We only want default routes added manually to this machine ospf_metric2 = 1000; preference = 150; accept "Accepted default route from kernel"; } reject; }; export all; # Install all Bird routes in kernel }; } I couldn't find any documentation that explained the role of 'preference' in the import process, so I was stuck on this for an embarrassing amount of time :blush: In the end, this blog https://mk16.de/blog/bird-preference/ and some tangential comments in mailing lists led me to try the explicit preference, and that solved the issue. Apologies if I missed this in the docs, but if it isn't already covered, perhaps a more detailed explanation of the way routes are compared for import into a table would help others. Again, thank you for Bird! I think we might build it into Ubuntu in some interesting ways :) Mark
Hi Mark, Actually the best route selection algorithm can be found here: https://bird.network.cz/?get_doc&v=20&f=bird-2.html#ss2.1 And the preference is clearly noticed there. Have you looked for it in some other place? If you describe how did you try to find it, I believe it might help the developers to better structure the documentation. Best regards, Alexander On Thu, Feb 8, 2024 at 11:41 AM Mark Shuttleworth <mark.shuttleworth@canonical.com> wrote:
Hi folks
Thank you for Bird, I have enjoyed working with it over the holidays for a personal project routing between offices. Along the way I got stuck on an issue that might catch other people as well, so I thought to describe it here in the hope that docs are updated to make the behaviour more clear.
I have a metropolitan area network between a few sites. Each of those sites has its own internet connection, but the MAN is much faster, so my goal was to route traffic to whichever exit point had the best bandwidth, using the different sites as backup internet gateways. At each site, I was testing the local connectivity (looking beyond simple link up to whether I could reach common sites) and then, if the link was good, adding a static route out from that location.
I wanted to use Bird to export and distribute those static routes to other sites, with custom metrics to prioritise the gateways that had the best bandwidth.
The expected behaviour for me was that a local, statically defined default route would be imported and have the custom metric set based on its bandwidth, and that the 'best' route would then win and be distributed by OSPF to the other sites.
The problem I ran into was that the implicit protocol preference comparison prevented the local route from even being considered at all. So once a route had propagated to a site with Bird, it could not be displaced by a 'better' static route, because of the relative preferences.
I ended up with a config like this, where the explicit preference solved the issue:
protocol kernel { learn; # Learn alien routes from the kernel ipv4 { # Connect protocol to IPv4 table by channel import filter { if net = 0.0.0.0/0 then { # We only want default routes added manually to this machine ospf_metric2 = 1000; preference = 150; accept "Accepted default route from kernel"; } reject; }; export all; # Install all Bird routes in kernel }; }
I couldn't find any documentation that explained the role of 'preference' in the import process, so I was stuck on this for an embarrassing amount of time :blush: In the end, this blog https://mk16.de/blog/bird-preference/ and some tangential comments in mailing lists led me to try the explicit preference, and that solved the issue. Apologies if I missed this in the docs, but if it isn't already covered, perhaps a more detailed explanation of the way routes are compared for import into a table would help others.
Again, thank you for Bird! I think we might build it into Ubuntu in some interesting ways :)
Mark
Hi Alexander On Thu, 8 Feb 2024 at 13:28, Alexander Zubkov <green@qrator.net> wrote:
Actually the best route selection algorithm can be found here: https://bird.network.cz/?get_doc&v=20&f=bird-2.html#ss2.1
Can't argue that it's there, but even with the benefit of hindsight it is a little terse :) Admittedly I am not a networking specialist, but perhaps that makes my feedback useful in terms of gaining new users and bringing Bird to a wider audience. My previous experience has been playing with Babel for this same personal project, and Bird is obviously a step up in complexity given its support for many different protocols. As an aside, it's great to see Babel support in Bird :) At the time, here's what I struggled with. Hopefully some of this will be helpful in shaping docs for new users. It would be useful early in the doc to have a visualisation of the relationship between a table such as master4 and a protocol, like kernel or ospf. I found drawing a picture for myself helped to make import vs export clear. Perhaps the docs can have a picture like that! My suggestion would be to make that visualisation include multiple protocols such as the kernel protocol and one of the routing protocols like RIP or OSPF, so that it can help explain how a route might move between routers (OSPF) and into the local machine routing table. I would walk through a full example, showing movement of routes in both directions from both protocols. Describe, step by step, how a kernel route would be learned and imported into a table like ipv4, and then exported by OSPF, and vice versa. Perhaps this is obvious to a routing expert but it wasn't obvious to me as someone playing with a proper routing daemon for the first time. The multi-protocol case (kernel protocol and ospf protocol) may be less interesting to someone who's focused on BGP or OSPF across many routers, but it seems to me to be critical to the new user experience. A new user is likely to have one, two or three routers, and they need to move routes between them AND get those routes installed into the kernel, so they have to understand the multi-protocol interaction dynamics. This is where things like preference, which are not things one can learn about just by reading about OSPF or RIP, come into play. The part of the doc you refer to is critical - it covers the main deal which is 'which routes from the various protocols will move into position where they can be actioned', but it's also very terse, and it's just a paragraph or two in a long stretch of detail. It also uses the term 'preference' twice, and I'm not sure those two uses refer to the same thing. It says 'Preferences of the routes are compared' and 'Source protocol instance preferences are compared' but neither 'route preference' nor 'source protocol instance preference' has been defined before this text. I'm still not sure how I would go and look at a setup with birdc and find out easily what the route preference or source protocol instance preference would be. So I think all of this warrants a section on its own, early in the overall documentation, that walks through the behaviours of protocols, channels, and tables, and how they interact. It's notable that other people have found a need to try and explain this more clearly themselves - digging around to compose this email I found another example at https://blog.kintone.io/entry/bird which may be helpful to you. It would also help if logging of decisions wrt import or export captured more of the rationale for the decision. As I dug into this I saw messages like 'route rejected by protocol', but no rationale for that rejection (it turned out to be the preference). Because the relationship between tables and protocols, import and export is unclear initially, it would be helpful to be explicit in these log messages. Would it be possible to say something like this: * <TRACE> 10.2.21.0/24 <http://10.2.21.0/24> from kernel import to master4 accepted** <TRACE> **10.2.34.0/24 <http://10.2.34.0/24> from * *master4 export to ospf1 rejected because protocol preference 100<30* In this example I'm replacing angle brackets with explicit import / export, and listing the table names explicitly, to try and make it easier for a new user to follow that log and not be confused. I'm trying to show direction in the order - from kernel import to master4, and from master4 export to ospf1. The reason for the rejection would help to then dig in to details like protocol preferences. In debugging with birdc, I found the layouts rather inscrutable and could not find documentation. For example: bird> show route Table master4: 0.0.0.0/0 unicast [ospf1 2024-01-31] * E2 (150/40/900) [192.168.12.2] via 192.168.4.248 on bond-lan weight 1 via 192.168.4.249 on bond-lan weight 1 192.168.254.48/30 unicast [ospf1 2024-01-31] * I (150/40) [192.168.254.57] via 192.168.4.248 on bond-lan weight 1 via 192.168.4.249 on bond-lan weight 1 What is each of those fields? It would be nice if the docs for the protocol also explained how birdc represents routes from that protocol. In this case, the docs at https://bird.network.cz/?get_doc&v=20&f=bird-6.html#ss6.11 could show the various permutations of an OSPF route, and point out where preference and metrics are (it took a lot of googling to understand "E2 (150/40/900)" :) Similarly, while preference is a critical element of the table import / export route selection algorithm, it's hard to know what it actually is. For example, where is the preference of this protocol instance? bird> show ospf ospf1: RFC1583 compatibility: disabled Stub router: No RT scheduler tick: 1 Number of areas: 1 Number of LSAs in DB: 32 Area: 0.0.0.0 (0) [BACKBONE] Stub: No NSSA: No Transit: No Number of interfaces: 3 Number of neighbors: 4 Number of adjacent neighbors: 4 Apologies if this is newbie stuff, and hopefully it helps you get your future newbies through that learning process faster with fewer questions to your list :) Mark
Hello Mark, you are having a lot of useful suggestions. There are even more thoughts alike yours, regarding documentation. And indeed you are right that our documentation is kinda good as a reference manual but it kinda sucks if you are a newbie. The thing is, we have quite a long backlog and some technological debt to resolve (see e.g. multithreading or our still-non-existent API), and all of these are very time-consuming. Writing good documentation, or better to say, creating good paedagogical materials for self-learning, is time-consuming as well. And even though we are very much in favor of having better documentation, the aforementioned problems have somehow higher priority than documentation (money being the main reason). Anyway, I'll later try to split your suggestions into independent pieces and create our internal issues for each of them, to not lose them. Now to the suggestions (inline): On 2024-02-09 06:13, Mark Shuttleworth wrote:
Admittedly I am not a networking specialist, but perhaps that makes my feedback useful in terms of gaining new users and bringing Bird to a wider audience. My previous experience has been playing with Babel for this same personal project, and Bird is obviously a step up in complexity given its support for many different protocols. As an aside, it's great to see Babel support in Bird :)
This is actually a very good position to read the docs and see what makes you confused. I'm very happy that you didn't freak out.
At the time, here's what I struggled with. Hopefully some of this will be helpful in shaping docs for new users.
It would be useful early in the doc to have a visualisation of the relationship between a table such as master4 and a protocol, like kernel or ospf. I found drawing a picture for myself helped to make import vs export clear. Perhaps the docs can have a picture like that! My suggestion would be to make that visualisation include multiple protocols such as the kernel protocol and one of the routing protocols like RIP or OSPF, so that it can help explain how a route might move between routers (OSPF) and into the local machine routing table. I would walk through a full example, showing movement of routes in both directions from both protocols. Describe, step by step, how a kernel route would be learned and imported into a table like ipv4, and then exported by OSPF, and vice versa. Perhaps this is obvious to a routing expert but it wasn't obvious to me as someone playing with a proper routing daemon for the first time.
We indeed don't have many pictures in the documentation and we shall add some. Anyway, this reminds me of … you are a visual learner, more than a textual one, aren't you? Would you appreciate a video tutorial, explaining these things step by step?
The multi-protocol case (kernel protocol and ospf protocol) may be less interesting to someone who's focused on BGP or OSPF across many routers, but it seems to me to be critical to the new user experience. A new user is likely to have one, two or three routers, and they need to move routes between them AND get those routes installed into the kernel, so they have to understand the multi-protocol interaction dynamics. This is where things like preference, which are not things one can learn about just by reading about OSPF or RIP, come into play.
The part of the doc you refer to is critical - it covers the main deal which is 'which routes from the various protocols will move into position where they can be actioned', but it's also very terse, and it's just a paragraph or two in a long stretch of detail. It also uses the term 'preference' twice, and I'm not sure those two uses refer to the same thing. It says 'Preferences of the routes are compared' and 'Source protocol instance preferences are compared' but neither 'route preference' nor 'source protocol instance preference' has been defined before this text. I'm still not sure how I would go and look at a setup with birdc and find out easily what the route preference or source protocol instance preference would be.
So I think all of this warrants a section on its own, early in the overall documentation, that walks through the behaviours of protocols, channels, and tables, and how they interact. It's notable that other people have found a need to try and explain this more clearly themselves - digging around to compose this email I found another example at https://blog.kintone.io/entry/bird which may be helpful to you.
This probably needs much more rewriting, considering that there are also MPLS rules now, EVPN upcoming … good point from you.
It would also help if logging of decisions wrt import or export captured more of the rationale for the decision. As I dug into this I saw messages like 'route rejected by protocol', but no rationale for that rejection (it turned out to be the preference). Because the relationship between tables and protocols, import and export is unclear initially, it would be helpful to be explicit in these log messages. Would it be possible to say something like this: /<TRACE> 10.2.21.0/24 <http://10.2.21.0/24> from kernel import to master4 accepted //<TRACE> //10.2.34.0/24 <http://10.2.34.0/24> from //master4 export to ospf1 rejected because protocol preference 100<30 / In this example I'm replacing angle brackets with explicit import / export, and listing the table names explicitly, to try and make it easier for a new user to follow that log and not be confused. I'm trying to show direction in the order - from kernel import to master4, and from master4 export to ospf1. The reason for the rejection would help to then dig in to details like protocol preferences.
People actually quite often parse the logs so we don't have so much space to change the format, but you're right that the log format is very dense and sometimes misleading. Regarding the "because something" part … it's quite challenging to actually achieve this but I'll make an issue for it and we'll see later what can be done with that. Especially with the BIRD 3 multithreaded architecture, there may be some funny corner cases and race conditions which have to be checked for.
In debugging with birdc, I found the layouts rather inscrutable and could not find documentation. For example:
bird> show route Table master4: 0.0.0.0/0 <http://0.0.0.0/0> unicast [ospf1 2024-01-31] * E2 (150/40/900) [192.168.12.2] via 192.168.4.248 on bond-lan weight 1 via 192.168.4.249 on bond-lan weight 1 192.168.254.48/30 <http://192.168.254.48/30> unicast [ospf1 2024-01-31] * I (150/40) [192.168.254.57] via 192.168.4.248 on bond-lan weight 1 via 192.168.4.249 on bond-lan weight 1
What is each of those fields? It would be nice if the docs for the protocol also explained how birdc represents routes from that protocol. In this case, the docs at https://bird.network.cz/?get_doc&v=20&f=bird-6.html#ss6.11 <https://bird.network.cz/?get_doc&v=20&f=bird-6.html#ss6.11> could show the various permutations of an OSPF route, and point out where preference and metrics are (it took a lot of googling to understand "E2 (150/40/900)" :)
Similarly, while preference is a critical element of the table import / export route selection algorithm, it's hard to know what it actually is. For example, where is the preference of this protocol instance?
bird> show ospf ospf1: RFC1583 compatibility: disabled Stub router: No RT scheduler tick: 1 Number of areas: 1 Number of LSAs in DB: 32 Area: 0.0.0.0 (0) [BACKBONE] Stub: No NSSA: No Transit: No Number of interfaces: 3 Number of neighbors: 4 Number of adjacent neighbors: 4
Documenting these is one solution (but you have to look it up), creating a web-browser-friendly interface is another one. (Both can be done.) And with our new shiny API (which is still mainly on the drawing board), the web-browser interface will be possible to create quite smoothly. Also, there are so-called looking-glasses, which are basically web interfaces over BIRD with added features. See e.g. Alice-LG: https://github.com/alice-lg/alice-lg With that, I'm quite convinced that we're gonna have some web-browser interface with all of the routes and parameters visualised, well styled and explained with context help. (Funding of such an effort, or even just getting a commercial support package to actually steer our development this way, would be very much appreciated, though.)
Apologies if this is newbie stuff, and hopefully it helps you get your future newbies through that learning process faster with fewer questions to your list :)
You did the best thing you could, and thank you for all your thoughts. If you find any other suggestion, feel free to write it. Have a nice day! Maria -- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
Maria Matejka via Bird-users <bird-users@network.cz> writes:
Hello Mark,
you are having a lot of useful suggestions. There are even more thoughts alike yours, regarding documentation. And indeed you are right that our documentation is kinda good as a reference manual but it kinda sucks if you are a newbie.
The thing is, we have quite a long backlog and some technological debt to resolve (see e.g. multithreading or our still-non-existent API), and all of these are very time-consuming. Writing good documentation, or better to say, creating good paedagogical materials for self-learning, is time-consuming as well. And even though we are very much in favor of having better documentation, the aforementioned problems have somehow higher priority than documentation (money being the main reason).
Have you considered applying for the recently announced Google Season of Docs? It seems this would be an excellent fit for that programme! https://opensource.googleblog.com/2024/02/announcing-google-season-of-docs-2... Happy to pitch in to help answer questions such a doc writer may have if the application is successful! :) -Toke
participants (4)
-
Alexander Zubkov -
Maria Matejka -
Mark Shuttleworth -
Toke Høiland-Jørgensen