BIRD Journey to Threads: Chapter zero of a blogpost series
Hello! As we're working on multithreading support in BIRD, there is a need to document this. There is a user documentation (which almost doesn't change) and a programmer's documentation composed mostly of comments spread across the source code. There is also a kind-of high-level documentation to describe how are the BIRD's internal processes changing. If you're just an ordinary user, you don't need to care much about that. We're sharing this mostly for people who know (or want to know) how BIRD works internally and how it is going to change. The zeroth chapter is here, other chapters should follow soon: https://en.blog.nic.cz/2021/03/15/bird-journey-to-threads-chapter-0-the-reas... (Content: An introduction and summary of previous major BIRD structural changes.) I deeply appreciate your feedback and questions. Maria
Hey Maria. Great so see the work being done on multithreading! I do have one question but not sure if it's related. I asked a while back if bird would ever support any sort of 'structured output' - i.e. adding the possibility of json or some other output which would make it easier to get the right data. I believe you mentioned it was getting looked at but multithreading was coming first. Any movement on that though? Thanks Darren On Mon, 15 Mar 2021 at 06:45, Maria Matejka <maria.matejka@nic.cz> wrote:
Hello!
As we're working on multithreading support in BIRD, there is a need to document this. There is a user documentation (which almost doesn't change) and a programmer's documentation composed mostly of comments spread across the source code.
There is also a kind-of high-level documentation to describe how are the BIRD's internal processes changing. If you're just an ordinary user, you don't need to care much about that. We're sharing this mostly for people who know (or want to know) how BIRD works internally and how it is going to change.
The zeroth chapter is here, other chapters should follow soon:
https://en.blog.nic.cz/2021/03/15/bird-journey-to-threads-chapter-0-the-reas...
(Content: An introduction and summary of previous major BIRD structural changes.)
I deeply appreciate your feedback and questions.
Maria
Hello! Yes, it is going to happen. The order should be (roughly) as follows: 0) basic multithreading support (already implemented) 1) structural changes in route export enabling table access from multiple threads 2) conversion of 'show route' to the new route export code 3) json route formatter Maria On March 15, 2021 4:15:57 PM GMT+01:00, Darren O'Connor <mellow.drifter@gmail.com> wrote:
Hey Maria.
Great so see the work being done on multithreading! I do have one question but not sure if it's related. I asked a while back if bird would ever support any sort of 'structured output' - i.e. adding the possibility of json or some other output which would make it easier to get the right data. I believe you mentioned it was getting looked at but multithreading was coming first. Any movement on that though?
Thanks Darren
On Mon, 15 Mar 2021 at 06:45, Maria Matejka <maria.matejka@nic.cz> wrote:
Hello!
As we're working on multithreading support in BIRD, there is a need to document this. There is a user documentation (which almost doesn't change) and a programmer's documentation composed mostly of comments spread across the source code.
There is also a kind-of high-level documentation to describe how are the BIRD's internal processes changing. If you're just an ordinary user, you don't need to care much about that. We're sharing this mostly for people who know (or want to know) how BIRD works internally and how it is going to change.
The zeroth chapter is here, other chapters should follow soon:
https://en.blog.nic.cz/2021/03/15/bird-journey-to-threads-chapter-0-the-reas...
(Content: An introduction and summary of previous major BIRD
structural
changes.)
I deeply appreciate your feedback and questions.
Maria
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Hello, Maria and everyone on the list... This is Great News! I'm so happy to receive that mail! That part about "Filter Rework" made me a bit anxious! I don't know exactly means the part: "we decided to preprocess the filter internal structure to another structure which is much easier to execute"... But, to me, it sounds like fragmenting, even more, the "test, mark, and actuate" on the filters... Right? Talking about BGP, I love Bird because only with it I know how to do what I call "2 cycles filtering"... - I create several Small Tests, where each route is tested and marked somehow... - On each Peer-Type, import or export(but mostly on import), and I define which tests will be called to be executed on that route, and live their mark on each route. - I here I test, Mask-lenght, AS-Set, AS-Origin, AS-Path, foreigner-communities, internal-communities, a lot of Regex, and all those multiple complex things... - And at the end of the filter-police I do an "IF" that tests on a combined And+Or the marks lived by the previous tests, and based on that do the accept, reject, add-communities, adjust MED/Local-Pref/Etc... That thing with "Lua scripts" that you mentioned, would be something like what I described? Thanks in advance! Em seg., 15 de mar. de 2021 às 07:44, Maria Matejka <maria.matejka@nic.cz> escreveu:
Hello!
As we're working on multithreading support in BIRD, there is a need to document this. There is a user documentation (which almost doesn't change) and a programmer's documentation composed mostly of comments spread across the source code.
There is also a kind-of high-level documentation to describe how are the BIRD's internal processes changing. If you're just an ordinary user, you don't need to care much about that. We're sharing this mostly for people who know (or want to know) how BIRD works internally and how it is going to change.
The zeroth chapter is here, other chapters should follow soon:
https://en.blog.nic.cz/2021/03/15/bird-journey-to-threads-chapter-0-the-reas...
(Content: An introduction and summary of previous major BIRD structural changes.)
I deeply appreciate your feedback and questions.
Maria
-- Douglas Fernando Fischer Engº de Controle e Automação
Hello! On 3/15/21 6:07 PM, Douglas Fischer wrote:
That part about "Filter Rework" made me a bit anxious! I don't know exactly means the part: "we decided to preprocess the filter internal structure to another structure which is much easier to execute"... But, to me, it sounds like fragmenting, even more, the "test, mark, and actuate" on the filters... Right?
This is a work that has been already included in v2.0.x for x ≥ 4 as far as I can remember. The filter is simply converted (in config time) from an infix structure with lots of pointers to a stack-oriented postfix bytecode. That bytecode is somehow similar to Lua. We also thought about using Lua itself as a filtering language, anyway we had problems with Lua stack initialization. A simple run of an empty filter took at least 2 us on our testing machine, compared to 35 ns of our empty filter.
Talking about BGP, I love Bird because only with it I know how to do what I call "2 cycles filtering"...
- I create several Small Tests, where each route is tested and marked somehow... - On each Peer-Type, import or export(but mostly on import), and I define which tests will be called to be executed on that route, and live their mark on each route. - I here I test, Mask-lenght, AS-Set, AS-Origin, AS-Path, foreigner-communities, internal-communities, a lot of Regex, and all those multiple complex things... - And at the end of the filter-police I do an "IF" that tests on a combined And+Or the marks lived by the previous tests, and based on that do the accept, reject, add-communities, adjust MED/Local-Pref/Etc...
This stays the same, the filter language didn't change, we just reworked the interpreter. To add to this, there are also custom route attributes. It is quite common to do the small test on import, store the results to BGP communities, decide on export based on them and then filter them out. For this use, you can define your own route attributes and you don't mess with the communities at all. (And these are also faster than community list manipulation.) We are thinking about more filtering features like iterating over the community lists or multipath nexthops. Don't worry, the filters are going to stay the same or better. Thank you for your input! Maria
I'm trying to understand a bit deeper the new concepts of multithreading of Bird. You mentioned: "and a programmer's documentation composed mostly of comments spread across the source code" So I decided to dig a bit on the code available at https://gitlab.nic.cz/labs/bird/ to look at the comments, try to understand the concepts... But I could not find any tags earlier than 1 year. No releases bigger than 2.0.7. No Milestones. Is reasonable to consider that this innovative part of the code is still no publicly available? How someone that "want to know how BIRD works internally and how it is going to change" could look a bit more directly to that part of the code changes? Em seg., 15 de mar. de 2021 às 15:02, Maria Matejka <maria.matejka@nic.cz> escreveu:
Hello!
On 3/15/21 6:07 PM, Douglas Fischer wrote:
That part about "Filter Rework" made me a bit anxious! I don't know exactly means the part: "we decided to preprocess the filter internal structure to another structure which is much easier to execute"... But, to me, it sounds like fragmenting, even more, the "test, mark, and actuate" on the filters... Right?
This is a work that has been already included in v2.0.x for x ≥ 4 as far as I can remember. The filter is simply converted (in config time) from an infix structure with lots of pointers to a stack-oriented postfix bytecode.
That bytecode is somehow similar to Lua. We also thought about using Lua itself as a filtering language, anyway we had problems with Lua stack initialization. A simple run of an empty filter took at least 2 us on our testing machine, compared to 35 ns of our empty filter.
Talking about BGP, I love Bird because only with it I know how to do what I call "2 cycles filtering"...
- I create several Small Tests, where each route is tested and marked somehow... - On each Peer-Type, import or export(but mostly on import), and I define which tests will be called to be executed on that route, and live their mark on each route. - I here I test, Mask-lenght, AS-Set, AS-Origin, AS-Path, foreigner-communities, internal-communities, a lot of Regex, and all those multiple complex things... - And at the end of the filter-police I do an "IF" that tests on a combined And+Or the marks lived by the previous tests, and based on that do the accept, reject, add-communities, adjust MED/Local-Pref/Etc...
This stays the same, the filter language didn't change, we just reworked the interpreter. To add to this, there are also custom route attributes. It is quite common to do the small test on import, store the results to BGP communities, decide on export based on them and then filter them out. For this use, you can define your own route attributes and you don't mess with the communities at all. (And these are also faster than community list manipulation.)
We are thinking about more filtering features like iterating over the community lists or multipath nexthops. Don't worry, the filters are going to stay the same or better.
Thank you for your input!
Maria
-- Douglas Fernando Fischer Engº de Controle e Automação
Hello! Well, there are several branches spread across the repository, keeping different stages of development, mostly buggy or badly designed in some way, with cherry-picks and rebases everywhere. To see something that seems to be useful (and mostly final) even for now, look at the commit c78f88c91c7b75f530d70d503fa011c20f16f278 (yes, not even a branch, just a commit deep in a repository). The locking code in that branch may get simplified before release. There is also a branch named 'guernsey' (for completely unrelated reasons; yes, it is chaotic) where you can see the (unfinished) next chapter of this series together with lots of other stuff related to route storage. This branch may be rebased or otherwise amended with no warning. There is also some code not to be used, testing some design ideas, like parts of branches 'mq-async-export' and 'jersey'. We may cherry-pick some changes from there. Parts of the code are also still not written (e.g. most of the changes in route export) and I'm going to code these after writing the documentation, instead of documenting what the code does. You are also right that the last version is currently 2.0.7; the pace of ordinary releases has dropped down a bit as we spend quite a lot of time thinking about multithreading design. Maria On 3/16/21 12:36 PM, Douglas Fischer wrote:
I'm trying to understand a bit deeper the new concepts of multithreading of Bird.
You mentioned: "and a programmer's documentation composed mostly of comments spread across the source code"
So I decided to dig a bit on the code available at https://gitlab.nic.cz/labs/bird/ <https://gitlab.nic.cz/labs/bird/> to look at the comments, try to understand the concepts... But I could not find any tags earlier than 1 year. No releases bigger than 2.0.7. No Milestones.
Is reasonable to consider that this innovative part of the code is still no publicly available?
How someone that "want to know how BIRD works internally and how it is going to change" could look a bit more directly to that part of the code changes?
Em seg., 15 de mar. de 2021 às 15:02, Maria Matejka <maria.matejka@nic.cz <mailto:maria.matejka@nic.cz>> escreveu:
Hello!
On 3/15/21 6:07 PM, Douglas Fischer wrote: > That part about "Filter Rework" made me a bit anxious! > I don't know exactly means the part: > "we decided to preprocess the filter internal structure to another > structure which is much easier to execute"... > But, to me, it sounds like fragmenting, even more, the "test, mark, and > actuate" on the filters... Right?
This is a work that has been already included in v2.0.x for x ≥ 4 as far as I can remember. The filter is simply converted (in config time) from an infix structure with lots of pointers to a stack-oriented postfix bytecode.
That bytecode is somehow similar to Lua. We also thought about using Lua itself as a filtering language, anyway we had problems with Lua stack initialization. A simple run of an empty filter took at least 2 us on our testing machine, compared to 35 ns of our empty filter.
> Talking about BGP, I love Bird because only with it I know how to do > what I call "2 cycles filtering"... > > - I create several Small Tests, where each route is tested and marked > somehow... > - On each Peer-Type, import or export(but mostly on import), and I > define which tests will be called to be executed on that route, and live > their mark on each route. > - I here I test, Mask-lenght, AS-Set, AS-Origin, AS-Path, > foreigner-communities, internal-communities, a lot of Regex, and all > those multiple complex things... > - And at the end of the filter-police I do an "IF" that tests on a > combined And+Or the marks lived by the previous tests, and based on that > do the accept, reject, add-communities, adjust MED/Local-Pref/Etc...
This stays the same, the filter language didn't change, we just reworked the interpreter. To add to this, there are also custom route attributes. It is quite common to do the small test on import, store the results to BGP communities, decide on export based on them and then filter them out. For this use, you can define your own route attributes and you don't mess with the communities at all. (And these are also faster than community list manipulation.)
We are thinking about more filtering features like iterating over the community lists or multipath nexthops. Don't worry, the filters are going to stay the same or better.
Thank you for your input!
Maria
-- Douglas Fernando Fischer Engº de Controle e Automação
participants (4)
-
Darren O'Connor -
Douglas Fischer -
Maria Matejka -
Maria Matějka