Hello!

...

...

There is also a blogpost about performance comparison we have recently done between 3.0-alpha0 and 2.0.8 (the previous single-threaded version) and you can read there also other articles about the multi-threaded version.

… and here is the blogpost.

https://en.blog.nic.cz/2022/02/21/bird-journey-to-threads-chapter-3%c2%bd-ro...

Thanks for the clear writeup. I like to read about this progress (I really should read the dedicated blog about the multithreaded stuff). I can't wait for it to be stable and have a stable 3.0 release. I know this is way off still, but a guy can dream, right? :).

To be honest, writing these blogposts is a really good method to do several things at once: * design documentation * PR * rethinking once more whether the implementation is correct * fixing an awful lot of bugs due to repeated code inspection * learning even better English In fact, the stable 3.0 release is not so far from now on. Most of the work is already done. The reason for this is simple: BIRD 1.x and 2.x is super-optimized to run in single-threaded environment. This has made BIRD somehow competitive even when others came with SMP. The drawback of these optimizations is that I had to untangle them all at once before even attempting to run something in its own thread. Since the release, we have already discovered two major bugs, both of these show pretty well how everything in BIRD is internally entangled with everything else. 1. ROA check in filters called while "show route" CLI command. Thanks to DE-CIX for reporting this. There is a major internal deadlock-prevention principle: Locks have a partial ordering. A thread may lock if all its current locks are strictly greater than the wanted lock. To do ROA check, you have to lock the ROA table. To do "show route", you have to lock the table holding these routes. All table locks are incomparable and this call will simply fail. When regular import and export is done, no table is locked and the filter may look into any other table freely. (The export case took a serious amount of rework.) I mistakenly thought that CLI could be left as-is for now so it simply walks over the table while holding the table lock. This bug can't be fixed easily in any way without a complete rewrite of "show route". It is already in plan, no big problem, I'm using this bug to illustrate the internal entanglement. 2. MRT dumping. Found internally while discussing BMP merger. I forgot to think about this at all. Massive MRT dumping may lead to race conditions and undebuggable corefiles as there is no locking involved between MRT and BGP. The locking can't be added to the code as-is, it would lead to locking priority inversion, therefore MRT must also get some minor rework (hopefully not so big as with "show route") to call 3.0 stable.

Good luck with further optimizing the multithreaded code, this can't be an easy task.

Thank you. The optimization itself is actually of the easier tasks in the overall rework. In fact, most of the work was like this: 1. this internal API is thread-unsafe, how to fix it 2. it violates any possible thread-safeness principle, needa rework 3. this unremarkable routine was abusing the old API, goto 2 4. finally the API is OK, let's rework everything around 5. this unremarkable routine causes locking priority inversion 6. this change depends on another change because of something 7. temporary branch and commit, let's do the other change first 8. goto 1 Thankfully, we're (probably) out of this deadloop and the previously mentioned bugs shouldn't trigger it back. Most of the actual optimization has been postponed and I now feel quite confident that the worst part (of this journey) is already behind us. I'm honestly looking forward to the optimization. It's much fun and rewarding to see the results immediately after the change. To make it crystal clear, 3.0 stable won't be much optimized in the algorithmic or data structure way, this will happen later on. Anyway, thank you for all your wishes. Everyone, be safe these times. Maria

Hello! On 3/4/22 11:14 PM, Toke Høiland-Jørgensen wrote:

Maria Matejka <maria.matejka@nic.cz> writes:

...

To make it crystal clear, 3.0 stable won't be much optimized in the algorithmic or data structure way, this will happen later on. Anyway, thank you for all your wishes.

One question: What changes (if any) are necessary for the protocols to run in multithreaded mode? I wouldn't mind taking a look at adding support to the Babel protocol if you'd be interested in taking patches at this stage?

Simply, almost none compared to other changes inside BIRD. One of my goals is to keep the protocols untouched if possible. What is needed, as far as I can remember (list may be incomplete): * check that there is no global variable inside Babel (including static variables inside functions) * check that nothing inside Babel touches directly any global data structure and no quirk in Nest touches Babel directly * check that all timers are stopped before protocol sets itself DOWN (this won't be needed in future, maybe) * in config.Y, request to run the protocol in its own loop * set the SKF_THREAD flag for all sockets * explicitly lock by birdloop_enter() and birdloop_leave() in CLI commands like "show babel ..." For now, you can look at RPKI or BGP diff, anyway I'm going to write a blogpost specifically about internal protocol API, where to lock and how. Therefore, I think the most convenient time to do the Babel patching will be after publishing that blogpost, if you don't mind waiting. I won't refuse patches now, yet I think the best approach is postponing the work to the time when the blogpost is out and the API gets its final shape and description. I estimate that time to several months from now as there are some tasks (as described in my previous email) with higher priority. Maria

...

...

One question: What changes (if any) are necessary for the protocols to run in multithreaded mode? I wouldn't mind taking a look at adding support to the Babel protocol if you'd be interested in taking patches at this stage?

Simply, almost none compared to other changes inside BIRD. One of my goals is to keep the protocols untouched if possible. What is needed, as far as I can remember (list may be incomplete):

* check that there is no global variable inside Babel (including static variables inside functions) * check that nothing inside Babel touches directly any global data structure and no quirk in Nest touches Babel directly * check that all timers are stopped before protocol sets itself DOWN (this won't be needed in future, maybe) * in config.Y, request to run the protocol in its own loop * set the SKF_THREAD flag for all sockets * explicitly lock by birdloop_enter() and birdloop_leave() in CLI commands like "show babel ..."

Thanks for the pointers!Those bits seem simple enough. The first two items should be OK for Babel, the others need patches (but simple ones).

If you find anything too complicated or ugly, please let us know. It is intended to be simple. I suppose Babel is OK to fit into the current model "one thread per protocol instance". Multithreading inside one protocol is supported, yet it isn't much convenient for now.

...

For now, you can look at RPKI or BGP diff, anyway I'm going to write a blogpost specifically about internal protocol API, where to lock and how. Therefore, I think the most convenient time to do the Babel patching will be after publishing that blogpost, if you don't mind waiting.

I won't refuse patches now, yet I think the best approach is postponing the work to the time when the blogpost is out and the API gets its final shape and description. I estimate that time to several months from now as there are some tasks (as described in my previous email) with higher priority.

Right, let's see if I get around to playing with this before then. If I do, I may open a pull request with the fixes to Babel; I don't mind rebasing if you do end up changing the requirements along the way.

Fine, thank you! Maria

Ooowwww! I've read this 2 times... And it was a lot confusing to me! I can't even imagine the day-to-day of dealing with it... I will read that again in a couple of days and reprocess it. Thank you very much for your big effort! Em qui., 26 de mai. de 2022 às 17:52, Maria Matejka <maria.matejka@nic.cz> escreveu:

Hello!

On 5/26/22 3:48 PM, Douglas Fischer wrote:

...

Hello all!

Any news from the front of version 3?

TL;DR: We're working on it, there is a lot of heavy lifting done and another lot of heavy lifting still to be done.

You can also watch my RIPE 84 talk from last week here: https://ripe84.ripe.net/archives/video/748/

DISCLAIMER: We don'ŧ promise anything specific, this is a work in progress and may change without any notice. I'm writing this summary not only for you (and the list) to know what is in the queue, but primarily for myself to have something for future to look behind and roll on the floor laughing how naïve I was in May 2022. And also to remind myself that I should first write down all the planned work before really estimating the time needed.

Anyway, here is the current state dump, written from my perspective.

Once upon a time, there was a version 3.0-alpha0 which has been released and found broken soon afterwards. It has three major flaws which we know about right now:

1. it can't be easily merged with version 2.0.9 (I tried to do it, believe me, it is impossible.)

1A. so let's create the next version 3 by merging its commits into 2.0.9 by short intervals and fix what is broken by these merges [IN PROGRESS] 1B. something in the original branch looks weird, let's do it better as we now know the goal better [IN PROGRESS] 1C. some of these changes deserve to be merged back into the 2.0.x branch to avoid future merge conflicts [IN PROGRESS] 1D. oops, now there is the commit with original import/export table rework which I don't want to merge at all [jump to 3]. 1E. dozens of following commits to be merged and fixed [PENDING]

2. MRT dumps are insecure and will crash your BIRD [PENDING] (Found out during some internal team call after the release.)

2A. needs a special service layer 2B. this has to wait until tables are properly threaded 2C. it is better to wait for BMP to be merged to do multithreading of both of these monitoring services at once

3. calling ROA check from a filter in 'show route' crashes BIRD. (Reported by DE-CIX soon after release. Thanks for testing!)

3A. this is due to lock ordering violation 3B. we wanted to rework the show route anyway to not lock the table while formatting the route, so let's do it [PENDING] 3C. but there is also a significant performance problem in channel auxiliary tables (import / export) – the original rework made them more of a full-size table, which adds lots of unwanted overhead [PENDING] 3D. in fact, the import table may be done by storing the pre-filter route attributes "under" the actual attributes; filter modifications would create an overlay over the original route attributes [PENDING] 3E. the export table also deserves some rework and it would help performance a lot [PENDING] 3F. well, it would be less work in total to first do the import / export table rework and then the show-route rework [PENDING] 3G. the current route attribute implementation is a two-layer mess originating in version 1 where everything was an IP route, let's squash these into one layer [IN PROGRESS] 3H. also the nexthop resolving procedures and flowspec validation procedures kinda suck, let's make them more obvious, transparent and thread-friendly [IN PROGRESS] 3I. well, also the current "extended" attribute layer doesn't scale well, let's fix it [IN PROGRESS] 3J. and finally also the type-value structures are different in filters and attributes, let's fix it [IN PROGRESS]

The appropriate branches are (named in a weird way, as common): * haugesund --> this should become the -alpha1, shouldn't rebase * haugesund-to-2.0 --> this should merge into 2.0.x soon, code review and other feedback pending, rebases may occur when some bug is found to fix the original commit rather than adding a fix commit * haugesund-types --> here the 3A-3J is being done, it is my working branch and it receives rebases every other day

It's quite a lot of work but still just a little compared to the amount of work needed before to get -alpha0 to release. Most of the tasks depend on the previous ones deeply so it will still take several weeks until something can be offloaded. For now, these branches are quite a one-woman-show (I don't like that).

Out of the spotlight, Santiago has fixed several bugs in v2.0.9 and there may be v2.0.10 fixing these bugs soon. He's also preparing BMP to be merged for v2.0.11, AFAIK.

I hope you aren't too scared by all of this. There is just some technological debt and quirky implementations of later features and after releasing 3.0-alpha1, I hope for no more major flaws.

Live then happily ever after!

Maria

-- Douglas Fernando Fischer Engº de Controle e Automação

a) Could you please talk a bit more about the cgroups recommendation you have made? P.S.: I feel that this can be related to the use BIRD in containers(maybe dockers) that I think can be a good future to BIRD.

Just a strong recommendation to make administrative measures to keep yourself able to maintain the machine. BIRD may spawn lots of threads and you probably don't want your SSH to become unresponsive. For illustration, a multitable route server configuration with 1000 peers spawns 3002 threads if I count correctly. I don't know whether Docker is a good idea to run BIRD in, I neither recommend nor discourage it.

b) About the non-blocking reconfiguration. A couple months ago I was chatting with a friend about how using conf files splitted and with includes can be a good way to avoid miss configurations. Mostly when you have a tem working on that config, and also part of that configuration been generated by automatic mechanisms.

During your talk(on video) I thought that splitting conf files, mostly variables like prefix-list and AS-Path Regex that are unique by each peer, cloud be a way to reduce the workload of parsing configs on reconfiguration.

Imagine a file with several prefixes that did not changed since last reconfiguration... Why spent clocks re-parsing it? Maybe keeping the already parsed files with a stamp of "this was good last time was checked".

Yes, I Know that most of those included conf-files has interdependent variables. But, maybe, with the correct definition of how would need to be limitation of contents of a partial config file to enjoy the "feature" of economic-reconfig, this could be efficient.

First, this would need quite a substantial rework of config as we suppose that all data in config is allocated from one pool and when reconfiguration is finished, we dispose of them at once. Second, I'm afraid this kind of constraint on what is allowed for include is be a can of really huge and disgusting worms. Maybe simply just the prefix trees, but … … but third, there should be some extension of RTR-RPKI protocol to allow for this kind of data to be fed into a routing daemon externally. For now, we'd like to stick with this approach to get rid of these blocks of constants at all. Hope this is a sufficient explanation for you. Maria

Hello! As there are some merges done, here is a short update.

3G. the current route attribute implementation is a two-layer mess originating in version 1 where everything was an IP route, let's squash these into one layer [IN PROGRESS]

Done (at least for now), will also comment the updated data structures publicly in a separate blogpost soon.

The appropriate branches are (named in a weird way, as common): * haugesund --> this should become the -alpha1, shouldn't rebase * haugesund-to-2.0 --> this should merge into 2.0.x soon, code review and other feedback pending, rebases may occur when some bug is found to fix the original commit rather than adding a fix commit * haugesund-types --> here the 3A-3J is being done, it is my working branch and it receives rebases every other day

The haugesund-types branch has been merged into haugesund and the development will continue branching from there. Maria -- Sent from my Android device with K-9 Mail. Please excuse my brevity.

Hello! I'd like to share some information on what has been done until now. My update is based on the "roadmap" I've shared with you on May 26 this year: http://trubka.network.cz/pipermail/bird-users/2022-May/016139.html On 5/26/22 10:52 PM, Maria Matejka wrote:

Anyway, here is the current state dump, written from my perspective.

Once upon a time, there was a version 3.0-alpha0 which has been released and found broken soon afterwards. It has three major flaws which we know about right now:

1. it can't be easily merged with version 2.0.9 (I tried to do it, believe me, it is impossible.)

1A. so let's create the next version 3 by merging its commits into 2.0.9 by short intervals and fix what is broken by these merges [IN PROGRESS] 1B. something in the original branch looks weird, let's do it better as we now know the goal better [IN PROGRESS] 1C. some of these changes deserve to be merged back into the 2.0.x branch to avoid future merge conflicts [IN PROGRESS]

1C. is prepared now in the "backport" branch. Needs some code review and comments but yes, it is there.

1D. oops, now there is the commit with original import/export table rework which I don't want to merge at all [jump to 3]. 1E. dozens of following commits to be merged and fixed [PENDING]

2. MRT dumps are insecure and will crash your BIRD [PENDING] (Found out during some internal team call after the release.)

2A. needs a special service layer 2B. this has to wait until tables are properly threaded 2C. it is better to wait for BMP to be merged to do multithreading of both of these monitoring services at once

Not yet done but soon on the table.

3. calling ROA check from a filter in 'show route' crashes BIRD. (Reported by DE-CIX soon after release. Thanks for testing!)

3A. this is due to lock ordering violation 3B. we wanted to rework the show route anyway to not lock the table while formatting the route, so let's do it [PENDING]

Done.

3C. but there is also a significant performance problem in channel auxiliary tables (import / export) – the original rework made them more of a full-size table, which adds lots of unwanted overhead [PENDING]

Partially fixed by making the import table just a lower layer of route attributes. The attribute implementation deserves some optimizations yet to be done.

3D. in fact, the import table may be done by storing the pre-filter route attributes "under" the actual attributes; filter modifications would create an overlay over the original route attributes [PENDING]

Done and seems working.

3E. the export table also deserves some rework and it would help performance a lot [PENDING]

Reworked to store the routes right before sending in BGP, therefore _after_ applying after-export modifications of attributes, e.g. attribute stripping, nexthop replacement or ASN appending. This will be quite a major behavior change between version 2 and 3, yet we think storing the exported routes in this place is much more useful than where it was before.

3F. well, it would be less work in total to first do the import / export table rework and then the show-route rework [PENDING]

Well, done interleaved somehow.

3G. the current route attribute implementation is a two-layer mess originating in version 1 where everything was an IP route, let's squash these into one layer [IN PROGRESS]

Done.

3H. also the nexthop resolving procedures and flowspec validation procedures kinda suck, let's make them more obvious, transparent and thread-friendly [IN PROGRESS] 3I. well, also the current "extended" attribute layer doesn't scale well, let's fix it [IN PROGRESS] 3J. and finally also the type-value structures are different in filters and attributes, let's fix it [IN PROGRESS]

These three are still pending.

The appropriate branches are (named in a weird way, as common): * haugesund --> this should become the -alpha1, shouldn't rebase

Renamed to thread-next as we are now quite sure that it is the way to go.

* haugesund-to-2.0 --> this should merge into 2.0.x soon, code review and other feedback pending, rebases may occur when some bug is found to fix the original commit rather than adding a fix commit

Renamed to backport.

It's quite a lot of work but still just a little compared to the amount of work needed before to get -alpha0 to release. Most of the tasks depend on the previous ones deeply so it will still take several weeks until something can be offloaded. For now, these branches are quite a one-woman-show (I don't like that).

I'd like to say that it still seems slow from my POV, yet we're moving forwards. Let's see how much of coding opportunities is there during this summer as work performance always drops during the holiday season. Naively, I thought we might have got 3.0-alpha1 like now or so, yet it's now more like mid-September. I'll be anyway very upset if we miss the opportunity to announce 3.0-alpha1 during the next RIPE session (autumn 2022, should be in Beograd). Maria