(added bird-users@ back, I didn't mean to remove them before)

Sorry that I missed that you'd already added the pull request. I merged it. we do need a way to specify exabgp vs bird, but that's just another thing to add.

I agree about containers and versions. Just haven't had time to think it through carefully. 

The policy support is what bgperf had before I forked it. I don't use it, I don't know how to usefully test with it. 

One more feature I'd like to see, yet I'm not sure how to do this
properly – to make the measurement more accurate, the target should not
only run on its own machine, but bgperf with all the testers should also
watch their load, issuing warning when the bgperf suite may be the
bottleneck. This will probably help testing some hardware routers in a
somehow comparable way.

I agree. My best quick hack for this is to make sure that there are extra resources on the CPU when running. That's why bgperf now tracks %idle. 

As you can probably tell,  bgperf isn't a great architecture now, and I'm trying to get useful results as quickly as possible. So much could be done here.

thanks a lot!

Justin

On Tue, Aug 31, 2021 at 2:30 AM Maria Matejka <maria.matejka@nic.cz> wrote:
Hello!

On 8/31/21 1:29 AM, Justin Pietsch wrote:
> I meant to ask if you'd compared exabgp to bird as testers? I'll try to
> compare later if I have some time.
Well, I have no exact numbers, I just implemented the bird tester and it
seems to eat less memory and less time. You could probably run the same
batch first with exabgp testers and then with bird testers, comparing
how much this affects the results.

> On Mon, Aug 30, 2021 at 4:27 PM Justin Pietsch <jpietsch@gmail.com
> <mailto:jpietsch@gmail.com>> wrote:
>
>     This is fantastic!!!! Can you submit those changes as PRs to my
>     bgperf? I'm working on moving it to a more permanent place. Then we
>     can track issues! :)

Well, yes, the tester exchange is already there pending:

        https://github.com/jopietsch/bgperf/pull/1

and it seems that there are two more from somebody else.

The tester exchange is anyway quite stupid and may need some similar
code to MRT testers instead of what is there now.

>     your suggestion for specifying branches used to work in bgperf, but
>     I broke it in trying to get everything working. I moved some of the
>     targets to just download prebuilt docker images, which makes it
>     harder. Also with batch I'm not sure how to have multiple versions
>     of the same container. Just more things to figure out, but that all
>     makes sense.

I think it would be better to have separate containers for different
branches, yet I couldn't figure out how to do that, except for brutal
solutions which were too disgusting to implement.

Anyway, when BIRD is used as tester, it shouldn't imho change with the
target, which is probably the main reason why I prefer separate
containers for branches.

One more thing – when running update -c, the container should also pull,
not only checkout, imho. (Haven't had time to fix this yet.)

>         * fixed policy configuration in BIRD (no clue whether it works with
>         other stacks)

This may need thorough examination first to formally define what the
policies should mean. It would be really stupid to compare route
filtering with different policies in each stack.

One more feature I'd like to see, yet I'm not sure how to do this
properly – to make the measurement more accurate, the target should not
only run on its own machine, but bgperf with all the testers should also
watch their load, issuing warning when the bgperf suite may be the
bottleneck. This will probably help testing some hardware routers in a
somehow comparable way.

Maria