The question raised by colleague Irene reminded me of a topic that may or may not be the focus of BIRD's development. I imagine that the biggest supporters of SMP/Multi-Core/Thread-Safe evolution on BIRD are Operators of Route-Servers of large IXPs, and operators of large-scale Route-Reflectors. Although BFD has its greatest use in the transport network and Underlay, it is increasingly common to see the use of BFD in BGP Internet. I'm personally overly excited about what BIRD version 3 is demonstrating in terms of vertical scalability. But I keep imagining that, even having scalability in the BGP engine, it is almost prohibitive to use BFD in a scenario with a thousand BGP Peers. Is there any view from the IBRD development team for this matter? Or even... Is there any open project focused on BFD that can address this? And going a little further... In a quick Brainstorming... What would be the techniques that could help a Route-Server or Route-Reflector with a thousand BGP Peers to also support a thousand BFD sessions effectively and efficiently? Perhaps resort to some hardware-based off-loading method? Would eBPF help this in any way? -- Douglas Fernando Fischer Engº de Controle e Automação
Douglas Fischer <fischerdouglas@gmail.com> writes:
The question raised by colleague Irene reminded me of a topic that may or may not be the focus of BIRD's development.
I imagine that the biggest supporters of SMP/Multi-Core/Thread-Safe evolution on BIRD are Operators of Route-Servers of large IXPs, and operators of large-scale Route-Reflectors.
Although BFD has its greatest use in the transport network and Underlay, it is increasingly common to see the use of BFD in BGP Internet.
I'm personally overly excited about what BIRD version 3 is demonstrating in terms of vertical scalability.
But I keep imagining that, even having scalability in the BGP engine, it is almost prohibitive to use BFD in a scenario with a thousand BGP Peers.
Is there any view from the IBRD development team for this matter? Or even... Is there any open project focused on BFD that can address this?
And going a little further... In a quick Brainstorming... What would be the techniques that could help a Route-Server or Route-Reflector with a thousand BGP Peers to also support a thousand BFD sessions effectively and efficiently? Perhaps resort to some hardware-based off-loading method? Would eBPF help this in any way?
On this last point: it would probably be fairly straight-forward to implement the echo reflector bit of BFP in eBPF, say in XDP: you'd just populate a map with the session data, parse the packet as it comes in and check against that, sending out a reply directly. I imagine that would scale fairly well. Can't really do the other direction, though (we don't have a way to synthesise packet transmission from inside eBPF yet), and obviously some state propagation would be needed. But I imagine it would be possible to at least speed things up somewhat using eBPF :) -Toke
On Fri, Apr 01, 2022 at 08:44:50AM -0300, Douglas Fischer wrote:
The question raised by colleague Irene reminded me of a topic that may or may not be the focus of BIRD's development.
I imagine that the biggest supporters of SMP/Multi-Core/Thread-Safe evolution on BIRD are Operators of Route-Servers of large IXPs, and operators of large-scale Route-Reflectors.
Although BFD has its greatest use in the transport network and Underlay, it is increasingly common to see the use of BFD in BGP Internet.
I'm personally overly excited about what BIRD version 3 is demonstrating in terms of vertical scalability.
But I keep imagining that, even having scalability in the BGP engine, it is almost prohibitive to use BFD in a scenario with a thousand BGP Peers.
Is there any view from the IBRD development team for this matter? Or even... Is there any open project focused on BFD that can address this?
Hmm, that is a good point. It would make sense to have multiple BFD threads, but i think that it is more a question of improving I/O loop performance in BIRD, as thousand peers with 100ms period is about 10 kpps UDP rate, which should be manageable even from a single thread. We should make some effort to do some benchmarking for BFD. -- Elen sila lumenn' omentielvo Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
Hello all! Just passing here to see if any moves on this BFD Scaling occurred. This week some friends that are involved in the operation of a really big IXP told me that they were having problems with some "funny" participants of its IXP that adjusted their BGP Timer to numbers like 5/15. On an environment with thousands of peers, I'm sure you can imagine the CPU impact of that. Now you are probably asking: What that has to do with Scale the BFD capacities on BIRD? Well... Those 'j'enius are probably adjusting the timers to have some kind of control of some communication issue occurs between his router and the RS. They just "forget" that on that level of reduction, it compromises the processing capacity of the RS. If BFD engine could support session for every participant at IXP, or at least for those that wants that kind of resource. Then would be reasonable to lock the timers of BGP Session in 60/180. Em sex., 1 de abr. de 2022 às 13:41, Ondrej Zajicek <santiago@crfreenet.org> escreveu:
On Fri, Apr 01, 2022 at 08:44:50AM -0300, Douglas Fischer wrote:
The question raised by colleague Irene reminded me of a topic that may or may not be the focus of BIRD's development.
I imagine that the biggest supporters of SMP/Multi-Core/Thread-Safe evolution on BIRD are Operators of Route-Servers of large IXPs, and operators of large-scale Route-Reflectors.
Although BFD has its greatest use in the transport network and Underlay, it is increasingly common to see the use of BFD in BGP Internet.
I'm personally overly excited about what BIRD version 3 is demonstrating in terms of vertical scalability.
But I keep imagining that, even having scalability in the BGP engine, it is almost prohibitive to use BFD in a scenario with a thousand BGP Peers.
Is there any view from the IBRD development team for this matter? Or even... Is there any open project focused on BFD that can address this?
Hmm, that is a good point. It would make sense to have multiple BFD threads, but i think that it is more a question of improving I/O loop performance in BIRD, as thousand peers with 100ms period is about 10 kpps UDP rate, which should be manageable even from a single thread. We should make some effort to do some benchmarking for BFD.
-- Elen sila lumenn' omentielvo
Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
-- Douglas Fernando Fischer Engº de Controle e Automação
Douglas there was a workshop on "BFD at IXes" at the recent Euro-IX Forum in Tampere. These are the parameters the participants agreed upon * Interval: 500ms - 1500ms * Multiplier: 3 or 5 * Passive: on * Idle_tx: 3x Interval * Auth: none Greetings Arnold On 18.06.2022 13:47, Douglas Fischer wrote:
Hello all!
Just passing here to see if any moves on this BFD Scaling occurred.
This week some friends that are involved in the operation of a really big IXP told me that they were having problems with some "funny" participants of its IXP that adjusted their BGP Timer to numbers like 5/15. On an environment with thousands of peers, I'm sure you can imagine the CPU impact of that.
Now you are probably asking: What that has to do with Scale the BFD capacities on BIRD?
Well... Those 'j'enius are probably adjusting the timers to have some kind of control of some communication issue occurs between his router and the RS. They just "forget" that on that level of reduction, it compromises the processing capacity of the RS.
If BFD engine could support session for every participant at IXP, or at least for those that wants that kind of resource. Then would be reasonable to lock the timers of BGP Session in 60/180.
Em sex., 1 de abr. de 2022 às 13:41, Ondrej Zajicek <santiago@crfreenet.org <mailto:santiago@crfreenet.org>> escreveu:
On Fri, Apr 01, 2022 at 08:44:50AM -0300, Douglas Fischer wrote: > The question raised by colleague Irene reminded me of a topic that may or > may not be the focus of BIRD's development. > > I imagine that the biggest supporters of SMP/Multi-Core/Thread-Safe > evolution on BIRD are Operators of Route-Servers of large IXPs, and > operators of large-scale Route-Reflectors. > > Although BFD has its greatest use in the transport network and Underlay, it > is increasingly common to see the use of BFD in BGP Internet. > > I'm personally overly excited about what BIRD version 3 is demonstrating in > terms of vertical scalability. > > But I keep imagining that, even having scalability in the BGP engine, it is > almost prohibitive to use BFD in a scenario with a thousand BGP Peers. > > Is there any view from the IBRD development team for this matter? > Or even... Is there any open project focused on BFD that can address this?
Hmm, that is a good point. It would make sense to have multiple BFD threads, but i think that it is more a question of improving I/O loop performance in BIRD, as thousand peers with 100ms period is about 10 kpps UDP rate, which should be manageable even from a single thread. We should make some effort to do some benchmarking for BFD.
-- Elen sila lumenn' omentielvo
Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org <mailto:santiago@crfreenet.org>) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net <http://wwwkeys.pgp.net>) "To err is human -- to blame it on a computer is even more so."
-- Douglas Fernando Fischer Engº de Controle e Automação
-- Keep calm, keep distance, keep connected! Arnold Nipper email: arnold@nipper.de mobile: +49 172 2650958
Hello Arnold Thank you for the information! Are there any public docs or references of that workshop? Sincerely, what caught my attention was the "Auth: none" part. On a room with more than thousand persons, confirm if the voice you rear is really from the person you think it is makes sense to me. But the good part is that the interval is not crazily small as some wanted. This reduces the number of pps the deamon has to deal with. Using the biggest IXP in number os participants as a reference. Assuming all participants active BFD, there are 2500 BFD session per server per protocol(v4+v6) Assuming that the minimum interval(500ms) is always used(worst case scenario). 2750*2*=11.000pps I don't know if that is feasible for an engine without any improvements of performance. Em qui., 23 de jun. de 2022 às 08:52, Arnold Nipper <arnold@nipper.de> escreveu:
Douglas
there was a workshop on "BFD at IXes" at the recent Euro-IX Forum in Tampere.
These are the parameters the participants agreed upon
* Interval: 500ms - 1500ms * Multiplier: 3 or 5 * Passive: on * Idle_tx: 3x Interval * Auth: none
Greetings Arnold
On 18.06.2022 13:47, Douglas Fischer wrote:
Hello all!
Just passing here to see if any moves on this BFD Scaling occurred.
This week some friends that are involved in the operation of a really big IXP told me that they were having problems with some "funny" participants of its IXP that adjusted their BGP Timer to numbers like 5/15. On an environment with thousands of peers, I'm sure you can imagine the CPU impact of that.
Now you are probably asking: What that has to do with Scale the BFD capacities on BIRD?
Well... Those 'j'enius are probably adjusting the timers to have some kind of control of some communication issue occurs between his router and the RS. They just "forget" that on that level of reduction, it compromises the processing capacity of the RS.
If BFD engine could support session for every participant at IXP, or at least for those that wants that kind of resource. Then would be reasonable to lock the timers of BGP Session in 60/180.
Em sex., 1 de abr. de 2022 às 13:41, Ondrej Zajicek <santiago@crfreenet.org <mailto:santiago@crfreenet.org>> escreveu:
On Fri, Apr 01, 2022 at 08:44:50AM -0300, Douglas Fischer wrote: > The question raised by colleague Irene reminded me of a topic that may or > may not be the focus of BIRD's development. > > I imagine that the biggest supporters of SMP/Multi-Core/Thread-Safe > evolution on BIRD are Operators of Route-Servers of large IXPs, and > operators of large-scale Route-Reflectors. > > Although BFD has its greatest use in the transport network and Underlay, it > is increasingly common to see the use of BFD in BGP Internet. > > I'm personally overly excited about what BIRD version 3 is demonstrating in > terms of vertical scalability. > > But I keep imagining that, even having scalability in the BGP engine, it is > almost prohibitive to use BFD in a scenario with a thousand BGP Peers. > > Is there any view from the IBRD development team for this matter? > Or even... Is there any open project focused on BFD that can address this?
Hmm, that is a good point. It would make sense to have multiple BFD threads, but i think that it is more a question of improving I/O loop performance in BIRD, as thousand peers with 100ms period is about 10 kpps UDP rate, which should be manageable even from a single thread. We should make some effort to do some benchmarking for BFD.
-- Elen sila lumenn' omentielvo
Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org <mailto:santiago@crfreenet.org>) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net <http://wwwkeys.pgp.net>) "To err is human -- to blame it on a computer is even more so."
-- Douglas Fernando Fischer Engº de Controle e Automação
-- Keep calm, keep distance, keep connected!
Arnold Nipper email: arnold@nipper.de mobile: +49 172 2650958
-- Douglas Fernando Fischer Engº de Controle e Automação
On 23.06.2022 23:41, Douglas Fischer wrote:
Are there any public docs or references of that workshop?
Not yet, Douglas
Sincerely, what caught my attention was the "Auth: none" part. On a room with more than thousand persons, confirm if the voice you rear is really from the person you think it is makes sense to me.
Well, on an IX LAN, you should know how is talking to you ;-) Requring auth doesn't add any security IMO.
But the good part is that the interval is not crazily small as some wanted. This reduces the number of pps the deamon has to deal with.
Using the biggest IXP in number os participants as a reference. Assuming all participants active BFD, there are 2500 BFD session per server per protocol(v4+v6) Assuming that the minimum interval(500ms) is always used(worst case scenario). 2750*2*=11.000pps I don't know if that is feasible for an engine without any improvements of performance.
The smaller IXes may go for 500ms. The bigger ones probably will not do. And unlike BGP, BFD always goes with the higher interval timer. Arnold
Em qui., 23 de jun. de 2022 às 08:52, Arnold Nipper <arnold@nipper.de <mailto:arnold@nipper.de>> escreveu:
Douglas
there was a workshop on "BFD at IXes" at the recent Euro-IX Forum in Tampere.
These are the parameters the participants agreed upon
* Interval: 500ms - 1500ms * Multiplier: 3 or 5 * Passive: on * Idle_tx: 3x Interval * Auth: none
Greetings Arnold
On 18.06.2022 13:47, Douglas Fischer wrote: > Hello all! > > Just passing here to see if any moves on this BFD Scaling occurred. > > This week some friends that are involved in the operation of a really > big IXP told me that they were having problems with some "funny" > participants of its IXP that adjusted their BGP Timer to numbers like 5/15. > On an environment with thousands of peers, I'm sure you can imagine the > CPU impact of that. > > Now you are probably asking: > What that has to do with Scale the BFD capacities on BIRD? > > Well... > Those 'j'enius are probably adjusting the timers to have some kind of > control of some communication issue occurs between his router and the RS. > They just "forget" that on that level of reduction, it compromises the > processing capacity of the RS. > > If BFD engine could support session for every participant at IXP, or at > least for those that wants that kind of resource. > Then would be reasonable to lock the timers of BGP Session in 60/180. > > > Em sex., 1 de abr. de 2022 às 13:41, Ondrej Zajicek > <santiago@crfreenet.org <mailto:santiago@crfreenet.org> <mailto:santiago@crfreenet.org <mailto:santiago@crfreenet.org>>> escreveu: > > On Fri, Apr 01, 2022 at 08:44:50AM -0300, Douglas Fischer wrote: > > The question raised by colleague Irene reminded me of a topic > that may or > > may not be the focus of BIRD's development. > > > > I imagine that the biggest supporters of SMP/Multi-Core/Thread-Safe > > evolution on BIRD are Operators of Route-Servers of large IXPs, and > > operators of large-scale Route-Reflectors. > > > > Although BFD has its greatest use in the transport network and > Underlay, it > > is increasingly common to see the use of BFD in BGP Internet. > > > > I'm personally overly excited about what BIRD version 3 is > demonstrating in > > terms of vertical scalability. > > > > But I keep imagining that, even having scalability in the BGP > engine, it is > > almost prohibitive to use BFD in a scenario with a thousand BGP > Peers. > > > > Is there any view from the IBRD development team for this matter? > > Or even... Is there any open project focused on BFD that can > address this? > > Hmm, that is a good point. It would make sense to have multiple BFD > threads, but i think that it is more a question of improving I/O loop > performance in BIRD, as thousand peers with 100ms period is about 10 > kpps > UDP rate, which should be manageable even from a single thread. We > should > make some effort to do some benchmarking for BFD. > > -- > Elen sila lumenn' omentielvo > > Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org <mailto:santiago@crfreenet.org> > <mailto:santiago@crfreenet.org <mailto:santiago@crfreenet.org>>) > OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, > wwwkeys.pgp.net <http://wwwkeys.pgp.net> <http://wwwkeys.pgp.net <http://wwwkeys.pgp.net>>) > "To err is human -- to blame it on a computer is even more so." > > > > -- > Douglas Fernando Fischer > Engº de Controle e Automação
-- Keep calm, keep distance, keep connected!
Arnold Nipper email: arnold@nipper.de <mailto:arnold@nipper.de> mobile: +49 172 2650958
-- Douglas Fernando Fischer Engº de Controle e Automação
-- Keep calm, keep distance, keep connected! Arnold Nipper email: arnold@nipper.de mobile: +49 172 2650958
Arnold Nipper пишет 24.06.2022 12:32:
On 23.06.2022 23:41, Douglas Fischer wrote:
Are there any public docs or references of that workshop?
Not yet, Douglas
Sincerely, what caught my attention was the "Auth: none" part. On a room with more than thousand persons, confirm if the voice you rear is really from the person you think it is makes sense to me.
Well, on an IX LAN, you should know how is talking to you ;-) Requring auth doesn't add any security IMO.
But the good part is that the interval is not crazily small as some wanted. This reduces the number of pps the deamon has to deal with.
Using the biggest IXP in number os participants as a reference. Assuming all participants active BFD, there are 2500 BFD session per server per protocol(v4+v6) Assuming that the minimum interval(500ms) is always used(worst case scenario). 2750*2*=11.000pps I don't know if that is feasible for an engine without any improvements of performance.
The smaller IXes may go for 500ms. The bigger ones probably will not do. And unlike BGP, BFD always goes with the higher interval timer.
Hi, It also up for customers wishes. We provide selective BFD timers. Some of IXP members local , some 1000+ kilometers away. Some "requires" sub-second failure detection (you need to think about your infrastructure to support this). 95+% agree with our default values. Wbr, Mikhail.
Arnold
Em qui., 23 de jun. de 2022 às 08:52, Arnold Nipper <arnold@nipper.de <mailto:arnold@nipper.de>> escreveu:
Douglas
there was a workshop on "BFD at IXes" at the recent Euro-IX Forum in Tampere.
These are the parameters the participants agreed upon
* Interval: 500ms - 1500ms * Multiplier: 3 or 5 * Passive: on * Idle_tx: 3x Interval * Auth: none
Greetings Arnold
On 18.06.2022 13:47, Douglas Fischer wrote: > Hello all! > > Just passing here to see if any moves on this BFD Scaling occurred. > > This week some friends that are involved in the operation of a really > big IXP told me that they were having problems with some "funny" > participants of its IXP that adjusted their BGP Timer to numbers like 5/15. > On an environment with thousands of peers, I'm sure you can imagine the > CPU impact of that. > > Now you are probably asking: > What that has to do with Scale the BFD capacities on BIRD? > > Well... > Those 'j'enius are probably adjusting the timers to have some kind of > control of some communication issue occurs between his router and the RS. > They just "forget" that on that level of reduction, it compromises the > processing capacity of the RS. > > If BFD engine could support session for every participant at IXP, or at > least for those that wants that kind of resource. > Then would be reasonable to lock the timers of BGP Session in 60/180. > > > Em sex., 1 de abr. de 2022 às 13:41, Ondrej Zajicek > <santiago@crfreenet.org <mailto:santiago@crfreenet.org> <mailto:santiago@crfreenet.org <mailto:santiago@crfreenet.org>>> escreveu: > > On Fri, Apr 01, 2022 at 08:44:50AM -0300, Douglas Fischer wrote: > > The question raised by colleague Irene reminded me of a topic > that may or > > may not be the focus of BIRD's development. > > > > I imagine that the biggest supporters of SMP/Multi-Core/Thread-Safe > > evolution on BIRD are Operators of Route-Servers of large IXPs, and > > operators of large-scale Route-Reflectors. > > > > Although BFD has its greatest use in the transport network and > Underlay, it > > is increasingly common to see the use of BFD in BGP Internet. > > > > I'm personally overly excited about what BIRD version 3 is > demonstrating in > > terms of vertical scalability. > > > > But I keep imagining that, even having scalability in the BGP > engine, it is > > almost prohibitive to use BFD in a scenario with a thousand BGP > Peers. > > > > Is there any view from the IBRD development team for this matter? > > Or even... Is there any open project focused on BFD that can > address this? > > Hmm, that is a good point. It would make sense to have multiple BFD > threads, but i think that it is more a question of improving I/O loop > performance in BIRD, as thousand peers with 100ms period is about 10 > kpps > UDP rate, which should be manageable even from a single thread. We > should > make some effort to do some benchmarking for BFD. > > -- > Elen sila lumenn' omentielvo > > Ondrej 'Santiago' Zajicek (email: santiago@crfreenet.org <mailto:santiago@crfreenet.org> > <mailto:santiago@crfreenet.org <mailto:santiago@crfreenet.org>>) > OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, > wwwkeys.pgp.net <http://wwwkeys.pgp.net> <http://wwwkeys.pgp.net <http://wwwkeys.pgp.net>>) > "To err is human -- to blame it on a computer is even more so." > > > > -- > Douglas Fernando Fischer > Engº de Controle e Automação
-- Keep calm, keep distance, keep connected!
Arnold Nipper email: arnold@nipper.de <mailto:arnold@nipper.de> mobile: +49 172 2650958
-- Douglas Fernando Fischer Engº de Controle e Automação
On Fri, 24 Jun 2022, 22:34 Mikhail Grishin, <magr@ripn.net> wrote:
Arnold Nipper пишет 24.06.2022 12:32:
On 23.06.2022 23:41, Douglas Fischer wrote:
Sincerely, what caught my attention was the "Auth: none" part. On a room with more than thousand persons, confirm if the voice you rear is really from the person you think it is makes sense to me.
Well, on an IX LAN, you should know how is talking to you ;-) Requring auth doesn't add any security IMO.
Not to mention it only affects BFD, not the BGP session it supports. You aren't affecting anything of value by targeting unauthenticated BFD. It also up for customers wishes. We provide selective BFD timers.
Some of IXP members local , some 1000+ kilometers away. Some "requires" sub-second failure detection (you need to think about your infrastructure to support this).
Those people are silly. Sub-second failure detection is fine when you're talking about an MPLS tunnel with precomputed secondary paths or fast reroute, but this is BGP. Your network is very unlikely to reconverge in under a second after a BGP session goes down if there are more than a handful of prefixes, as everything has to recalculate best routes etc. But hey, it probably fixes *someone's* use case... M
participants (6)
-
Arnold Nipper -
Douglas Fischer -
Matthew Walster -
Mikhail Grishin -
Ondrej Zajicek -
Toke Høiland-Jørgensen