Hello,
I had quite a hard time coming up with a subject line for this email.
I hope "BGP Session Stateful Failover across multiple BIRD instances" is sufficiently self-descriptive.
I should mention upfront that I am not aware if this functionality already exists in BIRD (or any other open-source routing daemon).
The goal is to optimize availability in a scenario where BGP sessions are disaggregated from the actual forwarding boxes.
Scenario 1—the most basic one—is straightforward:
- Anycast for a set of IPv4 and IPv6 addresses, propagated via IGP.
- Downstream BGP sessions configured as eBGP multihop, pointing to these Anycast IPs.
- Each node is configured with 100% of the peers and their respective policies.
- Metrics, such as IGP cost, determine which Anycast instance a given BGP client connects to.
- These nodes handle the BGP heavy lifting—managing route exchanges and applying filters for the downstream peers.
- They then propagate these routes to the forwarding devices, providing them in the most streamlined (filtered/optimized) form possible.
Up to this point—as long as the network remains stable—everything works well.
The problem arises when an event causes a specific set of downstream clients to switch from one Route Reflector (RR) node to another.
Initially, I assumed that this kind of "BGP session roaming" would primarily be triggered by the unavailability of a server running the Route Reflector. However, that is rarely the case.
More often, changes within the IGP—for various reasons—cause those BGP clients to shift to a different node.
This leads to session flaps, which in turn cause traffic flaps and significant churn during route recalculations.
I even tried to prevent the IGP from steering traffic from a given downstream client to a different node... but that only made things worse.
The conclusion I’ve reached is that the healthiest approach would be for a BGP session to be resumed from a different node, in such a way that the BGP client remains unaware that this has occurred.
---
Having outlined the problem and the desired solution...
I’m not sure how to actually implement this!
I don’t know if BIRD supports this, and if it does, I’m not sure how to configure it.
I am aware that "graceful restart" is a prerequisite for this kind of implementation.
However, I honestly don’t know how to make this "Anycast TCP" concept work within the context of BGP.
I imagine it would require some form of copying or replication of the active BGP session state table to the nodes that are currently in a passive state.
Logically, this would also likely necessitate a quorum mechanism to determine which node is the active one.
---
I am sharing this scenario—the problem and the intended solution—with you all.
I would welcome any guidance or suggestions as to whether you have encountered a solution for this issue.
P.S.1: I suspect that companies like Google and AWS, which utilize disaggregation in their peering scenarios, likely already implement some form of stateful failover for BGP sessions across multiple boxes.
P.S.2: There is a "Scenario 2" involving IP-in-IP tunnels and PBR (Policy-Based Routing) to make it appear as though the Reflector is directly connected to the EdgePeer.
However, I believe detailing that scenario would only add unnecessary complexity and distract from the core issue: stateful BGP session failover.
Thank you in advance.
-- Douglas Fernando Fischer
Engº de Controle e Automação