Good morning, I was wondering, what is the best practice is for running 2 data centers, each with one uplink and a dark fiber in between them? I think it's pretty much a standard situation, but will show our setup and assumptions as well: upstream 1 upstream 2 | | dc1--darkfiber----dc 2 Additional each DC has 2 routers: dc1: dc2: |---router1 ------- router 1 | / | / | /--------/ | router2---------router2 | | |---------------------| (both routers are connected to each other and both routers are connected to each upstream) The objective is to stay online in each DC as long as possible, so in theory: - upstream 1 can die or - upstream 2 can die or - the darkfiber can die And in each of the situations, both DCs should still be reachable from outside and both be reaching itself. Question 1: Is "direct;" is the right protocol for all links? As all links are layer 2 connections, we have configured all links to be direct. However this causes the "Invalid NEXT_HOP attribute" in various situations. Question 2: Is "next hop self ebgp;" the correct answer to the Invalid NEXT_HOP attribute? This way we rewrote routes from upstreams and generally speaking it seems to work. However, this still fails iBGP routes: router1.dc1 cannot reach a network in dc2 that is configured to be statically routed to another host in dc2. My assumption is that this is the problem of the "gateway direct" resolution, even though the documentation kind of indicates that this should be resolved via the next hop of the dc2. Originally I'd have thought that not even "next hop self ebgp" is required, as the next hop attribute is coming from the peering router, so the route should be directly reachable. Question 3: Is "not direct" (aka multiphop) the right thing for iBGP? So our dcs are directly connected vi layer 2, but the default for iBGP is multihop. If we omit the "direct" keyword, the result is that no routes are in the end imported from the other DC and that we get various warnings like the following in syslog: Dec 16 00:58:35 router2 daemon.warn bird: Next hop address 2a0a:e5c0:1:8::5 resolvable through recursive route for 2a0a:e5c0:1:8::/64 So at this stage it's a bit unclear to me, how proper iBGP redistribution / next hop resolution should work with bird. Question 4: How to (not) announce umbrella networks? So both data centers are reachable within the 2a0a:e5c0::/29 prefix and all routers have a statement route 2a0a:e5c0::/29 unreachable; in the static protocol. However if the dark fiber goes down, it is not clear that / how the upstreams should / will propagate the smaller (per DC) /48s. If we remove the full /29 announcement, we might run into the problem that a packet is being sent out through one upstream and returns back to us and thus creates a routing loop. While the last question is eBGP specific, the main question of this thread still is: how do you correctly distribute internal and external routes via iBGP in your DCs? Thanks for any hints in the right direction. Best regards, Nico -- Modern, affordable, Swiss Virtual Machines. Visit www.datacenterlight.ch