Hello! I'm new to using Bird and BGP, but I'm learning. I've recently acquired an IPv4/24, and I've been working on building an Anycast network for CDN use. Right now I have a 15 datacenter network from a VPS provider (which uses Bird as it's recommended daemon: https://www.vultr.com/docs/configuring-bgp-on-vultr). Using their configuration example, I was able to get an anycast network up without any modification. That part is working great! The problem I have now is figuring out how to improve the routing. For example, Comcast users in Portland, OR are being sent to Los Angeles, even though the closest server is in Seattle (confirmed with traceroute). The underlying BGP problem seems to be that Comcast sees multiple routes as 2 AS routes (7922 20473), so it doesn't understand that the Seattle instance is preferred here. There are many more examples like this I've found, and I'm trying to figure out what an approach might be to try to improve this. Is there a way to improve routing using bird export filters or some other mechanism I'm not familiar with? Can I use Bird to steer certain traffic to closer servers, or to only announce routes to regional BGP servers? Or is this largely out of my control and I need to accept these non-ideal routes? I've also been told I might be able to use BGP communities for this, but I've never worked with communities directly, so I don't understand how they would help me here. This is the list of communities I have available to me: https://www.vultr.com/docs/as20473-bgp-customer-guide Thank you in advance for any ideas or code examples here. There is zero documentation on tuning Anycast out there (even the old school BGP books I've skimmed through barely mention anycast). Here is my current bird.conf: log syslog all; router id **MY_IP**; protocol bgp vultr { local as 395409; source address **MY_IP**; import none; export all; graceful restart on; multihop 2; neighbor **BGP_IP** as 64515; password "**OBSCURED**"; } protocol static { route 198.51.233.0/24 via **MY_IP**; } protocol device { scan time 5; } protocol direct { interface "neocities*"; import all; } -Kyle
On 21 Sep 2016, at 22:35, Kyle Drake <kyle@kyledrake.net> wrote:
I've recently acquired an IPv4/24, and I've been working on building an Anycast network for CDN use
Normally you would use anycast to get you to a DNS server (which doesn't have to be that near), then a geographic DNS server to get you to the right CDN element. -- Alex Bligh
Normally you would use anycast to get you to a DNS server (which doesn't have to be that near), then a geographic DNS server to get you to the right CDN element.
That's what I was doing previously, but I need to control the IPs for the CDN, and I only have the budget for one /24, so I'm trying to make the best of it. Aside from some occasional weird routing, the network has worked really well. State has not been an issue for what we're doing (short-lived HTTP connections). I'm just trying to see what the extent of my powers to control weird routing are. It's odd to see Comcast cold-potatoing connections to the wrong routes, sometimes on the other side of the continent ( https://gist.github.com/kyledrake/7a4cd36ea276ec3134b4a51a42a37f48). I'm wondering if there is a way to configure Bird to help steer these sorts of things a little better, even if it's on a case-by-case or region-by-region level. My apologies if these are all dumb questions. Again, not much anycast documentation out there (I'm planning to improve this later by putting together a web resource for people doing this). -Kyle
These decisions are entirely made by Comcast's equipment - you have very little influence over their routing choices, especially because you do not peer directly with them. They do seem a bit nonsensical - Comcast's internal metrics should know that SEA is closer to PDX than LAX is, and should route accordingly, but this may be done for capacity reasons facing Choopa in Seattle or on the PDX-SEA links, an observation that the majority of Choopa traffic is destined for LA, and Seattle would be a detour, or who knows why else. You could ask Vultr to raise this with Comcast and see if you can get an adjustment or answer, some providers are responsive to this kind of request, but only from a direct customer/peer. There is one mechanism built into BGP for influencing route metrics - MED, but this is only transitive to the neighbour AS, so Comcast wouldn't see your MEDs, and it's very common to ignore them anyway. The other mechanism for influencing this kind of thing is through BGP community signalling, though this method is usually pretty blunt. Typically you can suppress announcements toward a specific AS entirely, or prepend toward them (lengthening the AS path and affecting BGP route selection); usually you can also modify the localpref of your announcement within your peer's network. The goal being, more or less, to encourage remote networks to avoid the problem AS/link entirely. If you're trying to avoid Comcast's messed up internal routing, your only real option is not to announce toward Comcast at all, and hope that Comcast will hand it off quickly to someone that routes it better. I wouldn't really count on it. Prepending might work, if Choopa and Comcast are peers, but if Choopa is a customer, Comcast will likely local-pref their routes above their peering/transit, and AS path length can't defeat this. Vultr indicates they support some BGP communities to do this kind of manipulation: https://www.vultr.com/docs/as20473-bgp-customer-guide Comcast might also support something similar, though it is common to strip communities before passing the routes upstream. You might be able to work with Vultr to have them signal to Comcast that your announcements are backup only or the like, or do so yourself and the Vultr/Choopa routers pass this along. But again, this has the same effect as not announcing to them at all - they will choose a *different* route that doesn't connect directly to Choopa, but it may not be better. More or less you're at their mercy. K Quoting Kyle Drake <kyle@kyledrake.net>:
Normally you would use anycast to get you to a DNS server (which doesn't have to be that near), then a geographic DNS server to get you to the right CDN element.
That's what I was doing previously, but I need to control the IPs for the CDN, and I only have the budget for one /24, so I'm trying to make the best of it. Aside from some occasional weird routing, the network has worked really well. State has not been an issue for what we're doing (short-lived HTTP connections).
I'm just trying to see what the extent of my powers to control weird routing are. It's odd to see Comcast cold-potatoing connections to the wrong routes, sometimes on the other side of the continent ( https://gist.github.com/kyledrake/7a4cd36ea276ec3134b4a51a42a37f48). I'm wondering if there is a way to configure Bird to help steer these sorts of things a little better, even if it's on a case-by-case or region-by-region level.
My apologies if these are all dumb questions. Again, not much anycast documentation out there (I'm planning to improve this later by putting together a web resource for people doing this).
-Kyle
Thanks for the detailed information, this is very informative. I realize now that this question was probably off topic for the mailing list and I apologize to everyone for that. After talking with someone about it, I realized that the problem was that my BGP provider (http://bgp.he.net/AS20473) was indeed direct peering with Comcast in a few datacenters but not all their datacenters, causing a heavy traffic bias to a few locations. I used one of their BGP communities to AS prepend Comcast to discourage direct peering and I'll see if that helps spread the traffic more evenly (my guess is that it will, the IP transit routes tend to be doing much better). Thanks again! This is how my bird.conf ended up looking: log syslog all; router id **MY_IP**; protocol bgp vultr { local as 395409; source address **MY_IP**; import none; export all; graceful restart on; multihop 2; neighbor **BGP_IP** as 64515; password "**OBSCURED**"; export filter { bgp_community = -empty-; bgp_community.add((64603,7922)); # 64603 = Prepend 3x to Comcast ASN (7922) accept; }; } protocol static { route 198.51.233.0/24 via **MY_IP**; } protocol device { scan time 5; } protocol direct { interface "neocities*"; import all; } On Wed, Sep 21, 2016 at 4:41 PM, Keenan Tims <ktims@gotroot.ca> wrote:
These decisions are entirely made by Comcast's equipment - you have very little influence over their routing choices, especially because you do not peer directly with them. They do seem a bit nonsensical - Comcast's internal metrics should know that SEA is closer to PDX than LAX is, and should route accordingly, but this may be done for capacity reasons facing Choopa in Seattle or on the PDX-SEA links, an observation that the majority of Choopa traffic is destined for LA, and Seattle would be a detour, or who knows why else. You could ask Vultr to raise this with Comcast and see if you can get an adjustment or answer, some providers are responsive to this kind of request, but only from a direct customer/peer.
There is one mechanism built into BGP for influencing route metrics - MED, but this is only transitive to the neighbour AS, so Comcast wouldn't see your MEDs, and it's very common to ignore them anyway.
The other mechanism for influencing this kind of thing is through BGP community signalling, though this method is usually pretty blunt. Typically you can suppress announcements toward a specific AS entirely, or prepend toward them (lengthening the AS path and affecting BGP route selection); usually you can also modify the localpref of your announcement within your peer's network. The goal being, more or less, to encourage remote networks to avoid the problem AS/link entirely. If you're trying to avoid Comcast's messed up internal routing, your only real option is not to announce toward Comcast at all, and hope that Comcast will hand it off quickly to someone that routes it better. I wouldn't really count on it. Prepending might work, if Choopa and Comcast are peers, but if Choopa is a customer, Comcast will likely local-pref their routes above their peering/transit, and AS path length can't defeat this.
Vultr indicates they support some BGP communities to do this kind of manipulation:
https://www.vultr.com/docs/as20473-bgp-customer-guide
Comcast might also support something similar, though it is common to strip communities before passing the routes upstream. You might be able to work with Vultr to have them signal to Comcast that your announcements are backup only or the like, or do so yourself and the Vultr/Choopa routers pass this along. But again, this has the same effect as not announcing to them at all - they will choose a *different* route that doesn't connect directly to Choopa, but it may not be better.
More or less you're at their mercy.
K
Quoting Kyle Drake <kyle@kyledrake.net>:
Normally you would use anycast to get you to a DNS server (which doesn't have to be that near), then a geographic DNS server to get you to the right CDN element.
That's what I was doing previously, but I need to control the IPs for the
CDN, and I only have the budget for one /24, so I'm trying to make the best of it. Aside from some occasional weird routing, the network has worked really well. State has not been an issue for what we're doing (short-lived HTTP connections).
I'm just trying to see what the extent of my powers to control weird routing are. It's odd to see Comcast cold-potatoing connections to the wrong routes, sometimes on the other side of the continent ( https://gist.github.com/kyledrake/7a4cd36ea276ec3134b4a51a42a37f48). I'm wondering if there is a way to configure Bird to help steer these sorts of things a little better, even if it's on a case-by-case or region-by-region level.
My apologies if these are all dumb questions. Again, not much anycast documentation out there (I'm planning to improve this later by putting together a web resource for people doing this).
-Kyle
participants (3)
-
Alex Bligh -
Keenan Tims -
Kyle Drake