Hi all. I had bird installed on a server with one outside connection (eth2). That's been working fine. When I add in the second connection, for whatever reason I can't get it added to the routing. I'm curious if anyone has any suggestions? Here are some of the details: The New connection is 10 Gig Fiber (Intel X520, ixgbe driver) - eth5. Cogent can ping my port directly from their router, and I can ping their router ip from my server using -S to specify the Cogent Router port. Sudo ip route show table main | grep eth5 only gives the following info (IP addresses generalized, but correct in my system). 1.1.1.152/29 dev eth5 proto kernel scope link src 1.1.1.154 So assume my ubuntu router has IP address for eth5: 1.1.1.154 Cogent IP address: 1.1.1.153 Network: 1.1.1.152/29 I can ping: ping -S 1.1.1.154 1.1.1.153 And outside, from my home computer, I can ping: ping 1.1.1.154 and I get no response, but watching on my server with ipconfig -a, I see RX packets number increase. I'm not sure if this is directly bird related, but I think it is... Any tips or ideas would be greatly appreciated. Thanks!
$quoted_author = "dspazman@epicup.com" ;
I had bird installed on a server with one outside connection (eth2). That's been working fine. When I add in the second connection, for whatever reason I can't get it added to the routing. I'm curious if anyone has any suggestions?
You are probably running into a combination of your default route pointing out your existing provider and their strict uRPF blocking source IP's from your Cogent range.
I'm not sure if this is directly bird related, but I think it is... Any tips or ideas would be greatly appreciated.
This is not bird related. If you want this to work you need to route packets with a Cogent source IP out via your Cogent link. For that you'll need to do some policy based routing using the source IP as the filter. e.g. http://muchtall.com/2007/08/06/24/ cheers Marty
Sorry, should have been a little clearer. I have tried setting up my routing like: http://lartc.org/howto/lartc.rpdb.multiple-links.html And my bird setup is like: https://git.nic.cz/redmine/projects/bird/wiki/BGP_filtering (it is down right now, but you can use google cache to still see it) for the two /29 IP networks I have, one from Cogent, one from HE. I also have a /22 that I am announcing over BGP. The end result is to have a BGP router using two connections to announce my /22 using my ASN, from the bird router I need to be able to send out requests from my /22 over both interfaces (I have them weighted in the bird config file so one gets more traffic). I guess what I am asking is if this sounds like it is the correct setup? I don't need someone to babysit me through the whole process, I just want to make sure I'm not going down a rabbit hole and completely missing something. I'll be done with bugging the whole list now. If I could get a confirmation still that that is the correct route to take, that would be great. Setting up 2 routing tables like the first article, and then setting up bird like the BGP filtering article. I just didn't know if I should be setting up the 2 routing tables, or if the decision on the outbound routing was entirely bird. If anyone wanted to help more and contact me directly I could provide more specific information as well, but that's not required, and I don't want to be any trouble to the group in general. Thanks again! -----Original Message----- From: "Martin Barry" <marty@supine.com> Sent: Sunday, December 4, 2011 6:29am To: bird-users@atrey.karlin.mff.cuni.cz Subject: Re: Routing issues $quoted_author = "dspazman@epicup.com" ;
I had bird installed on a server with one outside connection (eth2). That's been working fine. When I add in the second connection, for whatever reason I can't get it added to the routing. I'm curious if anyone has any suggestions?
You are probably running into a combination of your default route pointing out your existing provider and their strict uRPF blocking source IP's from your Cogent range.
I'm not sure if this is directly bird related, but I think it is... Any tips or ideas would be greatly appreciated.
This is not bird related. If you want this to work you need to route packets with a Cogent source IP out via your Cogent link. For that you'll need to do some policy based routing using the source IP as the filter. e.g. http://muchtall.com/2007/08/06/24/ cheers Marty
$quoted_author = "dspazman@epicup.com" ;
I have tried setting up my routing like: http://lartc.org/howto/lartc.rpdb.multiple-links.html
And my bird setup is like: https://git.nic.cz/redmine/projects/bird/wiki/BGP_filtering (it is down right now, but you can use google cache to still see it)
for the two /29 IP networks I have, one from Cogent, one from HE. I also have a /22 that I am announcing over BGP.
The end result is to have a BGP router using two connections to announce my /22 using my ASN, from the bird router I need to be able to send out requests from my /22 over both interfaces (I have them weighted in the bird config file so one gets more traffic).
I guess what I am asking is if this sounds like it is the correct setup? I don't need someone to babysit me through the whole process, I just want to make sure I'm not going down a rabbit hole and completely missing something.
You're on the right track! :-) My original answer relates only to traffic with a source address in your Cogent /29 as this might be filtered by HE. You might need to do something similar to traffic with a source address in your HE /29 as this might be filtered by Cogent. Traffic with a source address in your /22 shouldn't have that problem.
I'll be done with bugging the whole list now. If I could get a confirmation still that that is the correct route to take, that would be great. Setting up 2 routing tables like the first article, and then setting up bird like the BGP filtering article. I just didn't know if I should be setting up the 2 routing tables, or if the decision on the outbound routing was entirely bird. If anyone wanted to help more and contact me directly I could provide more specific information as well, but that's not required, and I don't want to be any trouble to the group in general.
Bird will accept, filter and sort the BGP routes before exporting them to the kernel. The source based routing mentioned above is just an override at the kernel level so packets from either /29 make it back out the respective link. cheers Marty
On Mon, Dec 05, 2011 at 07:38:30AM +1100, Martin Barry wrote:
The end result is to have a BGP router using two connections to announce my /22 using my ASN, from the bird router I need to be able to send out requests from my /22 over both interfaces (I have them weighted in the bird config file so one gets more traffic).
which would balance the outgoing traffic. however the incoming comes over based on bgp preferences(mainly as path) in the internet which results in a lot of asymetric routing. this will result in weird latency and problems with one line affecting all connections.
My original answer relates only to traffic with a source address in your Cogent /29 as this might be filtered by HE. You might need to do something similar to traffic with a source address in your HE /29 as this might be filtered by Cogent. Traffic with a source address in your /22 shouldn't have that problem.
the policy routing should probably still be setup to allow testing of a particular line because this way incoming and outgoing traffic do not have to come over the same line. mk
On Mon, Dec 05, 2011 at 07:38:30AM +1100, Martin Barry wrote:
The end result is to have a BGP router using two connections to announce my /22 using my ASN, from the bird router I need to be able to send out requests from my /22 over both interfaces (I have them weighted in the bird config file so one gets more traffic).
which would balance the outgoing traffic. however the incoming comes over based on bgp preferences(mainly as path) in the internet which results in a lot of asymetric routing. this will result in weird latency and problems with one line affecting all connections.
Can the bgp routing path be set so incoming traffic prefers one connection over the other, as well? I thought setting something like this: protocol ospf { import all; export filter { ospf_metric1 = 1000; if source = RTS_STATIC then accept; else reject; }; area 0 { interface "eth0" { cost 50; type broadcast; hello 5; retransmit 2; wait 10; dead 20; }; interface "eth1" { cost 5; type pointopoint; hello 5; retransmit 2; wait 10; dead 20; }; interface "*" { cost 1000; stub; }; }; } Would cause the path through the connection on eth0 to be 10x more prefered then the path through eth1, so incoming bandwidth should be split in about a 10-1 ratio. So if I have eth0 as a 10 Gig connection, and eth1 as a 1 Gig connection, between using that for incoming traffic, and a multipath rule for outgoing traffic like this: ip route add default scope global nexthop via 1.1.1.153 dev eth0 weight 10 nexthop via 2.2.2.81 dev eth1 weight 1 Would cause the two connections to be pretty uniformly used on incoming and outgoing traffic at a 10/1 ratio, allowing me to pretty fully saturate both lines?
My original answer relates only to traffic with a source address in your Cogent /29 as this might be filtered by HE. You might need to do something similar to traffic with a source address in your HE /29 as this might be filtered by Cogent. Traffic with a source address in your /22 shouldn't have that problem.
the policy routing should probably still be setup to allow testing of a particular line because this way incoming and outgoing traffic do not have to come over the same line. mk
On Dec 4, 2011, at 15:38, dspazman@epicup.com wrote:
Can the bgp routing path be set so incoming traffic prefers one connection over the other, as well? I thought setting something like this: [...]
The rest of the internet decides which direction to send traffic to your network. Generally if the peers are different companies, the only lever you really have to push is the PATH length on your outgoing announcements. Ask
On Sun, Dec 04, 2011 at 06:38:51PM -0500, dspazman@epicup.com wrote:
On Mon, Dec 05, 2011 at 07:38:30AM +1100, Martin Barry wrote:
The end result is to have a BGP router using two connections to announce my /22 using my ASN, from the bird router I need to be able to send out requests from my /22 over both interfaces (I have them weighted in the bird config file so one gets more traffic).
which would balance the outgoing traffic. however the incoming comes over based on bgp preferences(mainly as path) in the internet which results in a lot of asymetric routing. this will result in weird latency and problems with one line affecting all connections.
Can the bgp routing path be set so incoming traffic prefers one connection over the other, as well? I thought setting something like this:
protocol ospf { import all; export filter { ospf_metric1 = 1000; if source = RTS_STATIC then accept; else reject; };
area 0 { interface "eth0" { cost 50; type broadcast; hello 5; retransmit 2; wait 10; dead 20; };
interface "eth1" { cost 5; type pointopoint; hello 5; retransmit 2; wait 10; dead 20; }; interface "*" { cost 1000; stub; }; }; }
Would cause the path through the connection on eth0 to be 10x more prefered then the path through eth1, so incoming bandwidth should be split in about a 10-1 ratio. So if I have eth0 as a 10 Gig connection, and eth1 as a 1 Gig connection, between using that for incoming traffic, and a multipath rule for outgoing traffic like this:
ip route add default scope global nexthop via 1.1.1.153 dev eth0 weight 10 nexthop via 2.2.2.81 dev eth1 weight 1
Would cause the two connections to be pretty uniformly used on incoming and outgoing traffic at a 10/1 ratio, allowing me to pretty fully saturate both lines?
eh. not really. the thing you show here is ospf and the cost is simply the weight of the link that is taken into account when it computes the shortest path. there is no load balancing involved. bgp can loadbalance only by ip networks, and it is a manual process for long term setup, not per packet magic solution. hate to say it but setting it correctly requires a lot of reading and some experience. basically the best thing to do is just to set two bgp peers and then mess with the as path to balance the incoming traffic over those two connections. mk
$quoted_author = "dspazman@epicup.com" ;
Would cause the path through the connection on eth0 to be 10x more prefered then the path through eth1, so incoming bandwidth should be split in about a 10-1 ratio. So if I have eth0 as a 10 Gig connection, and eth1 as a 1 Gig connection, between using that for incoming traffic, and a multipath rule for outgoing traffic like this:
ip route add default scope global nexthop via 1.1.1.153 dev eth0 weight 10 nexthop via 2.2.2.81 dev eth1 weight 1
Would cause the two connections to be pretty uniformly used on incoming and outgoing traffic at a 10/1 ratio, allowing me to pretty fully saturate both lines?
Like the other Martin said, this doesn't work quite how you are thinking. BGP unfortunately has only a limited range of controls one can manipulate and often you have no control over what remote ASes are doing and hence can't control the inbound traffic as much as you would like (if at all). You'll need to read up more on the BGP selection criteria and how these work but the ones to focus on for... Inbound - prefix specificity - AS path and prepending thereof - Communities. Signals to your upstreams for things like 'no export' or prepending to specific ASes - MED. This is works best with multiple links to the same provider. Outbound - filtering - local preference cheers Marty
Let's say you have two routing tables (T1, T2), one for each interface (eth0, eth1), so you can route traffic out each one separately. https://git.nic.cz/redmine/projects/bird/wiki/BGP_example_2 I'm assuming there would have to be some changes to that BGP script to support both routing tables. I'm not sure how to support multiple routing tables with bird. Would someone be able to show that? Seems like it would be a useful example in general, and be a pretty simple modification to that example for someone who knows what they are doing. I didn't see anything on any of the examples on git.nic.cz that showed using multiple tables, as a configuration like http://lartc.org/howto/lartc.rpdb.multiple-links.html shows you'd be using for multiple outgoing connections, which you would typically have in a BGP situation. Though maybe https://git.nic.cz/redmine/projects/bird/wiki/Policy_routing explains it. I plan on keep looking into that example as well. Thanks! -----Original Message----- From: "Martin Barry" <marty@supine.com> Sent: Monday, December 5, 2011 11:53am To: bird-users@atrey.karlin.mff.cuni.cz Subject: Re: Routing issues $quoted_author = "dspazman@epicup.com" ;
Would cause the path through the connection on eth0 to be 10x more prefered then the path through eth1, so incoming bandwidth should be split in about a 10-1 ratio. So if I have eth0 as a 10 Gig connection, and eth1 as a 1 Gig connection, between using that for incoming traffic, and a multipath rule for outgoing traffic like this:
ip route add default scope global nexthop via 1.1.1.153 dev eth0 weight 10 nexthop via 2.2.2.81 dev eth1 weight 1
Would cause the two connections to be pretty uniformly used on incoming and outgoing traffic at a 10/1 ratio, allowing me to pretty fully saturate both lines?
Like the other Martin said, this doesn't work quite how you are thinking. BGP unfortunately has only a limited range of controls one can manipulate and often you have no control over what remote ASes are doing and hence can't control the inbound traffic as much as you would like (if at all). You'll need to read up more on the BGP selection criteria and how these work but the ones to focus on for... Inbound - prefix specificity - AS path and prepending thereof - Communities. Signals to your upstreams for things like 'no export' or prepending to specific ASes - MED. This is works best with multiple links to the same provider. Outbound - filtering - local preference cheers Marty
On Mon, Dec 05, 2011 at 07:53:03PM -0500, dspazman@epicup.com wrote:
Let's say you have two routing tables (T1, T2), one for each interface (eth0, eth1), so you can route traffic out each one separately.
https://git.nic.cz/redmine/projects/bird/wiki/BGP_example_2
I'm assuming there would have to be some changes to that BGP script to support both routing tables. I'm not sure how to support multiple routing tables with bird. Would someone be able to show that? Seems like it would be a useful example in general, and be a pretty simple modification to that example for someone who knows what they are doing.
I didn't see anything on any of the examples on git.nic.cz that showed using multiple tables, as a configuration like http://lartc.org/howto/lartc.rpdb.multiple-links.html shows you'd be using for multiple outgoing connections, which you would typically have in a BGP situation.
Though maybe https://git.nic.cz/redmine/projects/bird/wiki/Policy_routing explains it. I plan on keep looking into that example as well.
Yes, that is an example for using multiple routing tables. You can prefer one uplink in one table and the other in the second table. You can specify that internal traffic from eth0 go through the first one and from eth1 through the second one, but: - The OSPF config (mentioned before) does not automatically split that traffic, 'cost' work in a different way. - Even if traffic from eth0 and eth1 is sufficiently splited, that will hel just split outgoing traffic on uplinks, splitting incoming is harder. -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Tue, Dec 06, 2011 at 01:58:30PM +0100, Ondrej Zajicek wrote:
- Even if traffic from eth0 and eth1 is sufficiently splited, that will hel just split outgoing traffic on uplinks, splitting incoming is harder.
also the asymetric routing is a bother because problems on one line will affect many more connections and in case one line goes down the whole asymetric traffic dies until the failure gets detected and the internet routes around int. mk
I tried modifying the example. Here is the situation. No clients, just have my own block of IPs I want to announce (A.A.A.A/22). I also have two /29s, one from each provider (Y.Y.Y.A/29, Y.Y.Y.B/29) that contains the external IP address for each of my router's two ethernet interfaces, as assigned by each ISP. The two providers I'm trying to just run so one is preferred, but both can be used. The asymetric isn't a huge deal, so if it's simpler to take it out, that's ok also. Also, besides the bird setup, I ran the following rules: ip rule add iif eth2 table 1 ip rule add iif eth5 table 2 (This is on Ubuntu, btw, and my two ISP interfaces are eth2 (my less prefered one) and eth5 (my high speed 10 Gig fiber one). The problems / questions I ran into was this. 1) Does the table 1 / table 2 need to be declared in the underlying system first? I didn't think so, like in /etc/iproute2/rt_tables? Because the IP rules seemed to work for pinging out on the interface. 2) When I looked at the routing tables, I didn't see all the global routes, so it didn't seem like I was getting them all pushed to me with that configuration. The routing table only showed my routes for my interfaces (have 6, actually, 2 ISP the other 4 internal for my network), and the /22 (and I think the /29s). Is there something wrong with how the bgp protocol sections are set up with the import all / export all? As the current way before this I was getting the full BGP tables from provider A. 3) Is the ospf part needed or helpful? Is it a good practice to have the BGP router also run OSPF, in general? -----Original Message----- From: "Ondrej Zajicek" <santiago@crfreenet.org> Sent: Tuesday, December 6, 2011 7:58am To: dspazman@epicup.com Cc: bird-users@trubka.network.cz Subject: Re: Routing issues On Mon, Dec 05, 2011 at 07:53:03PM -0500, dspazman@epicup.com wrote:
Let's say you have two routing tables (T1, T2), one for each interface (eth0, eth1), so you can route traffic out each one separately.
https://git.nic.cz/redmine/projects/bird/wiki/BGP_example_2
I'm assuming there would have to be some changes to that BGP script to support both routing tables. I'm not sure how to support multiple routing tables with bird. Would someone be able to show that? Seems like it would be a useful example in general, and be a pretty simple modification to that example for someone who knows what they are doing.
I didn't see anything on any of the examples on git.nic.cz that showed using multiple tables, as a configuration like http://lartc.org/howto/lartc.rpdb.multiple-links.html shows you'd be using for multiple outgoing connections, which you would typically have in a BGP situation.
Though maybe https://git.nic.cz/redmine/projects/bird/wiki/Policy_routing explains it. I plan on keep looking into that example as well.
Yes, that is an example for using multiple routing tables. You can prefer one uplink in one table and the other in the second table. You can specify that internal traffic from eth0 go through the first one and from eth1 through the second one, but: - The OSPF config (mentioned before) does not automatically split that traffic, 'cost' work in a different way. - Even if traffic from eth0 and eth1 is sufficiently splited, that will hel just split outgoing traffic on uplinks, splitting incoming is harder. -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Tue, Dec 06, 2011 at 10:06:30AM -0500, dspazman@epicup.com wrote:
I tried modifying the example. Here is the situation.
No clients, just have my own block of IPs I want to announce (A.A.A.A/22). I also have two /29s, one from each provider (Y.Y.Y.A/29, Y.Y.Y.B/29) that contains the external IP address for each of my router's two ethernet interfaces, as assigned by each ISP. The two providers I'm trying to just run so one is preferred, but both can be used. The asymetric isn't a huge deal, so if it's simpler to take it out, that's ok also.
In that case i would suggest to forget multiple routing table and keep it simple, like in: https://git.nic.cz/redmine/projects/bird/wiki/BGP_example_2 (or the later filtering example) (and just insert several your ASNs to AS path in export filter to less preferred uplink)
Also, besides the bird setup, I ran the following rules: ip rule add iif eth2 table 1 ip rule add iif eth5 table 2
(This is on Ubuntu, btw, and my two ISP interfaces are eth2 (my less prefered one) and eth5 (my high speed 10 Gig fiber one).
I thought you want to using multiple routing tables to route your traffic from your two internal links. There is no reason to route differently traffic received from each uplink.
The problems / questions I ran into was this.
1) Does the table 1 / table 2 need to be declared in the underlying system first? I didn't think so, like in /etc/iproute2/rt_tables? Because the IP rules seemed to work for pinging out on the interface.
No
3) Is the ospf part needed or helpful?
If you use OSPF in your network, then yes, otherwise no. But your OSPF config seems to be completely pointless. I would suggest to read some general texts about OSPF, BGP and routing.
Is it a good practice to have the BGP router also run OSPF, in general?
I think it is, unless in trivial cases. But in most cases you do not have OSPF session with your provider. -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Tue, Dec 06, 2011 at 10:06:30AM -0500, dspazman@epicup.com wrote:
protocol static static_bgp { import all;
route A.A.A.A/22 reject; route Y.Y.Y.A/29 reject; route Y.Y.Y.B/29 reject; }
BTW, you want just the A.A.A.A/22 here, no reason to put here Y.Y.Y.A/29 and Y.Y.Y.B/29. -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
participants (5)
-
Ask Bjørn Hansen -
dspazman@epicup.com -
Martin Barry -
Martin Kraus -
Ondrej Zajicek