Hello, Thanks for your advice but I think that you misunderstood our issue. We have established iBGP session between two route servers with IP address in same ethernet broadcast domain. RS1 - 10.1.0.100/24 | iBGP | RS2 - 10.1.0.200/24 Each prefix is learned correctly in both RS. 31.13.244.0/24 via 10.1.0.252 on eth0 [R0_252 14:44] * (100) [AS60230i] via 10.1.0.252 on eth0 [*R0_100* 15:12 from 10.1.0.100] (100) [AS60230i] All members in the IXP have assigned IP address from 10.1.0.0/24. The traffic should not goes to both RS. They're used only to hold the BGP sessions to the members. I just think that this message is rather a cosmetic issue then real problem but I'm not sure. Thanks again. Best~ On 03/20/2014 11:49 AM, Frédéric LOUI wrote:

Hi Javor,

I do not have the whole set-up context therefore my interpretation might be might be wrong.

When you active (i)BGP between 2 BGP speakers. You should make sure that:

1) Either all point to point links connected ([/30 | /31] for IPv4) to each (i)BGP speaker can be reachable by both system * via static * or a routing protocol such as OSPF (use the passive interface knob)

2) Either by using NEXT-HOP-SELF in your iBGP session between RS1 and RS2 (and vice versa)

So this will guarantee that your BGP NEXT HOP is reachable and that might clear your INVALID NEXT HOP issue.

Hope this help. — Frédéric

Le 20 mars 2014 à 10:20, Javor Kliachev <jkliachev@neterra.net <mailto:jkliachev@neterra.net>> a écrit :

...

Hello,

Unfortunately, I still have no response from anybody. The messages continue to come into our logs. The iBGP session is UP since more than 1 week and till this moment we don't have complaints by some member.

But this messages are definitely strange and we would be happy if someone may give some advice according to their experience.

For any questions or need more information & debug, please feel free to contact me at anytime.

I really highly appreciate any help.

Thanks in advance.

Best~

On 03/07/2014 03:35 PM, Javor Kliachev wrote:

...

Hello,

We use BIRD as route server for a long time without any problems. We have both fully symmetric RS for redundancy.

Yesterday we decided to establish "internal BGP session" between both of them. The reason to do this is to achieve prefix symmetric on both RS. Some of our members hold BGP session only with one of our route servers.

But after establishing this BGP session we began to see the following message into our logs on both RS for total random prefixes on absolutely random intervals.

Here is part of my bird.log: 07-03-2014 14:19:41 R0_29: Invalid NEXT_HOP attribute in route 178.254.232.0/21 07-03-2014 14:19:41 R0_30: Invalid NEXT_HOP attribute in route 178.254.232.0/21 07-03-2014 14:18:52 R0_52: Invalid NEXT_HOP attribute in route 79.140.156.0/22 07-03-2014 14:18:52 R0_52: Invalid NEXT_HOP attribute in route 79.140.148.0/22 07-03-2014 14:18:52 R0_52: Invalid NEXT_HOP attribute in route 79.140.144.0/20 07-03-2014 14:18:52 R0_52: Invalid NEXT_HOP attribute in route 79.140.152.0/22 07-03-2014 14:18:52 R0_52: Invalid NEXT_HOP attribute in route 79.140.144.0/22 07-03-2014 14:18:52 R0_52: Invalid NEXT_HOP attribute in route 79.140.156.0/22 07-03-2014 14:18:52 R0_52: Invalid NEXT_HOP attribute in route 79.140.144.0/20 07-03-2014 14:18:52 R0_52: Invalid NEXT_HOP attribute in route 79.140.148.0/22 07-03-2014 14:18:52 R0_52: Invalid NEXT_HOP attribute in route 79.140.152.0/22 07-03-2014 14:18:52 R0_52: Invalid NEXT_HOP attribute in route 79.140.144.0/22 07-03-2014 14:16:39 R0_29: Invalid NEXT_HOP attribute in route 178.254.232.0/21 07-03-2014 14:16:39 R0_30: Invalid NEXT_HOP attribute in route 178.254.232.0/21 07-03-2014 14:01:39 R0_52: Invalid NEXT_HOP attribute in route 95.168.66.0/23 07-03-2014 14:01:39 R0_52: Invalid NEXT_HOP attribute in route 95.168.68.0/23 07-03-2014 13:52:02 R0_14: Invalid NEXT_HOP attribute in route 151.252.198.0/24 07-03-2014 13:51:57 R0_14: Invalid NEXT_HOP attribute in route 151.252.198.0/24 07-03-2014 13:29:36 R0_87: Invalid NEXT_HOP attribute in route 85.11.144.0/20 07-03-2014 13:29:28 R0_87: Invalid NEXT_HOP attribute in route 85.11.144.0/20 07-03-2014 13:29:15 R0_87: Invalid NEXT_HOP attribute in route 85.11.144.0/20 07-03-2014 13:29:06 R0_87: Invalid NEXT_HOP attribute in route 85.11.144.0/20 07-03-2014 13:28:14 R0_87: Invalid NEXT_HOP attribute in route 85.187.120.0/22 07-03-2014 13:28:14 R0_87: Invalid NEXT_HOP attribute in route 85.187.120.0/22 07-03-2014 13:19:36 R0_87: Invalid NEXT_HOP attribute in route 85.187.120.0/22 07-03-2014 13:19:31 R0_87: Invalid NEXT_HOP attribute in route 85.187.120.0/22 07-03-2014 13:19:31 R0_87: Invalid NEXT_HOP attribute in route 85.11.144.0/20 07-03-2014 13:19:31 R0_87: Invalid NEXT_HOP attribute in route 85.187.120.0/22 07-03-2014 13:19:22 R0_87: Invalid NEXT_HOP attribute in route 85.11.144.0/20 07-03-2014 13:13:43 R0_71: Invalid NEXT_HOP attribute in route 46.229.194.0/24 07-03-2014 13:13:43 R0_71: Invalid NEXT_HOP attribute in route 46.229.192.0/23 07-03-2014 13:13:42 R0_29: Invalid NEXT_HOP attribute in route 46.229.194.0/24 07-03-2014 13:13:42 R0_29: Invalid NEXT_HOP attribute in route 46.229.192.0/23 07-03-2014 13:06:22 R0_60: Invalid NEXT_HOP attribute in route 193.104.73.0/24 07-03-2014 13:03:54 R0_52: Invalid NEXT_HOP attribute in route 178.79.61.0/24 07-03-2014 12:54:56 R0_60: Invalid NEXT_HOP attribute in route 89.35.114.0/24 07-03-2014 12:54:56 R0_60: Invalid NEXT_HOP attribute in route 86.105.150.0/24 07-03-2014 12:54:56 R0_60: Invalid NEXT_HOP attribute in route 46.102.181.0/24 07-03-2014 12:54:56 R0_60: Invalid NEXT_HOP attribute in route 46.102.176.0/24 07-03-2014 12:54:56 R0_60: Invalid NEXT_HOP attribute in route 37.156.69.0/24 07-03-2014 12:54:25 R0_60: Invalid NEXT_HOP attribute in route 188.240.70.0/24 07-03-2014 12:54:25 R0_60: Invalid NEXT_HOP attribute in route 188.212.157.0/24 07-03-2014 12:34:43 R0_52: Invalid NEXT_HOP attribute in route 91.148.106.0/24 07-03-2014 12:27:48 R0_52: Invalid NEXT_HOP attribute in route 91.148.118.0/24 07-03-2014 12:16:33 R0_29: Invalid NEXT_HOP attribute in route 178.254.232.0/21 07-03-2014 12:16:33 R0_30: Invalid NEXT_HOP attribute in route 178.254.232.0/21 07-03-2014 12:09:39 R0_87: Invalid NEXT_HOP attribute in route 85.11.144.0/20 07-03-2014 12:09:10 R0_87: Invalid NEXT_HOP attribute in route 85.11.144.0/20 07-03-2014 12:08:57 R0_87: Invalid NEXT_HOP attribute in route 85.187.120.0/22 07-03-2014 12:08:50 R0_87: Invalid NEXT_HOP attribute in route 85.187.120.0/22 07-03-2014 12:08:49 R0_87: Invalid NEXT_HOP attribute in route 85.187.120.0/22 07-03-2014 12:02:41 R0_52: Invalid NEXT_HOP attribute in route 212.200.0.0/21 07-03-2014 11:47:01 R0_29: Invalid NEXT_HOP attribute in route 178.254.232.0/21 07-03-2014 11:47:01 R0_30: Invalid NEXT_HOP attribute in route 178.254.232.0/21 07-03-2014 10:55:50 R0_60: Invalid NEXT_HOP attribute in route 91.206.209.0/24

I had checked many times and make many tests if the next-hop is correct and everything looks fine but this messages comes permanently :)

We managed to troubleshoot the following behavior and relationship and see the following effect:

Our scenario:

0.100 0.200 RS1 <- iBGP -> RS2 | | | | TEST.AS ----------- 0.252

TEST.AS announce: 31.13.244.0/24

RS2 learn this prefix as:

31.13.244.0/24 via 10.1.0.252 on eth0 [R0_252 14:44] * (100) [AS60230i] via 10.1.0.252 on eth0 [*R0_100* 15:12 from 10.1.0.100] (100) [AS60230i]

As soon as we stop to announce 31.13.244.0/24 from TEST.AS to RS1, in the logs of RS1 begin to come the following message:

07-03-2014 15:10:09R0_252: Invalid NEXT_HOP attribute in route 31.13.244.0/24

But when we try to shutdown (administratively ) the session between TEST.AS and RS1 this behavior does not occur and everything seems normal.

I hope this information will be useful for resolving this strange issue.

Here is our configuration set on both RS. The real AS number & IPs has been changed for security reason :)

# RS2: table T65535

protocol pipe P65535 from iBGP_PIPES { description "RS1"; peer table T65535; export where RS_PIPE_OUT(); }

protocol bgp R0_100 from iBGP { description "0.100_iBGP_RS1"; source address 10.1.0.200; neighbor 10.1.0.100 as 65535; import all; export all; passive off; table T65535; route limit 10000; gateway direct; }

# RS1: table T65535

protocol pipe P65535 from iBGP_PIPES { description "RS2"; peer table T65535; export where RS_PIPE_OUT(); }

protocol bgp R0_200 from iBGP { description "0.200_iBGP_RS1"; source address 10.1.0.100; neighbor 10.1.0.200 as 65535; import all; export all; passive off; table T65535; route limit 10000; gateway direct; }

# show bgp summary shows the following: # RS2: ---------------------------------------------------------------- adv / rcv / limit --- 0.100_iBGP_RS1 10.1.0.100 65535 Mar06 Established 4527/3235/10000 --------------------------------------------------------------------------------------

# RS1: ---------------------------------------------------------------- adv / rcv / limit --- 0.200_iBGP_RS2 10.1.0.200 65535 Mar06 Established 3235/4527/10000 --------------------------------------------------------------------------------------

Any ideas or thoughts are highly appreciated!

Thanks in advance!

-- --- Find out about our new Cloud service - Cloudware.bg <http://cloudware.bg/?utm_source=email&utm_medium=signature&utm_content=link&utm_campaign=newwebsite> Access anywhere. Manage it yourself. Pay as you go. ------------------------------------------------------------------------ *Javor Kliachev* IP Engineer

Neterra Ltd. Telephone: +359 2 975 16 16 Fax: +359 2 975 34 36 www.neterra.net <http://www.neterra.net/>

-- --- Find out about our new Cloud service - Cloudware.bg <http://cloudware.bg/?utm_source=email&utm_medium=signature&utm_content=link&utm_campaign=newwebsite> Access anywhere. Manage it yourself. Pay as you go. ------------------------------------------------------------------------ *Javor Kliachev* IP Engineer

Neterra Ltd. Telephone: +359 2 975 16 16 Fax: +359 2 975 34 36 www.neterra.net <http://www.neterra.net/>

-- --- Find out about our new Cloud service - Cloudware.bg <http://cloudware.bg/?utm_source=email&utm_medium=signature&utm_content=link&utm_campaign=newwebsite> Access anywhere. Manage it yourself. Pay as you go. ------------------------------------------------------------------------ *Javor Kliachev* IP Engineer Neterra Ltd. Telephone: +359 2 975 16 16 Fax: +359 2 975 34 36 www.neterra.net <http://www.neterra.net>

Hello, Thanks for your detailed explanation. Today I tried the new version of BIRD with the current one setup with add path rx & rx but I'm observing the following things: 1) With adding: add paths rx|tx, at first time it looks works fine. Already more than one route path is advertised via iBGP session. All members learned properly it. 2) Till this moment I used the option "gateway direct" but with the new version I see the following output to return to CLI: root@bird-test-RS1:~# birdc configure BIRD 1.4.2 ready. Reading configuration from /usr/local/bird-patched/etc/bird.conf */usr/local/bird-patched/etc/AS65535, line 21: Multihop BGP cannot use direct gateway mode* Is this no longer supported with this combination or I trying to do something wrong. Here is my current config: /table T65535// // //protocol pipe P65535 from iBGP_PIPES {// // description "RS2";// // peer table T65535;// // export where RS_PIPE_OUT();// //}// // //protocol bgp R0_200 from iBGP {// // description "0.200_iBGP_RS1";// // source address 10.0.0.100;// // neighbor 10.0.0.200 as 65535;// // import all;// // export all;// // passive off;// // table T65535;// // route limit 10000;// // add paths tx;// // add paths rx;// //# gateway direct;// //}// / 3) With the new version over established iBGP session between two RS all prefixes comes with status "unreachable" but nevertheless, the routes are distributed properly to the other members RIBs. This is strange. Here is output about given IPv4 prefix: root@bird-test-RS1:~# birdc show route table T65535 78.130.178.0/24 all BIRD 1.4.2 ready. 78.130.178.0/24 *unreachable* [R0_200 12:02:41 from 10.0.0.200] * (100/-) [AS29030i] Type: BGP unicast univ BGP.origin: IGP BGP.as_path: 29030 BGP.next_hop: 10.0.0.22 BGP.med: 1000 BGP.local_pref: 100 BGP.community: (1,1) (1,1023) (64700,29030) (65400,0) (65400,65400) *unreachable* [R0_200 12:02:41 from 10.0.0.200] (100/-) [AS45007i] Type: BGP unicast univ BGP.origin: IGP BGP.as_path: 45007 BGP.next_hop: 10.0.0.21 BGP.med: 1000 BGP.local_pref: 100 BGP.community: (1,1) (1,1087) (64700,45007) (65400,0) (65400,65400) I would like to know if that is normal behavior? Thanks in advance! Best~ On 03/25/2014 11:57 PM, Ondrej Zajicek wrote:

On Thu, Mar 20, 2014 at 11:20:20AM +0200, Javor Kliachev wrote:

...

Hello,

Unfortunately, I still have no response from anybody. The messages continue to come into our logs. The iBGP session is UP since more than 1 week and till this moment we don't have complaints by some member.

...

Here is part of my bird.log: 07-03-2014 14:19:41 R0_29: Invalid NEXT_HOP attribute in route This message means that BIRD tries to send a route with NEXT_HOP attribute X to a neighbor with IP address X.

That happens because both iBGP and RS eBGP do not change NEXT_HOP, so if you have neighbor N connected to both:

RS1 --- RS2 \ / \ / \ / N

And it propagates the same route to both RS1 and RS2, then you have the same route both times in RS1 and RS2. Usually the directly received is prefered, but if for some reason the one received through IBGP is preferred (perhaps N sends a withdraw, which is propagated first to RS1 so directly received route disappears but the one received from RS2 by iBGP is still in RS1 for a moment), then it is scheduled for propagation back to N, but the check for NEXT_HOP denies it.

I am not sure now whether this is some non-standard situation or the check should be silent. Generally i would advise against connecting route servers through IBGP. You get most routes two times, therefore double memory requirements. Another reason is that IBGP (unless with ADD-PATH extension) will silently eliminate non-preferred routes, which would cause hidden unexpected problems.

-- --- Find out about our new Cloud service - Cloudware.bg <http://cloudware.bg/?utm_source=email&utm_medium=signature&utm_content=link&utm_campaign=newwebsite> Access anywhere. Manage it yourself. Pay as you go. ------------------------------------------------------------------------ *Javor Kliachev* IP Engineer Neterra Ltd. Telephone: +359 2 975 16 16 Fax: +359 2 975 34 36 www.neterra.net <http://www.neterra.net>

Hello, Many thanks for your working solution!!! As per your advise I've changed it and now everything is fine with the prefixes learned via iBGP. root@bird-test-RS1:~# birdc show route table T65535 78.130.178.0/24 all BIRD 1.4.2 ready. 78.130.178.0/24 via 10.0.0.22 on eth1 [R0_200 15:35:14 from 10.0.0.200] * (100) [AS29030i] Type: BGP unicast univ BGP.origin: IGP BGP.as_path: 29030 BGP.next_hop: 10.0.0.22 BGP.med: 1000 BGP.local_pref: 100 BGP.community: (1,1) (1,1023) (64700,29030) (65400,0) (65400,65400) via 10.0.0.21 on eth1 [R0_200 15:35:14 from 10.0.0.200] (100) [AS45007i] Type: BGP unicast univ BGP.origin: IGP BGP.as_path: 45007 BGP.next_hop: 10.0.0.21 BGP.med: 1000 BGP.local_pref: 100 BGP.community: (1,1) (1,1087) (64700,45007) (65400,0) (65400,65400) The prefix now is with status "*reachable" :) * Best~ On 04/08/2014 03:28 PM, MrBr @ GMail wrote:

Hi Javor Had same problem Use "direct" instead of "gateway direct""

On Tue, Apr 8, 2014 at 12:16 PM, Javor Kliachev <jkliachev@neterra.net <mailto:jkliachev@neterra.net>> wrote:

Hello,

Thanks for your detailed explanation.

Today I tried the new version of BIRD with the current one setup with add path rx & rx but I'm observing the following things:

1) With adding: add paths rx|tx, at first time it looks works fine. Already more than one route path is advertised via iBGP session. All members learned properly it.

2) Till this moment I used the option "gateway direct" but with the new version I see the following output to return to CLI:

root@bird-test-RS1:~# birdc configure BIRD 1.4.2 ready. Reading configuration from /usr/local/bird-patched/etc/bird.conf */usr/local/bird-patched/etc/AS65535, line 21: Multihop BGP cannot use direct gateway mode*

Is this no longer supported with this combination or I trying to do something wrong.

Here is my current config:

/table T65535// // //protocol pipe P65535 from iBGP_PIPES {// // description "RS2";// // peer table T65535;// // export where RS_PIPE_OUT();// //}// // //protocol bgp R0_200 from iBGP {// // description "0.200_iBGP_RS1";// / / source address 10.0.0.100;// // neighbor 10.0.0.200 as 65535;/ / // import all;// // export all;// // passive off;// // table T65535;// // route limit 10000;// / / add paths tx;// // add paths rx;// //# gateway direct;// //}// /

3) With the new version over established iBGP session between two RS all prefixes comes with status "unreachable" but nevertheless, the routes are distributed properly to the other members RIBs. This is strange.

Here is output about given IPv4 prefix:

root@bird-test-RS1:~# birdc show route table T65535 78.130.178.0/24 <http://78.130.178.0/24> all BIRD 1.4.2 ready. 78.130.178.0/24 <http://78.130.178.0/24> *unreachable* [R0_200 12:02:41 from 10.0.0.200] * (100/-) [AS29030i] Type: BGP unicast univ BGP.origin: IGP BGP.as_path: 29030 BGP.next_hop: 10.0.0.22 BGP.med: 1000 BGP.local_pref: 100 BGP.community: (1,1) (1,1023) (64700,29030) (65400,0) (65400,65400) *unreachable* [R0_200 12:02:41 from 10.0.0.200] (100/-) [AS45007i] Type: BGP unicast univ BGP.origin: IGP BGP.as_path: 45007 BGP.next_hop: 10.0.0.21 BGP.med: 1000 BGP.local_pref: 100 BGP.community: (1,1) (1,1087) (64700,45007) (65400,0) (65400,65400)

I would like to know if that is normal behavior?

Thanks in advance!

Best~

On 03/25/2014 11:57 PM, Ondrej Zajicek wrote:

...

On Thu, Mar 20, 2014 at 11:20:20AM +0200, Javor Kliachev wrote:

...

Hello,

Unfortunately, I still have no response from anybody. The messages continue to come into our logs. The iBGP session is UP since more than 1 week and till this moment we don't have complaints by some member.

...

Here is part of my bird.log: 07-03-2014 14:19:41 R0_29: Invalid NEXT_HOP attribute in route

This message means that BIRD tries to send a route with NEXT_HOP attribute X to a neighbor with IP address X.

That happens because both iBGP and RS eBGP do not change NEXT_HOP, so if you have neighbor N connected to both:

RS1 --- RS2 \ / \ / \ / N

And it propagates the same route to both RS1 and RS2, then you have the same route both times in RS1 and RS2. Usually the directly received is prefered, but if for some reason the one received through IBGP is preferred (perhaps N sends a withdraw, which is propagated first to RS1 so directly received route disappears but the one received from RS2 by iBGP is still in RS1 for a moment), then it is scheduled for propagation back to N, but the check for NEXT_HOP denies it.

I am not sure now whether this is some non-standard situation or the check should be silent. Generally i would advise against connecting route servers through IBGP. You get most routes two times, therefore double memory requirements. Another reason is that IBGP (unless with ADD-PATH extension) will silently eliminate non-preferred routes, which would cause hidden unexpected problems.

-- --- Find out about our new Cloud service - Cloudware.bg <http://cloudware.bg/?utm_source=email&utm_medium=signature&utm_content=link&utm_campaign=newwebsite> Access anywhere. Manage it yourself. Pay as you go. ------------------------------------------------------------------------ *Javor Kliachev* IP Engineer

Neterra Ltd. Telephone: +359 2 975 16 16 <tel:%2B359%202%20975%2016%2016> Fax: +359 2 975 34 36 <tel:%2B359%202%20975%2034%2036> www.neterra.net <http://www.neterra.net>

-- --- Find out about our new Cloud service - Cloudware.bg <http://cloudware.bg/?utm_source=email&utm_medium=signature&utm_content=link&utm_campaign=newwebsite> Access anywhere. Manage it yourself. Pay as you go. ------------------------------------------------------------------------ *Javor Kliachev* IP Engineer Neterra Ltd. Telephone: +359 2 975 16 16 Fax: +359 2 975 34 36 www.neterra.net <http://www.neterra.net>