<div dir="ltr">Hi,<div><br></div><div>I have a setup that looks like this:</div><div><br></div><div> upstream1 upstream2 upstream3 upstream4</div><div> | | | |</div><div><pre class="gmail-result--content--pre" style="margin-top:0px;margin-bottom:0px;padding:0px;overflow:hidden;line-height:1.2em;color:rgb(0,0,0)"> +------------------+ +-----------------+
| bgp1 rr | | bgp2 rr |
| ebgp to upstreams |ebgp to upstreams|
| ibgp to bgp2 | |ibgp to bgp1 |
+------------------+ +-----------------+
| |
+-----------+ +-----------+
| location a| ------------- | location b|
+---switch--+ +---switch--+
| |
| |
+-------------------+ +-----------------------+
| pppoe serv 1 | | pppoe serv 2 |
| rr client of bgp1 | | rr client of bgp1 |
| rr client of bgp2 | | rr client of bgp2 |
- +-------------------+ +-----------------------+
</pre></div><div> </div><div>BGP1 and BGP2 are ebgp peering with upstream providers.</div><div>BGP1 and BGP2 are ibgp peering between themselves to exchange routes and act as route reflectors towards PPPOE SERV 1 and PPPOE SERV 2.</div><div>There is an area 0 in ospf with broadcast between BGP1/BGP2/PPPOE SERV1/SERV2.</div><div>On PPPOE SERV1/SERV2 Bird is used to redistribute pppoe /32 addresses of clients.</div><div>I put everything in the single routing table but mark prefixes with communites to know where they come from and to filter based on that.</div><div><br></div><div>By default PPPOE SERV1/2 receives only a default route from both BGP1 and BGP2.</div><div><br></div><div>With such multipath default route, a packet on PPPOE SERV1 destined for UPSTREAM 1 can end up at BGP2 (depends how it is hashed) where it is routed then to BGP1. </div><div><br></div><div>I wanted to optimize this by feeding routes from BGP1/BGP2 to PPPOE SERV1/ SERV2. Now such unnecessary jumping back and forth between BGP1/BGP2 of packets from PPPOE SERVERS would be kept to a minimum.</div><div><br></div><div>However with the following setup, where I now feed /24 and lower routes from a combination of the upstream providers to PPPOE Servers (about 800k routes) I face performance issues that causes a mild packet loss on the PPPOE Servers (which is not present if only default routes are used).</div><div><br></div><div>To give an example:</div><div><br></div><div>PPPOE SERV1 has about 2000 PPPOE interfaces of the form:</div><div><br></div><div>1535: subpppoe.1423: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc htb state UNKNOWN mode DEFAULT group default qlen 3<br> link/ppp<br></div><div><br></div><div>and is using ACCEL-PPP software for the PPPOE client termination.</div><div><br></div><div>The server has an 8 core cpu.</div><div><br></div><div>The network card has 6 RSS queues that are pinned to CPU 0,1,2,3,4,5,7 via /proc/irq/XXX/snmp_affinity_list.</div><div><br></div><div>Bird is pinned to the last unused CPU via: taskset -cp 8 $(pgrep bird)</div><div><br></div><div>The ACCEL-PPP termination software is using 6 threads internally and I have pinned the entire process to the first 7 CPUs: taskset -cp 0-7 $(pgrep accel-pppd)</div><div><br></div><div>In essence Bird should have its own CPU to perform route calculations without affecting any packet routing.</div><div><br></div><div>When the 800k routes are fed to the server from the BGP route reflectors, on CPU 8; Bird is periodically jumping to 100% cpu usage for a brief moment (that is expected, calculating routes), but at the same time, the ACCEL-PPPD daemon process is also jumping to very high values (about 80%ish).<br></div><div><br></div><div>This causes a brief packet loss on the PPPOE client interfaces.</div><div><br></div><div>I wonder, is there anything I can do to prevent this from happening or is this a combination of route calculation/interface scanning that causes some sort of brief lockup in processing on the PPPOE daemon?</div><div><br></div><div>By sticking to default routes on the PPPOE SERVERs, Bird does ocassionally jump to a high CPU usage, but the PPPOE daemon doesn't and I can't detect any packet loss on the PPPOE Servers.</div><div><br></div><div>Any suggestion appreciated. </div><div><br></div><div>Thank you.</div><div><br></div><div>-- I have included some sample Bird configs from the machines:</div><div><br></div><div>BGP1 to BGP2 config sample:</div><div><br></div><div>protocol static myas_routes {<br> ipv4;<br> route 1.2.3.4.0/24 blackhole { preference = 1; };<br> route 4.3.2.1.0/24 blackhole { preference = 1; };</div><div>}</div><div>protocol bgp ibgp_from_bgp1_to_bgp2 {<br> local as 1111;<br> neighbor <loopback ip of bgp2> as 1111;<br> source address <loopback ip of bgp1>;<br> rr cluster id 0.0.0.1;<br><br> ipv4 {<br> next hop self;<br> export filter { /* exports only RTS_BGP routes and "myas_routes" object */ }<br> import filter { /* imports all but sets a lower preference to routes that originate from BGP2's rr clients */ }<br> };<br>}<br></div><div><br></div><div>Sample config of BGP1 to PPPOE SERV1:</div><div><br></div><div>protocol bgp rr_bgp1_to_pppoe_serv1 {<br> disabled no;<br> local as 1111;<br> neighbor <loopback ip of pppoe serv 1> as 1111;<br> source address <loopback ip of bgp1> ;<br> rr client;<br> rr cluster id 0.0.0.1;<br> ipv4 {<br> next hop self ebgp;<br> export filter { <exports /24 and lower routes from ebgp upstream 1,2,3,4> };<br> import filter { <imports /32 pppoe client addresses from pppoe serv1> };<br> };<br> };<br>}<br></div><div><br></div><div>Sample config of PPPOE SERV1 to BGP1:</div><div><br></div><div>protocol kernel {<br> merge paths yes;<br> learn yes;<br> ipv4 {<br> import all;<br> export all;<br> };<br>}<br><br>protocol device { <br>}<br><br>protocol direct {<br> ipv4; <br>}<br></div><div><br></div><div>protocol bgp rr_pppoe_serv1_to_bgp1 {<br> local as 1111;<br> neighbor <loopback ip of bgp1> as 1111;<br> source address <loopback ip of pppoe serv 1>;<br><br> ipv4 {<br> next hop self;<br> export filter {</div><div> if source = RTS_DEVICE || source = RTS_STATIC_DEVICE || source = RTS_STATIC || source = RTS_INHERIT then {<br> accept;<br> }<br> reject;<br> };<br> import all;<br> };<br>}<br></div><div><br></div><div><br></div></div>