Bird performance issue with many PPPOE interfaces and a large routing table.

5 Jul 2022

      Hi,

I have a setup that looks like this:

                    upstream1      upstream2             upstream3
 upstream4
                       |                       |
 |                      |

           +------------------+         +-----------------+
           |      bgp1 rr     |         |    bgp2 rr      |
           |  ebgp to upstreams         |ebgp to upstreams|
           |  ibgp to bgp2    |         |ibgp to bgp1     |
           +------------------+         +-----------------+
                   |                            |
              +-----------+                +-----------+
              | location a| -------------  | location b|
              +---switch--+                +---switch--+
                    |                            |
                    |                            |
              +-------------------+      +-----------------------+
              | pppoe serv 1      |      |    pppoe serv 2       |
              | rr client of bgp1 |      |    rr client of bgp1  |
              | rr client of bgp2 |      |    rr client of bgp2  |
-             +-------------------+      +-----------------------+

BGP1 and BGP2 are ebgp peering with upstream providers.
BGP1 and BGP2 are ibgp peering between themselves to exchange routes and
act as route reflectors towards PPPOE SERV 1 and PPPOE SERV 2.
There is an area 0 in ospf with broadcast between BGP1/BGP2/PPPOE
SERV1/SERV2.
On PPPOE SERV1/SERV2 Bird is used to redistribute pppoe /32 addresses of
clients.
I put everything in the single routing table but mark prefixes with
communites to know where they come from and to filter based on that.

By default PPPOE SERV1/2 receives only a default route from both BGP1 and
BGP2.

With such multipath default route, a packet on PPPOE SERV1 destined for
UPSTREAM 1 can end up at BGP2 (depends how it is hashed) where it is routed
then to BGP1.

I wanted to optimize this by feeding routes from BGP1/BGP2 to PPPOE SERV1/
SERV2. Now such unnecessary jumping back and forth between BGP1/BGP2 of
packets from PPPOE SERVERS would be kept to a minimum.

However with the following setup, where I now feed /24 and lower routes
from a combination of the upstream providers to PPPOE Servers (about 800k
routes) I face performance issues that causes a mild packet loss on the
PPPOE Servers (which is not present if only default routes are used).

To give an example:

PPPOE SERV1 has about 2000 PPPOE interfaces of the form:

1535: subpppoe.1423: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492
qdisc htb state UNKNOWN mode DEFAULT group default qlen 3
    link/ppp

and is using ACCEL-PPP software for the PPPOE client termination.

The server has an 8 core cpu.

The network card has 6 RSS queues that are pinned to CPU 0,1,2,3,4,5,7 via
/proc/irq/XXX/snmp_affinity_list.

Bird is pinned to the last unused CPU via: taskset -cp 8 $(pgrep bird)

The ACCEL-PPP termination software is using 6 threads internally and I have
pinned the entire process to the first 7 CPUs: taskset -cp 0-7 $(pgrep
accel-pppd)

In essence Bird should have its own CPU to perform route calculations
without affecting any packet routing.

When the 800k routes are fed to the server from the BGP route reflectors,
on CPU 8; Bird is periodically jumping to 100% cpu usage for a brief moment
(that is expected, calculating routes), but at the same time, the
ACCEL-PPPD daemon process is also jumping to very high values (about
80%ish).

This causes a brief packet loss on the PPPOE client interfaces.

I wonder, is there anything I can do to prevent this from happening or is
this a combination of route calculation/interface scanning that causes some
sort of brief lockup in processing on the PPPOE daemon?

By sticking to default routes on the PPPOE SERVERs, Bird does
ocassionally jump to a high CPU usage, but the PPPOE daemon doesn't and I
can't detect any packet loss on the PPPOE Servers.

Any suggestion appreciated.

Thank you.

-- I have included some sample Bird configs from the machines:

BGP1 to BGP2 config sample:

protocol static myas_routes {
        ipv4;
        route 1.2.3.4.0/24 blackhole { preference = 1; };
        route 4.3.2.1.0/24 blackhole { preference = 1; };
}
protocol bgp ibgp_from_bgp1_to_bgp2 {
        local as 1111;
        neighbor <loopback ip of bgp2> as 1111;
        source address <loopback ip of bgp1>;
        rr cluster id 0.0.0.1;

        ipv4 {
            next hop self;
            export filter { /* exports only RTS_BGP routes  and
"myas_routes" object */ }
            import filter { /* imports all but sets a lower preference to
routes that originate from BGP2's rr clients */ }
        };
}

Sample config of BGP1 to PPPOE SERV1:

protocol bgp rr_bgp1_to_pppoe_serv1 {
        disabled no;
        local as 1111;
        neighbor <loopback ip of pppoe serv 1> as 1111;
        source address <loopback ip of bgp1> ;
        rr client;
        rr cluster id 0.0.0.1;
        ipv4 {
                next hop self ebgp;
                export filter { <exports /24 and lower routes from ebgp
upstream 1,2,3,4> };
                import filter { <imports /32 pppoe client addresses from
pppoe serv1> };
           };
        };
}

Sample config of PPPOE SERV1 to BGP1:

protocol kernel {
    merge paths yes;
    learn yes;
    ipv4 {
        import all;
        export all;
    };
}

protocol device {
}

protocol direct {
    ipv4;
}

protocol bgp rr_pppoe_serv1_to_bgp1 {
    local as 1111;
    neighbor <loopback ip of bgp1> as 1111;
    source address <loopback ip of pppoe serv 1>;

    ipv4 {
        next hop self;
        export filter {
            if source = RTS_DEVICE || source = RTS_STATIC_DEVICE || source
= RTS_STATIC || source = RTS_INHERIT then {
                accept;
            }
            reject;
        };
        import all;
    };
}

Bird performance issue with many PPPOE interfaces and a large routing table.

LU