BIRD on physical / virtual server.
Hi, what is your experience with installing bird on virtual server (ESXi)? Is there any limitation of this kind of deployment for example problems with performance etc.? Regards, Mike
Hi Mike, Make sure you use paravirtualised network adapters and your hypervisor isn't overcommitted too much. And... good metrics. Carefully monitor iowait, load, memory and such in the VM and adjust specs as needed. Besides that you're fine. K. On 08-07-2020 08:05, Mike Neo wrote:
what is your experience with installing bird on virtual server (ESXi)? Is there any limitation of this kind of deployment for example problems with performance etc.?
Hi, There is a known issue with with Open vSwitch (OVS) performance with BGP. Even without OVS we hit IRQ (too many) issue on "physical" network in our KVM environment. Official OVS quote:
We'd accept patches to improve OVS's routing table code. It's not designed to scale to 1,800,000 routes. We'd also take code to suppress the routing table code in cases where it isn't actually needed, since it's not always needed. But we can't take a patch to just delete it; I'm sure you understand. I tried to apply this patch at that time, but was already useless for newer versions:
https://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20161123/5379... <https://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20161123/5379b333/attachment.bin> Our workaround was to scale VM with 3 vCPU-s, since our average system load is 1.5 for BGP. You can see what is happening: [root@bgp1 ~]# top ... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 654 root 10 -10 1284492 1.0g 20276 R 98.0 27.0 2513:01 ovs-vswitchd 16 root 20 0 0 0 0 S 2.0 0.0 24:45.60 ksoftirqd/1 [root@bgp1 ~]# ip route show ... 1.0.0.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird 1.0.4.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird 1.0.4.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird 1.0.5.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird Routes being constantly added and deleted: [root@bgp1 ~]# ip monitor ... Deleted 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium Deleted 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium Deleted 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium Deleted 68.69.37.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird 68.69.37.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 2.16.70.0/23 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 88.221.28.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 23.50.188.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 92.122.68.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 88.221.100.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 92.123.208.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird ..... Regards, saso
On 8 Jul 2020, at 08:05, Mike Neo <neomikemac@gmail.com> wrote:
Hi,
what is your experience with installing bird on virtual server (ESXi)? Is there any limitation of this kind of deployment for example problems with performance etc.?
Regards, Mike
Hi Mike, Saso is correct; we bumped into that one too. However, as long as you're running BIRD in a VM and OVS is outside the VM there's no issue. K. On 08-07-2020 08:58, Saso Tavcar wrote:
Hi,
There is a known issue with with Open vSwitch (OVS) performance with BGP. Even without OVS we hit IRQ (too many) issue on "physical" network in our KVM environment.
Official OVS quote:
/We'd accept patches to improve OVS's routing table code. It's not />/designed to scale to 1,800,000 routes. We'd also take code to suppress />/the routing table code in cases where it isn't actually needed, since />/it's not always needed. But we can't take a patch to just delete it; />/I'm sure you understand./ I tried to apply this patch at that time, but was already useless for newer versions:
_https://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20161123/5379...
Our workaround was to scale VM with 3 vCPU-s, since our average system load is 1.5 for BGP.
You can see what is happening:
[root@bgp1 ~]# top ... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND * 654 root 10 -10 1284492 1.0g 20276 R 98.0 27.0 2513:01 ovs-vswitchd * 16 root 20 0 0 0 0 S 2.0 0.0 24:45.60 ksoftirqd/1
[root@bgp1 ~]# ip route show ... 1.0.0.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird 1.0.4.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird 1.0.4.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird 1.0.5.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird
Routes being constantly added and deleted:
[root@bgp1 ~]# ip monitor ... Deleted 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium Deleted 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium Deleted 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium Deleted 68.69.37.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird 68.69.37.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 2.16.70.0/23 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 88.221.28.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 23.50.188.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 92.122.68.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 88.221.100.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 92.123.208.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird .....
Regards, saso
On 8 Jul 2020, at 08:05, Mike Neo <neomikemac@gmail.com <mailto:neomikemac@gmail.com>> wrote:
Hi,
what is your experience with installing bird on virtual server (ESXi)? Is there any limitation of this kind of deployment for example problems with performance etc.?
Regards, Mike
Great, thanks! śr., 8 lip 2020 o 09:19 Kees Meijs <kees@nefos.nl> napisał(a):
Hi Mike,
Saso is correct; we bumped into that one too.
However, as long as you're running BIRD in a VM and OVS is outside the VM there's no issue.
K.
On 08-07-2020 08:58, Saso Tavcar wrote:
Hi,
There is a known issue with with Open vSwitch (OVS) performance with BGP. Even without OVS we hit IRQ (too many) issue on "physical" network in our KVM environment.
Official OVS quote:
* We'd accept patches to improve OVS's routing table code. It's not *>* designed to scale to 1,800,000 routes. We'd also take code to suppress *>* the routing table code in cases where it isn't actually needed, since *>* it's not always needed. But we can't take a patch to just delete it; *>* I'm sure you understand.*
I tried to apply this patch at that time, but was already useless for newer versions:
*https://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20161123/5379... <https://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20161123/5379b333/attachment.bin>*
Our workaround was to scale VM with 3 vCPU-s, since our average system load is 1.5 for BGP.
You can see what is happening:
[root@bgp1 ~]# top ... PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
* 654 root 10 -10 1284492 1.0g 20276 R 98.0 27.0 2513:01 ovs-vswitchd * 16 root 20 0 0 0 0 S 2.0 0.0 24:45.60 ksoftirqd/1
[root@bgp1 ~]# ip route show ... 1.0.0.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird 1.0.4.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird 1.0.4.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird 1.0.5.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird
Routes being constantly added and deleted:
[root@bgp1 ~]# ip monitor ... Deleted 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium Deleted 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium Deleted 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium 2620:11d:6000::/42 via 2a01:xxx:xxx::1 dev t2-v26-ha proto bird metric 1024 pref medium Deleted 68.69.37.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird 68.69.37.0/24 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird 103.115.180.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 2.16.70.0/23 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 88.221.28.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 23.50.188.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 92.122.68.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 88.221.100.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird Deleted 92.123.208.0/22 via 89.212.xx.xx dev t2-v24-ha proto bird .....
Regards, saso
On 8 Jul 2020, at 08:05, Mike Neo <neomikemac@gmail.com> wrote:
Hi,
what is your experience with installing bird on virtual server (ESXi)? Is there any limitation of this kind of deployment for example problems with performance etc.?
Regards, Mike
participants (3)
-
Kees Meijs -
Mike Neo -
Saso Tavcar