[3.1.1 on EL8] CPU 100% with NetworkManager
Hi there, When I upgrade bird from 2.16.2 to 3.1.1 today, it became unstable that it often causes CPU usage to 100% for bird, and later also for NetworkManager, and there are some notable syslogs: 6月 02 16:04:28 zx-router bird[2197971]: I/O loop cycle took 14046.255 ms for 43 events 6月 02 16:04:48 zx-router bird[2197971]: I/O loop cycle took 15307.277 ms for 42 events 6月 02 16:04:57 zx-router bird[2197971]: I/O loop cycle took 8562.921 ms for 43 events 6月 02 16:05:16 zx-router bird[2197971]: I/O loop cycle took 9799.271 ms for 85 events 6月 02 16:05:25 zx-router bird[2197971]: I/O loop cycle took 9330.265 ms for 80 events 6月 02 16:05:30 zx-router bird[2197971]: I/O loop cycle took 5004.867 ms for 44 events 6月 02 16:06:09 zx-router bird[2197971]: I/O loop cycle took 38759.438 ms for 84 events 6月 02 16:06:17 zx-router bird[2197971]: I/O loop cycle took 5465.396 ms for 6 events 6月 02 16:06:23 zx-router bird[2197971]: I/O loop cycle took 5906.858 ms for 44 events 6月 02 16:06:37 zx-router bird[2197971]: I/O loop cycle took 12896.032 ms for 9 events 6月 02 16:06:56 zx-router bird[2197971]: I/O loop cycle took 8217.969 ms for 81 events 6月 02 16:07:08 zx-router bird[2197971]: I/O loop cycle took 9012.775 ms for 25 events 6月 02 16:07:26 zx-router bird[2197971]: I/O loop cycle took 11506.570 ms for 45 events 6月 02 16:07:56 zx-router bird[2197971]: I/O loop cycle took 30014.580 ms for 85 events 6月 02 16:08:05 zx-router bird[2197971]: I/O loop cycle took 6333.289 ms for 6 events 6月 02 16:08:14 zx-router bird[2197971]: I/O loop cycle took 8751.816 ms for 42 events 6月 02 16:08:29 zx-router bird[2197971]: I/O loop cycle took 13017.064 ms for 46 events 6月 02 16:08:51 zx-router bird[2197971]: I/O loop cycle took 13223.357 ms for 80 events 6月 02 16:09:44 zx-router bird[2197971]: I/O loop cycle took 17730.328 ms for 5 events 6月 02 16:10:37 zx-router bird[2197971]: I/O loop cycle took 8061.950 ms for 8 events 6月 02 16:10:46 zx-router bird[2197971]: I/O loop cycle took 9008.065 ms for 65 events 6月 02 16:10:54 zx-router bird[2197971]: I/O loop cycle took 6035.965 ms for 4 events 6月 02 16:11:03 zx-router bird[2197971]: I/O loop cycle took 9959.243 ms for 67 events I'm exporting full table to kernel with NetworkManager running on RHEL8 related systems (CentOS/Alma/Rocky 8). There is no problem before upgrading.
bird 3.1.2 seems to be far better, but still about half minutes high CPU usage: 6月 02 17:27:52 zx-router bird[2232668]: I/O loop cycle took 7108.812 ms for 26 events 6月 02 17:28:11 zx-router bird[2232668]: I/O loop cycle took 5419.657 ms for 15 events 6月 02 17:28:17 zx-router bird[2232668]: I/O loop cycle took 5445.657 ms for 80 events it's still worse comparing with 2.16.2. Zhang Xun 发件人: zx@zxinc.org 发送时间: 2025-06-02 17:11 收件人: bird-users 主题: [3.1.1 on EL8] CPU 100% with NetworkManager Hi there, When I upgrade bird from 2.16.2 to 3.1.1 today, it became unstable that it often causes CPU usage to 100% for bird, and later also for NetworkManager, and there are some notable syslogs: 6月 02 16:04:28 zx-router bird[2197971]: I/O loop cycle took 14046.255 ms for 43 events 6月 02 16:04:48 zx-router bird[2197971]: I/O loop cycle took 15307.277 ms for 42 events 6月 02 16:04:57 zx-router bird[2197971]: I/O loop cycle took 8562.921 ms for 43 events 6月 02 16:05:16 zx-router bird[2197971]: I/O loop cycle took 9799.271 ms for 85 events 6月 02 16:05:25 zx-router bird[2197971]: I/O loop cycle took 9330.265 ms for 80 events 6月 02 16:05:30 zx-router bird[2197971]: I/O loop cycle took 5004.867 ms for 44 events 6月 02 16:06:09 zx-router bird[2197971]: I/O loop cycle took 38759.438 ms for 84 events 6月 02 16:06:17 zx-router bird[2197971]: I/O loop cycle took 5465.396 ms for 6 events 6月 02 16:06:23 zx-router bird[2197971]: I/O loop cycle took 5906.858 ms for 44 events 6月 02 16:06:37 zx-router bird[2197971]: I/O loop cycle took 12896.032 ms for 9 events 6月 02 16:06:56 zx-router bird[2197971]: I/O loop cycle took 8217.969 ms for 81 events 6月 02 16:07:08 zx-router bird[2197971]: I/O loop cycle took 9012.775 ms for 25 events 6月 02 16:07:26 zx-router bird[2197971]: I/O loop cycle took 11506.570 ms for 45 events 6月 02 16:07:56 zx-router bird[2197971]: I/O loop cycle took 30014.580 ms for 85 events 6月 02 16:08:05 zx-router bird[2197971]: I/O loop cycle took 6333.289 ms for 6 events 6月 02 16:08:14 zx-router bird[2197971]: I/O loop cycle took 8751.816 ms for 42 events 6月 02 16:08:29 zx-router bird[2197971]: I/O loop cycle took 13017.064 ms for 46 events 6月 02 16:08:51 zx-router bird[2197971]: I/O loop cycle took 13223.357 ms for 80 events 6月 02 16:09:44 zx-router bird[2197971]: I/O loop cycle took 17730.328 ms for 5 events 6月 02 16:10:37 zx-router bird[2197971]: I/O loop cycle took 8061.950 ms for 8 events 6月 02 16:10:46 zx-router bird[2197971]: I/O loop cycle took 9008.065 ms for 65 events 6月 02 16:10:54 zx-router bird[2197971]: I/O loop cycle took 6035.965 ms for 4 events 6月 02 16:11:03 zx-router bird[2197971]: I/O loop cycle took 9959.243 ms for 67 events I'm exporting full table to kernel with NetworkManager running on RHEL8 related systems (CentOS/Alma/Rocky 8). There is no problem before upgrading.
Hi Zhang, could you please provide us with more information about your situation? Could you send us your configuration file? What does your routing table look like? Are there any specific network events when BIRD complains? Happy routing, Jana Jana Babovakova (she/her) | BIRD Tech Support | CZ.NIC, z.s.p.o. On 02. 06. 25 11:31, zx@zxinc.org wrote:
bird 3.1.2 seems to be far better, but still about half minutes high CPU usage: 6月 02 17:27:52 zx-router bird[2232668]: I/O loop cycle took 7108.812 ms for 26 events 6月 02 17:28:11 zx-router bird[2232668]: I/O loop cycle took 5419.657 ms for 15 events 6月 02 17:28:17 zx-router bird[2232668]: I/O loop cycle took 5445.657 ms for 80 events
it's still worse comparing with 2.16.2.
Zhang Xun
发件人:zx@zxinc.org 发送时间: 2025-06-02 17:11 收件人: bird-users 主题: [3.1.1 on EL8] CPU 100% with NetworkManager Hi there,
When I upgrade bird from 2.16.2 to 3.1.1 today, it became unstable that it often causes CPU usage to 100% for bird, and later also for NetworkManager, and there are some notable syslogs:
6月 02 16:04:28 zx-router bird[2197971]: I/O loop cycle took 14046.255 ms for 43 events 6月 02 16:04:48 zx-router bird[2197971]: I/O loop cycle took 15307.277 ms for 42 events 6月 02 16:04:57 zx-router bird[2197971]: I/O loop cycle took 8562.921 ms for 43 events 6月 02 16:05:16 zx-router bird[2197971]: I/O loop cycle took 9799.271 ms for 85 events 6月 02 16:05:25 zx-router bird[2197971]: I/O loop cycle took 9330.265 ms for 80 events 6月 02 16:05:30 zx-router bird[2197971]: I/O loop cycle took 5004.867 ms for 44 events 6月 02 16:06:09 zx-router bird[2197971]: I/O loop cycle took 38759.438 ms for 84 events 6月 02 16:06:17 zx-router bird[2197971]: I/O loop cycle took 5465.396 ms for 6 events 6月 02 16:06:23 zx-router bird[2197971]: I/O loop cycle took 5906.858 ms for 44 events 6月 02 16:06:37 zx-router bird[2197971]: I/O loop cycle took 12896.032 ms for 9 events 6月 02 16:06:56 zx-router bird[2197971]: I/O loop cycle took 8217.969 ms for 81 events 6月 02 16:07:08 zx-router bird[2197971]: I/O loop cycle took 9012.775 ms for 25 events 6月 02 16:07:26 zx-router bird[2197971]: I/O loop cycle took 11506.570 ms for 45 events 6月 02 16:07:56 zx-router bird[2197971]: I/O loop cycle took 30014.580 ms for 85 events 6月 02 16:08:05 zx-router bird[2197971]: I/O loop cycle took 6333.289 ms for 6 events 6月 02 16:08:14 zx-router bird[2197971]: I/O loop cycle took 8751.816 ms for 42 events 6月 02 16:08:29 zx-router bird[2197971]: I/O loop cycle took 13017.064 ms for 46 events 6月 02 16:08:51 zx-router bird[2197971]: I/O loop cycle took 13223.357 ms for 80 events 6月 02 16:09:44 zx-router bird[2197971]: I/O loop cycle took 17730.328 ms for 5 events 6月 02 16:10:37 zx-router bird[2197971]: I/O loop cycle took 8061.950 ms for 8 events 6月 02 16:10:46 zx-router bird[2197971]: I/O loop cycle took 9008.065 ms for 65 events 6月 02 16:10:54 zx-router bird[2197971]: I/O loop cycle took 6035.965 ms for 4 events 6月 02 16:11:03 zx-router bird[2197971]: I/O loop cycle took 9959.243 ms for 67 events
I'm exporting full table to kernel with NetworkManager running on RHEL8 related systems (CentOS/Alma/Rocky 8). There is no problem before upgrading.
Hello Jana, I see the same message with BIRD 3.1.2, however on Red Hat Enterprise Linux 9.5: Jun 2 15:42:09 bird[72494]: I/O loop cycle took 28009.143 ms for 10 events The message unfortunately leaves open for me whether this is bad or good or if I even need to worry. I did not see this in BIRD 2.16 before. I already provided my BIRD configuration related to the segmentation fault in 3.1.1 on Tue, 20 May 2025 17:14:33 +0200 (the segmentation fault from 3.1.1 seems to be gone with 3.1.2, even don't know which changelog entry is related to that - but thank you for fixing this!). Regards, Robert On Mon, 02 Jun 2025, Jana Babovakova via Bird-users wrote:
Hi Zhang,
could you please provide us with more information about your situation? Could you send us your configuration file? What does your routing table look like? Are there any specific network events when BIRD complains?
Happy routing, Jana
Jana Babovakova (she/her) | BIRD Tech Support | CZ.NIC, z.s.p.o.
On 02. 06. 25 11:31, zx@zxinc.org wrote:
bird 3.1.2 seems to be far better, but still about half minutes high CPU usage: 6月 02 17:27:52 zx-router bird[2232668]: I/O loop cycle took 7108.812 ms for 26 events 6月 02 17:28:11 zx-router bird[2232668]: I/O loop cycle took 5419.657 ms for 15 events 6月 02 17:28:17 zx-router bird[2232668]: I/O loop cycle took 5445.657 ms for 80 events
it's still worse comparing with 2.16.2.
Zhang Xun
发件人:zx@zxinc.org 发送时间: 2025-06-02 17:11 收件人: bird-users 主题: [3.1.1 on EL8] CPU 100% with NetworkManager Hi there,
When I upgrade bird from 2.16.2 to 3.1.1 today, it became unstable that it often causes CPU usage to 100% for bird, and later also for NetworkManager, and there are some notable syslogs:
6月 02 16:04:28 zx-router bird[2197971]: I/O loop cycle took 14046.255 ms for 43 events 6月 02 16:04:48 zx-router bird[2197971]: I/O loop cycle took 15307.277 ms for 42 events 6月 02 16:04:57 zx-router bird[2197971]: I/O loop cycle took 8562.921 ms for 43 events 6月 02 16:05:16 zx-router bird[2197971]: I/O loop cycle took 9799.271 ms for 85 events 6月 02 16:05:25 zx-router bird[2197971]: I/O loop cycle took 9330.265 ms for 80 events 6月 02 16:05:30 zx-router bird[2197971]: I/O loop cycle took 5004.867 ms for 44 events 6月 02 16:06:09 zx-router bird[2197971]: I/O loop cycle took 38759.438 ms for 84 events 6月 02 16:06:17 zx-router bird[2197971]: I/O loop cycle took 5465.396 ms for 6 events 6月 02 16:06:23 zx-router bird[2197971]: I/O loop cycle took 5906.858 ms for 44 events 6月 02 16:06:37 zx-router bird[2197971]: I/O loop cycle took 12896.032 ms for 9 events 6月 02 16:06:56 zx-router bird[2197971]: I/O loop cycle took 8217.969 ms for 81 events 6月 02 16:07:08 zx-router bird[2197971]: I/O loop cycle took 9012.775 ms for 25 events 6月 02 16:07:26 zx-router bird[2197971]: I/O loop cycle took 11506.570 ms for 45 events 6月 02 16:07:56 zx-router bird[2197971]: I/O loop cycle took 30014.580 ms for 85 events 6月 02 16:08:05 zx-router bird[2197971]: I/O loop cycle took 6333.289 ms for 6 events 6月 02 16:08:14 zx-router bird[2197971]: I/O loop cycle took 8751.816 ms for 42 events 6月 02 16:08:29 zx-router bird[2197971]: I/O loop cycle took 13017.064 ms for 46 events 6月 02 16:08:51 zx-router bird[2197971]: I/O loop cycle took 13223.357 ms for 80 events 6月 02 16:09:44 zx-router bird[2197971]: I/O loop cycle took 17730.328 ms for 5 events 6月 02 16:10:37 zx-router bird[2197971]: I/O loop cycle took 8061.950 ms for 8 events 6月 02 16:10:46 zx-router bird[2197971]: I/O loop cycle took 9008.065 ms for 65 events 6月 02 16:10:54 zx-router bird[2197971]: I/O loop cycle took 6035.965 ms for 4 events 6月 02 16:11:03 zx-router bird[2197971]: I/O loop cycle took 9959.243 ms for 67 events
I'm exporting full table to kernel with NetworkManager running on RHEL8 related systems (CentOS/Alma/Rocky 8). There is no problem before upgrading.
Hello Robert, On Mon, Jun 02, 2025 at 03:56:06PM +0200, Robert Scheck wrote:
I see the same message with BIRD 3.1.2, however on Red Hat Enterprise Linux 9.5:
Jun 2 15:42:09 bird[72494]: I/O loop cycle took 28009.143 ms for 10 events
The message unfortunately leaves open for me whether this is bad or good or if I even need to worry. I did not see this in BIRD 2.16 before.
It looks like BIRD 3 spends more time in kernel table handling than BIRD 2 does. That may be harmless but it is worth checking on our side.
I already provided my BIRD configuration related to the segmentation fault in 3.1.1 on Tue, 20 May 2025 17:14:33 +0200 (the segmentation fault from 3.1.1 seems to be gone with 3.1.2, even don't know which changelog entry is related to that - but thank you for fixing this!).
Gonna check your config again, thanks! Maria -- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
participants (4)
-
Jana Babovakova -
Maria Matejka -
Robert Scheck -
zx@zxinc.org