another bug: system time change
Hi Ondrejs, I think I found another bug. For some mysterious reason our system time jumped forward (more than 30.000 seconds) on one of our route-servers. I don't know why this happened, but I suspect a broken ntp server could have caused this. Nevetheless, this was reason enough for BIRD to drop the BGP peerings: You see the log-entries from the wtachdog, which is run every minute, all of a sudden the time jumps to 10:08 UTC, and BIRD brings down BGP sessions. Are the session hold timers dependent on the system time? Apr 9 01:33:01 rs2 /USR/SBIN/CRON[25129]: (root) CMD (/usr/local/sbin/watchdog.sh) Apr 9 01:34:01 rs2 /USR/SBIN/CRON[25136]: (root) CMD (/usr/local/sbin/watchdog.sh) Apr 9 01:35:01 rs2 /USR/SBIN/CRON[25143]: (root) CMD (/usr/local/sbin/watchdog.sh) Apr 9 10:08:28 rs2 bird: R8596x189: Error: Hold timer expired Apr 9 10:08:28 rs2 bird: R8596x189: BGP session closed Apr 9 10:08:28 rs2 bird: R8596x189: State changed to flush Apr 9 10:08:28 rs2 bird: R3330x113: Error: Hold timer expired Apr 9 10:08:28 rs2 bird: R3330x113: BGP session closed Apr 9 10:08:28 rs2 bird: R3330x113: State changed to flush Apr 9 10:08:28 rs2 bird: R1853x104: Error: Hold timer expired Wolfgang -- www.vix.at | www.aco.net wh@univie.ac.at | WH844-RIPE Vienna University Computer Center
On Fri, Apr 09, 2010 at 10:22:05AM +0200, Wolfgang Hennerbichler wrote:
Hi Ondrejs,
I think I found another bug. For some mysterious reason our system time jumped forward (more than 30.000 seconds) on one of our route-servers. I don't know why this happened, but I suspect a broken ntp server could have caused this. Nevetheless, this was reason enough for BIRD to drop the BGP peerings:
You see the log-entries from the wtachdog, which is run every minute, all of a sudden the time jumps to 10:08 UTC, and BIRD brings down BGP sessions. Are the session hold timers dependent on the system time?
That is really strange. On Linux 2.6, we use monotonic timers, which shouldn't be affected by system time change. If monotonic timers are not available (on Linux 2.4), we use system time but we detect time jumps and ignore them. -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Apr 9, 2010, at 10:49 , Ondrej Zajicek wrote:
On Fri, Apr 09, 2010 at 10:22:05AM +0200, Wolfgang Hennerbichler wrote:
Hi Ondrejs,
I think I found another bug. For some mysterious reason our system time jumped forward (more than 30.000 seconds) on one of our route-servers. I don't know why this happened, but I suspect a broken ntp server could have caused this. Nevetheless, this was reason enough for BIRD to drop the BGP peerings:
You see the log-entries from the wtachdog, which is run every minute, all of a sudden the time jumps to 10:08 UTC, and BIRD brings down BGP sessions. Are the session hold timers dependent on the system time?
That is really strange. On Linux 2.6, we use monotonic timers, which shouldn't be affected by system time change. If monotonic timers are not available (on Linux 2.4), we use system time but we detect time jumps and ignore them.
Hm. Now this is strange indeed. I run linux 2.6.33.1 (amd64) - but it is a virtualized host (with xen). Maybe it was xen's fault, but the logs don't reveal much as you see. Hm. this sucks.
-- Elen sila lumenn' omentielvo
Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
-- www.vix.at | www.aco.net wh@univie.ac.at | WH844-RIPE Vienna University Computer Center
On Fri, Apr 09, 2010 at 10:49:15AM +0200, Wolfgang Hennerbichler wrote:
Hm. Now this is strange indeed. I run linux 2.6.33.1 (amd64) - but it is a virtualized host (with xen). Maybe it was xen's fault, but the logs don't reveal much as you see. Hm. this sucks.
If there is a virtualization involved, then a bug in the virtualized monotonic timers is the simplest explanation. -- Elen sila lumenn' omentielvo Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
On Apr 9, 2010, at 14:08 , Ondrej Zajicek wrote:
On Fri, Apr 09, 2010 at 10:49:15AM +0200, Wolfgang Hennerbichler wrote:
Hm. Now this is strange indeed. I run linux 2.6.33.1 (amd64) - but it is a virtualized host (with xen). Maybe it was xen's fault, but the logs don't reveal much as you see. Hm. this sucks.
If there is a virtualization involved, then a bug in the virtualized monotonic timers is the simplest explanation.
yes. might be. damn :)
-- Elen sila lumenn' omentielvo
Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
-- www.vix.at | www.aco.net wh@univie.ac.at | WH844-RIPE Vienna University Computer Center
Hello, This is my first time so sorry if I have not done this right. I have seen this problem before. I have seen this in Linux when the BIOS synchronises its time with the OS. The Linux OS could have been fine and have had a valid NTP source, but periodically there is some form of synchronisation that occurs between the BIOS and the OS and weird things can happen. This is probably nothing to do with bird. PS; A Virtual BIOS does exactly the same thing. Ie. the real BIOS time gets synchronised with all the Virtual BIOS clocks (one for each VM), which each in turn synchronise with their respective guest. Check all your BIOS times (physical BIOS and virtual Guest BIOS'). Hope this helps. Andy. -----Original Message----- From: owner-bird-users@atrey.karlin.mff.cuni.cz [mailto:owner-bird-users@atrey.karlin.mff.cuni.cz] On Behalf Of Wolfgang Hennerbichler Sent: 09 April 2010 09:49 To: Ondrej Zajicek Cc: Bird Users Subject: Re: another bug: system time change On Apr 9, 2010, at 10:49 , Ondrej Zajicek wrote:
On Fri, Apr 09, 2010 at 10:22:05AM +0200, Wolfgang Hennerbichler wrote:
Hi Ondrejs,
I think I found another bug. For some mysterious reason our system time jumped forward (more than 30.000 seconds) on one of our route-servers. I don't know why this happened, but I suspect a broken ntp server could have caused this. Nevetheless, this was reason enough for BIRD to drop the BGP peerings:
You see the log-entries from the wtachdog, which is run every minute, all of a sudden the time jumps to 10:08 UTC, and BIRD brings down BGP sessions. Are the session hold timers dependent on the system time?
That is really strange. On Linux 2.6, we use monotonic timers, which shouldn't be affected by system time change. If monotonic timers are not available (on Linux 2.4), we use system time but we detect time jumps and ignore them.
Hm. Now this is strange indeed. I run linux 2.6.33.1 (amd64) - but it is a virtualized host (with xen). Maybe it was xen's fault, but the logs don't reveal much as you see. Hm. this sucks.
-- Elen sila lumenn' omentielvo
Ondrej 'SanTiago' Zajicek (email: santiago@crfreenet.org) OpenPGP encrypted e-mails preferred (KeyID 0x11DEADC3, wwwkeys.pgp.net) "To err is human -- to blame it on a computer is even more so."
-- www.vix.at | www.aco.net wh@univie.ac.at | WH844-RIPE Vienna University Computer Center Tel: +43 1 4277-14031 | Fax: -9140 Monitor Computer Systems Limited Company Registration Number: NI 17805 Registered Office: 3 Pine Crest, Holywood, North Down, Northern Ireland BT18 9ED
On 09.04.2010 10:22 Wolfgang Hennerbichler wrote
Hi Ondrejs,
I think I found another bug. For some mysterious reason our system time jumped forward (more than 30.000 seconds) on one of our route-servers. I don't know why this happened, but I suspect a broken ntp server could have caused this. Nevetheless, this was reason enough for BIRD to drop the BGP peerings:
You see the log-entries from the wtachdog, which is run every minute, all of a sudden the time jumps to 10:08 UTC, and BIRD brings down BGP sessions. Are the session hold timers dependent on the system time?
Apr 9 01:33:01 rs2 /USR/SBIN/CRON[25129]: (root) CMD (/usr/local/sbin/watchdog.sh) Apr 9 01:34:01 rs2 /USR/SBIN/CRON[25136]: (root) CMD (/usr/local/sbin/watchdog.sh) Apr 9 01:35:01 rs2 /USR/SBIN/CRON[25143]: (root) CMD (/usr/local/sbin/watchdog.sh) Apr 9 10:08:28 rs2 bird: R8596x189: Error: Hold timer expired Apr 9 10:08:28 rs2 bird: R8596x189: BGP session closed Apr 9 10:08:28 rs2 bird: R8596x189: State changed to flush Apr 9 10:08:28 rs2 bird: R3330x113: Error: Hold timer expired Apr 9 10:08:28 rs2 bird: R3330x113: BGP session closed Apr 9 10:08:28 rs2 bird: R3330x113: State changed to flush Apr 9 10:08:28 rs2 bird: R1853x104: Error: Hold timer expired
Is this really a BIRD bug? I guess it is expected behaviour to drop a session if you _think_ you missed keepalives The bug is w/ the underlying OS. Virtualisation and keeping time really seems to be a problem. For this and other reasons we refrain from using a VM for RS. Arnold -- Arnold Nipper / nIPper consulting, Sandhausen, Germany email: arnold@nipper.de phone: +49 6224 9259 299 mobile: +49 172 2650958 fax: +49 6224 9259 333
On Apr 9, 2010, at 12:44 , Arnold Nipper wrote:
Is this really a BIRD bug? I guess it is expected behaviour to drop a session if you _think_ you missed keepalives
Time Counters within a Daemon should NEVER be dependent on the system time, so yes, I would consider this a bug of BIRD, but it seems that BIRD has implemented this correctly, as Ondrej said.
The bug is w/ the underlying OS. Virtualisation and keeping time really seems to be a problem. For this and other reasons we refrain from using a VM for RS.
We'll see how this works out.
Arnold -- Arnold Nipper / nIPper consulting, Sandhausen, Germany email: arnold@nipper.de phone: +49 6224 9259 299 mobile: +49 172 2650958 fax: +49 6224 9259 333
-- www.vix.at | www.aco.net wh@univie.ac.at | WH844-RIPE Vienna University Computer Center
participants (4)
-
Andrew Lemin -
Arnold Nipper -
Ondrej Zajicek -
Wolfgang Hennerbichler