Graceful shutdown request signal
Hello, I run bird on a system which uses systemd as a service supervisor, and would like to implement graceful restart in a way which works well with it. In particular, what I'd like to do is: • If I restart the bird service (e.g. for an upgrade), Bird performs a graceful restart • If I manually stop bird (systemctl stop bird2), shutdown the system, or any other action which is liable to cause a longer period of downtime, perform a graceful restart (for BGP at least) Ideally systemd would have an ExecStopRestart option which could be used to distinguish between stops and restarts, but unfortunately it doesn't (I might submit a feature request for this regardless). However, it does support `KillSignal and RestartKillSignal options.` `So therefore I was wondering how the Bird developers and users would feel about introducing a signal (SIGUSR2 maybe?) which can be used to request that Bird perform a graceful restart?` `- Erin`
Hello! Well, it's a pity that systemd doesn't allow for custom operations – in such case you could call "systemd graceful bird2" or "systemd restart bird2"… Anyway, feel free to implement it, it should be like 10 lines of code. Sigusr2 is probably ok. Then it'll be on anybody to choose whether to restart gracefully or hardly by systemd. Maria On 20 June 2023 20:20:50 CEST, Erin Shepherd <bird-users@erinshepherd.net> wrote:
Hello,
I run bird on a system which uses systemd as a service supervisor, and would like to implement graceful restart in a way which works well with it.
In particular, what I'd like to do is: • If I restart the bird service (e.g. for an upgrade), Bird performs a graceful restart • If I manually stop bird (systemctl stop bird2), shutdown the system, or any other action which is liable to cause a longer period of downtime, perform a graceful restart (for BGP at least) Ideally systemd would have an ExecStopRestart option which could be used to distinguish between stops and restarts, but unfortunately it doesn't (I might submit a feature request for this regardless). However, it does support `KillSignal and RestartKillSignal options.`
`So therefore I was wondering how the Bird developers and users would feel about introducing a signal (SIGUSR2 maybe?) which can be used to request that Bird perform a graceful restart?`
`- Erin` -- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
I ended up picking this up and implementing it, and in my experiments it seems to be working fine Patch attached - Erin On Tue, 20 Jun 2023, at 21:07, Maria Matejka via Bird-users wrote:
Hello!
Well, it's a pity that systemd doesn't allow for custom operations – in such case you could call "systemd graceful bird2" or "systemd restart bird2"…
Anyway, feel free to implement it, it should be like 10 lines of code. Sigusr2 is probably ok. Then it'll be on anybody to choose whether to restart gracefully or hardly by systemd.
Maria
On 20 June 2023 20:20:50 CEST, Erin Shepherd <bird-users@erinshepherd.net> wrote:
Hello,
I run bird on a system which uses systemd as a service supervisor, and would like to implement graceful restart in a way which works well with it.
In particular, what I'd like to do is: • If I restart the bird service (e.g. for an upgrade), Bird performs a graceful restart • If I manually stop bird (systemctl stop bird2), shutdown the system, or any other action which is liable to cause a longer period of downtime, perform a graceful restart (for BGP at least) Ideally systemd would have an ExecStopRestart option which could be used to distinguish between stops and restarts, but unfortunately it doesn't (I might submit a feature request for this regardless). However, it does support `KillSignal and RestartKillSignal options.`
`So therefore I was wondering how the Bird developers and users would feel about introducing a signal (SIGUSR2 maybe?) which can be used to request that Bird perform a graceful restart?`
`- Erin` -- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
Hi Erin, On Tue, Jun 20, 2023 at 08:20:50PM +0200, Erin Shepherd wrote:
I run bird on a system which uses systemd as a service supervisor, and would like to implement graceful restart in a way which works well with it.
I'm also interested in getting this working. I'm wondering how graceful restart is supposed to behave to begin with though. Last time I just tried `birdc graceful restart` I was surprised that this actually makes bird exit with rv=0 instead of ... well actually restarting. Is that normal or a bug in the Debian packaging/systemd service? I suppose if you're supposed to use graceful restart just before rebooting the system it makes sense but at the time I just wanted to do a full restart to update the running bird executable without causing traffic disruption and was rather surprised when it didn't come back up.
In particular, what I'd like to do is: • If I restart the bird service (e.g. for an upgrade), Bird performs a graceful restart • If I manually stop bird (systemctl stop bird2), shutdown the system, or any other action which is liable to cause a longer period of downtime, perform a graceful restart (for BGP at least)
When you want to tickle special behaviour out of systemd I found it best to not try and do everything in one unit. Using the dependency system you can stop another unit before bird.service gets stopped, this can then call birdc graceful restart, like this: bird-graceful-restart.service: [Unit] After=bird.service Requires=bird.service [Service] Type=oneshot RemainAfterExit=yes ExecStop=birdc graceful restart [Install] WantedBy=bird.service That way you can enable/disable system wide graceful bird restart by `systemctl enable bird-graceful-restart.service`. The BindsTo+After means stopping bird.service (which is part of restart) will first stop bird-graceful-restart.service, which can then casually signal bird to perform graceful restart without a race (I think). WanteBy=bird.service is needed as ExecStop will only run when a service has actually been started. Also some special handling for the case where bird is already gone (crashed or exited) and birdc would fail might be needed. Excercise for the reader :) Doing some quick testing this actually seems to do what I want. Somewhat unexpectedly `systemctl restart bird.servce` with the above enabled does then actually do what you'd expect since restart is just stop+start. Bird will exit on getting the graceful restart command so systemd can't try to stop it some more but the subsequent start will make it come back up. Neat. On system shutdown only the stop part will happen so bird exiting is actully useful now and that too should work as expected.
Ideally systemd would have an ExecStopRestart option which could be used to distinguish between stops and restarts, but unfortunately it doesn't (I might submit a feature request for this regardless). However, it does support `KillSignal and RestartKillSignal options.`
I've been perplexed by this in the past also. Don't even get me started on the fact that ExecStop= runs even when a service already terminated on it's own, to like stop it some more... :) The semantics are really bonkers sometimes but it's actually rather flexible if you look at the nuaunces of the different dependency types a bit. --Daniel
Hello Daniel, On 21 June 2023 01:03:50 CEST, "Daniel Gröber" <dxld@darkboxed.org> wrote:
Hi Erin,
On Tue, Jun 20, 2023 at 08:20:50PM +0200, Erin Shepherd wrote:
I run bird on a system which uses systemd as a service supervisor, and would like to implement graceful restart in a way which works well with it.
I'm also interested in getting this working. I'm wondering how graceful restart is supposed to behave to begin with though. Last time I just tried `birdc graceful restart` I was surprised that this actually makes bird exit with rv=0 instead of ... well actually restarting.
Is that normal or a bug in the Debian packaging/systemd service?
You're supposed to start BIRD again on your terms. It's a misleading command wording, kind of. I don't know whether it's a good idea to actually exec from the shutdown cycle. Will think about it. It may be a nice feature.
I suppose if you're supposed to use graceful restart just before rebooting the system it makes sense but at the time I just wanted to do a full restart to update the running bird executable without causing traffic disruption and was rather surprised when it didn't come back up.
Yes, these two different use cases were what we were choosing from when implementing this.
In particular, what I'd like to do is: • If I restart the bird service (e.g. for an upgrade), Bird performs a graceful restart • If I manually stop bird (systemctl stop bird2), shutdown the system, or any other action which is liable to cause a longer period of downtime, perform a graceful restart (for BGP at least)
When you want to tickle special behaviour out of systemd I found it best to not try and do everything in one unit. Using the dependency system you can stop another unit before bird.service gets stopped, this can then call birdc graceful restart, like this:
bird-graceful-restart.service:
[Unit] After=bird.service Requires=bird.service [Service] Type=oneshot RemainAfterExit=yes ExecStop=birdc graceful restart [Install] WantedBy=bird.service
That way you can enable/disable system wide graceful bird restart by `systemctl enable bird-graceful-restart.service`.
The BindsTo+After means stopping bird.service (which is part of restart) will first stop bird-graceful-restart.service, which can then casually signal bird to perform graceful restart without a race (I think). WanteBy=bird.service is needed as ExecStop will only run when a service has actually been started.
Also some special handling for the case where bird is already gone (crashed or exited) and birdc would fail might be needed. Excercise for the reader :)
Doing some quick testing this actually seems to do what I want. Somewhat unexpectedly `systemctl restart bird.servce` with the above enabled does then actually do what you'd expect since restart is just stop+start. Bird will exit on getting the graceful restart command so systemd can't try to stop it some more but the subsequent start will make it come back up. Neat.
On system shutdown only the stop part will happen so bird exiting is actully useful now and that too should work as expected.
If there was somebody to implement this additional Systemd unit and provide it as a patch, we'd happily merge it. Thank you for all the ideas! Maria -- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
Hello Maria, Regarding restarts, I think the killer feature might be some sort of restart, when bird execs a new binary, keeping all the file descriptors open and its state somehow. So the new instance could transparently catch up with all the running sessions, etc. It can serialize the internal state somehow and then reinitialize it from that. But I'm afraid it would require a great effort to implement something like that. On Wed, Jun 21, 2023, 08:58 Maria Matejka via Bird-users < bird-users@network.cz> wrote:
Hello Daniel,
On 21 June 2023 01:03:50 CEST, "Daniel Gröber" <dxld@darkboxed.org> wrote:
Hi Erin,
On Tue, Jun 20, 2023 at 08:20:50PM +0200, Erin Shepherd wrote:
I run bird on a system which uses systemd as a service supervisor, and would like to implement graceful restart in a way which works well with it.
I'm also interested in getting this working. I'm wondering how graceful restart is supposed to behave to begin with though. Last time I just tried `birdc graceful restart` I was surprised that this actually makes bird exit with rv=0 instead of ... well actually restarting.
Is that normal or a bug in the Debian packaging/systemd service?
You're supposed to start BIRD again on your terms. It's a misleading command wording, kind of. I don't know whether it's a good idea to actually exec from the shutdown cycle. Will think about it. It may be a nice feature.
I suppose if you're supposed to use graceful restart just before rebooting the system it makes sense but at the time I just wanted to do a full restart to update the running bird executable without causing traffic disruption and was rather surprised when it didn't come back up.
Yes, these two different use cases were what we were choosing from when implementing this.
In particular, what I'd like to do is: • If I restart the bird service (e.g. for an upgrade), Bird performs a graceful restart • If I manually stop bird (systemctl stop bird2), shutdown the system, or any other action which is liable to cause a longer period of downtime, perform a graceful restart (for BGP at least)
When you want to tickle special behaviour out of systemd I found it best to not try and do everything in one unit. Using the dependency system you can stop another unit before bird.service gets stopped, this can then call birdc graceful restart, like this:
bird-graceful-restart.service:
[Unit] After=bird.service Requires=bird.service [Service] Type=oneshot RemainAfterExit=yes ExecStop=birdc graceful restart [Install] WantedBy=bird.service
That way you can enable/disable system wide graceful bird restart by `systemctl enable bird-graceful-restart.service`.
The BindsTo+After means stopping bird.service (which is part of restart) will first stop bird-graceful-restart.service, which can then casually signal bird to perform graceful restart without a race (I think). WanteBy=bird.service is needed as ExecStop will only run when a service has actually been started.
Also some special handling for the case where bird is already gone (crashed or exited) and birdc would fail might be needed. Excercise for the reader :)
Doing some quick testing this actually seems to do what I want. Somewhat unexpectedly `systemctl restart bird.servce` with the above enabled does then actually do what you'd expect since restart is just stop+start. Bird will exit on getting the graceful restart command so systemd can't try to stop it some more but the subsequent start will make it come back up. Neat.
On system shutdown only the stop part will happen so bird exiting is actully useful now and that too should work as expected.
If there was somebody to implement this additional Systemd unit and provide it as a patch, we'd happily merge it.
Thank you for all the ideas! Maria
-- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
Hello Alexander, you're right, catching up with all the running session and restoring the full internal state would be a nightmare. I think it is doable but too difficult (and labour-costly) to get it right. Regarding the need for graceful restart support anyway, I don't see much of a good reason to prefer this over a regular graceful restart. Maria On 6/21/23 13:41, Alexander Zubkov wrote:
Hello Maria,
Regarding restarts, I think the killer feature might be some sort of restart, when bird execs a new binary, keeping all the file descriptors open and its state somehow. So the new instance could transparently catch up with all the running sessions, etc. It can serialize the internal state somehow and then reinitialize it from that. But I'm afraid it would require a great effort to implement something like that.
On Wed, Jun 21, 2023, 08:58 Maria Matejka via Bird-users <bird-users@network.cz> wrote:
Hello Daniel,
On 21 June 2023 01:03:50 CEST, "Daniel Gröber" <dxld@darkboxed.org> wrote: >Hi Erin, > >On Tue, Jun 20, 2023 at 08:20:50PM +0200, Erin Shepherd wrote: >> I run bird on a system which uses systemd as a service supervisor, and >> would like to implement graceful restart in a way which works well with >> it. > >I'm also interested in getting this working. I'm wondering how graceful >restart is supposed to behave to begin with though. Last time I just tried >`birdc graceful restart` I was surprised that this actually makes bird exit >with rv=0 instead of ... well actually restarting. > >Is that normal or a bug in the Debian packaging/systemd service?
You're supposed to start BIRD again on your terms. It's a misleading command wording, kind of. I don't know whether it's a good idea to actually exec from the shutdown cycle. Will think about it. It may be a nice feature.
>I suppose if you're supposed to use graceful restart just before rebooting >the system it makes sense but at the time I just wanted to do a full >restart to update the running bird executable without causing traffic >disruption and was rather surprised when it didn't come back up.
Yes, these two different use cases were what we were choosing from when implementing this.
>> In particular, what I'd like to do is: >> • If I restart the bird service (e.g. for an upgrade), Bird performs a graceful restart >> • If I manually stop bird (systemctl stop bird2), shutdown the system, or any other action which is liable to cause a longer period of downtime, perform a graceful restart (for BGP at least) > >When you want to tickle special behaviour out of systemd I found it best to >not try and do everything in one unit. Using the dependency system you can >stop another unit before bird.service gets stopped, this can then call >birdc graceful restart, like this: > >bird-graceful-restart.service: > > [Unit] > After=bird.service > Requires=bird.service > [Service] > Type=oneshot > RemainAfterExit=yes > ExecStop=birdc graceful restart > [Install] > WantedBy=bird.service > >That way you can enable/disable system wide graceful bird restart by >`systemctl enable bird-graceful-restart.service`. > >The BindsTo+After means stopping bird.service (which is part of restart) >will first stop bird-graceful-restart.service, which can then casually >signal bird to perform graceful restart without a race (I >think). WanteBy=bird.service is needed as ExecStop will only run when a >service has actually been started. > >Also some special handling for the case where bird is already gone (crashed >or exited) and birdc would fail might be needed. Excercise for the reader :) > >Doing some quick testing this actually seems to do what I want. Somewhat >unexpectedly `systemctl restart bird.servce` with the above enabled does >then actually do what you'd expect since restart is just stop+start. Bird >will exit on getting the graceful restart command so systemd can't try to >stop it some more but the subsequent start will make it come back up. Neat. > >On system shutdown only the stop part will happen so bird exiting is >actully useful now and that too should work as expected.
If there was somebody to implement this additional Systemd unit and provide it as a patch, we'd happily merge it.
Thank you for all the ideas! Maria
-- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
-- Maria Matejka (she/her) | BIRD Team Leader | CZ.NIC, z.s.p.o.
participants (5)
-
Alexander Zubkov -
Daniel Gröber -
Erin Shepherd -
Maria Matejka -
Vincent Bernat