Hi BIRD users, Does anyone know whether a BGP shared secret can be rotated without incurring any network downtime? I did some testing with the BGP password functionality offered and it appears that any update to the BGP password configuration incurs a brief network outage with both existing/new connections. It seems like something about the way BIRD is restarting is leading to it pulling down learned routes immediately as opposed to letting them live according to the timeout setting. Does BIRD flush all routes it has learned when this configuration changes? Here is a brief excerpt to demonstrate the outage. Take note that the network disruption precisely matches the timestamp at which BIRD is reconfigured: # Logs from calico-node-4w4bv (10.95.14.104) 2022-06-29 18:26:18.836 [INFO][60] confd/client.go 1415: Trigger to recheck BGP peers following possible password update 2022-06-29 18:26:18.836 [INFO][60] confd/client.go 250: Recompute v1 BGP peerings 2022-06-29 18:26:18.836 [INFO][60] confd/client.go 949: Recompute BGP peerings: bird: Reconfiguration requested by SIGHUP 2022-06-29 18:26:18.844 [INFO][60] confd/resource.go 278: Target config /etc/calico/confd/config/bird.cfg has been updated due to change in key: /calico/bgp/v1/global bird: Reconfiguring bird: device1: Reconfigured bird: direct1: Reconfigured bird: Restarting protocol Mesh_10_95_14_105 bird: Mesh_10_95_14_105: Shutting down bird: Mesh_10_95_14_105: State changed to stop bird: Restarting protocol Mesh_10_95_14_110 bird: Mesh_10_95_14_110: Shutting down bird: Mesh_10_95_14_110: State changed to stop bird: Mesh_10_95_14_105: State changed to down bird: Mesh_10_95_14_105: Initializing bird: Mesh_10_95_14_105: Starting bird: Mesh_10_95_14_105: State changed to start bird: Mesh_10_95_14_110: State changed to down bird: Mesh_10_95_14_110: Initializing bird: Mesh_10_95_14_110: Starting bird: Mesh_10_95_14_110: State changed to start bird: Reconfigured bird: Mesh_10_95_14_110: Connected to table master bird: Mesh_10_95_14_110: State changed to feed bird: Mesh_10_95_14_110: State changed to up bird: Mesh_10_95_14_105: Connected to table master bird: Mesh_10_95_14_105: State changed to feed bird: Mesh_10_95_14_105: State changed to up 2022-06-29 18:26:36.079 [INFO][58] monitor-addresses/autodetection_methods.go 117: Using autodetected IPv4 address 10.95.14.104/26 on matching interface eth0 2022-06-29 18:26:54.537 [INFO][62] felix/summary.go 100: Summarising 9 dataplane reconciliation loops over 1m2.1s: avg=5ms longest=21ms (resync-filter-v4,resync-nat-v4) # Connection Tester Daemonset (hitting an echoserver twice a second or so) Wed Jun 29 18:26:17 UTC 2022 Successful echo server connection Wed Jun 29 18:26:17 UTC 2022 Wed Jun 29 18:26:18 UTC 2022 Successful echo server connection Wed Jun 29 18:26:18 UTC 2022 Wed Jun 29 18:26:18 UTC 2022 curl: (28) Connection timed out after 300 milliseconds Failed to connect to echo server Wed Jun 29 18:26:19 UTC 2022 Wed Jun 29 18:26:19 UTC 2022 curl: (28) Connection timed out after 300 milliseconds Failed to connect to echo server Wed Jun 29 18:26:20 UTC 2022 Wed Jun 29 18:26:20 UTC 2022 Successful echo server connection Wed Jun 29 18:26:20 UTC 2022 # For peer /host/10.95.14.81/ip_addr_v4 protocol bgp Mesh_10_95_14_81 from bgp_template { neighbor 10.95.14.81 as 64512; source address 10.95.14.82; # The local address we use for the TCP connection graceful restart time 1800; # This parameter seems to make no difference when changing BGP passwords password "LJiKASiglY+KafEwEn/cSmkiok0zHgpQq5EtYhYgoDcSQwKIpX22Tz7jOzX+"; } I have perused the RFCs for both BGP Graceful Restart (4724) & Secure BGP Sessions (2385) but haven't found a solid answer yet. When the password is changed it makes complete sense that any peers with the new password will refuse to accept any NEW routes received from peers using the old password and vice versa. I don't see the fundamental reason why TCP segments arriving with some unexpected hash necessitates that previously learned routes from that peer need to be flushed with no TTL, but the observance of the outage suggests that is what is happening. One would think that, in principle, it could wait to tear down existing routes until a configurable timeout (say the graceful restart) expires, providing a window in which we can change the password and maintain stable routing. I am relatively new to BGP and am using BIRD indirectly via Calico for container networking inside Kubernetes. I will of course take things up with the guys behind Calico, but is there anything in the BGP spec/BIRD implementation which fundamentally prevents network disruption free secret rotation? Let me know if there is any place I should look for more information on this or any debug logs which would be helpful. Thanks, Calvin