<div dir="ltr">Closing the loop on this:<div><br>A bit over a week in, any of the boxes on 2.0.7 consistently reproduce the issue (need `configure` issued twice to actually pick up the export policy changes) whereas the test boxes on 2.0.10 are still consistently behaving properly (single `configure` properly updates exports).</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 8, 2023 at 2:20 PM Hugo Slabbert <<a href="mailto:hugo.slabbert@menlosecurity.com">hugo.slabbert@menlosecurity.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">*nods*<div><br></div><div>We generally run distro packages unless we have specific requirements for newer features or fixes etc.</div><div><br></div><div>Current experiments show that the test routers on 2.0.7 started to reproduce this issue after about 3 days since BIRD's process start, whereas the test boxes we have on 2.0.10 have not yet reproduced the issue after about 3 days and 5 days, respectively.</div><div><br></div><div>It's always tough to prove a negative, but we'll keep it running through the weekend to validate and look at bumping to 2.0.10 across the fleet if we're still clear on the 2.0.10 boxes at that point.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Mar 8, 2023 at 2:17 PM Ross Tajvar <<a href="mailto:ross@tajvar.io" target="_blank">ross@tajvar.io</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div>By the way - compiling the most recent version of bird is very easy. So even though there's not a package for 2.0.12 for bullseye, I recommend just compiling with the same options as the bird package and running that.<br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 3, 2023, 2:07 PM Hugo Slabbert via Bird-users <<a href="mailto:bird-users@network.cz" target="_blank">bird-users@network.cz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Ah; thanks. Okay, I was misreading that as just referring to regular table filtering, not in conjunction with import/export. I had looked at <font face="monospace">`show symbols table`</font> and not seen any indication of it, but missed that these are present in the <font face="monospace">`show route export table <p.c>`</font> format regardless.<div><br></div><div>Thanks. That confirms that we do in fact see a difference there between the export table and the ad hoc route export view when this occurs, after a single call to `birdc configure` (scrubbed slightly here):</div><div><br></div><div><font face="monospace">```</font></div><div><font face="monospace">bird> show route export gw_085ea85_euwest2<br>bird> <br></font></div><div><font face="monospace">bird> show route export table gw_085ea85_euwest2.ipv4<br>Table export:<br><a href="http://57.140.1.0/24" rel="noreferrer" target="_blank">57.140.1.0/24</a> unicast [<name of static source> 16:41:41.058] * (100)<br> via <next hop> on bond0 onlink<br>```</font></div><div><br></div><div>We'll keep an eye here and validate if we do see this returning on 2.0.10 as well, or if 2.0.10 remains clear.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 3, 2023 at 10:18 AM Alexander Zubkov <<a href="mailto:green@qrator.net" rel="noreferrer" target="_blank">green@qrator.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Hi,<div dir="auto"><br></div><div dir="auto">It is documented in recent versions and on the bird's site too. Pay attention to this:</div><div dir="auto"><br></div><div dir="auto">[(import|export) table p.c]<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 3, 2023, 18:32 Hugo Slabbert via Bird-users <<a href="mailto:bird-users@network.cz" rel="noreferrer" target="_blank">bird-users@network.cz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Right, so,<div><br>I've gone ahead and enabled export tables on the channels for the relevant peers, per Alexander's suggestion for possibly getting additional visibility. I don't seem to spot any different views for route status, though. I don't see any particular docs on how to <i>view</i> export tables; does enabling export tables make a different view available to look at the export table contents specifically? Or does it just shift the behaviour so <font face="monospace">`show route export <protocol>`</font> displays the export table contents rather than a point-in-time evaluation of export filters for the specified neighbor?</div><div><br></div><div>Snippet showing export tables config enabled for the peers:<br></div><div><div><br><font face="monospace">```</font></div><div><font face="monospace">template bgp GATEWAY_v6 {<br> hold time 6;<br> startup hold time 20;<br> connect delay time 3;<br> connect retry time 6;<br> error wait time 3, 12;<br> med metric;<br> allow local as 1;<br><br> local fdff::4:2 as MENLO_ASN;<br><br> ipv6 {<br> export table on;<br> import filter GATEWAY_IMPORT_v6;<br> export filter GATEWAY_EXPORT_v6;<br> };<br> ipv4 {<br> export table on;<br> extended next hop on;<br> add paths rx;<br> import filter GATEWAY_IMPORT_v4;<br> export filter GATEWAY_EXPORT_v4;<br> };<br>}<br># ...<br>protocol bgp gw_085ea85_euwest2 from GATEWAY_v6 {<br> neighbor fdff::8005:f4d1 as 65000;<br>}<br>```</font></div><div><br>I don't see any different behaviour on the affected hosts, though. E.g. a host that just had <font face="monospace">`configure`</font> called once after setting the draining flag is showing these symptoms, showing nothing for <font face="monospace">`show route export <protocol>`</font>:<br></div><div><br></div><div><font face="monospace">```</font></div><div><font face="monospace">bird> show route export gw_085ea85_euwest2<br>bird><br>```</font><br><br></div><div>...but still showing exports under the protocol details:<br><br></div><div><font face="monospace">```</font></div><div><font face="monospace">bird> show protocols all gw_085ea85_euwest2<br>Name Proto Table State Since Info<br>gw_085ea85_euwest2 BGP --- up 2023-03-03 16:33:43 Established <br> BGP state: Established </font></div><div><font face="monospace"># ...</font></div><div><font face="monospace"> Channel ipv6<br> State: UP<br> Table: master6<br> Preference: 100<br> Input filter: GATEWAY_IMPORT_v6<br> Output filter: GATEWAY_EXPORT_v6<br> Routes: 2 imported, 2 exported, 1 preferred<br> Route change stats: received rejected filtered ignored accepted<br> Import updates: 3 0 1 0 2<br> Import withdraws: 0 0 --- 0 0<br> Export updates: 109 5 96 --- 8<br> Export withdraws: 2 --- --- --- 2<br> BGP Next hop: fdff::4:2<br> Channel ipv4<br> State: UP<br> Table: master4<br> Preference: 100<br> Input filter: GATEWAY_IMPORT_v4<br> Output filter: GATEWAY_EXPORT_v4<br> Routes: 12 imported, 1 exported, 0 preferred<br> Route change stats: received rejected filtered ignored accepted<br> Import updates: 12 0 0 0 12<br> Import withdraws: 0 0 --- 0 0<br> Export updates: 39 4 31 --- 4<br> Export withdraws: 0 --- --- --- 1<br> BGP Next hop: fdff::4:2 <br>```</font><br><br></div><div>Note this is still on 2.0.7. We've bumped some hosts to 2.0.10, but as indicated in the previous message, just a simple restart clears this issue from occurring. We've enabled the export table config on both a 2.0.7 and a 2.0.10 host, to be able to possibly spot if this reoccurs on the 2.0.10 host as well after a period. An example host on 2.0.7 showing this behaviour has been up for ~2 weeks. The box upgraded to 2.0.10 has had BIRD running for just ~16 hours at this point and is not yet showing any issues.<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 2, 2023 at 4:07 PM Hugo Slabbert <<a href="mailto:hugo.slabbert@menlosecurity.com" rel="noreferrer noreferrer" target="_blank">hugo.slabbert@menlosecurity.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">A slight update on this:<div><br></div><div><a href="https://gitlab.nic.cz/labs/bird/-/commit/3f477ccb03ed99cf6754baaca179fcf791bcda55" rel="noreferrer noreferrer" target="_blank">3f477ccb</a> does appear to be in 2.0.7, which we're running, so if that's the issue that may not be the problem.<br></div><div><br></div><div>This looked to be successful initially when upgrading to 2.0.10. But, I then checked a box that was still running 2.0.7 and where we could repro it. I simply restarted bird there, and then could no longer repro it.</div><div><br></div><div>So, just restarting bird at 2.0.7 was sufficient to clear the problem, at least temporarily, and the bump to 2.0.10 then wasn't a clear test, given that's obviously a fresh instance of bird running.</div><div><br></div><div>We'll try to validate if the problem eventually returns on the 2.0.7 box(es) after a restart, and if it does *not* return on the 2.0.10 instance, but we don't have a clear timeline at the moment on this if it's something that pops up "in a while" of bird running.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 2, 2023 at 2:54 PM Hugo Slabbert <<a href="mailto:hugo.slabbert@menlosecurity.com" rel="noreferrer noreferrer" target="_blank">hugo.slabbert@menlosecurity.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Was this perhaps <a href="https://gitlab.nic.cz/labs/bird/-/commit/3f477ccb03ed99cf6754baaca179fcf791bcda55" rel="noreferrer noreferrer" target="_blank">3f477ccb</a>?<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Filters: Function body comparison result now used.<br>Function bodies were compared in post-parse time, yet the result was not<br>used and the functions were incorrectly considered the same as before. </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Now the result is used to reload affected protocols.</blockquote></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 2, 2023 at 2:51 PM Hugo Slabbert <<a href="mailto:hugo.slabbert@menlosecurity.com" rel="noreferrer noreferrer" target="_blank">hugo.slabbert@menlosecurity.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">ah, right, apologies.<div><br></div><div>bird 2.0.7-4.1 on Debian 11.6, kernel 5.10.136-1<br></div><div><br></div><div>Looks like 2.0.7 was released Oct 16 2019 (<a href="https://bird.network.cz/?download" rel="noreferrer noreferrer" target="_blank">https://bird.network.cz/?download</a>), so a fair chance we might be hitting this? It looks like something from 2.0.10 is available from the bullseye backports, with the most recent being 2.0.12 in bookworm or sid. I'll look at pulling one of those in to validate.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">...where changes in functions sometimes got ignored.</blockquote><div><br></div><div>This might be reaching, but would that explain the difference between what's shown in route export status output versus what's actually being exported? </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 2, 2023 at 2:39 PM Maria Matejka via Bird-users <<a href="mailto:bird-users@network.cz" rel="noreferrer noreferrer" target="_blank">bird-users@network.cz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello!<br>
<br>
<br>
> We've tried adding a sleep between when the include snippet that changes <br>
> the DRAIN_NODE value is written and when we hit `birdc configure`, but <br>
> that doesn't appear to make any difference. If we execute `birdc <br>
> configure` *twice*, though, everything's fine: The actual exports are <br>
> stopped. That's true without any sleep or break between running <br>
> configure as well; literally just `birdc configure` back to back in the <br>
> script that manages this.<br>
> <br>
> We do not see any indication of issues in the `birdc configure` runs or <br>
> in BIRD's logs.<br>
<br>
You are not disclosing the version of BIRD you are using. I vaguely <br>
remember that we fixed this kind of bug several years ago where changes <br>
in functions sometimes got ignored.<br>
<br>
Thus if you are not using a recent BIRD version, you are probably <br>
hitting that old bug.<br>
<br>
Maria<br>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div></div></div>
</blockquote></div>
</blockquote></div>