<div dir="auto"><div>(copying back to the list as this went direct inadvertently)</div><div dir="auto"><br></div><div dir="auto"><div dir="auto">Gotcha, thanks.</div><div dir="auto"><br></div><div dir="auto">Having export tables available is a great idea to help debug this, thanks. We'll enable that on some dev nodes to see if we can validate things going wonky and to give an additional view. The protocol reload is also a helpful idea to probably use in conjunction with that, to validate how things look before/after. </div><br><div class="gmail_quote" dir="auto"><div dir="ltr" class="gmail_attr">On Thu, Mar 2, 2023, 16:26 Alexander Zubkov <<a href="mailto:green@qrator.net">green@qrator.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">> Is `show route export` showing what BIRD thinks should be exported based on current filters, or what it's actively sending to those peers?<div dir="auto"><br></div><div dir="auto">It should be the first variant. So the problem not that config is read incorrectly, but that bird is not reloading protocols with changed filters. As if the "soft" configure was called. You probably could also workaround it by calling reload on your protocols.</div><div dir="auto"><br></div><div dir="auto">You can also enable export tables on your bgp session channels to examine what was actually sent to the neighbor.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 2, 2023, 23:13 Hugo Slabbert via Bird-users <<a href="mailto:bird-users@network.cz" target="_blank" rel="noreferrer">bird-users@network.cz</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi folks,<div><br></div><div>We're seeing an issue where BIRD is reporting one thing for BGP route exports (via <font face="monospace">`show route export <protocol>`</font>) but actually doing something different.</div><div><br></div><div>We see this when we change some of the conditions around export. Our export filters include a function that evaluates as either true or false, for a global toggle of whether we should proceed on with any exports to relevant peers. Roughly it looks like this:</div><div><br></div><div><font face="monospace">```</font></div><div><font face="monospace">function do_export ()<br>bool DRAIN_NODE;<br>bool HEALTHY;<br>{<br> DRAIN_NODE = false;<br> HEALTHY = false;<br> include "includes/birdwatch/*";<br> return HEALTHY && !DRAIN_NODE;<br>}</font></div><div><font face="monospace">```</font></div><div><br></div><div>That function is then included as the first term in relevant export filters, e.g.:</div><div><br></div><div><font face="monospace">```</font></div><div><font face="monospace">filter PUBLIC_EXPORT_v4 {<br> if !do_export() then reject;<br># other export terms here</font></div><div><font face="monospace">}</font><br></div><div><font face="monospace">```</font></div><div><br></div><div>We have external programs that monitor for whether the host is indicated as drained, or if the process that's handling traffic for the box is healthy. If those conditions change, they add or remove config snippets at the `<font face="monospace">includes/birdwatch/` </font>directory to change the values of the <font face="monospace">DRAIN_NODE</font> or <font face="monospace">HEALTHY</font> bools, and we then call `<font face="monospace">birdc configure</font>` to reload the config and pick up the change to either stop exports or proceed on with them. Doing the common trick of adding/removing /32s from loopbacks or such for DIRECT export don't fit our use case, so this method provides external programs control over global export state for those filters.</div><div><br></div><div>This has worked cleanly for us so far. But, recently we've hit an issue. Assuming we start in a "non-drained" state where DRAIN_NODE is false and HEALTHY is true, so we're exporting routes as expected, we then:</div><div><br></div><div>* toggle DRAIN_NODE to be true</div><div>* execute <font face="monospace">`birdc configure`</font> to reload the configuration</div><div><br></div><div>When we look at exported routes, e.g. <span style="font-family:monospace">`</span><font face="monospace">show route export <bgp peer protocol>`,</font> BIRD shows no routes exported, as expected. However, those BGP peers definitely do still see routes advertised from BIRD. We are also using <a href="https://github.com/czerwonk/bird_exporter" rel="noreferrer noreferrer" target="_blank">https://github.com/czerwonk/bird_exporter</a>, and the <font face="monospace">`bird_protocol_prefix_export_count`</font> metrics for exported routes for the drained host remain at their previous levels. In other words:</div><div><br></div><div>* birdc says nothing is being exported</div><div>* but peers still are seeing prefixes from BIRD</div><div>* and bird_exporter still shows prefixes being exported from BIRD</div><div><br></div><div>Effectively, the <font face="monospace">`birdc show route export`</font> output is lying.</div><div><br></div><div>We've tried adding a sleep between when the include snippet that changes the DRAIN_NODE value is written and when we hit <font face="monospace">`birdc configure`</font>, but that doesn't appear to make any difference. If we execute <font face="monospace">`birdc configure`</font> *twice*, though, everything's fine: The actual exports are stopped. That's true without any sleep or break between running configure as well; literally just <font face="monospace">`birdc configure`</font> back to back in the script that manages this.</div><div><br></div><div>We do not see any indication of issues in the <font face="monospace">`birdc configure`</font> runs or in BIRD's logs.</div><div><br></div><div>The reverse also does occur, where if we start with the DRAIN_NODE set to true and are exporting nothing, then unset DRAIN_NODE and hit <font face="monospace">`birdc configure`</font>: <font face="monospace">`show route export`</font> shows things being exported, but neighbours actually don't hear any routes.</div><div><br></div><div>We have not seen this occur in our smaller dev environment, but it has shown up in our more scaled environment. The relevant hosts have a couple of upstream v4 and v6 peers feeding them defaults, with:</div><div><br></div><div>Import:</div><div>* 1x IPv4 default route per BGP upstream peer, 2x total</div><div>* 1x IPv6 default route per BGP upstream peer, 2x total</div><div>Export:</div><div>* 1-5 IPv4 /24s per BGP upstream peer</div><div>* 0-1 IPv6 /48s per BGP upstream peer</div><div><br></div><div>They have a larger number of downstream BGP peers, currently around 138 but that number can fluctuate. Each of those peers have an IPv4 and IPv6 channel, running over an IPv6 transport (<font face="monospace">`extended next hop on`</font><font face="arial, sans-serif"> for the ipv4 channel</font>). </div><div><br></div><div>At the moment those have:</div><div><br></div><div>Import:</div><div>* 2x IPv6 routes per peer (one /48, one /128)</div><div>* 63x IPv4 routes per peer (all /24s)</div><div>Export:</div><div>* 21x IPv6 routes per peer (all /128s)</div><div>* 1-5x IPv6 routes per peer (all /24s)</div><div><br></div><div>So this does end up with the boxes have ~290 BGP "protocols", between v4 and v6 channels combined, ~9300 routes imported (between v4 and v6, multiplied by peers), and ~3,100 routes exported (between v4 and v6, multiplied by peers).</div><div><br></div><div>Any thoughts on what could be causing BIRD to fail to apply the policy properly on the first <font face="monospace">`birdc configure`</font> but succeeding on the second? Or why the route export output is incorrect after the first <font face="monospace">`birdc configure`</font><font face="arial, sans-serif"> but the metrics exporter being correct? Is </font><font face="monospace">`show route export`</font><font face="arial, sans-serif"> showing what BIRD <i>thinks</i> should be exported based on current filters, or what it's actively sending to those peers? Or any thoughts on how we can further debug or trace this to figure out what's confusing BIRD?</font></div></div>
</blockquote></div>
</blockquote></div></div></div>