Probable cause: In context of T388610: Migrate production Elastic clusters to Opensearch, search traffic was rerouted to codfw, see https://sal.toolforge.org/log/Wp_0rpYB8tZ8Ohr0AONU and cirrus: explicitly route search traffic to codfw.

In turn, this might be a contributing factor to, or maybe even the whole cause of, the delay in the listTaskCounts maintenance script that we obsever in T394018: Link Recommendation Task pool data missing for some wikis.

@Michael can we move this to the maintenance board or the tracking column if we're waiting for more data in relation to T388610: Migrate production Elastic clusters to Opensearch?

In T394295#10844517, @DMburugu wrote:

@Michael can we move this to the maintenance board or the tracking column if we're waiting for more data in relation to T388610: Migrate production Elastic clusters to Opensearch?

Let's move it to maintenance for now, that sounds like the right place.

It seems to have gone down again to an even lower value. Still not clear how or why or:

Discovery-Search: if there were changes by you recently-ish that could have caused this, then it would be nice to understand. That being said, I don't think that these changes in latency by themselves require action at this point.

Search traffic moved between datacenters due to operational concerns, this causes latency effects as noted in the above. The latency difference is ~30ms per round trip to the search servers if the requests have to go cross-datacenter. Typically there are 2 round trips but that can vary for a number of reasons.

In T394295#10938963, @EBernhardson wrote:

Search traffic moved between datacenters due to operational concerns, this causes latency effects as noted in the above. The latency difference is ~30ms per round trip to the search servers if the requests have to go cross-datacenter. Typically there are 2 round trips but that can vary for a number of reasons.

Thanks, that is good to know! We see the effects of this pretty directly in our metrics. I'll add a note to the affected panels, but otherwise I don't think there is much more to do here.

Is there a Phabricator project against which these moves of traffic between datacenters are usually tagged? Then we could check that first ourselves the next time we see our latencies jump abruptly without us doing anything.

There isn't any particular phabricator task for tracking where search traffic currently flows to, i suppose it's tracked in config-master discovery but that's not human readable. The shift in traffic above was part of T370147. There will potentially be a similar shift when we move to opensearch 2, but thats probably 6+ months away. In general it's not particularlly common that we have to disable one of the two search clusters, but it happens from time to time.