@Clement_Goubert: Retries sound like the preferred solution if we can implement them in a sensible way. Could you point me to an example where they have been implemented particularly well?

I agree with prioritizing this as "High" because ideally we get some mitigation in place soon. Otherwise, we have a lot of alert-noise in our internal Growth engineering channel that distracts from actual errors and trains us in alert fatigue.

@Michael I'm not well versed enough in MediaWiki code to know where retries are implemented, but I imagine a loop with a try-catch for the search exception, exponential backoff and a limit for the number of retries would be the general pattern.

In T395247#10856502, @Michael wrote:

@Clement_Goubert: Retries sound like the preferred solution if we can implement them in a sensible way. Could you point me to an example where they have been implemented particularly well?

A good example of what a retry logic can look like is https://doc.wikimedia.org/wmflib/master/api/wmflib.decorators.html#wmflib.decorators.retry, which is used as base for our automation.

Change #1150711 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] mw::maintenance::growthexperiment: Ignore foreachwiki errors

https://gerrit.wikimedia.org/r/1150711

Change #1150711 merged by Clément Goubert:

[operations/puppet@production] mw::maintenance::growthexperiment: Ignore foreachwiki errors

https://gerrit.wikimedia.org/r/1150711

The loop for this job now ignores errors and continues, as it was in the old wrapper.

I have deleted the failed jobs to clear alerting.