@Michael, this follows the conversation on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1165480 and in our 1:1. I'm curious if you have anything to add here!

In T398592#10971502, @Urbanecm_WMF wrote:

@Michael, this follows the conversation on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/1165480 and in our 1:1. I'm curious if you have anything to add here!

Sounds good so far, thank you!

I'll mention T395249 here as prior context for this. And ServiceOps new in general, and @Clement_Goubert and @hnowlan in particular, are probably the right people to tell us whether this is a good idea and how to move it forward.

For now I'd recommend using FOREACHWIKI_IGNORE_ERRORS to restore previous behaviour. The issue with that is you won't get an alert unless the last wiki fails. This can be done in puppet by adding foreachwiki_ignore_errors => true to your job definition. I can do it for you if you decide on going this way.

In the future, when T341984: Update Kubernetes clusters to 1.31 has been rolled out to both datacenters after the next switchover, we will be able to define podFailurePolicies, which will allow us for example to catch some exit codes for a restart or the job, or treat them as fatal.

It seems like there are two different time horizons to this task, as initially framed:

1. In the near term, restoring the earlier behavior on mwmaint hosts (i.e., Problem #4) as a error-handling model that may be more appropriate for Growth-Team's use case. My understanding that this is possible using FOREACHWIKI_IGNORE_ERRORS per T398592#10971912, but please correct me if I'm wrong here.
2. Introducing a more sophisticated error handling model that, for example, allows different error conditions / exceptions to surface different exit status codes (i.e., changes to maintenance scripts themselves and / or the abstractions they're built on) and accompanying support in mw-cron for status-depending handling (Problem #3).

@Urbanecm_WMF - Does that sound right, and if so, has your team explored #1 for your use case?

@Urbanecm_WMF gentle follow-up, could you help us confirm the status of this?

Hello, thanks for the ping and apologies for my delayed response. We are aware of FOREACHWIKI_IGNORE_ERRORS and we make use of it for some of our periodic jobs. However, stopping the job entirely for most errors makes sense for us, as long as the errors that cause the abortion are not infrastructure-related.

I'm not sure whether we should migrate individual jobs to use FOREACHWIKI_IGNORE_ERRORS (and then migrate them back individually). Maybe it would make sense to (temporarily) revert the behaviour back to what it used to be (essentially, set FOREACHWIKI_IGNORE_ERRORS for all jobs by default), and once the more advanced model is ready, we can start reverting that change?

My main concern is that ignoring alerts is much easier for affected teams than having discussions like this one, and I'd like to avoid getting used to ignoring alerts.

Hope this makes sense.

I don't want to set FOREACHWIKI_IGNORE_ERRORS to true as default for all jobs because it allows us to surface failure modes that didn't before, and that is needed to gather information about the error codes maintenance scripts should throw on certain failures so we can handle them correctly with podFailurePolicies in the near future.

In the meantime, it's your call whether you want to use FOREACHWIKI_IGNORE_ERRORS for all your jobs or only some of them.

We'll keep you updated when we have something that can handle failures more precisely, there will probably be some work to be done either in the Maintenance framework or in individual scripts to differentiate exit codes, and maybe some puppet definitions to update, but we're not there yet.

In the meantime, I don't think there's much more to do for serviceops here, so I will put it on our Radar.