Restart CronJobs on failure of the service mesh
Open, LowPublic
Actions

Assigned To

None

Authored By

	Clement_Goubert
	Apr 3 2025, 2:42 PM

Description

Define a specific exit code in MWScript.php when the service mesh doesn't respond in time, so it can be caught by a podFailurePolicy and the Job can be retried (since it never actually started work, that is safe).

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	MWScript.php: exit code on mesh, longer timeout	operations/mediawiki-config	master	+6 -4

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved		dancy	T392693 Drop support for Debian Buster
Resolved		Clement_Goubert	T341560 Migrate mwmaint server functionality to mw-on-k8s
Resolved		Clement_Goubert	T341555 Implement periodic maintenance scripts for mw-on-k8s
Resolved		RLazarus	T341553 Allow running one-off scripts manually
Resolved		RLazarus	T348284 Handle sidecar containers in one-off Kubernetes jobs
Resolved		RLazarus	T359127 MW image version for maintenance scripts
Resolved		RLazarus	T367200 mw-script fails to render in CI
Resolved		RLazarus	T368966 Pipe stdin into one-off maintenance scripts on Kubernetes
Resolved		RLazarus	T369142 Show more useful information when mwscript-k8s fails to launch
Declined		None	T369143 Allow cleaning up specific mwscript-k8s runs
Resolved		RLazarus	T369175 mwscript-k8s --attach error: TypeError: 'NoneType' object is not iterable
Resolved	BUG REPORT	RLazarus	T369676 `mwscript-k8s --attach` seems to terminate IO after a few seconds without input
Resolved		RLazarus	T376099 --timeout flag for mwscript-k8s
Resolved		RLazarus	T376230 Support bringing text files into the container for one-off maintenance scripts
Resolved		Lucas_Werkmeister_WMDE	T376604 [PS] Update PropertySuggester update process for mwscript-k8s
Resolved		Arian_Bozorg	T377986 Q4 2024 update of Property Suggester data
Open		None	T376616 MWScript.php doesn't allow wikiless scripts without the .php suffix
Resolved		RLazarus	T376714 Evaluate running a statsd-exporter in the mw-script namespace
Resolved		RLazarus	T376776 mw-scripts SAL integration
Resolved		RLazarus	T377292 Support machine-readable output for mwscript-k8s
Resolved		RLazarus	T378429 Allow members of restricted to run maintenance scripts
Resolved		RLazarus	T378479 Allow using helper scripts inside of mwscript-k8s
Open		None	T378754 Persistent logging of mwscript-k8s runs
Open		None	T379675 Support output files in mwscript-k8s
Resolved		RLazarus	T380925 Support passing env variables to maintenance scripts in mwscript-k8s
Resolved		tstarling	T381251 MW maintenance scripts on k8s can't do internal HTTP requests
Duplicate		None	T382398 Mediawiki maint scripts using service proxied by the tls proxy might fail when running with mwscript-k8s
Invalid		None	T385818 mwscript-k8s fails when there are parenthesis used in arguments
Duplicate		RLazarus	T387127 mwscript-k8s purgeList does not reliably purge cached URLs
Resolved		Joe	T387208 Ensure tls-proxy container is started before launching main container
Open		RLazarus	T387268 Kubernetes maintenance script status UI
Resolved		Clement_Goubert	T389484 Create a mediawiki-cli image
Resolved		Scott_French	T389499 Refactor scap's kubernetes DeploymentsConfig to support selection of image kinds
Open		None	T394534 Visualise mw-script jobs
Resolved		RLazarus	T401737 Custom dblists for mwscript-k8s
Open		None	T412469 MediaWiki periodic job startupregistrystats-testwiki failed
Open		None	T411654 MediaWiki periodic job startupregistrystats failed
Open		None	T412052 MediaWiki periodic job centralauth-backfilllocalaccounts.php-metawiki failed
Open		None	T410764 MediaWiki periodic job startupregistrystats-mediawikiwiki failed
Resolved		JMeybohm	T341984 Update Kubernetes clusters to 1.31
Open		None	T390972 Restart CronJobs on failure of the service mesh

Event Timeline

Clement_Goubert created this task.Apr 3 2025, 2:42 PM

Change #1133935 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/mediawiki-config@master] MWScript.php: Specific exit code on mesh failure

https://gerrit.wikimedia.org/r/1133935

gerritbot added a project: Patch-For-Review.Apr 3 2025, 2:48 PM

podFailurePolicy isn't available on the kubernetes version currently running our production wikikube clusters.

Clement_Goubert added a parent task: T341984: Update Kubernetes clusters to 1.31.Apr 3 2025, 3:07 PM

Change #1133935 merged by jenkins-bot:

[operations/mediawiki-config@master] MWScript.php: exit code on mesh, longer timeout

https://gerrit.wikimedia.org/r/1133935

Mentioned in SAL (#wikimedia-operations) [2025-04-10T10:45:56Z] <cgoubert@deploy1003> Started scap sync-world: Backport for [[gerrit:1133935|MWScript.php: exit code on mesh, longer timeout (T390972 T387208)]]

Mentioned in SAL (#wikimedia-operations) [2025-04-10T10:54:08Z] <cgoubert@deploy1003> cgoubert: Backport for [[gerrit:1133935|MWScript.php: exit code on mesh, longer timeout (T390972 T387208)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2025-04-10T11:08:11Z] <cgoubert@deploy1003> Finished scap sync-world: Backport for [[gerrit:1133935|MWScript.php: exit code on mesh, longer timeout (T390972 T387208)]] (duration: 22m 15s)