Page MenuHomePhabricator
Create Task
Maniphest T378479

Allow using helper scripts inside of mwscript-k8s
Closed, ResolvedPublic
Actions

Description

As part of T377958: Add helper script functionality to our php images, I've deployed a script implementing the common helpers called by one-off script runners in the mediawiki image:

  • mwscript
  • foreachwiki
  • foreachwikiindblist
  • mwscriptwikiset (effectively the same as foreachwikiindblist)

Right now, mwscript-k8s only allows for passing the php script and its arguments directly, but we should provide a way for users to call these helpers (particularly both foreachwiki* and mwscriptwikiset) so that long loops can be executed inside a single Pod, limiting the number of spawned Jobs.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
mediawiki: Remove backwards compatibility path for running php directlyoperations/deployment-chartsmaster+1 -5
deployment_server: Call into the mwscript helper from mwscript-k8soperations/puppetproduction+7 -10
deployment_server: Add --dblist to mwscript-k8soperations/puppetproduction+28 -2
mediawiki: Allow varying the entrypoint through mwscript valuesoperations/deployment-chartsmaster+15 -4
deployment_server: Pass mwscript.command in mwscript-k8s valuesoperations/puppetproduction+7 -6
deployment_server: Use cli-image for mw-scriptoperations/puppetproduction+1 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

Clement_Goubert created this task.Oct 29 2024, 12:25 PM
ABran-WMF unsubscribed.Oct 29 2024, 12:29 PM
ABran-WMF subscribed.
Urbanecm_WMF added a comment.Oct 29 2024, 1:22 PM

Related note: I sometimes need to run a script on an arbitrary set of wikis (I have the set of wikis in a txt file, rather as a committed dblist). Currently, there are several things I can do to do so:

  1. Bash loop: something like while read wiki; do mwscript SCRIPT.php --wiki=$wiki ...; done < wikis.txt would run the script (with minimal overhead, as foreachwikiindblist and friends effectively are a bash loop)
  2. Create a temporary dblist: write a temporary file under mwmaint:/srv/mediawiki/dblists, then proceed to use foreachwikiindblist
  3. Actually commit the dblist: commit it to the operations/mediawiki-config repo, deploy the change and use foreachwikiindblist

With mwscript-k8s, the only option that remains to work unchanged is option 3. While it sometimes might be the most appropriate one, it often is not, especially when I'm virtually certain I will never need to work on the same wikiset again. Option 1 is theoretically still an option, but it adds a lot of overhead (as each script invocation creates a full pod). In addition to that, I might accidentally make the script invocations run in paralel to each other, without realising so (although passing -f should avoid that, at least in theory). Option 2 is no longer an option, as the dblists are sourced from the image, not the deployment host (control host, in this instance?).

Would it be possible to have mwscript-k8s introduce something lower level (like, "here is a list of wikis, run the script on all of them sequentially")? Once that is available, both the helper scripts and users can make use of that, whether the set of wikis comes from a dblist or from a different source.

Novem_Linguae subscribed.May 6 2025, 8:25 PM
Ladsgroup subscribed.May 7 2025, 5:11 PM
gerritbot added a comment.May 19 2025, 11:55 PM

Change #1147901 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] deployment_server: Use cli-image for mw-script

https://gerrit.wikimedia.org/r/1147901

gerritbot added a project: Patch-For-Review.May 19 2025, 11:55 PM
gerritbot added a comment.May 20 2025, 12:32 AM

Change #1147901 merged by RLazarus:

[operations/puppet@production] deployment_server: Use cli-image for mw-script

https://gerrit.wikimedia.org/r/1147901

Maintenance_bot removed a project: Patch-For-Review.May 20 2025, 1:30 AM
gerritbot added a comment.May 20 2025, 3:32 AM

Change #1147917 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] deployment_server: Pass mwscript.command in mwscript-k8s values

https://gerrit.wikimedia.org/r/1147917

gerritbot added a project: Patch-For-Review.May 20 2025, 3:32 AM

Change #1147918 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] mediawiki: Allow varying the entrypoint through mwscript values

https://gerrit.wikimedia.org/r/1147918

gerritbot added a comment.May 20 2025, 8:45 PM

Change #1148450 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] deployment_server: Add --dblist to mwscript-k8s

https://gerrit.wikimedia.org/r/1148450

gerritbot added a comment.May 20 2025, 11:43 PM

Change #1147917 merged by RLazarus:

[operations/puppet@production] deployment_server: Pass mwscript.command in mwscript-k8s values

https://gerrit.wikimedia.org/r/1147917

gerritbot added a comment.May 20 2025, 11:56 PM

Change #1147918 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Allow varying the entrypoint through mwscript values

https://gerrit.wikimedia.org/r/1147918

gerritbot added a comment.May 21 2025, 1:38 AM

Change #1148490 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] deployment_server: Call into the mwscript helper from mwscript-k8s

https://gerrit.wikimedia.org/r/1148490

gerritbot added a comment.May 21 2025, 1:43 AM

Change #1148491 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] mediawiki: Remove backwards compatibility path for running php directly

https://gerrit.wikimedia.org/r/1148491

gerritbot added a comment.May 21 2025, 1:57 AM

Change #1148450 merged by RLazarus:

[operations/puppet@production] deployment_server: Add --dblist to mwscript-k8s

https://gerrit.wikimedia.org/r/1148450

Novem_Linguae unsubscribed.May 21 2025, 1:58 AM
RLazarus closed this task as Resolved.May 21 2025, 2:22 AM
RLazarus claimed this task.
RLazarus added subscribers: Scott_French, RLazarus.

This is now supported via the --dblist flag! See https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Running_on_multiple_wikis_(the_safe_way).

Resolving, although there are three items of related followup work, which @Scott_French kindly offered to shepherd to completion:

There's also potential feature work mentioned in the commit message at https://gerrit.wikimedia.org/r/1148450: if expanddblist allowed for dblists outside of the canonical dblists directory, then a dblist could be uploaded with --file, making the situation described in T378479#10272539 a little better. Then mwscript-k8s could also grow a flag like --wikis=enwiki,frwiki to create and use an ad-hoc dblist in the ConfigMap. But the core of this functionality is done.

Stashbot added a comment.May 22 2025, 5:14 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-22T17:14:51Z] <swfrench@deploy1003> Started scap sync-world: Non-deploy scap run to pick up mw-script / mw-cron logging changes - T378479

gerritbot added a comment.May 27 2025, 3:05 PM

Change #1148490 merged by Scott French:

[operations/puppet@production] deployment_server: Call into the mwscript helper from mwscript-k8s

https://gerrit.wikimedia.org/r/1148490

gerritbot added a comment.May 29 2025, 5:36 PM

Change #1148491 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki: Remove backwards compatibility path for running php directly

https://gerrit.wikimedia.org/r/1148491

Stashbot added a comment.May 29 2025, 5:44 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-29T17:44:30Z] <swfrench@deploy1003> Started scap sync-world: Clear noop helmfile diffs from gerrit change r/1148491 - T378479

Stashbot added a comment.May 29 2025, 5:47 PM

Mentioned in SAL (#wikimedia-operations) [2025-05-29T17:46:59Z] <swfrench@deploy1003> Finished scap sync-world: Clear noop helmfile diffs from gerrit change r/1148491 - T378479 (duration: 02m 29s)

Scott_French added a comment.May 29 2025, 5:51 PM

All three followup items from T378479#10842481 have now been completed.

Maintenance_bot removed a project: Patch-For-Review.May 29 2025, 6:31 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits