I'm wondering if this is because of the lack of being in the deployment group and/or because of being in restricted?

I'm in deployment and was able to run the commands as mentioned

There seems to be something wrong with the command line parsing for members of restricted:

cgoubert@deploy1003:/srv/deployment-charts/helmfile.d/services/mw-script$ kubectl describe job mw-script.eqiad.k2ozj7tm
[...]
  Containers:          
   mediawiki-k2ozj7tm-app:           
    Image:      docker-registry.discovery.wmnet/restricted/mediawiki-multiversion-cli:2025-05-28-203847-publish-81
    Port:       <none>                          
    Host Port:  <none>
    Command:   
      /usr/bin/php <--  should be calling /usr/local/bin/foreachwikiindblist
    Args:       
      /srv/mediawiki/multiversion/MWScript.php
      s1                    
      Version.php

compared to members of deployment:

cgoubert@deploy1003:/srv/deployment-charts/helmfile.d/services/mw-script$ kubectl describe job mw-script.eqiad.yhh9i204
[...]
  Containers:                                                                                                  
   mediawiki-yhh9i204-app:
    Image:      docker-registry.discovery.wmnet/restricted/mediawiki-multiversion-cli:2025-05-28-203847-publish-81
    Port:       <none>               
    Host Port:  <none>                                                                                         
    Command:                                    
      /usr/local/bin/foreachwikiindblist
    Args:      
      s2                                                                                                       
      sql.php   
      u4c2025_tables.sql

Many thanks to @jrbs for the discussion out of band.

As @Clement_Goubert notes in T395521#10874927, the configuration of job/mw-script.eqiad.k2ozj7tm looks quite odd:

1. the command should be /usr/local/bin/foreachwikiindblist, not /usr/bin/php
2. the args list, which is otherwise "correct" in terms of what mwscript-k8s should generate in the --dblist case, has been prepended with /srv/mediawiki/multiversion/MWScript.php

I think the most likely way to arrive at something like this would be if using a version of the mediawiki helm chart prior to 0.8.17 (https://gerrit.wikimedia.org/r/1147918) merged May 20th.

How would that happen? The special casing we added for HELM_CACHE_HOME in the restricted-group case in https://gerrit.wikimedia.org/r/1138951. I'm pretty sure the helm cache needs updated on every run rather than only once, lest the user end up using a very stale mediawiki chart.

@jrbs - Could you try deleting ~/.cache/helm and then retrying a simple test like mwscript-k8s --comment="testing" --dblist="s1" --follow -- Version.php?

In T395521#10877806, @Scott_French wrote:

@jrbs - Could you try deleting ~/.cache/helm and then retrying a simple test like mwscript-k8s --comment="testing" --dblist="s1" --follow -- Version.php?

Seems to work:

foks@deploy1003:~$ mwscript-k8s --comment="testing" --dblist="s1" --follow -- Version.php
⏳ Starting Version.php on Kubernetes as job mw-script.eqiad.k74d6fo8 ...
👋 Setting up the Helm cache. Only need to do this the first time...
🚀 Job is running.
📜 Streaming logs:
Version.php: Running on s1
-----------------------------------------------------------------
enwiki
-----------------------------------------------------------------
enwiki MediaWiki version: 1.45.0-wmf.3 (built: 22:09, 26 May 2025)

Change #1152854 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] deployment_server: Update the local helm cache in mwscript-k8s

https://gerrit.wikimedia.org/r/1152854

Amazing - thank you very for confirming, @jrbs. I believe https://gerrit.wikimedia.org/r/1152854 should fix this, and should not pose a significant usability issue (the update operation is actually rather quick).

In T395521#10877883, @Scott_French wrote:

Amazing - thank you very for confirming, @jrbs. I believe https://gerrit.wikimedia.org/r/1152854 should fix this, and should not pose a significant usability issue (the update operation is actually rather quick).

Oh good find, I didn't think cache dir.

Change #1152854 merged by Scott French:

[operations/puppet@production] deployment_server: Update the local helm cache in mwscript-k8s

https://gerrit.wikimedia.org/r/1152854