Page MenuHomePhabricator
Create Task
Maniphest T376230

Support bringing text files into the container for one-off maintenance scripts
Closed, ResolvedPublic
Actions

Description

In T341553#10192994, @taavi wrote:

Is it possible to include a text file from disk in the container where a script runs in? Some scripts (like extensions/CentralAuth/maintenance/attachAccount.php) use those for lists of things to process.

We should be able to pull text files from the deployment host (where mwscript-k8s is executed) into a ConfigMap-provided Volume to make them available to the maintenance script.

As a workaround in the meantime for new and updated scripts, instead of reading from a file in the container, read from stdin and use mwscript-k8s --attach to pipe in the contents of the file on the deployment host.

To use scripts like attachAccount.php that take a filename on the command line, without modifying the code, passing --userlist php://stdin should work (not tested).

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
deployment_server: Add --file to mwscript-k8sRLazarusoperations/puppetproduction+55 -0
mediawiki: Support copying text files into mw-script containersRLazarusoperations/deployment-chartsmaster+30 -3
Customize query in gerrit

Related Objects
Search...

Event Timeline

RLazarus created this task.Oct 1 2024, 11:29 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 1 2024, 11:29 PM
RLazarus mentioned this in T341553: Allow running one-off scripts manually.Oct 1 2024, 11:31 PM
Urbanecm_WMF subscribed.Oct 10 2024, 6:36 PM

To use scripts like attachAccount.php that take a filename on the command line, without modifying the code, passing --userlist php://stdin should work (not tested).

I tested it, and I can't get it to work:

[urbanecm@deploy2002 ~]$ cat > users.txt
MediaWiki message delivery
[urbanecm@deploy2002 ~]$ mwscript-k8s --attach extensions/CentralAuth/maintenance/attachAccount.php -- --wiki=zhwiki --userlist=php://stdin < users.txt 
⏳ Starting extensions/CentralAuth/maintenance/attachAccount.php on Kubernetes as job mw-script.codfw.88buxkba ...
⏳ Waiting for the container to start...
🚀 Job is running.
📜 Attached to stdin/stdout:
error: unable to upgrade connection: container mediawiki-88buxkba-app not found in pod mw-script.codfw.88buxkba-h8lh8_mw-script
☠️ Command failed with status 1: /usr/bin/kubectl attach --quiet job/mw-script.codfw.88buxkba --container mediawiki-88buxkba-app -i
For logs (may not work) run:
K8S_CLUSTER=codfw KUBECONFIG=/etc/kubernetes/mw-script-deploy-codfw.config kubectl logs -f job/mw-script.codfw.88buxkba mediawiki-88buxkba-app
[urbanecm@deploy2002 ~]$ K8S_CLUSTER=codfw KUBECONFIG=/etc/kubernetes/mw-script-deploy-codfw.config kubectl logs -f job/mw-script.codfw.88buxkba mediawiki-88buxkba-app
ERROR - File not found: php://stdin
[urbanecm@deploy2002 ~]$

Reading from stdin doesn't seem to work in the old way either.

So for now, this kind of operation seems to be only possible in the old system.

RLazarus added a comment.EditedOct 10 2024, 6:44 PM

Ha, attachAccount.php specifically blocks this from working:

if ( !is_file( $list ) ) {
	$this->fatalError( "ERROR - File not found: {$list}" );
}

So, this specific script will need modification, but only to adjust that check (since is_file insists on a regular file and no other kind of stream). The subsequent fopen and fgets calls ought to work fine with php://stdin.

• bd808 subscribed.Oct 10 2024, 9:30 PM
RLazarus mentioned this in T377497: Functional replacement for importImages.php on Kubernetes.Oct 17 2024, 6:44 PM
gerritbot added a comment.Nov 1 2024, 2:09 AM

Change #1085506 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] mediawiki: Support copying text files into mw-script containers

https://gerrit.wikimedia.org/r/1085506

gerritbot added a project: Patch-For-Review.Nov 1 2024, 2:09 AM

Change #1085507 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] deployment_server: Add --file to mwscript-k8s

https://gerrit.wikimedia.org/r/1085507

gerritbot added a comment.Nov 5 2024, 12:03 AM

Change #1085506 merged by RLazarus:

[operations/deployment-charts@master] mediawiki: Support copying text files into mw-script containers

https://gerrit.wikimedia.org/r/1085506

gerritbot added a comment.Nov 5 2024, 12:11 AM

Change #1085507 merged by RLazarus:

[operations/puppet@production] deployment_server: Add --file to mwscript-k8s

https://gerrit.wikimedia.org/r/1085507

Maintenance_bot removed a project: Patch-For-Review.Nov 5 2024, 12:31 AM
RLazarus closed this task as Resolved.Nov 5 2024, 12:49 AM

This is now supported, and documented at https://wikitech.wikimedia.org/wiki/Maintenance_scripts#Input_from_a_file.

@Urbanecm_WMF If you get a chance to try it out, let me know how it works for you.

Urbanecm_WMF added a comment.Nov 27 2024, 6:14 PM

Seems to work perfectly now, thanks!

[urbanecm@deploy2002 ~]$ cat > users.txt
MediaWiki message delivery
[urbanecm@deploy2002 ~]$ mwscript-k8s --file users.txt -f extensions/CentralAuth/maintenance/attachAccount.php -- --wiki=cswiki --userlist users.txt 
⏳ Starting extensions/CentralAuth/maintenance/attachAccount.php on Kubernetes as job mw-script.codfw.w1hr2123 ...
⏳ Waiting for the container to start...
🚀 Job is running.
📜 Streaming logs:
CentralAuth account attach for: MediaWiki message delivery
ATTACHING: MediaWiki message delivery@cswiki
[2024-11-27 18:13:34] processed: 1 (5.7/sec); ok: 0 (0.0%); attached: 1 (100.0%); partial: 0 (0.0%); failed: 0 (0.0%); missing: 0 (0.0%);
done.
[urbanecm@deploy2002 ~]$
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits