Page MenuHomePhabricator

Command-line wrapper for interacting with core's docker-compose stack
Open, MediumPublic

Description

Now that a base docker-compose.yml file exists in core (T238224), I'm interested to explore the idea of a wrapper CLI tool that facilitate working with the docker-compose stack; this could be useful for both novice and expert users.

Other projects (https://github.com/physikerwelt/mediawiki-docker, https://github.com/addshore/mediawiki-docker-dev) have convenience bash scripts to do things like adding a new site, dropping a database and reinstalling (and handling copying over old LocalSettings values) etc. In theory those projects would be able to use a common wrapper CLI tool as well.

Rather than bash scripts I'd propose to use Go (inspired by the wrapper script that local-charts folks created) so we'd have a portable binary that can be installed easily without other dependencies.

As a starting point it would probably be useful for the tool (let's call it for example mw) to allow for things like:

  • mw docker init -- set up a docker-compose.yml.override with an interactive guide (or flags for programmatic usage) to defining database type (sqlite, mysql, postgres) and php version
    • could also create a .env file which can be used by shared components (like fresh-node when running selenium tests)
    • could also prompt to clone mediawiki if run in a directory that doesn't look like a mediawiki directory
  • mw docker install -- installs MediaWiki; prompts to back up and restore LocalSettings.php if one exists
  • mw docker status for showing extensions / skins present on file system and/or if they are loaded via LocalSettings?
  • mw docker get for downloading skins/extensions
  • mw docker enable/disable for adding/removing lines to LocalSettings.php for enabling/disabling skins/extensions
  • mw docker start simple wrapper around docker-compose up
  • mw docker stop simple wrapper around docker-compose stop
  • mw docker destroy brings down the environment, removes any volumes, and deletes the .env file
  • mw docker shell opens a bash shell in the mediawiki service container

Event Timeline

kostajh created this task.Feb 25 2020, 2:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 25 2020, 2:52 PM

@jeena @brennen curious about your thoughts on this (and the others I added as subscribers too).

Perhaps this work could be combined with T99268? This way it could be a "sub command" of sorts like mw docker <command>

Whoops... not sure what happened to that task, it was "RfC: Create a proper command-line runner for MediaWiki maintenance tasks"

This sounds amazing! I really enjoy the CLI tools provided by Addshore's docker wrapper, and adding on ways to manage extensions would be incredible too. I assume it wouldn't mimic vagrant roles, e.g. do everything you need to do to get CentralAuth working? This is totally fine; just a helper to download the extension would still be quite nifty.

Could we also add mw shell to get you into the container itself? The more we dumb it down, the better, in my opinion.

brennen triaged this task as Medium priority.Feb 25 2020, 7:17 PM
brennen added a subscriber: mmodell.

Yeah, I'm interested in this. Based on experience with local-charts (and umpteen other projects), wrapping up common operations in a CLI makes life much easier for everybody. A few thoughts follow.

Go seems fine to me, unless anyone has super strong feelings that MediaWiki developers would rather see something in PHP or Python. @mmodell might want to weigh in on the one he wrote for local-charts.

A pattern I like is the git-style thing with subcommands where, e.g., if mw-foo exists in your path it can be invoked as mw foo, which gives people options for extending things in the language of their choice.

Perhaps this work could be combined with T99268? This way it could be a "sub command" of sorts like mw docker <command>

I'm also curious if there're other needs we should leave room for with this, though it shouldn't get bogged down in trying to be all things to all people.

At any rate it'd be cool to leave open the option of using the same root command for the eventual heavy-weight environment (the kind of thing the Makefile and the Go CLI do for local-charts right now). So mw docker as a base for the docker-compose stuff seems like a good idea.

I like to check if a new utility name steps on anything likely to be installed in the general Linux/Unix CLI namespace. I took a quick look at all the binaries in Debian and it looks pretty safe:

12:12:53 brennen@inertia:/var/lib/apt/lists ✯ grep '^mw' ~/used_names.txt              
mwaw2epub
mwaw2html
mwaw2odf
mwaw2raw
mwaw2text
mwc
mwc-ace
mwcontam
mwfilter
mwhatis
mwic
mwlQuery
mwm
mwrank
mwrap

Whoops... not sure what happened to that task, it was "RfC: Create a proper command-line runner for MediaWiki maintenance tasks"

I think that one is something meant to be executed with PHP, so I could see this tool interacting with that but not necessarily that they would be the same thing.

I assume it wouldn't mimic vagrant roles, e.g. do everything you need to do to get CentralAuth working? This is totally fine; just a helper to download the extension would still be quite nifty.

Yeah, probably not, at least not initially, it sounds quite complicated :)

Could we also add mw shell to get you into the container itself? The more we dumb it down, the better, in my opinion.

Sure!

I'm also curious if there're other needs we should leave room for with this, though it shouldn't get bogged down in trying to be all things to all people.

Yes I'm sure other use cases could come up (e.g. automating the setup and install of ElasticSearch) but starting with something simple would probably be good. FWIW while I've been thinking about this for a while, my motivation in filing the task was from helping a colleague get set up with MediaWiki Docker (success!) and seeing how there are some things we could do with a wrapper script to make the process even easier.

Another thing I thought of is something like mw patch-get {gerrit-url} which would figure out if it's a core or extension/skin patch, then run git review on it, and also a sub command or flag that would allow you to use Quibble in a container to run all the zuul clone commands to get every extension/skin/core at the correct commit for reproducing a build.

At any rate it'd be cool to leave open the option of using the same root command for the eventual heavy-weight environment (the kind of thing the Makefile and the Go CLI do for local-charts right now). So mw docker as a base for the docker-compose stuff seems like a good idea.

Hmm, yes potentially, although I would worry about that adding a lot of complexity. Certainly open to it though!

Go seems fine to me, unless anyone has super strong feelings that MediaWiki developers would rather see something in PHP or Python. @mmodell might want to weigh in on the one he wrote for local-charts.

PHP has advantages in that it would be a lot easier for people to contribute to and a framework like symfony/console is really straightforward to work with. On the other hand it assumes a local PHP install and I would prefer not to do that to make things easier on less technical end users and also for CI environments. Personally I would prefer to avoid Python, I know it's possible to distribute a standalone binary from a python project but Go seems a lot more straightforward for this type of task.

I like to check if a new utility name steps on anything likely to be installed in the general Linux/Unix CLI namespace. I took a quick look at all the binaries in Debian and it looks pretty safe:

Thanks for checking that!

PHP has advantages in that it would be a lot easier for people to contribute to and a framework like symfony/console is really straightforward to work with. On the other hand it assumes a local PHP install and I would prefer not to do that to make things easier on less technical end users and also for CI environments. Personally I would prefer to avoid Python, I know it's possible to distribute a standalone binary from a python project but Go seems a lot more straightforward for this type of task.

All good points and pretty much why we landed on Go for local-charts I think.

At any rate it'd be cool to leave open the option of using the same root command for the eventual heavy-weight environment (the kind of thing the Makefile and the Go CLI do for local-charts right now). So mw docker as a base for the docker-compose stuff seems like a good idea.

Hmm, yes potentially, although I would worry about that adding a lot of complexity. Certainly open to it though!

Yeah, I think I'd just advocate using a subcommand so as to leave the top level namespace open to other logical groupings, but not worrying too much about implementation of anything else at the moment. mw just seems like a great namespace for developer-facing tools of this type.

kostajh updated the task description. (Show Details)Feb 26 2020, 11:30 AM

I started some prototyping work at https://github.com/kostajh/mw, I can post an update here after experimenting a bit more. It doesn't seem like there's a great Go library for interacting with docker-compose but it might not be necessary either; and the golang SDK for docker seems pretty good.

kostajh updated the task description. (Show Details)Feb 26 2020, 2:12 PM
greg awarded a token.Feb 26 2020, 3:59 PM
brennen moved this task from Backlog to Watching on the User-brennen board.Feb 26 2020, 5:51 PM
jeena added a comment.Feb 26 2020, 9:57 PM

My only concern would be scope creep as far as making what was agreed at techconf to be a simple solution in mw/core to something much bigger, but after re-reading I understand this as a separate project that could be used in conjunction with a docker-compose environment.

mw docker status for showing extensions / skins present on file system and/or if they are loaded via LocalSettings?

I wonder if we could come up with a better name for this, like extensions list, or something.

mw docker enable/disable for adding/removing lines to LocalSettings.php for enabling/disabling skins/extensions

Can you elaborate more on this part? How much will it do?

My only concern would be scope creep as far as making what was agreed at techconf to be a simple solution in mw/core to something much bigger, but after re-reading I understand this as a separate project that could be used in conjunction with a docker-compose environment.

yes, exactly :)

Can you elaborate more on this part? How much will it do?

It would just add/remove wfLoad{Extension/Skin}( 'ExtensionOrSkinName' ); to LocalSettings.php. I know that may sound trivial but it's not necessarily the case for everyone.


The prototype so far lets you run mw docker start, which will

  • run docker-compose up
  • check if composer dependencies are installed (if not, it prompts you to do it and runs the command for you)
  • check if the host is Linux, and if so create an override file and export MW_DOCKER_UID/MW_DOCKER_GID to avoid permissions issues

mw docker stop brings the stack down.

Minimum viable release IMO would include the install command; then DEVELOPERS.md could become much shorter and that documentation could get shifted to on-wiki.

kostajh moved this task from Backlog to In progress on the User-kostajh board.Mar 2 2020, 10:28 AM

@kostajh: I'd like to collaborate on this as I already made some progress on a very similar tool in local-charts. I had mostly worked out idempotent command execution which is very useful for this kind of tooling. I think I can port that code over pretty easily.

I think we should create a repo in gerrit for this project and that means we need a name for it ;)

@kostajh: I'd like to collaborate on this as I already made some progress on a very similar tool in local-charts. I had mostly worked out idempotent command execution which is very useful for this kind of tooling. I think I can port that code over pretty easily.

cool, yes I'd love to collaborate on this! We could probably use MediaWiki-Docker as the project, but maybe add #cli or something as a component to organize tasks?

I think we should create a repo in gerrit for this project and that means we need a name for it ;)

I filed a request a few days ago to create mw in gerrit, probably under the releng namespace, but please feel free to update that request.

RHo added a subscriber: RHo.Mar 3 2020, 12:12 PM

@mmodell @brennen so far I've seen a few support requests around $MW_DOCKER_UID and $MW_DOCKER_GID. The existing code in mediawiki-tools-cli sets these values automatically for Linux hosts. Even though the tool is in very early stages maybe it's mature enough for us to ship since mw docker start and mw docker stop work with correctly setting up docker-compose.override.yml and setting those two variables; then we could adjust DEVELOPERS.md to tell people to download this tool. The only piece missing from it currently is mw docker install-mediawiki which shouldn't be difficult to add. Then the "quickstart" section can basically be replaced with mw docker start (it will prompt you to install composer dependencies if they are not downloaded already, and will do it for you) and mw docker install-mediawiki.

How can we distribute the application binary?

@kostajh: I agree we should release early (and release often?). As for distributing the binary, I guess a binary uploaded to https://releases.wikimedia.org along with a signature hash?