Page MenuHomePhabricator

Port mediawiki-docker-dev "mwdd" v1 cli in to go
Open, Needs TriagePublic

Description

@Addshore plans to build a (partial) mock-up of the mwdd "v1" cli in go, for inclusion in the MediaWiki-Docker cli.

@mmodell will code review and help with filling in some functionality of the commands.

Event Timeline

mmodell created this task.May 21 2020, 2:58 PM
Restricted Application added a project: User-Addshore. · View Herald TranscriptMay 21 2020, 2:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
mmodell updated the task description. (Show Details)
mmodell moved this task from Backlog to In Progress on the MediaWiki-Docker board.
mmodell added a subscriber: kostajh.

Change 597864 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/tools/cli@master] WIP DNM port mwdd v1 to this go repo

https://gerrit.wikimedia.org/r/597864

If I could propose a slightly different take on this (although I might be missing context, sorry for missing the meeting yesterday):

MDD provides, as a default, database replication, PhpMyAdmin, Grafana, Redis, and Nginx proxy, in a (at least somewhat) opinionated way. While I would like to see as an end goal that mw cli could be used to create this type of setup, I don't think I would agree that it should be the default setup. I would love to instead provide an interactive setup tool that can provide generate the relevant docker-compose.override.yml configuration (or reading from multiple pre-defined docker-compose.yml files) depending on what the user wants: pick the PHP version, database engine, if there should be db replication, if there should be redis, etc. If we have a really good configuration setup tool then we can make it easier to for users to mix and match according to what they need. This can also allow us to better support development on lower resourced machines.

I agree that it shouldn't be the default setup too, in the v1 branch of mwdd this is reduced to proxy, dns, mediawiki and db.

depending on what the user wants: pick the PHP version, database engine, if there should be db replication, if there should be redis, etc.

This is again the goal of mwdd v1 and also what is in the WIP go patch. You can for example:

  • mw srv create mw # this will create a proxy, dns, mediawiki and db currently (db could be removed if people really want sqlite, and that should be easily possible)
  • mw srv create redis # creates redis
  • mw src create phpmy admin # creates phpmyadmin

So the system is as you describe, a composition of various different separate compose yml files, rather than a default bloated system with no flex.

One of the things in the call that was discussed is now close or far apart this more advanced setup should be vs not using the CLI tool and just using the docker-compose file in core.
The general consensus was that this should not at all replace the docker-compose file in core, and going that route should be separate.

I agree that it shouldn't be the default setup too, in the v1 branch of mwdd this is reduced to proxy, dns, mediawiki and db.

depending on what the user wants: pick the PHP version, database engine, if there should be db replication, if there should be redis, etc.

This is again the goal of mwdd v1 and also what is in the WIP go patch. You can for example:

  • mw srv create mw # this will create a proxy, dns, mediawiki and db currently (db could be removed if people really want sqlite, and that should be easily possible)
  • mw srv create redis # creates redis
  • mw src create phpmy admin # creates phpmyadmin

So the system is as you describe, a composition of various different separate compose yml files, rather than a default bloated system with no flex.

One of the things in the call that was discussed is now close or far apart this more advanced setup should be vs not using the CLI tool and just using the docker-compose file in core.
The general consensus was that this should not at all replace the docker-compose file in core, and going that route should be separate.

Cool, thanks for clarifying.

  • mw srv create mw # this will create a proxy, dns, mediawiki and db currently (db could be removed if people really want sqlite, and that should be easily possible)

The proxy and DNS are needed for facilitating inter-container network requests?

The proxy and DNS are needed for facilitating inter-container network requests?

So the proxy is to make the user facing interaction nicer, so that the whole environment can be exposed with a single port for HTTP rather than needing to keep a custom port mapping up to date and thus remember .
This also makes all of the dev environments more similar and require less wiring.
Services currently then choose the VHOST that they want to expose themself via.
This does require updating the hosts file of a system to make it all work nicely, but that solution has worked fairly well with mwdd so far and can be made even easier with a better CLI tool
Whenever a new service is added that has a new host that is not in your hosts file the tool can prompt you to add it or add it for you.

As for the DNS, this was mainly added so that mediawiki an other services internal to the docker network can easily use the same vhosts that are in the proxy, without needing to go via the host network or have an extra bridge.
Some of the main use cases of this are for example wikibase requiring API communication with a second mediawiki install. There are of course other things that also need this.
This is another point in favour of using a proxy however, as again, doing this only with the provided "service" dns entries provided by docker-composer and ports would be complex.

Addshore added a comment.EditedMay 22 2020, 2:34 PM

In PS9 of https://gerrit.wikimedia.org/r/c/597864 the bulk of the mwdd v1 system works to some degree, so you can already run it, setup mediawiki and play around
Still a bit to go, but the main picture is now there..

Right now I continued work on the patches that I started a few days ago, which also alter the current "docker" command.

Would everyone prefer if I leave that alone for now and alter this patch to instead introduce a separate "mwdd" command initially?

Addshore renamed this task from Mock up the mediawiki-docker-dev "mwdd" cli in go to Mock up the mediawiki-docker-dev "mwdd " cli in go.May 22 2020, 2:49 PM
Addshore renamed this task from Mock up the mediawiki-docker-dev "mwdd " cli in go to Mock up the mediawiki-docker-dev "mwdd" v1 cli in go.

Would everyone prefer if I leave that alone for now and alter this patch to instead introduce a separate "mwdd" command initially?

It's fine with me if we add subcommands to docker, e.g. mw docker {cmd}. Ideally mw docker would be the overview / help command that lists all the subcommands.

Would everyone prefer if I leave that alone for now and alter this patch to instead introduce a separate "mwdd" command initially?

It's fine with me if we add subcommands to docker, e.g. mw docker {cmd}. Ideally mw docker would be the overview / help command that lists all the subcommands.

I guess the plan would then be to have some commands other than docker at some point? :)
Just checking as commands like this are starting to get a little long mwcli docker service mediawiki start.
But perhaps if interacting with docker services can live under docker then I could remove the service bit

I guess the plan would then be to have some commands other than docker at some point? :)

Right, I think for example the medium/heavyweight local environment (local-charts) may move into this project (e.g. mw local-charts ...) and I could also see adding e.g. a download utility for mw get SomeExtension or extension management mw enable/disable SomeExtension.

I would keep the service part of the subcommand, but for starting up the entire local stack (which may have multiple services) we could probably keep the existing mw docker start and mw docker stop commands, and those can be adjusted internally to call the relevant mw docker service {serviceName} start/stop commands.

I guess the plan would then be to have some commands other than docker at some point? :)

Right, I think for example the medium/heavyweight local environment (local-charts) may move into this project (e.g. mw local-charts ...) and I could also see adding e.g. a download utility for mw get SomeExtension or extension management mw enable/disable SomeExtension.

Gotcha

I would keep the service part of the subcommand, but for starting up the entire local stack (which may have multiple services) we could probably keep the existing mw docker start and mw docker stop commands, and those can be adjusted internally to call the relevant mw docker service {serviceName} start/stop commands.

Would you want to keep the cli being able to interact directly with the thiner mediawiki-docker docker-compose file in core?

Would you want to keep the cli being able to interact directly with the thiner mediawiki-docker docker-compose file in core?

I think that would be nice, but we could also reconsider the presence of the docker-compose.yml file in core once the mw CLI is ready for use. Is the main problem the lack of compatibility between the mediawiki-docker-dev version 2 compose files and core's version 3 file? If so I think we should do the (unfortunately, tedious) work of converting the docker-dev compose files to version 3.

Would you want to keep the cli being able to interact directly with the thiner mediawiki-docker docker-compose file in core?

I think that would be nice, but we could also reconsider the presence of the docker-compose.yml file in core once the mw CLI is ready for use. Is the main problem the lack of compatibility between the mediawiki-docker-dev version 2 compose files and core's version 3 file? If so I think we should do the (unfortunately, tedious) work of converting the docker-dev compose files to version 3.

That is indeed a major factor.
We should also confirm if, for example, 2.2 files work with 2.3 too, or even if 3.0 works with 3.1/
If different docker-compose files need to be the exact same version then having any sort of distributed collection of files won't really work.

There are also some other points, such as looking the services in the file up to internal DNS etc, however I think that could be programmed around by the CLI adding these bits to files that it pulls in.

We should also confirm if, for example, 2.2 files work with 2.3 too, or even if 3.0 works with 3.1/

If one file declares its version as 2.2 and the other says 2.3, docker-compose will refuse to process them together.

If different docker-compose files need to be the exact same version then having any sort of distributed collection of files won't really work.

Right. I think we should avoid this, until / unless there's a compelling case for supporting it.

Change 615478 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/tools/cli@master] Add 'docker ps' command

https://gerrit.wikimedia.org/r/615478

Addshore renamed this task from Mock up the mediawiki-docker-dev "mwdd" v1 cli in go to Port mediawiki-docker-dev "mwdd" v1 cli in to go.Aug 18 2020, 7:31 AM

Change 615478 merged by jenkins-bot:
[mediawiki/tools/cli@master] Add 'docker status' command

https://gerrit.wikimedia.org/r/615478