There is a sample docker-compose.yml at https://www.mediawiki.org/wiki/Docker/Hub#Using_for_MediaWiki_Development is there anything more that needs to be done here?

Change 550708 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[mediawiki/core@master] Proof-of-concept Docker development environment

https://gerrit.wikimedia.org/r/550708

In T238224#5660541, @dbarratt wrote:

There is a sample docker-compose.yml at https://www.mediawiki.org/wiki/Docker/Hub#Using_for_MediaWiki_Development is there anything more that needs to be done here?

Yes, quite a bit IMO:

• commit to core
• support Redis, Selenium, Parsoid XDebug
• Figure out which image(s) we want to use
• Improve discoverability and documentation
• Document how to support overriding/extending services with the override file
• Dogfooding by at a minimum using this setup with CI

Why would the Dockerfile in core be for development and not production?

Regardless, this seems out of scope for MediaWiki-Docker (which perhaps should be renamed if this gets committed).

Why would the Dockerfile in core be for development and not production?

My understanding is that we're still some distance from having a Dockerfile to use in production. Having a "production-like" image that has some shared settings/packages with what will eventually be used in production seems like a more easily attainable short term goal.

Believe this isn't a CPT thing doesn't belong in MW core should probably be a separate repo. We think this is a RelEng thing but we are happy to provide feedback if needed.

I've been a bit bogged down in note-taking and not reading mail, so I'm coming to this discussion a bit late, but I want to note that there was a strong signal during tech conference discussion that MW core is a good place for a baseline entry point for the development environment, and we (RelEng) are very interested in the advantages of that approach. I think there are a bunch of unanswered questions about the specifics, but I'd rather not see that particular notion die right out of the gate.

For a somewhat different take on this, I hacked together https://github.com/kostajh/mediawiki-dev-env earlier today; the README has some more details there. While there are limitations to PHP's built-in server, its ease of use and performance on macOS / Windows compared to a container approach seems worth exploring.

Change 555914 had a related patch set uploaded (by Kosta Harlan; owner: Kosta Harlan):
[releng/dev-images@master] Add php-fpm/apache image for use with MediaWiki Dev Env

https://gerrit.wikimedia.org/r/555914

@jeena @brennen @hashar @awight @Krinkle something I'm interested in pursuing as an alternative approach to T225218 is to reduce the scope of what Quibble does. Currently it has a Dockerfile that contains all the languages (php, python, nodeJS) and tools (mysql server, selenium, chromedriver, php etc) needed for running all kinds of tests.

IMHO we could reduce complexity and maintenance burden by having Quibble run on the host (ideally with one of the packaging tools for python that can generate a standalone binary), and then quibble would be responsible for issuing a docker-compose up -d with the services needed for a test run (see also T234902), followed by e.g. docker-compose exec app php vendor/bin/phpunit [...] or docker-compose exec selenium npm run selenium-test. If Quibble runs on the host then I think modifying to interact with a docker-compose stack is not terribly difficult.

Even better, Quibble could use the same docker-compose.yml in MediaWiki core that developers use for local development, so reproducing CI failures becomes a bit easier.

In T238224#5701695, @kostajh wrote:

For a somewhat different take on this, I hacked together https://github.com/kostajh/mediawiki-dev-env earlier today; the README has some more details there. While there are limitations to PHP's built-in server, its ease of use and performance on macOS / Windows compared to a container approach seems worth exploring.

I'm ready to give up on this. The single thread limitation on PHP's built-in webserver means, for example, that Parsoid/PHP will be impossible to run.

In T238224#5723042, @kostajh wrote:

I'm ready to give up on this. The single thread limitation on PHP's built-in webserver means, for example, that Parsoid/PHP will be impossible to run.

You could maybe run two containers?

@dbarratt sorry, my comment was too vague. I was referring to the setup specific to that repo which has the idea of running PHP on the host (no Docker) and other services (DB, ElasticSearch, Redis, etc) in containers as needed. And, I'm not giving up on it entirely; I think a workaround would be to run two host-based instances using the same repo and tell instance 1 that the API endpoint is on instance 2, that way we don't end up blocking the loop.

Some more thoughts on this:

I think shipping a basic[1] docker-compose.yml is a nice start, without worrying too much about extensibility. Advanced users could use a docker-compose.override.yml to modify to their needs.

Looking farther out, I think it could be nice to have a CLI tool (let's call it mw for purposes of illustration) one could download for setting up a development or CI environment in a given MediaWiki repository. You could use the tool to mw init a repository with a configuration (.mw.yaml), and choose from a handful of predefined recipes (so, not the a la carte style in MW-Vagrant, which while really cool, can result in breakages due to incompatibilities with ingredients). And at a base level it would be nice if one could specify if PHP should run on the host or in a Docker container, because on the host is so much faster on macOS and also a lot easier to setup XDebug, run various tools, etc. And you could use mw php as a wrapper to execute your php commands in the running Docker container or on the host with a given set of environment variables (for example env variables which define how to connect to an ElasticSearch service, see also T173955 and this experimental code).

Different jobs in CI could then use a given .mw.yaml configuration for deciding which recipe should be used for spinning up the stack.

[1] after thinking about it I think I would define "basic" as a single container with PHP-FPM & Apache with XDebug enabled and a SQLite database

I sometimes wonder if, perhaps it is best to train developers on the basics of docker rather than attempting to abstract it away form them.

For instance, locally I have this Dockerfile:

FROM mediawiki:1.34

# Development Tools
RUN apt-get update && apt-get install -y \
		  sqlite3 \
    --no-install-recommends && rm -r /var/lib/apt/lists/*

# SQLite preferences
RUN { \
      echo '.mode column'; \
      echo '.headers ON'; \
    } > ~/.sqliterc;

# Debugging, but slows everything down.
# RUN pecl install xdebug \
#     && docker-php-ext-enable xdebug

and then I use this docker-compose.yml file:

version: '3.2'
services:
  web:
    build: ./
    ports:
      - 8888:80
    volumes:
      - ./html:/var/www/html:cached
      - database:/var/www/data

  ## Email Testing
  smtp:
    image: sj26/mailcatcher
    ports:
      - 1080:1080
volumes:
  database:

and that works really well on macOS.

In fact, if you don't need XDebug you could simplify it to just using this docker-compose.yml:

version: '3.2'
services:
  web:
    image: mediawiki
    ports:
      - 8888:80
    volumes:
      - ./html:/var/www/html:cached
      - database:/var/www/data
volumes:
  database:

Change 568165 had a related patch set uploaded (by Brennen Bearnes; owner: Brennen Bearnes):
[mediawiki/core@master] blubber.yml: Use renamed base image for dev images

https://gerrit.wikimedia.org/r/568165

Change 568165 merged by jenkins-bot:
[mediawiki/core@master] blubber.yml: Use renamed base image for dev images

https://gerrit.wikimedia.org/r/568165

Change 555914 abandoned by Kosta Harlan:
Add php-fpm/apache image for use with MediaWiki Dev Env

Reason:
due to https://gerrit.wikimedia.org/r/c/releng/dev-images/ /568140

https://gerrit.wikimedia.org/r/555914

I think I misunderstood @kchapman's comment from T238224#5661298. I did not read it as "we are opposed to this being in core". If that's the case, then maybe there could be an RfC process to discuss it.

In the meantime, we could have a separate repo and instruct new developers to use curl/wget to download the docker-compose.yml and supporting scripts into their MediaWiki core directory.

Tentatively retagging per T245167.

Proposed announcement e-mail: https://etherpad.wikimedia.org/p/T238224-announcement

Feel free to edit mercilessly.

In T238224#5899646, @Jdforrester-WMF wrote:

Proposed announcement e-mail: https://etherpad.wikimedia.org/p/T238224-announcement

Feel free to edit mercilessly.

+1, I like it (modulo the use of en_UK pluralization of a team of people :P)

Re the issue of "mw/core or not":

Keeping this in mw/core itself is the goal to ensure that diversions over time do not creep in. That was one of the major issues with other attempts (eg: MW-Vagrant) that caused continual headaches.

Having it here allows us to more easily do (and surface to developers more readily) good hygiene things such as adding CI jobs that run and verify the stability of the environment to catch issues pre-merge that could break the environment for our developers.

This was discussed heavily at TechConf this year and the consensus in the room was to keep whatever-local-dev-tooling within mw/core itself. See, for example T234632, especially the "action items" section at the end of the notes.

Adding this to mw/core is Engineering Productivity's plan.

In T238224#5899646, @Jdforrester-WMF wrote:

Proposed announcement e-mail: https://etherpad.wikimedia.org/p/T238224-announcement

Feel free to edit mercilessly.

Looks good! I made some very light edits.

In T238224#5899732, @greg wrote:

Keeping this in mw/core itself is the goal to ensure that diversions over time do not creep in. That was one of the major issues with other attempts (eg: MW-Vagrant) that caused continual headaches.

Having it here allows us to more easily do (and surface to developers more readily) good hygiene things such as adding CI jobs that run and verify the stability of the environment to catch issues pre-merge that could break the environment for our developers.

A CI job could do that with it being separate, much as it already does for certain extensions on core changes. Manual verification requires that the people manually verifying actually use it, which again doesn't seem to require it in core.

This was discussed heavily at TechConf this year and the consensus in the room was to keep whatever-local-dev-tooling within mw/core itself. See, for example T234632, especially the "action items" section at the end of the notes.

I replied on that task at the time, but I didn't get a convincing response there either.

But I get the sense that further discussion isn't going to get one, so I'm just going to unsubscribe now.

A CI job could do that with it being separate, much as it already does for certain extensions on core changes. Manual verification requires that the people manually verifying actually use it, which again doesn't seem to require it in core.

Yes, that's true, it could be done that way. And for now at least there is no CI job that would verify that the docker-compose config proposed for core actually works (site can be installed, XDebug functions properly, etc).

IMO a good next step here would be to reuse the images referenced in the docker-compose.yml for Quibble (T234902), and to use the proposed default configuration for core as the default Quibble config for jobs which use SQLite as a backend.

Change 550708 merged by jenkins-bot:
[mediawiki/core@master] Add configuration for Docker based development environment

https://gerrit.wikimedia.org/r/550708