Page MenuHomePhabricator

Get payments wiki working on docker
Open, LowPublic1 Estimated Story Points

Description

DoD
Get Payments wiki working on docker
(added later: in a way that's easily reproducible and useful for all FR-Tech developers as an initial part of a Docker-based development environment)

some notes:
https://etherpad.wikimedia.org/p/fr-tech-docker

Event Timeline

DStrine created this task.Sep 15 2020, 8:05 PM
DStrine removed a project: Epic.
DStrine moved this task from Triage to Current Sprint on the Fundraising-Backlog board.
DStrine set the point value for this task to 1.Sep 15 2020, 8:09 PM
DStrine triaged this task as Medium priority.Sep 16 2020, 3:17 PM
AndyRussG added a project: MediaWiki-Docker.EditedSep 16 2020, 5:35 PM

Hi! Adding the MediaWiki-Docker tag for visibility and coordination with related projects...

General notes:

  • This is part of a larger task to set up a fully functional Docker environment for FR-Tech development. See the parent task, T262971, for details.
  • We'd like to eventually get everything the vagrant fundraising role provides:
    • Mediawiki and extensions used for payments wiki (which developers will clone from git, so not expected to be included in a shared Docker image).
    • A bunch of other non-Mediawiki stuff, including Redis, Civicrm, miscelaneous fundraising tools (see the parent task).
  • Should have the same versions of things as FR production, which currently runs buster and PHP 7.3.
  • Other vagrant roles we currently use include https and mailcatcher.
  • Should not use multiwiki (used for the current vagrant fundraising role).
  • Needs MySQL (could run in a separate container).
  • Needs Xdebug.

For the Mediawiki parts of this (which this task is about) we'd like to coordinate with other work on Docker-based Mediawiki developer setup. In the docker-compose.yml in the MW core repo, I see these two images are used: stretch-php72-fpm-apache2-xdebug:0.5.0 and stretch-php72-jobrunner:0.0.1.

So, as a first step, I was just wondering what the process is for creating and maintaining those images, and how we can coordinate to create and maintain similar images based on buster.

Other things to coordinate on, I think, would be https, MySQL and mailcatcher.

Thanks so much in advance!!

@AndyRussG this patch will provide a PHP 7.3 image, but it's still on Stretch. Are there Buster specific things that you need?

The default environment already has XDebug and DEVELOPERS.md has some notes on configuring it for your setup. For the other things you mention, you could try a docker-compose.override.yml file (place it in the root of the MediaWiki core repo, so it sits along side docker-compose.yml) with these contents:

version: '3.7'
services:
  mail:
    image: mailhog/mailhog
    ports:
      - "8025:8025"
  database:
    image: mariadb
    environment:
        MYSQL_ALLOW_EMPTY_PASSWORD: 1
    volumes:
        - /dbdata:/var/lib/mysql
  redis:
    image: redis
volumes:
   dbdata:
     driver: local

(I provided a setup for MailHog, since I use that, but I assume Mailcatcher would be a similar process if you want to use that. See the docs for a snippet you need to put in your LocalSettings.php also)

Then you'd install MediaWiki with a command like: docker-compose exec mediawiki php maintenance/install.php --server http://localhost:8080 --dbuser=root --dbserver=database --lang en --scriptpath=/w --pass dockerpass mediawiki admin.

Overall the approach would be to try to manage the additional services (Redis, Mail handling, MySQL etc) in the docker-compose.override.yml file, and to maintain documentation for setup and individual commands that need to run, repositories that need to be cloned, etc as a tutorial/guide under the mw:MediaWiki-Docker page (see https://www.mediawiki.org/wiki/MediaWiki-Docker/Extension for a list)

DStrine updated the task description. (Show Details)Sep 28 2020, 8:29 PM

@kostajh thanks so much for this!

I've been poking about in the dev-images repository, learning about how it works, and also learning a lot about Docker... The patch you provided and the code you posted in your previous comment have been a huge help!

I think, for FR-Tech, we may actually need to take a different approach... In summary, FR-Tech has a separate production cluster, with a different version of Mediawiki (though it isn't actually used as a wiki) plus a bunch of other unique services and apps. In addition, we don't have as complete a staging setup as the main production cluster, so keeping developers' local setups closely in line with Fundraising production can be important.

The approach I'd like to try is as follows:

  • Use docker-pkg to create and version Docker images for FR-Tech development setup.
  • Use the dev-images repo for dockerfile and other config needed to create those images.
  • Instead of using existing Mediawiki development images in that repo, have a separate, specialized image (or images) for Fundraising. That way, we can tweak versions and setup as needed to keep things closely in line with the Fundraising cluster, without worrying about how to fit our config in with the more general dev image config.

Another reason for doing it this way would be to avoid re-creating in Docker the unwieldly complex, universally configurable setup we had with Vagrant. In Vagrant, there ended up being too many roles that could interact in many complex ways, and it was hard to figure out what was going on just by looking at the puppet code.

By having a separate Fundraising "corner" in the dev-images repo, we can make setup more linear, and we can still keep conventions generally in sync with those used in the rest of the repo, as they continue to evolve.

Anyway, this is motivation behind the proposed changes I'll send to the dev-images repo. (However, the first change is actually just a generic buster image, similar to the stretch one, and I imagine it could be used in the future for other buster-based environments.)

What do you think?

Thanks so much again!!

P.S. I also have a bunch of questions about why things are done the way they are in existing dev images... I'll send those soon, hope that's ok...!

Change 632171 had a related patch set uploaded (by AndyRussG; owner: AndyRussG):
[releng/dev-images@master] Create base Buster image

https://gerrit.wikimedia.org/r/632171

Change 632173 had a related patch set uploaded (by AndyRussG; owner: AndyRussG):
[releng/dev-images@master] Create fundraising-buster-php73-apache2-xdebug image

https://gerrit.wikimedia.org/r/632173

Here's a WIP repo for setting up Fundraising dev environment.

AndyRussG updated the task description. (Show Details)Tue, Oct 20, 7:17 PM

Change 635361 had a related patch set uploaded (by AndyRussG; owner: AndyRussG):
[releng/dev-images@master] Create buster-rsyslog image

https://gerrit.wikimedia.org/r/635361

Just completed the logging setup for this. (See the updated repo.)

As noted on the etherpad, the following bits and pieces remain for this task:

  • Make tests pass STARTED!
  • Smashpig config under payments
  • Test XDebug
  • License for fundraising-dev repo
  • Templates for content next to form
  • Private data/payment provider keys in LocalSettings.php
  • Mysql Workbench (optional)
AndyRussG added a comment.EditedTue, Oct 27, 7:42 PM

To help support review and retro of this task, here's a summary of what went into it:

  • Learn about Docker in general and docker-compose.
  • Learn about best practices for Docker-based dev environments.
  • Scope out requirements and general approach for a Docker-based FR-Tech development environment. (Some notes on this in the parent epic task.)
  • Learn and evaluate existing WMF tools for creating and versioning Docker images. Decision: We're using the WMF's docker-pkg tool and dev-images repo.
  • Figure out how standard Mediawiki dev images and Docker dev environments work, and decide what to re-use from them. Decision: We'll create FR-specific images and Docker setup, imitating a few things from the standard dev image setup. Details:
    • By creating our FR-specific images, we gain flexibility, simplicity and agility for updating the dev setup.
    • Don't use fpm for PHP, because it's just an extra complication.
    • Don't use a separate job runner image, because Medawiki for Fundraising doesn't need it.
    • Omit PlatformSettings.php used in standard dev images, because it fragments settings, making them harder to find and understand.
    • Copy some elements of Apache config from the standard dev images.
  • Set up Apache:
    • HTTPS with self-signed cert baked into the image
    • PHP
    • Send Apache logs outside the container.
  • Decide how to set up networking inside the application, and look into exposing payments and civi with different hostnames. Decision: It's not simple, so for now they'll just use different ports.
  • Create a workable logging setup. Decision: Run rsyslog in the payments container and send logs to a logger container that exposes them on the host. Details:
    • Messages to syslog are not easily exposable by Docker containers.
    • After digging into our Payments logging setup, we found it's not possible to reconfigure how and where logs our are sent, without changes to the actual code.
    • Figured out how to send messages on UPD port 514 outside the container using iptables, only to find discover that instead of UDP a Unix socket is used.
    • Finally got rsyslogd to run inside Payments with the correct setup, without running it as root.
    • For the logger container, looked into which rsyslog image to use. Decision: We create our own rsyslog image, because it's smaller, more secure, and gives us more control over the result.
    • Set up the logging container to receive messages from other containers and expose them on the host in separate files, based on the source.
  • Decide out how to run Mediawiki's install script, and how to provide our FR-specific LocalSettings.php to the container and the host, and keep it updated across the team.
  • Set up database container, using the same version of mariadb as production. In the setup script (next point) provide an option to fully reset database contents.
  • Write a bash script to automatically set up the dev environment. (This avoids time-consuming and error-prone copying of commands and config file contents, and will help reproduce issues across the team.)
  • Script to run phpunit tests on payments.
  • Wade through numerous gotchas (like the mysterious failure of single quotes in docker-compose.yml and having to wait for the database to be ready).
  • Write documentation for other developers about how to use all this.

Ehhhh I think that's most of it!!!

brennen moved this task from Backlog to Watching on the User-brennen board.
brennen added a subscriber: brennen.
DStrine lowered the priority of this task from Medium to Low.Tue, Oct 27, 9:17 PM