Page MenuHomePhabricator

Create integration test env for wmfmariadbpy
Open, MediumPublic

Description

wmfmariadbpy contains a number of scripts which interact and make changes to clusters of mariadb. This is in desperate need of a way to do integration testing, as otherwise it's very hard to have confidence that changes will actually work correctly in production.

Ideally this integration env is easy to set up on a local workstation, as well as being identical to the CI environment.

Proposal:

  • Use docker-compose to spin up a bunch of mariadb docker containers
  • Have a setup script that configures replication between the containers
  • Run integration tests against this cluster, resetting it between tests

Now that i've looked at dbdeployer, updated proposal:

  • Have a docker image with dbdeployer installed
  • Have "env" scripts to build/run this docker image, and download mysql/mariadb binary tarballs for it.
  • Have integration test scripts that connect to the docker container, use dbdeployer to configure the test env, and then run an integration test against it.

Event Timeline

Kormat created this task.Oct 12 2020, 10:01 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 12 2020, 10:01 AM
LSobanski moved this task from Triage to Ready on the DBA board.
LSobanski added a subscriber: LSobanski.
Aklapper renamed this task from Create intengration test env for wmfmariadbpy to Create integration test env for wmfmariadbpy.Oct 12 2020, 10:19 AM
jbond added a subscriber: jbond.Oct 12 2020, 10:54 AM

From yesterday discussion:

  • we do not run arbitrary images
  • the CI job usually just run an image using Docker, the image itself has the environment and usually an entrypoint which takes care of fetching the patch and invoking the normalized test commands (eg: tox for a python project).
  • we might have a way to run multiple containers in parallel, but that is afaik not in place

That being said, you could start by having the testsuite to create and spin the database processes. I have done something similar with Quibble, the MediaWiki test runner: it initializes an empty data dir, manages a MySQL process and adds grants. The code can be seen at:

https://gerrit.wikimedia.org/g/integration/quibble/+/d73b654fee81384e66124fdac22dbef7b94c7ca1/quibble/backend.py#197

Or in short something such as:

ROOTDIR=$(mktemp -d)
# initialize
mysql_install_db --datadir=$ROOTDIR --user=$USER
# start on a socket
/usr/sbin/mysqld --skip-networking \
  --datadir=$ROOTDIR \
  --log-error=$ROOTDIR/log \
  --pid-file=$ROOTDIR/pid \
  --socket=$ROOTDIR/socket

Which can be accessed via:

mysql --user=root --socket=$ROOTDIR/socket

Surely replace root with whatever has appropriate admin access.

Then, given the mysqld server is available, the testsuite itself can take care of spinning up fresh databases for each test suite / test and have a guarantee it is entirely clean. The CI job uses docker-registry.wikimedia.org/releng/tox-mysqld:0.5.0-s2 which is based on Debian Stretch and does include the packages mysql-server and percona-toolkit.

That might be a good start, we can even have a dedicated job that invokes a specific tox environment such as integration by having the dedicated job to run tox -e integration.

We should then be able to add multiple jobs using different MariaDB versions which can be done by creating a Docker image for each version. That would not let you easily change the version (the images would have to be rebuild whenever the version has to be upgraded), but I imagine we only care about a small set of versions and we could afford that pain.

JMeybohm triaged this task as Medium priority.Oct 13 2020, 10:08 AM

I just came across this: https://github.com/openark/orchestrator/blob/master/docs/ci-env.md#run-orchestrator-with-environment

It seems like it's pretty much exactly what i'm looking for; we might even be able to reuse parts of it.

Change 643693 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/wmfmariadbpy@master] integration: Very early PoC on dockerizing dbdeployer

https://gerrit.wikimedia.org/r/643693

Kormat updated the task description. (Show Details)Nov 26 2020, 10:04 AM
Kormat moved this task from Ready to In progress on the DBA board.

Change 643693 merged by jenkins-bot:
[operations/software/wmfmariadbpy@master] integration: Very early PoC on dockerizing dbdeployer

https://gerrit.wikimedia.org/r/643693

Change 644231 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/wmfmariadbpy@master] test: Start implementation of integration-env

https://gerrit.wikimedia.org/r/644231

Change 644231 merged by jenkins-bot:
[operations/software/wmfmariadbpy@master] test: Start implementation of integration-env

https://gerrit.wikimedia.org/r/644231

Change 646634 had a related patch set uploaded (by Kormat; owner: Kormat):
[operations/software/wmfmariadbpy@master] integration: Complete framework for running basic tests

https://gerrit.wikimedia.org/r/646634

Change 646634 merged by jenkins-bot:
[operations/software/wmfmariadbpy@master] integration: Complete framework for running basic tests

https://gerrit.wikimedia.org/r/646634