Page MenuHomePhabricator

Run wmfmariadbpy integration test suite on CI
Closed, DeclinedPublic

Description

operations/software/wmfmariadbpy has an integration test suite. We should have it run on CI whenever a patch is proposed.

Event Timeline

Change 621982 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: add percona-toolkit to tox-myxqld

https://gerrit.wikimedia.org/r/621982

Change 621982 merged by jenkins-bot:
[integration/config@master] docker: add percona-toolkit to tox-myxqld

https://gerrit.wikimedia.org/r/621982

Change 621983 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: wmfmariadbpy use image with percona-toolkit

https://gerrit.wikimedia.org/r/621983

Change 621762 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/software/wmfmariadbpy@master] Run integration tests on CI

https://gerrit.wikimedia.org/r/621762

Change 621983 merged by jenkins-bot:
[integration/config@master] jjb: wmfmariadbpy use image with percona-toolkit

https://gerrit.wikimedia.org/r/621983

hashar triaged this task as Medium priority.Aug 24 2020, 8:27 AM
hashar moved this task from Backlog to Repo setup on the Continuous-Integration-Config board.

I guess the bulk of the work will be done in the tox configuration / test suite: https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/621762/

hashar renamed this task from Run wmfmariadbpy on CI to Run wmfmariadbpy integration test suite on CI.Sep 4 2020, 9:22 AM

Change 624622 had a related patch set uploaded (by Hashar; owner: Hashar):
[operations/software/wmfmariadbpy@master] test: TestOnlineSchemaChanger missed analyze config

https://gerrit.wikimedia.org/r/624622

Change 621762 abandoned by Hashar:
[operations/software/wmfmariadbpy@master] Run integration tests on CI

Reason:

https://gerrit.wikimedia.org/r/621762

Change 624622 merged by jenkins-bot:
[operations/software/wmfmariadbpy@master] test: TestOnlineSchemaChanger missed analyze config

https://gerrit.wikimedia.org/r/624622

hashar added a subscriber: jcrespo.

The CI configuration has been changed and the job Docker image no more contains the mysqld or percona-toolkit Debian packages https://gerrit.wikimedia.org/r/c/integration/config/+/662992 (which explicitly goes against this task).

I have proposed a patch to run tox -e integration on CI https://gerrit.wikimedia.org/r/c/operations/software/wmfmariadbpy/+/621762 , though it did not get any review and I apparently abandoned as part of a bulk action. For the record the suite failed with:

09:31:09 Executing 'alter table `osc_test_table` add column osc_test_column int'
09:31:09 Executing 'alter table osc_test_table add column osc_test_column int'
09:31:09 ERROR 1060: Duplicate column name 'osc_test_column'

@Kormat and @jcrespo is there any interest in having CI to run tox -e integration for wmfmariadbpy or should I just discard this task?

@hashar: i've been working on a new integration framework for wmfmariadbpy (T265266: Create integration test env for wmfmariadbpy). As there doesn't seem to be a way to meet our requirements* in CI, we should close this task, and just live with having to run the integration tests manually locally.

Specifically:

  • have it be easy to run the integration tests locally (which effectively implies running the tests against a docker container. that would require docker-in-docker in CI)
  • run against different versions of mariadb (this involves maybe 5GB of compressed mariadb binary tarballs, which there's no good way to handle in CI afaics)

I am very happy with @Kormat proposals and the direction she is taking the repo (dbdeployer, docker), so I really don't have any further input here, I trust her 100%.

Change 954718 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] dockerfiles: remove percona-toolkit from tox-mysqld

https://gerrit.wikimedia.org/r/954718

Change 954718 merged by jenkins-bot:

[integration/config@master] dockerfiles: remove percona-toolkit from tox-mysqld

https://gerrit.wikimedia.org/r/954718