Quibble MariaDB should use a tmpfs as a datadir
Closed, ResolvedPublic

Description

Something we have lost in the migration from Nodepool to Quibble is MySQL running off a tmpfs. That dramatically speed up the MediaWiki test suite.

Quibble already drops the data directory at the end of the execution.

Maybe docker run has a way to spawn a tmpfs for us.

hashar created this task.Aug 30 2018, 4:38 PM
hashar added subscribers: thcipriani, dduvall.EditedAug 30 2018, 5:47 PM

We recently started to mount $WORKSPACE/tmp to the CI Docker container as /tmp (done with @dduvall and @thcipriani ). That was to workaround an issue with disk space on the Docker partition.

Quibble spawns MySQL on /tmp and Timo noticed it used to be on tmpfs.

Seems like on Linux Docker as support for a tmpfs volume mount: https://docs.docker.com/storage/tmpfs/ so maybe we can use that? It seems we just need to:

docker run  --tmpfs /tmp
NOTE: that might cause memory issues on the CI Docker slaves.
Krinkle added a subscriber: Krinkle.Sep 1 2018, 1:43 AM

@hashar Thanks for finding this. I'm hoping this can be prioritised given it is one of the factors causing MediaWiki builds to take 20-25min this week instead of the 5-7 minutes we had last month, before the Quibble/Docker migration.

Change 457070 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[integration/config@master] Have quibble jobs mount /tmp as tmpfs

https://gerrit.wikimedia.org/r/457070

From https://gerrit.wikimedia.org/r/#/c/integration/config/+/457070/:

Quibble learned to point the database directory to some specificy place instead of hardcoding it to /tmp . https://gerrit.wikimedia.org/r/#/c/integration/quibble/+/458826/ would let us do something like:

docker run --tmpfs /workspace/db releng/quibble --db-dir /workspace/db

And we will get a tmpfs solely for the database.

I will cut a release of Quibble and update the CI stack whenever I am done with upgrading to Firefox 60 and Chromium 69 T203902

CI now runs out of Quibble 0.0.26 which has --db-dir

Change 457070 abandoned by Hashar:
Have all docker jobs use tmpfs for /tmp

Reason:
This is obsolete, we will instead solely point the database dir to a tmpfs. See T203181 comments

https://gerrit.wikimedia.org/r/457070

Change 461108 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Use a 256MB tmpfs for Quibble database directory

https://gerrit.wikimedia.org/r/461108

hashar closed this task as Resolved.Sep 18 2018, 12:00 PM
hashar claimed this task.

Should be good now, using a 256MB tmpfs.

Change 461108 merged by jenkins-bot:
[integration/config@master] Use a 256MB tmpfs for Quibble database directory

https://gerrit.wikimedia.org/r/461108

I have abandoned the change https://gerrit.wikimedia.org/r/#/c/integration/config/+/457070/ which was passing --tmpfs /tmp to docker run, but forgot to redeploy the jenkins job. That has side effects beside mjolnir such as T204919 which is pycrypto not being installable.

I am refreshing all affected jobs to properly revert:

apps-android-wikipedia-tox-docker
cergen-tox-docker
certcentral-tox-docker
composer-hhvm-docker
composer-package-hhvm-docker
composer-package-php55-docker
composer-package-php70-docker
composer-package-php71-docker
composer-package-php72-docker
composer-php55-docker
composer-php56-docker
composer-php70-docker
conftool-tox-docker
cumin-tox-docker
eventlogging-tox-docker
fundraising-tools-tox-docker
integration-composer-check-php55-docker
integration-config-tox-docker
integration-quibble-tox-publish
labs-striker-tox-docker
labs-tools-heritage-tox-docker
labs-tox-docker
mwgate-composer-hhvm-docker
mwgate-composer-package-hhvm-docker
mwgate-composer-package-php55-docker
mwgate-composer-package-php70-docker
mwgate-composer-php55-docker
mwgate-composer-php70-docker
mwgate-tox-docker
operations-mw-config-composer-test-docker
poolcounter-tox-docker
pywikibot-CommonsDelinker-tox-docker
pywikibot-core-tox-doc-docker
pywikibot-core-tox-docker
pywikibot-core-tox-nose34-docker
pywikibot-core-tox-nose-docker
pywikibot-FLOSSbot-tox-docker
pywikibot-wikibase-tox-docker
pywikibot-xqbot-tox-docker
tox-docker
translatewiki-composer-hhvm-docker
wikiba.se-composer-php56-docker
wikimedia-fundraising-crm-composer-php56-docker
wmf-utils-tox-docker

Mentioned in SAL (#wikimedia-releng) [2018-09-20T11:21:21Z] <hashar> Refreshing jenkins jobs to get rid of docker run option "--tmp /tmpfs" . It is mounted with 'noexec' which causes various jobs to fail. | T203181 and T204919

contint1001$ grep -l '\--tmpfs /tmp' /var/lib/jenkins/jobs/*/config.xml|cut -d\/ -f1
contint1001$