Page MenuHomePhabricator
Create Task
Maniphest T217654

Merge blocker: The table 'l10n_cache' is full in quibble-vendor-mysql-hhvm-docker
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Looking at https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/38008/console for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ContentTranslation/+/480896 I see weird test failures.

In the log artifacts I see:
Error: 1114 The table 'l10n_cache' is full (/workspace/db/quibble-mysql-ay4501pw/socket)
Followed by
Error: 2006 MySQL server has gone away (/workspace/db/quibble-mysql-ay4501pw/socket)

See also https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/38008/artifact/log/mw-dberror.log

Impact

  • Unable to merge changes in certain repositories, including mediawiki/extensions/ContentTranslation.

Details

SubjectRepoBranchLines +/-
jjb: raises quibble db tmpfs from 256M to 320Mintegration/configmaster+8 -8
Customize query in gerrit

Related Objects

Event Timeline

Nikerabbit created this task.Mar 5 2019, 1:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 5 2019, 1:39 PM
Nikerabbit mentioned this in T216689: Merge blocker: quibble-vendor-mysql-hhvm-docker in gate fails for most merges (exit status -11).Mar 5 2019, 1:42 PM

"In most cases, ERROR 1114 results from lack of disk space."

Jakob_WMDE subscribed.Mar 5 2019, 3:40 PM
WMDE-leszek subscribed.Mar 5 2019, 4:05 PM
Nikerabbit triaged this task as Unbreak Now! priority.Mar 6 2019, 6:37 AM
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptMar 6 2019, 6:37 AM
KartikMistry subscribed.Mar 6 2019, 6:37 AM
hashar subscribed.Mar 6 2019, 9:26 AM

On first though I checked the disk space usage. From https://grafana-labs.wikimedia.org/dashboard/db/labs-project-board?panelId=18&fullscreen&orgId=1&var-project=integration&var-server=integration-slave-docker-1052&from=now-2d&to=now

slavedocker1052disk.png (665×1 px, 81 KB)

So apparently it had plenty of disk space.

Then I realized the database is on a tmpfs. That is done by running the Quibble container with:

--tmpfs /workspace/db:size=256M --db-dir /workspace/db

So somehow the 256MBytes tmpfs is exhausted.

PHPUnit uses HashBagOStuff. The rest use SqlBagOStuff for the l10n cache. Grepping the logs:

37938/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-24mbjsc4/socket)
37938/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-24mbjsc4/socket)
37943/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-fwkwvea3/socket)
37943/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-fwkwvea3/socket)
38000/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-k2ypv3gd/socket)
38000/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-k2ypv3gd/socket)
38008/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-ay4501pw/socket)
38008/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-ay4501pw/socket)
38013/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-xdtuah3n/socket)
38013/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-xdtuah3n/socket)
38014/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-xdwusvgp/socket)
38014/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-xdwusvgp/socket)
38026/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-fbjscllz/socket)
38026/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-fbjscllz/socket)
38052/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-bqbdkmkx/socket)
38052/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-bqbdkmkx/socket)
38059/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-cmlp1phy/socket)
38059/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-cmlp1phy/socket)
38090/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-ymoft17f/socket)
38090/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-ymoft17f/socket)
38111/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-56ze5z_m/socket)
38111/archive/log/mw-debug-www.log:[DBQuery] SQL ERROR: The table 'l10n_cache' is full (/workspace/db/quibble-mysql-56ze5z_m/socket)

All of those are solely for the ContentTranslation extension. The first build failure was for https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/ContentTranslation/+/494316/ on March 4th. So that is fairly recent.

The job has lot of mediawiki extensions injected, so the issue might be related to another extension.

I don't know whether we always have been at the limit and started to overflow it or whether some recently introduced change causes the l10n cache to suddenly by way too big.

hashar added a comment.Mar 6 2019, 9:41 AM

Grepping for caches being build:

For web requests:

LocalisationCache::isExpired(fr): cache missing, need to make one
LocalisationCache::isExpired(ar): cache missing, need to make one
LocalisationCache::isExpired(jp): cache missing, need to make one
LocalisationCache::isExpired(zh): cache missing, need to make one
LocalisationCache::isExpired(zh-hans): cache missing, need to make one
LocalisationCache::isExpired(nl): cache missing, need to make one
LocalisationCache::isExpired(ml): cache missing, need to make one
LocalisationCache::isExpired(hi): cache missing, need to make one
LocalisationCache::isExpired(en-gb): cache missing, need to make one
LocalisationCache::isExpired(mis): cache missing, need to make one
LocalisationCache::isExpired(ruq): cache missing, need to make one
LocalisationCache::isExpired(ruq-latn): cache missing, need to make one
LocalisationCache::isExpired(ruq-cyrl): cache missing, need to make one

For CLI scripts:

$ egrep '(Start command line script|LocalisationCache::isExpired)' mw-debug-cli.log
Start command line script maintenance/update.php
LocalisationCache::isExpired(en): cache missing, need to make one
Start command line script maintenance/rebuildLocalisationCache.php
LocalisationCache::isExpired(en): cache missing, need to make one
Start command line script tests/phpunit/phpunit.php
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(fr): cache missing, need to make one
LocalisationCache::isExpired(de-ch): cache missing, need to make one
LocalisationCache::isExpired(de): cache missing, need to make one
LocalisationCache::isExpired(ace): cache missing, need to make one
LocalisationCache::isExpired(anp): cache missing, need to make one
LocalisationCache::isExpired(bo): cache missing, need to make one
LocalisationCache::isExpired(fa): cache missing, need to make one
LocalisationCache::isExpired(gan): cache missing, need to make one
LocalisationCache::isExpired(haw): cache missing, need to make one
LocalisationCache::isExpired(krj): cache missing, need to make one
LocalisationCache::isExpired(ln): cache missing, need to make one
LocalisationCache::isExpired(lzh): cache missing, need to make one
LocalisationCache::isExpired(lzz): cache missing, need to make one
LocalisationCache::isExpired(nn): cache missing, need to make one
LocalisationCache::isExpired(pt): cache missing, need to make one
LocalisationCache::isExpired(sma): cache missing, need to make one
LocalisationCache::isExpired(sv): cache missing, need to make one
LocalisationCache::isExpired(ty): cache missing, need to make one
LocalisationCache::isExpired(udm): cache missing, need to make one
LocalisationCache::isExpired(vi): cache missing, need to make one
LocalisationCache::isExpired(zh-hans): cache missing, need to make one
LocalisationCache::isExpired(zh-hant): cache missing, need to make one
LocalisationCache::isExpired(qqx): cache missing, need to make one
LocalisationCache::isExpired(ml): cache missing, need to make one
LocalisationCache::isExpired(he): cache missing, need to make one
LocalisationCache::isExpired(ru): cache missing, need to make one
LocalisationCache::isExpired(arz): cache missing, need to make one
LocalisationCache::isExpired(not-a-real-code): cache missing, need to make one
LocalisationCache::isExpired(kaa): cache missing, need to make one
LocalisationCache::isExpired(pl): cache missing, need to make one
LocalisationCache::isExpired(nl): cache missing, need to make one
LocalisationCache::isExpired(de-formal): cache missing, need to make one
LocalisationCache::isExpired(blargh): cache missing, need to make one
LocalisationCache::isExpired(punja): cache missing, need to make one
LocalisationCache::isExpired(castellano): cache missing, need to make one
LocalisationCache::isExpired(hayeren): cache missing, need to make one
LocalisationCache::isExpired(kartuli): cache missing, need to make one
LocalisationCache::isExpired(qartuli): cache missing, need to make one
LocalisationCache::isExpired(nihongo): cache missing, need to make one
LocalisationCache::isExpired(valencia): cache missing, need to make one
LocalisationCache::isExpired(chinese): cache missing, need to make one
LocalisationCache::isExpired(finish): cache missing, need to make one
LocalisationCache::isExpired(ar): cache missing, need to make one
LocalisationCache::isExpired(df): cache missing, need to make one
LocalisationCache::isExpired(ga): cache missing, need to make one
LocalisationCache::isExpired(es): cache missing, need to make one
LocalisationCache::isExpired(sr): cache missing, need to make one
LocalisationCache::isExpired(it): cache missing, need to make one
LocalisationCache::isExpired(mk): cache missing, need to make one
LocalisationCache::isExpired(ca): cache missing, need to make one
LocalisationCache::isExpired(hu): cache missing, need to make one
LocalisationCache::isExpired(tg): cache missing, need to make one
LocalisationCache::isExpired(nb): cache missing, need to make one
LocalisationCache::isExpired(da): cache missing, need to make one
LocalisationCache::isExpired(be-tarask): cache missing, need to make one
LocalisationCache::isExpired(ast): cache missing, need to make one
LocalisationCache::isExpired(ko): cache missing, need to make one
LocalisationCache::isExpired(el): cache missing, need to make one
LocalisationCache::isExpired(th): cache missing, need to make one
LocalisationCache::isExpired(cs): cache missing, need to make one
LocalisationCache::isExpired(id): cache missing, need to make one
LocalisationCache::isExpired(uk): cache missing, need to make one
LocalisationCache::isExpired(ja): cache missing, need to make one
LocalisationCache::isExpired(ba): cache missing, need to make one
LocalisationCache::isExpired(gl): cache missing, need to make one
LocalisationCache::isExpired(tr): cache missing, need to make one
LocalisationCache::isExpired(be): cache missing, need to make one
LocalisationCache::isExpired(ms): cache missing, need to make one
LocalisationCache::isExpired(lv): cache missing, need to make one
LocalisationCache::isExpired(bn): cache missing, need to make one
LocalisationCache::isExpired(mr): cache missing, need to make one
LocalisationCache::isExpired(eo): cache missing, need to make one
LocalisationCache::isExpired(en-gb): cache missing, need to make one
LocalisationCache::isExpired(de-at): cache missing, need to make one
LocalisationCache::isExpired(tlh): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(qqx): cache missing, need to make one
LocalisationCache::isExpired(la): cache missing, need to make one
LocalisationCache::isExpired(hu): cache missing, need to make one
LocalisationCache::isExpired(kaa): cache missing, need to make one
LocalisationCache::isExpired(pt): cache missing, need to make one
LocalisationCache::isExpired(yue): cache missing, need to make one
LocalisationCache::isExpired(eo): cache missing, need to make one
LocalisationCache::isExpired(de): cache missing, need to make one
LocalisationCache::isExpired(es): cache missing, need to make one
LocalisationCache::isExpired(bo): cache missing, need to make one
LocalisationCache::isExpired(or): cache missing, need to make one
LocalisationCache::isExpired(fr): cache missing, need to make one
LocalisationCache::isExpired(lzh): cache missing, need to make one
LocalisationCache::isExpired(de-formal): cache missing, need to make one
LocalisationCache::isExpired(zh-hant): cache missing, need to make one
LocalisationCache::isExpired(zh-cn): cache missing, need to make one
LocalisationCache::isExpired(zh-tw): cache missing, need to make one
LocalisationCache::isExpired(zh-hk): cache missing, need to make one
LocalisationCache::isExpired(zh-sg): cache missing, need to make one
LocalisationCache::isExpired(zh-mo): cache missing, need to make one
LocalisationCache::isExpired(zh-my): cache missing, need to make one
LocalisationCache::isExpired(ii): cache missing, need to make one
LocalisationCache::isExpired(kk-latn): cache missing, need to make one
LocalisationCache::isExpired(kk-arab): cache missing, need to make one
LocalisationCache::isExpired(kk-kz): cache missing, need to make one
LocalisationCache::isExpired(kk-tr): cache missing, need to make one
LocalisationCache::isExpired(kk-cn): cache missing, need to make one
LocalisationCache::isExpired(unknown): cache missing, need to make one
LocalisationCache::isExpired(nl): cache missing, need to make one
LocalisationCache::isExpired(zh): cache missing, need to make one
LocalisationCache::isExpired(zh-hans): cache missing, need to make one
LocalisationCache::isExpired(kk-cyrl): cache missing, need to make one
LocalisationCache::isExpired(gan-hant): cache missing, need to make one
LocalisationCache::isExpired(it): cache missing, need to make one
LocalisationCache::isExpired(en-gb): cache missing, need to make one
LocalisationCache::isExpired(sr-ec): cache missing, need to make one
LocalisationCache::isExpired(ar): cache missing, need to make one
LocalisationCache::isExpired(nb): cache missing, need to make one
LocalisationCache::isExpired(nn): cache missing, need to make one
LocalisationCache::isExpired(qqq): cache missing, need to make one
LocalisationCache::isExpired(en-ca): cache missing, need to make one
LocalisationCache::isExpired(qug): cache missing, need to make one
LocalisationCache::isExpired(ru): cache missing, need to make one
LocalisationCache::isExpired(he): cache missing, need to make one
LocalisationCache::isExpired(fa): cache missing, need to make one
LocalisationCache::isExpired(arc): cache missing, need to make one
LocalisationCache::isExpired(sv): cache missing, need to make one
LocalisationCache::isExpired(my): cache missing, need to make one
LocalisationCache::isExpired(pi): cache missing, need to make one
LocalisationCache::isExpired(ne): cache missing, need to make one
LocalisationCache::isExpired(kn): cache missing, need to make one
LocalisationCache::isExpired(ace): cache missing, need to make one
LocalisationCache::isExpired(anp): cache missing, need to make one
LocalisationCache::isExpired(gan): cache missing, need to make one
LocalisationCache::isExpired(haw): cache missing, need to make one
LocalisationCache::isExpired(krj): cache missing, need to make one
LocalisationCache::isExpired(ln): cache missing, need to make one
LocalisationCache::isExpired(lzz): cache missing, need to make one
LocalisationCache::isExpired(sma): cache missing, need to make one
LocalisationCache::isExpired(ty): cache missing, need to make one
LocalisationCache::isExpired(udm): cache missing, need to make one
LocalisationCache::isExpired(vi): cache missing, need to make one
LocalisationCache::isExpired(af): cache missing, need to make one
LocalisationCache::isExpired(bar): cache missing, need to make one
LocalisationCache::isExpired(de-at): cache missing, need to make one
LocalisationCache::isExpired(nrm): cache missing, need to make one
LocalisationCache::isExpired(war): cache missing, need to make one
LocalisationCache::isExpired(ceb): cache missing, need to make one
LocalisationCache::isExpired(uk): cache missing, need to make one
LocalisationCache::isExpired(ms): cache missing, need to make one
LocalisationCache::isExpired(ko): cache missing, need to make one
LocalisationCache::isExpired(de-ch): cache missing, need to make one
LocalisationCache::isExpired(some-language): cache missing, need to make one
LocalisationCache::isExpired(ku): cache missing, need to make one
LocalisationCache::isExpired(ku-latn): cache missing, need to make one

Which is a lot of languages to hold :( By grepping for PHPUnitCommand as well:

Start command line script tests/phpunit/phpunit.php
LocalisationCache::isExpired(en): cache missing, need to make one
LocalisationCache::isExpired(fr): cache missing, need to make one
LocalisationCache::isExpired(de-ch): cache missing, need to make one
LocalisationCache::isExpired(de): cache missing, need to make one
LocalisationCache::isExpired(ace): cache missing, need to make one
LocalisationCache::isExpired(anp): cache missing, need to make one
LocalisationCache::isExpired(bo): cache missing, need to make one
LocalisationCache::isExpired(fa): cache missing, need to make one
LocalisationCache::isExpired(gan): cache missing, need to make one
LocalisationCache::isExpired(haw): cache missing, need to make one
LocalisationCache::isExpired(krj): cache missing, need to make one
LocalisationCache::isExpired(ln): cache missing, need to make one
LocalisationCache::isExpired(lzh): cache missing, need to make one
LocalisationCache::isExpired(lzz): cache missing, need to make one
LocalisationCache::isExpired(nn): cache missing, need to make one
LocalisationCache::isExpired(pt): cache missing, need to make one
LocalisationCache::isExpired(sma): cache missing, need to make one
LocalisationCache::isExpired(sv): cache missing, need to make one
LocalisationCache::isExpired(ty): cache missing, need to make one
LocalisationCache::isExpired(udm): cache missing, need to make one
LocalisationCache::isExpired(vi): cache missing, need to make one
LocalisationCache::isExpired(zh-hans): cache missing, need to make one
LocalisationCache::isExpired(zh-hant): cache missing, need to make one
[PHPUnitCommand] Start test WikibaseQuality\ConstraintReport\Tests\Maintenance\ImportConstraintEntitiesTest::testImportEntityFromJson
LocalisationCache::isExpired(es): cache missing, need to make one
LocalisationCache::isExpired(sr): cache missing, need to make one
LocalisationCache::isExpired(it): cache missing, need to make one
LocalisationCache::isExpired(mk): cache missing, need to make one
LocalisationCache::isExpired(ca): cache missing, need to make one
LocalisationCache::isExpired(hu): cache missing, need to make one
LocalisationCache::isExpired(tg): cache missing, need to make one
LocalisationCache::isExpired(nb): cache missing, need to make one
LocalisationCache::isExpired(da): cache missing, need to make one
LocalisationCache::isExpired(be-tarask): cache missing, need to make one
LocalisationCache::isExpired(ast): cache missing, need to make one
LocalisationCache::isExpired(ko): cache missing, need to make one
LocalisationCache::isExpired(el): cache missing, need to make one
LocalisationCache::isExpired(th): cache missing, need to make one
LocalisationCache::isExpired(cs): cache missing, need to make one
LocalisationCache::isExpired(id): cache missing, need to make one
LocalisationCache::isExpired(uk): cache missing, need to make one
LocalisationCache::isExpired(ja): cache missing, need to make one
LocalisationCache::isExpired(ba): cache missing, need to make one
LocalisationCache::isExpired(gl): cache missing, need to make one
LocalisationCache::isExpired(tr): cache missing, need to make one
LocalisationCache::isExpired(be): cache missing, need to make one
LocalisationCache::isExpired(ms): cache missing, need to make one
LocalisationCache::isExpired(lv): cache missing, need to make one
LocalisationCache::isExpired(bn): cache missing, need to make one
LocalisationCache::isExpired(mr): cache missing, need to make one
LocalisationCache::isExpired(eo): cache missing, need to make one
LocalisationCache::isExpired(en-gb): cache missing, need to make one
[PHPUnitCommand] Start test Wikibase\Lib\Tests\LanguageFallbackChainFactoryTest::testNewFromLanguage with data set #8
LocalisationCache::isExpired(zh-hant): cache missing, need to make one
LocalisationCache::isExpired(zh-cn): cache missing, need to make one
LocalisationCache::isExpired(zh-tw): cache missing, need to make one
LocalisationCache::isExpired(zh-hk): cache missing, need to make one
LocalisationCache::isExpired(zh-sg): cache missing, need to make one
LocalisationCache::isExpired(zh-mo): cache missing, need to make one
LocalisationCache::isExpired(zh-my): cache missing, need to make one
hashar added a comment.Mar 6 2019, 9:42 AM

And my next question is: should we just raise the tmpfs size (currently 256MBytes) or look at MediaWiki phpunit suite so that it stops accumulating l10n cache over and over?

Nikerabbit added a comment.Mar 6 2019, 10:44 AM

This seems to be appearing at random tables now: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ContentTranslation/+/486008 – that doesn't mean l10n_cache isn't the driver for it though. ContentTranslation pulls a bunch of extensions as dependencies which make it even bigger. It also wastes a lot of space which is something that I have proposed to reconsider a while ago (cannot find the relevant task right now)

How about configuring LocalisationCache to write to the disk instead of the DB? Would that help anything or just shift the issue?

Krinkle added projects: Wikimedia-production-error (ARCHIVED -- Shared Build Failure), Release-Engineering-Team.Mar 6 2019, 6:06 PM
Krinkle added subscribers: dduvall, greg.
Krinkle edited subscribers, added: Krinkle; removed: greg, dduvall.

(sorry, unintentional.

Krinkle added a comment.Mar 6 2019, 6:07 PM

@Nikerabbit That would solve it, I think. The workspace/src and cache directories afaik are not on tmpfs and thus not RAM-bound. There'd be plenty of space for it there.

Krinkle moved this task from Untriaged to In progress on the Wikimedia-production-error (ARCHIVED -- Shared Build Failure) board.Mar 6 2019, 6:10 PM
Krinkle updated the task description. (Show Details)Mar 8 2019, 12:18 AM
Krinkle updated the task description. (Show Details)
Nikerabbit added a comment.Mar 11 2019, 9:53 AM

I was checking LocalSettings.php from https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/38008/artifact/log/ and it seems it would be sufficient to configure $wgCacheDirectory to a suitable location. wgLocalisationCacheConf is not set which means it will default to CDB files if the cache directory is configured, otherwise the l10n_cache table.

hashar added a comment.Mar 12 2019, 10:55 AM

Sorry I was wrong in my previous comment mentioning PHPUnit high use. It is being run from CLI. Another view is:

  1 cli.log:[caches] LocalisationCacheBulkLoad: using store LCStoreDB
  1 cli.log:[caches] LocalisationCache: using store LCStoreDB
 30 cli.log:[caches] LocalisationCache: using store LCStoreNull
832 www.log:[caches] LocalisationCache: using store LCStoreDB

Quibble invokes rebuildLocalisationCache (cc23bfd7caddf1bb1fe164648f49b9daaad04edb) after running update.php for mediaiwiki and extensions. I did that because update.php does not populate the cache (T22471) and that caused the first web queries to slow down/racing when updating the same l10n table in the database T196347. That is for the context.

Quibble only builds the en cache, which at least solved the issue I encountered with SQLite.

Potentially using a cdb file would be faster to create and read from compared to the database? The appropriate setting ($wLocalisationCacheConf ?) can be set directly into mediawiki/core configuration file includes/DevelopmentSettings.php which is loaded by CI. I guess one can set it up to file to use CDB files. It is worth a try.

I would assume the cache directory would default to $IP/cache.

Krinkle added a comment.EditedMar 12 2019, 5:52 PM

This does not belong in DevelopmentSettings.php. It has no relation to debugging or development. I recommend adding it to quibble/local-settings.php instead, where other CI settings are.

Also, remember this is unbreak now. Lang Eng are unable to merge any patches for over a week now.

• ggellerman subscribed.Mar 12 2019, 6:24 PM
hashar added a comment.Mar 13 2019, 8:35 AM

This does not belong in DevelopmentSettings.php. It has no relation to debugging or development. I recommend adding it to quibble/local-settings.php instead, where other CI settings are.

I will eventually remove those PHP files from Quibble entirely in favor of mediawiki/core include/DevelopmentSettings.php or really any other mechanism developers would favor as long as it is not hardcoded in Quibble or the CI Docker container. Having to cut a new release of Quibble or rebuilding containers for a config change does not really scale well :-]

There are talk about being able to inject configuration dynamically when starting MediaWiki (eg via environment variables). That would probably become a requirement as we shift toward running MediaWiki in containers on Kubernetes.

Hence why I am proposing to tweak the config directly into mediawiki/core.

hashar added a comment.Mar 13 2019, 9:50 AM

Anyway I went to reproduce on my local machine to track down what exactly is going. I ran the tests while watching the database directory growing up. The magic commands:

install -d -m 777 cache
install -d -m 777 workspace

exec docker run \
	--init --rm \
	--volume /home/hashar/projects:/srv/git:ro \
	--volume "$(pwd)/cache:/cache" \
	--volume "$(pwd)/workspace:/workspace" \
	--env ZUUL_PROJECT=mediawiki/extensions/ContentTranslation \
	--env ZUUL_URL=https://gerrit.wikimedia.org/r/p \
	--env ZUUL_BRANCH=master \
	docker-registry.wikimedia.org/releng/quibble-stretch-php72:0.0.28-1 \
	--skip npm-test \
	--packages-source vendor \
	--db mysql \
	--db-dir /workspace/db \
	--dump-db-postrun \
	mediawiki/core mediawiki/skins/Vector mediawiki/vendor mediawiki/extensions/ContentTranslation mediawiki/skins/MinervaNeue mediawiki/extensions/AbuseFilter mediawiki/extensions/AntiSpoof mediawiki/extensions/ArticlePlaceholder mediawiki/extensions/BetaFeatures mediawiki/extensions/Capiunto mediawiki/extensions/CentralAuth mediawiki/extensions/CheckUser mediawiki/extensions/CirrusSearch mediawiki/extensions/Cite mediawiki/extensions/CodeEditor mediawiki/extensions/Echo mediawiki/extensions/EducationProgram mediawiki/extensions/Elastica mediawiki/extensions/EventLogging mediawiki/extensions/GeoData mediawiki/extensions/GuidedTour mediawiki/extensions/JsonConfig mediawiki/extensions/LiquidThreads mediawiki/extensions/MassMessage mediawiki/extensions/MobileApp mediawiki/extensions/MobileFrontend mediawiki/extensions/PdfHandler mediawiki/extensions/PropertySuggester mediawiki/extensions/Renameuser mediawiki/extensions/Scribunto mediawiki/extensions/SiteMatrix mediawiki/extensions/SyntaxHighlight_GeSHi mediawiki/extensions/TemplateData mediawiki/extensions/TimedMediaHandler mediawiki/extensions/TitleBlacklist mediawiki/extensions/UniversalLanguageSelector mediawiki/extensions/UserMerge mediawiki/extensions/VisualEditor mediawiki/extensions/WikiEditor mediawiki/extensions/Wikibase mediawiki/extensions/WikibaseLexeme mediawiki/extensions/WikibaseMediaInfo mediawiki/extensions/WikibaseQualityConstraints mediawiki/extensions/WikimediaBadges mediawiki/extensions/WikimediaEvents mediawiki/extensions/ZeroBanner mediawiki/extensions/ZeroPortal mediawiki/extensions/cldr

mysql first creates a few files when the database is initialized. With MariaDB 10 I got:

mkdir -p /tmp/dummydb
mysql_install_db --datadir=/tmp/dummydb --user="$USER"
$ find /tmp/dummydb/ -size +50k -printf '%k 1kblocks %f\n'
92 1kblocks help_keyword.MYD
96 1kblocks innodb_table_stats.ibd
96 1kblocks innodb_index_stats.ibd
396 1kblocks help_topic.MYD
96 1kblocks gtid_slave_pos.ibd
49152 1kblocks ib_logfile1
49152 1kblocks ib_logfile0
12288 1kblocks ibdata1
$ du /tmp/dummydb/
8	/tmp/dummydb/performance_schema
1092	/tmp/dummydb/mysql
111716	/tmp/dummydb/

So just on initialization, MySQL already consumes almost half of the 256MBytes tmpfs.

Then after install:

db fileSize
ibdata176 MBytes
ib_logfile048 MBytes
ib_logfile148 MBytes
Total172 MBytes

The disk usage consumption then roughly looks like:

`l10n_cache.ibdTotalDescription
12 MB201 MBafter phpunit database less tests, before Selenium
68 MB258 MBafter Selenium tests
gerritbot subscribed.Mar 13 2019, 10:01 AM

Change 496134 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: raises quibble db tmpfs from 256M to 320M

https://gerrit.wikimedia.org/r/496134

gerritbot added a project: Patch-For-Review.Mar 13 2019, 10:01 AM
Stashbot subscribed.Mar 13 2019, 10:03 AM

Mentioned in SAL (#wikimedia-releng) [2019-03-13T10:03:11Z] <hashar> Bump Quibble tmpfs disk space used to hold the database from 256MBytes to 320Mbytes. l10n_cache causes an overflow # T217654

gerritbot added a comment.Mar 13 2019, 10:11 AM

Change 496134 merged by jenkins-bot:
[integration/config@master] jjb: raises quibble db tmpfs from 256M to 320M

https://gerrit.wikimedia.org/r/496134

hashar lowered the priority of this task from Unbreak Now! to High.Mar 13 2019, 10:33 AM

The bump from 256M to 320M must be good enough and I have updated the Jenkins jobs. Lowering priority to High.

Right now we can have up to 5 jobs in parallel, so that is a memory pressure going from 5 * 256MB = 1280 MB to 5 * 320MB = 1600 MB. Or a 320MB increase. Looking at the instances memory usage: that fits right now.

If it is still a problem, we will want to bump the tmpfs size further but that might affect the instances running the jobs.

I have filled T218196: Optimize MySQL settings for MediaWiki CI / Quibble to initialize a smaller database, optimize settings and find the right size for the tmpfs.

Stashbot added a comment.Mar 13 2019, 10:39 AM

Mentioned in SAL (#wikimedia-releng) [2019-03-13T10:39:42Z] <hashar> Triggered tests for ContentTranslation changes that had label:verified=-1 # T217654

hashar claimed this task.Mar 13 2019, 10:46 AM
Restricted Application edited projects, added Release-Engineering-Team (Kanban); removed Release-Engineering-Team. · View Herald TranscriptMar 13 2019, 10:46 AM
hashar closed this task as Resolved.Mar 13 2019, 7:55 PM

Seems good now. The builds are still failing due to T26689: Email all commenters when a followup rev is being committed though.

A follow up action is to tweak mysql settings which is T218196: Optimize MySQL settings for MediaWiki CI / Quibble

Krinkle mentioned this in T218196: Optimize MySQL settings for MediaWiki CI / Quibble.Mar 15 2019, 10:46 PM
Liuxinyu970226 unsubscribed.Mar 16 2019, 1:22 PM
• Petar.petkovic removed a project: Patch-For-Review.Mar 19 2019, 8:07 AM
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits