| Krinkle | |
| Apr 16 2015, 5:16 AM |
| F116040: Screen_Shot_2015-04-17_at_15.22.41.png | |
| Apr 17 2015, 2:43 PM |
| F116042: Screen_Shot_2015-04-17_at_15.24.14.png | |
| Apr 17 2015, 2:43 PM |
After looking into T96229, @ori and @tstarling suggested changing MySQL storage to use tmpfs for /var/lib/mysql.
Tim confirmed that the mw-install-mysql.sh script is I/O limited.
Related:
http://jotschi.de/2014/02/03/high-performance-mysql-testdatabase.html
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| contint: Put mysql db on tmpfs for role::ci::slave::labs | operations/puppet | production | +52 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Krinkle | T96229 Creating MySQL tables for MediaWiki sometimes stalled on I/O for several minutes | |||
| Resolved | Krinkle | T95971 Apache on Jenkins slave takes over 30s to respond (QUnit/AJAX "Test timed out") | |||
| Resolved | Krinkle | T96249 MySQL tuning on CI slaves (tracking) | |||
| Resolved | Krinkle | T96230 Switch MySQL storage to tmpfs | |||
| Resolved | JanZerebecki | T126699 mediawiki jobs fail intermittently with "mw-teardown-mysql.sh: Can't revoke all privileges" |
The disk I/O on labs are not that nice on labs and I think Precise instances have slightly lower I/O capabilities than Trusty ones. Instances runs on different compute nodes which might have different I/O load as well.
I have created T96249 as a tracking task. We should further tune the innodb settings as well.
Change 204528 had a related patch set uploaded (by Krinkle):
contint: Put mysql db on tmpfs for role::ci::slave::labs
Change 204528 had a related patch set uploaded (by Krinkle):
contint: Put mysql db on tmpfs for role::ci::slave::labs
Running slave-scripts/bin/mw-install-mysql.sh and slave-scripts/bin/mw-teardown-mysql.sh alternatingly on a slave with /var/lib/mysql as tmpfs and on another slave without tmpfs did not show any notable difference. I ran it several dozen times. On both nodes it took about 5-10 seconds most times.
Since we can't reproduce the stalling of a minute from T96229 reliably, we'll have to see after deployment whether that stalling was caused by an I/O bottleneck in the mysql datadir. If it's still there, we can try investigating further. Perhaps mysql tmpdir comes into play (which is still disk-bound, defaulting to /tmp).
While installation had little to no difference, test execution did go notably faster (as expected). Using:
php phpunit.php --with-phpunitdir /srv/deployment/integration/phpunit/vendor/phpunit/phpunit --exclude-group Broken,ParserFuzz,Stub includes/PrefixSearchTest.php
integration-slave-trusty-1014 (using regular disk for mysql datadir; depooled; no jobs running)
integration-slave-trusty-1012 (using tmpfs for mysql datadir; depooled; no jobs running)
This was rolled out between 17:20 and 18:00 on 2014-04-16. I've took samples from jobs for MediaWiki core master and wmf branches (e.g. REL1_23 is not comparable). I also excluded builds that ran on the slave currently being used for libeatmydata (T96308).
https://integration.wikimedia.org/ci/job/mediawiki-phpunit-zend/
Before:
After:
https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/
Before:
After:
The trend shows that build times are shorter and more stable (fewer extremes). The arrow indicates the switch. This switch appears further back on the HHVM build graph because those are triggered more often (we only run Zend builds during the gate pipeline).
For Precise/Zend:
Change 204528 merged by Filippo Giunchedi:
contint: Put mysql db on tmpfs for role::ci::slave::labs