Page MenuHomePhabricator

Dozens of jobs failing on integration-slave-trusty-1012 because chmod fails for /tmp/jenkins-2
Closed, ResolvedPublic

Description

https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/22440/console
https://integration.wikimedia.org/ci/job/npm/40750/console

13:39:09 chmod: changing permissions of ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-2’: Operation not permitted


13:39:09 + /srv/deployment/integration/slave-scripts/bin/global-teardown.sh
13:39:09 rm: cannot remove ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-2/lessphp_c19h7h8ykkgkggc4cggksw0o0gsw0wk.lesscache’: Permission denied
13:39:09 rm: cannot remove ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-2/lessphp_49yj39cu3xycsgo48wgg04g80wggw0c.lesscache’: Permission denied
13:39:09 rm: cannot remove ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-2/lessphp_oioi2uyu1msgo08s4kgkgwgw4g040k4.lesscache’: Permission denied
13:39:09 rm: cannot remove ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-2/lessphp_c25rq4csau0c4kco0oc0soo00oso0ck.lesscache’: Permission denied
13:39:09 rm: cannot remove ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-2/lessphp_22pcjs2d0dwgccwwk4g840s4oggowcs.lesscache’: Permission denied
13:39:09 Build step 'Execute a set of scripts' marked build as failure

How to fix

On integration-saltmaster as root. Looks for files owned by www-data

salt -v '*slave*' cmd.run 'find /mnt/home/jenkins-deploy/tmpfs -user www-data'

Passing --delete will get rid of them. Since jobs might be running, you might want to limit deletion to files older than 10 minutes with -cmin 10:

salt -v '*slave*' cmd.run 'find /mnt/home/jenkins-deploy/tmpfs -user www-data -mmin +10 -delete'

Details

Event Timeline

Krinkle created this task.Dec 8 2015, 2:30 PM
Krinkle claimed this task.
Krinkle raised the priority of this task from to Needs Triage.
Krinkle updated the task description. (Show Details)
Krinkle added subscribers: Krinkle, hashar.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptDec 8 2015, 2:30 PM
Krinkle closed this task as Resolved.Dec 8 2015, 2:42 PM

Not sure what job left this behind, but I've just sudo rm -rf'ed it manually for now on integration-slave-trusty-1012.

hashar reopened this task as Open.EditedDec 9 2015, 2:07 PM
hashar added subscribers: dcausse, KartikMistry.

Happened again

https://integration.wikimedia.org/ci/job/cxserver-deploy-npm/113/console which ran on integration-slave-trusty-1011

$ ls -ld /mnt/home/jenkins-deploy/tmpfs/jenkins-0
drwxr-xr-x 2 www-data www-data 60 Dec  9 10:47 /mnt/home/jenkins-deploy/tmpfs/jenkins-0
$ ls -l /mnt/home/jenkins-deploy/tmpfs/jenkins-0
total 960
-rw-r--r-- 1 www-data www-data 979017 Dec  9 10:47 l10n_cache-en.cdb

Ran on same slave integration-slave-trusty-1011 https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/22565/console

chmod: changing permissions of ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-0’: Operation not permitted
hashar triaged this task as High priority.Dec 9 2015, 2:11 PM

Fixed with:

ssh integration-slave-trusty-1011.integration.eqiad.wmflabs sudo rm -fR /mnt/home/jenkins-deploy/tmpfs/jenkins-0

The root cause is most probably due to T120356 . The qunit jobs now generate the localization cache under the tmpfs dir. Apparently the dir is not created / chowned to belong to jerkins-bot. I suppose we do not create the dir before MediaWiki is hit, thus the dir ends up belonging to www-data. Then the chmod 777 is done by jenkins-bot and fails.

Cleaned again jenkins-3 on integration-slave-trusty-1016.eqiad.wmflabs.

hashar added a comment.Dec 9 2015, 3:39 PM

So seems mwext-Wikibase-qunit does a few curl requests to the web host before the tmpfs directory had a change to be created by jenkins-bot and chmod 777 happen.

The dir is usually created by global-setup.sh but it is invoked after the curl requests. The build flow from JJB definition:

zuul-cloner
composer
mw-install-sqlite
apply setting functions
update

Then we invoke qunit-karma which does:

  • prepare-localhost
    • source mw-set-env-localhost.sh
      • Inject conf in LocalSettings.php
      • ln -s workspace /srv/localhost/..
      • curl Special:BlankPage
      • curl /load.php?debug=true&modules=startup&only=scripts

At this point the tmpfs dir has not been created / made 777. I believe one of the curl command ends up triggering the LocalizationCache which fallback its cache dir to TMPDIR (which is set) and does a wfMkdirParents(). Thus the dir would belong to www-data

Then the flow continue:

  • some inline shell:
    • source mw-set-env-localhost.sh
    • source npm-setup.sh
      • source global-setup.sh
        • source global-set-env.sh
          • export TMPDIR=/mnt/home/jenkins-deploy/tmpfs/jenkins-3
        • mkdir -p "$TMPDIR"
        • chmod 777 "$TMPDIR"

At this point the command fails because $TMPDIR belongs to www-data

  • grunt karma:main

At the end of the build we invoke:

  • mw-teardown.sh
    • source global-teardown.sh
      • source global-set-env.sh
        • export TMPDIR=/mnt/home/jenkins-deploy/tmpfs/jenkins-3
      • rm -rf "$TMPDIR"

Which would fail as well.

So in short, we need to ensure prepare-localhost calls global-setup.sh before the curl queries are conducted.

If the problem is in mwext-Wikibase-qunit but not in mwext-qunit-composer, the first can just be replaced by the later.

hashar added a comment.Dec 9 2015, 4:53 PM

Seems mwext-Wikibase-qunit has the issue. I haven't looked at mwext-qunit-composer though but it might well suffer from the same issue (i.e. prepare-localhost called before global-setup.sh is invoked to create it).

mwext-Wikibase-qunit has some specific macro to inject Wikibase related settings in LocalSettings.php. But maybe it is no more needed?

Yea all qunit jobs have that problem as they all use the qunit-karma macro.

Krinkle added a comment.EditedDec 9 2015, 5:44 PM

So seems mwext-Wikibase-qunit does a few curl requests to the web host before the tmpfs directory had a change to be created by jenkins-bot and chmod 777 happen.
The dir is usually created by global-setup.sh but it is invoked after the curl requests. The build flow from JJB definition:

That's incorrect. global-setup.sh is already run much earlier as part of mw-install-sqlite. The flow is as follows.

  • wikidata.yaml#mwext-{ext-name}-qunit
    • zuul-cloner-extdeps
    • wd-mw-composer-merged-install
    • mw-install-sqlite. This runs mw-setup.sh which includes global-setup.sh and chmod 777
    • ...
    • qunit-karma

The problem has nothing to do with the macro order. That's been fine and working for over a year without issues. If that were the problem, it also wouldn't be a race condition.

As stated in the opening post, the problem is unrelated to the commands being executed. It is not failing on curl or php, it is failing in the actual script that sets and created the temporary directory.

That script is failing because the directory already exists and contains files from a previous build that used the same job executor number. So there is a regression in our Jenkins jobs that is causing some jobs to not clean up their temporary directory before Jenkins slave starts a new build in the same executor.

Krinkle removed Krinkle as the assignee of this task.Dec 9 2015, 5:53 PM
Krinkle set Security to None.
JanZerebecki added a comment.EditedDec 9 2015, 6:24 PM

outdated comment

Change 258140 had a related patch set uploaded (by JanZerebecki):
Work around race condition in TMPDIR creation/cleanup

https://gerrit.wikimedia.org/r/258140

Even if it is true that post-build runs concurrent with the next build (which I can't reproduce yet), why would that cause chmod problems? The creator of the directory is the same as the one cleaning it up in this case. Shouldn't be a problem.

Change 258140 abandoned by JanZerebecki:
Avoid race condition with EXECUTOR_NUMBER for TMPDIR

https://gerrit.wikimedia.org/r/258140

I misunderstood your comment T120824#1865989 .

I just marked integration-slave-trusty-1011 as offline because all jobs on it were failing.

hashar updated the task description. (Show Details)Dec 11 2015, 10:46 AM

I have put integration-slave-trusty-1011 after deleting all the tmpfs leftover files with:

salt --show-timeout '*slave*' cmd.run 'rm -fR /mnt/home/jenkins-deploy/tmpfs/jenkins-?/*'

Random thoughts from this morning

To follow up on @JanZerebecki , the post build action do block the build and keep the executor busy until completion. So there are never two builds running concurrently on the same executor.

With the hypothesis that our job always have the tmp dir created very early and belonging to jenkins-deploy with mode 777, when the Apache running process creates material in the directory jenkins-deploy would be able to delete them since it own the dir. Eg:

# Create jenkins-deploy  tmp dir
$ sudo -u jenkins-deploy mkdir /tmp/bla
$ sudo -u jenkins-deploy chmod 777 /tmp/bla

# Create a file as Apache user
$ sudo -u www-data touch /tmp/bla/www-data.txt

# Result
$ ls -ld /tmp/bla
drwxrwxrwx 2 jenkins-deploy root 4096 Dec 11 11:50 bla/
$ ls -l /tmp/bla
-rw-r--r-- 1 www-data       www-data 0 Dec 11 11:50 www-data.txt

# Deletion works:
$ rm -v -fR /tmp/bla
removed ‘/tmp/bla/www-data.txt’
removed directory: ‘/tmp/bla/’
$

What we can see is the temp directory belong to www-data. That is why I supposed the curl to Special:Blankpage is creating it and global-setup.sh was missing. But Timo confirmed it is run (the call doesn't show up in the console because the invoking script has set +x).

So under the hypothesis the jobs always create the tmp dir has jenkins-deploy, how do we explain it to be magically owned by www-data?

$ ls -ld /mnt/home/jenkins-deploy/tmpfs/jenkins-0
drwxr-xr-x 2 www-data www-data 60 Dec  9 10:47 /mnt/home/jenkins-deploy/tmpfs/jenkins-0

Timo is right that it all worked fine until now. A change I made recently was to point the LocalizationCache to use the tmpfs directory instead of the shared /tmp (subject to race condition between jobs).

That leads me to suspect the LocalizationCache code chown the cache destination, but there are no chmod() calls in MediaWiki core. The cache does initialize the directory though with:

if ( !file_exists( $this->directory ) ) {
     if ( !wfMkdirParents( $this->directory, null, __METHOD__ ) ) {

On gallium I have looked for mediawiki-* jobs having:

rm: cannot remove ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-1/lessphp_2116pao1z3hcc88400gc04so40ogcc8.lesscache’: Permission denied

I used grep -lR 2116pao1z3hcc8840 /var/lib/jenkins/jobs/mediawiki-* did some sorting:

Dec 11 07:36 mediawiki-extensions-qunit/builds/22834/log
Dec 11 08:27 mediawiki-extensions-qunit/builds/22835/log
Dec 11 09:31 mediawiki-phpunit-hhvm/builds/20611/log
Dec 11 09:48 mediawiki-core-qunit/builds/55051/log
Dec 11 09:59 mediawiki-extensions-qunit/builds/22838/log
Dec 11 10:30 mediawiki-phpunit-hhvm/builds/20614/log

I got the whole history of builds from /var/log/jenkins/jenkins.log and jenkins.log.1:

1Dec 10, 2015 07:04:31 AM integration-slave-trusty-1017_exec-2 scheduling npm build #41264
2Dec 10, 2015 07:16:45 AM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47309
3Dec 10, 2015 07:26:42 AM integration-slave-trusty-1017_exec-2 scheduling jshint build #5130
4Dec 10, 2015 07:39:43 AM integration-slave-trusty-1017_exec-0 scheduling npm build #41269
5Dec 10, 2015 07:48:47 AM integration-slave-trusty-1017_exec-1 scheduling translatewiki-puppetlint-lenient build #2440
6Dec 10, 2015 07:48:57 AM integration-slave-trusty-1017_exec-1 scheduling translatewiki-puppetlint-strict build #2438
7Dec 10, 2015 07:50:57 AM integration-slave-trusty-1017_exec-2 scheduling translatewiki-puppetlint-lenient build #2442
8Dec 10, 2015 07:51:54 AM integration-slave-trusty-1017_exec-1 scheduling translatewiki-puppet-validate build #4721
9Dec 10, 2015 08:25:09 AM integration-slave-trusty-1017_exec-1 scheduling npm build #41272
10Dec 10, 2015 08:41:58 AM integration-slave-trusty-1017_exec-3 scheduling mediawiki-extensions-hhvm build #42431
11Dec 10, 2015 08:44:09 AM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47310
12Dec 10, 2015 09:08:38 AM integration-slave-trusty-1017_exec-2 scheduling npm build #41276
13Dec 10, 2015 09:19:27 AM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-typos build #47315
14Dec 10, 2015 09:32:54 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-qunit build #22715
15Dec 10, 2015 09:40:27 AM integration-slave-trusty-1017_exec-1 scheduling mwext-Wikibase-client-tests-mysql-hhvm build #7035
16Dec 10, 2015 10:06:07 AM integration-slave-trusty-1017_exec-0 scheduling npm build #41281
17Dec 10, 2015 10:46:34 AM integration-slave-trusty-1017_exec-0 scheduling mediawiki-extensions-qunit build #22716
18Dec 10, 2015 11:03:16 AM integration-slave-trusty-1017_exec-2 scheduling npm build #41286
19Dec 10, 2015 11:05:43 AM integration-slave-trusty-1017_exec-0 scheduling mwext-Wikibase-client-tests-sqlite-hhvm build #7033
20Dec 10, 2015 11:21:47 AM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-puppetlint-strict build #34443
21Dec 10, 2015 11:30:02 AM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-client-tests-mysql-hhvm build #7038
22Dec 10, 2015 11:31:23 AM integration-slave-trusty-1017_exec-3 scheduling mediawiki-phpunit-hhvm-composer build #1584
23Dec 10, 2015 11:37:20 AM integration-slave-trusty-1017_exec-3 scheduling mwext-mw-selenium-composer build #255
24Dec 10, 2015 11:40:22 AM integration-slave-trusty-1017_exec-1 scheduling npm build #41294
25Dec 10, 2015 11:51:22 AM integration-slave-trusty-1017_exec-3 scheduling mediawiki-extensions-qunit build #22718
26Dec 10, 2015 11:51:22 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-phpunit-hhvm build #20565
27Dec 10, 2015 11:51:22 AM integration-slave-trusty-1017_exec-0 scheduling mediawiki-core-qunit build #55004
28Dec 10, 2015 11:51:47 AM integration-slave-trusty-1017_exec-1 scheduling npm build #41297
29Dec 10, 2015 12:02:33 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22719
30Dec 10, 2015 12:41:30 PM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-repo-tests-sqlite-hhvm build #7038
31Dec 10, 2015 12:41:59 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-extensions-hhvm build #42437
32Dec 10, 2015 12:42:28 PM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-client-tests-mysql-hhvm build #7041
33Dec 10, 2015 12:43:32 PM integration-slave-trusty-1017_exec-1 scheduling mwext-Wikibase-repo-tests-mysql-hhvm build #7048
34Dec 10, 2015 12:44:19 PM integration-slave-trusty-1017_exec-0 scheduling npm build #41309
35Dec 10, 2015 12:58:08 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47317
36Dec 10, 2015 12:59:25 PM integration-slave-trusty-1017_exec-2 scheduling mwext-qunit build #8895
37Dec 10, 2015 01:09:08 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47319
38Dec 10, 2015 01:17:03 PM integration-slave-trusty-1017_exec-2 scheduling mwext-mw-selenium-composer build #259
39Dec 10, 2015 01:17:03 PM integration-slave-trusty-1017_exec-3 scheduling mwext-Wikibase-client-tests-mysql-hhvm build #7043
40Dec 10, 2015 01:18:46 PM integration-slave-trusty-1017_exec-2 scheduling mwext-CirrusSearch-whitespaces build #832
41Dec 10, 2015 01:25:49 PM integration-slave-trusty-1017_exec-0 scheduling mwext-mw-selenium-composer build #260
42Dec 10, 2015 01:25:49 PM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-client-tests-sqlite-hhvm build #7040
43Dec 10, 2015 01:30:26 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-hhvm build #42439
44Dec 10, 2015 01:31:21 PM integration-slave-trusty-1017_exec-3 scheduling operations-puppet-puppetlint-strict build #34447
45Dec 10, 2015 01:37:53 PM integration-slave-trusty-1017_exec-0 scheduling operations-puppet-typos build #47323
46Dec 10, 2015 01:38:04 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47324
47Dec 10, 2015 01:38:15 PM integration-slave-trusty-1017_exec-3 scheduling operations-puppet-puppetlint-strict build #34452
48Dec 10, 2015 01:45:33 PM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-typos build #47327
49Dec 10, 2015 02:03:45 PM integration-slave-trusty-1017_exec-2 scheduling translatewiki-puppetlint-lenient build #2444
50Dec 10, 2015 02:05:04 PM integration-slave-trusty-1017_exec-3 scheduling operations-puppet-typos build #47329
51Dec 10, 2015 02:31:18 PM integration-slave-trusty-1017_exec-1 scheduling integration-config-puppet-validate build #212
52Dec 10, 2015 02:32:51 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-puppetlint-strict build #34458
53Dec 10, 2015 02:40:18 PM integration-slave-trusty-1017_exec-3 scheduling mwext-qunit-composer build #163
54Dec 10, 2015 02:44:03 PM integration-slave-trusty-1017_exec-2 scheduling mwext-testextension-hhvm-composer build #467
55Dec 10, 2015 02:55:34 PM integration-slave-trusty-1017_exec-1 scheduling wikidata-query-rdf build #753
56Dec 10, 2015 03:02:05 PM integration-slave-trusty-1017_exec-1 scheduling mwext-Wikibase-repo-tests-mysql-hhvm build #7051
57Dec 10, 2015 03:08:20 PM integration-slave-trusty-1017_exec-1 scheduling mwext-Wikibase-repo-tests-sqlite-hhvm build #7044
58Dec 10, 2015 03:09:59 PM integration-slave-trusty-1017_exec-0 scheduling mwext-mw-selenium-composer build #263
59Dec 10, 2015 03:09:59 PM integration-slave-trusty-1017_exec-3 scheduling mwext-Wikibase-repo-tests-sqlite-hhvm build #7045
60Dec 10, 2015 03:12:41 PM integration-slave-trusty-1017_exec-1 scheduling npm build #41321
61Dec 10, 2015 03:12:42 PM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-client-tests-sqlite-hhvm build #7044
62Dec 10, 2015 03:17:52 PM integration-slave-trusty-1017_exec-0 scheduling mwext-mw-selenium-composer build #265
63Dec 10, 2015 03:17:52 PM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-client-tests-mysql-hhvm build #7049
64Dec 10, 2015 03:20:40 PM integration-slave-trusty-1017_exec-3 scheduling npm build #41323
65Dec 10, 2015 03:20:40 PM integration-slave-trusty-1017_exec-0 scheduling mwext-Wikibase-qunit build #15836
66Dec 10, 2015 03:21:17 PM integration-slave-trusty-1017_exec-1 scheduling npm build #41324
67Dec 10, 2015 03:21:17 PM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-repo-tests-sqlite-hhvm build #7049
68Dec 10, 2015 03:25:54 PM integration-slave-trusty-1017_exec-2 scheduling npm build #41325
69Dec 10, 2015 03:25:54 PM integration-slave-trusty-1017_exec-3 scheduling mwext-Wikibase-repo-tests-sqlite-hhvm build #7050
70Dec 10, 2015 03:26:03 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-hhvm build #42440
71Dec 10, 2015 03:30:31 PM integration-slave-trusty-1017_exec-0 scheduling npm build #41327
72Dec 10, 2015 03:31:35 PM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-repo-tests-mysql-hhvm build #7059
73Dec 10, 2015 03:33:47 PM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-typos build #47335
74Dec 10, 2015 03:37:34 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47337
75Dec 10, 2015 03:38:28 PM integration-slave-trusty-1017_exec-3 scheduling mwext-mw-selenium-composer build #270
76Dec 10, 2015 03:42:32 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-gate build #33490
77Dec 10, 2015 03:45:33 PM integration-slave-trusty-1017_exec-2 scheduling wikidata-query-rdf build #754
78Dec 10, 2015 03:50:21 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-hhvm build #42441
79Dec 10, 2015 03:52:56 PM integration-slave-trusty-1017_exec-2 scheduling npm build #41333
80Dec 10, 2015 03:56:42 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-qunit build #22725
81Dec 10, 2015 04:09:35 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22726
82Dec 10, 2015 04:14:11 PM integration-slave-trusty-1017_exec-2 scheduling npm build #41336
83Dec 10, 2015 04:15:22 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47339
84Dec 10, 2015 04:16:25 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-hhvm build #42446
85Dec 10, 2015 04:16:38 PM integration-slave-trusty-1017_exec-1 scheduling jsduck build #3432
86Dec 10, 2015 04:19:08 PM integration-slave-trusty-1017_exec-1 scheduling mwext-qunit build #8897
87Dec 10, 2015 04:21:33 PM integration-slave-trusty-1017_exec-1 scheduling npm build #41340
88Dec 10, 2015 04:22:25 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22733
89Dec 10, 2015 04:22:25 PM integration-slave-trusty-1017_exec-3 scheduling npm build #41341
90Dec 10, 2015 04:23:39 PM integration-slave-trusty-1017_exec-3 scheduling mwext-VisualEditor-jsduck build #2790
91Dec 10, 2015 04:25:58 PM integration-slave-trusty-1017_exec-0 scheduling mediawiki-core-phpcs build #4025
92Dec 10, 2015 04:26:25 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-qunit build #22736
93Dec 10, 2015 04:26:38 PM integration-slave-trusty-1017_exec-3 scheduling mwext-VisualEditor-jsduck build #2791
94Dec 10, 2015 04:27:17 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-hhvm build #42454
95Dec 10, 2015 04:30:33 PM integration-slave-trusty-1017_exec-0 scheduling mwext-Wikibase-repo-tests-mysql-hhvm build #7062
96Dec 10, 2015 04:30:33 PM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-client-tests-sqlite-hhvm build #7052
97Dec 10, 2015 04:33:55 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-hhvm build #42457
98Dec 10, 2015 04:34:05 PM integration-slave-trusty-1017_exec-1 scheduling parsoidsvc-source-roundtrip-test-check build #5398
99Dec 10, 2015 04:36:05 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-extensions-hhvm build #42459
100Dec 10, 2015 04:42:49 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47342
101Dec 10, 2015 04:47:25 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47343
102Dec 10, 2015 04:49:21 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-extensions-hhvm build #42460
103Dec 10, 2015 04:49:33 PM integration-slave-trusty-1017_exec-2 scheduling npm build #41351
104Dec 10, 2015 04:54:43 PM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-typos build #47344
105Dec 10, 2015 05:04:26 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-extensions-qunit build #22744
106Dec 10, 2015 05:08:24 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-hhvm build #42462
107Dec 10, 2015 05:32:45 PM integration-slave-trusty-1017_exec-2 scheduling mwext-qunit build #8905
108Dec 10, 2015 05:37:13 PM integration-slave-trusty-1017_exec-3 scheduling operations-puppet-typos build #47345
109Dec 10, 2015 05:43:14 PM integration-slave-trusty-1017_exec-3 scheduling operations-puppet-typos build #47346
110Dec 10, 2015 05:50:36 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-hhvm build #42465
111Dec 10, 2015 05:52:23 PM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-typos build #47347
112Dec 10, 2015 06:20:03 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47353
113Dec 10, 2015 06:22:50 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-hhvm build #42468
114Dec 10, 2015 06:22:50 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-core-phpcs build #4027
115Dec 10, 2015 06:37:44 PM integration-slave-trusty-1017_exec-1 scheduling translatewiki-puppetlint-lenient build #2445
116Dec 10, 2015 06:46:29 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-extensions-qunit build #22752
117Dec 10, 2015 06:51:53 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-hhvm build #42471
118Dec 10, 2015 06:51:53 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-core-qunit build #55009
119Dec 10, 2015 06:59:10 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-core-jsduck build #12293
120Dec 10, 2015 07:08:52 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-qunit build #22756
121Dec 10, 2015 07:08:52 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-core-phpcs build #4029
122Dec 10, 2015 07:09:00 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-hhvm build #42474
123Dec 10, 2015 07:09:10 PM integration-slave-trusty-1017_exec-0 scheduling mediawiki-phpunit-hhvm build #20573
124Dec 10, 2015 07:16:25 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-core-jsduck-publish build #5189
125Dec 10, 2015 07:18:55 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-core-doxygen-publish build #12788
126Dec 10, 2015 07:31:31 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22765
127Dec 10, 2015 07:35:47 PM integration-slave-trusty-1017_exec-3 scheduling jsduck build #3438
128Dec 10, 2015 07:36:25 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-hhvm build #42484
129Dec 10, 2015 07:36:25 PM integration-slave-trusty-1017_exec-3 scheduling mwext-VisualEditor-npm build #10771
130Dec 10, 2015 07:44:47 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-extensions-hhvm build #42485
131Dec 10, 2015 07:44:47 PM integration-slave-trusty-1017_exec-1 scheduling mwext-VisualEditor-jsduck build #2798
132Dec 10, 2015 07:47:44 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-core-phpcs build #4032
133Dec 10, 2015 07:54:39 PM integration-slave-trusty-1017_exec-3 scheduling parsoidsvc-deploy-npm-0.10 build #398
134Dec 10, 2015 08:05:53 PM integration-slave-trusty-1017_exec-2 scheduling npm build #41368
135Dec 10, 2015 08:06:31 PM integration-slave-trusty-1017_exec-3 scheduling operations-puppet-puppetlint-strict build #34483
136Dec 10, 2015 08:07:17 PM integration-slave-trusty-1017_exec-1 scheduling mwext-VisualEditor-jsduck build #2799
137Dec 10, 2015 08:12:56 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47358
138Dec 10, 2015 08:13:21 PM integration-slave-trusty-1017_exec-0 scheduling mediawiki-extensions-qunit build #22771
139Dec 10, 2015 08:13:51 PM integration-slave-trusty-1017_exec-2 scheduling npm build #41371
140Dec 10, 2015 08:15:24 PM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-puppetlint-strict build #34487
141Dec 10, 2015 08:19:13 PM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-typos build #47362
142Dec 10, 2015 08:19:51 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47363
143Dec 10, 2015 08:23:07 PM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-puppetlint-strict build #34492
144Dec 10, 2015 08:27:40 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47367
145Dec 10, 2015 08:28:29 PM integration-slave-trusty-1017_exec-3 scheduling operations-puppet-typos build #47368
146Dec 10, 2015 08:28:59 PM integration-slave-trusty-1017_exec-3 scheduling operations-puppet-typos build #47369
147Dec 10, 2015 08:29:08 PM integration-slave-trusty-1017_exec-2 scheduling apps-android-wikipedia-test build #1031
148Dec 10, 2015 08:30:39 PM integration-slave-trusty-1017_exec-3 scheduling npm build #41373
149Dec 10, 2015 08:30:50 PM integration-slave-trusty-1017_exec-0 scheduling apps-android-wikipedia-test build #1032
150Dec 10, 2015 08:41:11 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-extensions-qunit build #22772
151Dec 10, 2015 08:47:27 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-gate build #33505
152Dec 10, 2015 08:59:26 PM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-typos build #47370
153Dec 10, 2015 09:07:45 PM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-typos build #47372
154Dec 10, 2015 09:17:51 PM integration-slave-trusty-1017_exec-2 scheduling apps-android-wikipedia-test build #1033
155Dec 10, 2015 09:20:22 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-core-jsduck-publish build #5191
156Dec 10, 2015 09:22:04 PM integration-slave-trusty-1017_exec-3 scheduling operations-puppet-typos build #47375
157Dec 10, 2015 09:33:39 PM integration-slave-trusty-1017_exec-0 scheduling mediawiki-gate build #33506
158Dec 10, 2015 09:34:51 PM integration-slave-trusty-1017_exec-2 scheduling npm build #41382
159Dec 10, 2015 09:37:21 PM integration-slave-trusty-1017_exec-1 scheduling npm build #41384
160Dec 10, 2015 09:39:42 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-extensions-qunit build #22778
161Dec 10, 2015 09:41:50 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-qunit build #22780
162Dec 10, 2015 09:41:56 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22781
163Dec 10, 2015 09:46:55 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-hhvm build #42499
164Dec 10, 2015 09:48:04 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-phpunit-hhvm build #20577
165Dec 10, 2015 09:57:43 PM integration-slave-trusty-1017_exec-2 scheduling mwext-VisualEditor-npm build #10785
166Dec 10, 2015 10:01:33 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-hhvm build #42503
167Dec 10, 2015 10:05:54 PM integration-slave-trusty-1017_exec-1 scheduling mwext-VisualEditor-publish build #874
168Dec 10, 2015 10:06:46 PM integration-slave-trusty-1017_exec-2 scheduling oojs-ui-jsduck-publish build #379
169Dec 10, 2015 10:06:58 PM integration-slave-trusty-1017_exec-3 scheduling mwext-mw-selenium build #2392
170Dec 10, 2015 10:07:16 PM integration-slave-trusty-1017_exec-0 scheduling npm build #41388
171Dec 10, 2015 10:07:58 PM integration-slave-trusty-1017_exec-0 scheduling mwext-Flow-jsduck-publish build #731
172Dec 10, 2015 10:08:29 PM integration-slave-trusty-1017_exec-0 scheduling operations-puppet-typos build #47376
173Dec 10, 2015 10:14:12 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22788
174Dec 10, 2015 10:30:57 PM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-typos build #47378
175Dec 10, 2015 10:42:57 PM integration-slave-trusty-1017_exec-1 scheduling mwext-VisualEditor-npm build #10788
176Dec 10, 2015 10:43:07 PM integration-slave-trusty-1017_exec-2 scheduling mwext-VisualEditor-jsduck build #2810
177Dec 10, 2015 10:45:56 PM integration-slave-trusty-1017_exec-2 scheduling npm build #41391
178Dec 10, 2015 10:52:54 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22792
179Dec 10, 2015 10:52:54 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-phpunit-hhvm build #20580
180Dec 10, 2015 10:53:52 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-hhvm build #42510
181Dec 10, 2015 11:01:07 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22794
182Dec 10, 2015 11:01:07 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-phpunit-hhvm build #20582
183Dec 10, 2015 11:02:08 PM integration-slave-trusty-1017_exec-2 scheduling mwext-VisualEditor-npm build #10790
184Dec 10, 2015 11:03:29 PM integration-slave-trusty-1017_exec-0 scheduling mediawiki-core-qunit build #55022
185Dec 10, 2015 11:07:56 PM integration-slave-trusty-1017_exec-1 scheduling parsoidsvc-source-npm-0.10 build #3882
186Dec 10, 2015 11:09:47 PM integration-slave-trusty-1017_exec-3 scheduling npm build #41397
187Dec 10, 2015 11:10:05 PM integration-slave-trusty-1017_exec-0 scheduling npm build #41398
188Dec 10, 2015 11:10:19 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-core-phpcs build #4038
189Dec 10, 2015 11:13:43 PM integration-slave-trusty-1017_exec-3 scheduling npm build #41400
190Dec 10, 2015 11:14:03 PM integration-slave-trusty-1017_exec-0 scheduling parsoidsvc-source-npm-0.10 build #3883
191Dec 10, 2015 11:15:41 PM integration-slave-trusty-1017_exec-3 scheduling mwext-VisualEditor-npm build #10791
192Dec 10, 2015 11:15:41 PM integration-slave-trusty-1017_exec-2 scheduling mwext-VisualEditor-jsduck build #2812
193Dec 10, 2015 11:15:44 PM integration-slave-trusty-1017_exec-1 scheduling parsoidsvc-source-parse-tool-check build #5409
194Dec 10, 2015 11:17:14 PM integration-slave-trusty-1017_exec-2 scheduling parsoidsvc-source-roundtrip-test-check build #5403
195Dec 10, 2015 11:17:47 PM integration-slave-trusty-1017_exec-3 scheduling npm build #41402
196Dec 10, 2015 11:18:31 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-hhvm build #42518
197Dec 10, 2015 11:18:38 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-extensions-qunit build #22800
198Dec 10, 2015 11:23:31 PM integration-slave-trusty-1017_exec-2 scheduling npm build #41407
199Dec 10, 2015 11:24:12 PM integration-slave-trusty-1017_exec-1 scheduling npm build #41408
200Dec 10, 2015 11:24:54 PM integration-slave-trusty-1017_exec-3 scheduling mwext-qunit build #8910
201Dec 10, 2015 11:25:18 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22803
202Dec 10, 2015 11:25:37 PM integration-slave-trusty-1017_exec-2 scheduling mwext-qunit build #8911
203Dec 10, 2015 11:26:26 PM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22805
204Dec 10, 2015 11:27:59 PM integration-slave-trusty-1017_exec-3 scheduling mwext-qunit build #8913
205Dec 10, 2015 11:28:35 PM integration-slave-trusty-1017_exec-2 scheduling npm build #41415
206Dec 10, 2015 11:32:36 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-qunit build #22808
207Dec 10, 2015 11:35:49 PM integration-slave-trusty-1017_exec-2 scheduling npm build #41420
208Dec 10, 2015 11:45:48 PM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-qunit build #22809
209Dec 10, 2015 11:52:45 PM integration-slave-trusty-1017_exec-3 scheduling npm build #41422
210Dec 11, 2015 12:00:21 AM integration-slave-trusty-1017_exec-3 scheduling mediawiki-core-phpcs build #4045
211Dec 11, 2015 12:00:32 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-phpunit-hhvm-composer build #1592
212Dec 11, 2015 12:02:55 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-gate build #33531
213Dec 11, 2015 12:03:37 AM integration-slave-trusty-1017_exec-0 scheduling npm build #41425
214Dec 11, 2015 12:15:22 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-core-doxygen-publish build #12792
215Dec 11, 2015 12:35:26 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-qunit build #22813
216Dec 11, 2015 12:35:27 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-core-qunit build #55032
217Dec 11, 2015 12:40:46 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-core-qunit build #55033
218Dec 11, 2015 12:42:46 AM integration-slave-trusty-1017_exec-3 scheduling mwext-qunit build #8916
219Dec 11, 2015 12:46:07 AM integration-slave-trusty-1017_exec-3 scheduling mediawiki-core-phpcs build #4049
220Dec 11, 2015 12:46:07 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-phpunit-hhvm-composer build #1593
221Dec 11, 2015 12:55:12 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-hhvm build #42533
222Dec 11, 2015 12:55:12 AM integration-slave-trusty-1017_exec-3 scheduling mediawiki-phpunit-hhvm build #20596
223Dec 11, 2015 01:03:51 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-core-phpcs build #4051
224Dec 11, 2015 01:03:51 AM integration-slave-trusty-1017_exec-3 scheduling mediawiki-core-qunit build #55036
225Dec 11, 2015 01:04:32 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-selenium-integration build #141
226Dec 11, 2015 01:07:30 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-qunit build #22818
227Dec 11, 2015 01:07:31 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22819
228Dec 11, 2015 01:09:58 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-core-phpcs build #4055
229Dec 11, 2015 01:09:58 AM integration-slave-trusty-1017_exec-0 scheduling mediawiki-core-jsduck build #12300
230Dec 11, 2015 01:09:58 AM integration-slave-trusty-1017_exec-3 scheduling mediawiki-core-npm build #8201
231Dec 11, 2015 01:11:10 AM integration-slave-trusty-1017_exec-0 scheduling mediawiki-extensions-hhvm build #42539
232Dec 11, 2015 01:21:01 AM integration-slave-trusty-1017_exec-1 scheduling npm build #41440
233Dec 11, 2015 01:21:18 AM integration-slave-trusty-1017_exec-3 scheduling mediawiki-core-phpcs build #4057
234Dec 11, 2015 01:21:18 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-core-qunit build #55042
235Dec 11, 2015 01:23:02 AM integration-slave-trusty-1017_exec-0 scheduling mediawiki-extensions-hhvm build #42541
236Dec 11, 2015 01:23:02 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-phpunit-hhvm build #20604
237Dec 11, 2015 01:29:05 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-core-jsduck-publish build #5196
238Dec 11, 2015 01:31:01 AM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47385
239Dec 11, 2015 01:41:00 AM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47386
240Dec 11, 2015 01:49:10 AM integration-slave-trusty-1017_exec-2 scheduling mwext-qunit build #8920
241Dec 11, 2015 01:57:15 AM integration-slave-trusty-1017_exec-3 scheduling mediawiki-gate build #33544
242Dec 11, 2015 02:02:17 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-phpunit-hhvm build #20606
243Dec 11, 2015 02:08:52 AM integration-slave-trusty-1017_exec-1 scheduling mwext-qunit build #8922
244Dec 11, 2015 02:08:52 AM integration-slave-trusty-1017_exec-3 scheduling npm build #41445
245Dec 11, 2015 02:30:06 AM integration-slave-trusty-1017_exec-3 scheduling parsoidsvc-source-roundtrip-test-check build #5407
246Dec 11, 2015 02:34:54 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22827
247Dec 11, 2015 02:35:45 AM integration-slave-trusty-1017_exec-3 scheduling parsoidsvc-source-npm-0.10 build #3889
248Dec 11, 2015 02:35:54 AM integration-slave-trusty-1017_exec-2 scheduling parsoidsvc-source-roundtrip-test-check build #5409
249Dec 11, 2015 02:52:09 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-hhvm build #42545
250Dec 11, 2015 03:06:39 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-gate build #33551
251Dec 11, 2015 03:29:54 AM integration-slave-trusty-1017_exec-3 scheduling mwext-VisualEditor-jsduck build #2813
252Dec 11, 2015 03:40:20 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22831
253Dec 11, 2015 03:40:20 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-phpunit-hhvm build #20608
254Dec 11, 2015 04:27:35 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-core-jslint build #13222
255Dec 11, 2015 05:18:42 AM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47390
256Dec 11, 2015 06:01:59 AM integration-slave-trusty-1017_exec-3 scheduling npm build #41455
257Dec 11, 2015 06:19:26 AM integration-slave-trusty-1017_exec-3 scheduling parsoidsvc-source-roundtrip-test-check build #5415
258Dec 11, 2015 06:39:10 AM integration-slave-trusty-1017_exec-2 scheduling parsoidsvc-source-roundtrip-test-check build #5416
259Dec 11, 2015 06:47:35 AM integration-slave-trusty-1017_exec-1 scheduling parsoidsvc-deploy-parse-tool-check build #519
260Dec 11, 2015 06:48:23 AM integration-slave-trusty-1017_exec-0 scheduling mwext-qunit build #8927
261Dec 11, 2015 07:12:05 AM integration-slave-trusty-1017_exec-2 scheduling npm build #41461
262Dec 11, 2015 07:36:11 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-extensions-qunit build #22834
263Dec 11, 2015 08:26:35 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22835
264Dec 11, 2015 08:47:13 AM integration-slave-trusty-1017_exec-1 scheduling npm build #41466
265Dec 11, 2015 09:25:27 AM integration-slave-trusty-1017_exec-2 scheduling npm build #41467
266Dec 11, 2015 09:31:18 AM integration-slave-trusty-1017_exec-0 scheduling mediawiki-phpunit-hhvm build #20611
267Dec 11, 2015 09:32:29 AM integration-slave-trusty-1017_exec-1 scheduling mwext-Wikibase-repo-tests-mysql-hhvm build #7063
268Dec 11, 2015 09:48:14 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-core-qunit build #55051
269Dec 11, 2015 09:59:33 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22838
270Dec 11, 2015 10:00:42 AM integration-slave-trusty-1017_exec-2 scheduling mwext-testextension-hhvm build #795
271Dec 11, 2015 10:29:58 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-phpunit-hhvm build #20614
272Dec 11, 2015 10:38:53 AM integration-slave-trusty-1017_exec-1 scheduling mwext-Wikibase-repo-tests-mysql-hhvm build #7064
273Dec 11, 2015 10:38:53 AM integration-slave-trusty-1017_exec-0 scheduling mwext-Wikibase-qunit build #15844
274Dec 11, 2015 10:42:30 AM integration-slave-trusty-1017_exec-3 scheduling mwext-mw-selenium-composer build #275
275Dec 11, 2015 10:42:30 AM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-qunit build #15845
276Dec 11, 2015 10:46:00 AM integration-slave-trusty-1017_exec-1 scheduling mediawiki-extensions-qunit build #22840
277Dec 11, 2015 10:46:00 AM integration-slave-trusty-1017_exec-2 scheduling mediawiki-core-jsduck build #12306
278Dec 11, 2015 10:46:00 AM integration-slave-trusty-1017_exec-3 scheduling mediawiki-core-npm build #8207
279Dec 11, 2015 10:50:11 AM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-typos build #47392
280Dec 11, 2015 10:56:15 AM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-client-tests-mysql-hhvm build #7060
281Dec 11, 2015 11:11:47 AM integration-slave-trusty-1017_exec-2 scheduling php-composer-test-hhvm build #2666
282Dec 11, 2015 11:12:41 AM integration-slave-trusty-1017_exec-2 scheduling npm build #41475
283Dec 11, 2015 11:13:50 AM integration-slave-trusty-1017_exec-1 scheduling npm build #41476
284Dec 11, 2015 11:13:50 AM integration-slave-trusty-1017_exec-2 scheduling mwext-Wikibase-client-tests-mysql-hhvm build #7061
285Dec 11, 2015 11:15:59 AM integration-slave-trusty-1017_exec-3 scheduling mwext-WikiTwidget-jslint build #6
286Dec 11, 2015 11:50:22 AM integration-slave-trusty-1017_exec-1 scheduling cxserver-deploy-npm build #116
287Dec 11, 2015 12:17:35 PM integration-slave-trusty-1017_exec-3 scheduling mediawiki-core-jslint build #13224
288Dec 11, 2015 12:22:28 PM integration-slave-trusty-1017_exec-1 scheduling operations-puppet-typos build #47394
289Dec 11, 2015 12:23:55 PM integration-slave-trusty-1017_exec-2 scheduling operations-puppet-typos build #47395

Note the _exec# suffix is the Gearman worker, it is most probably different from Jenkins own EXECUTOR_NUMBER.

One of the job behind must have created the tmpdir as www-data.

Based on previous comment forensic, I have looked back at an issue reported earlier:

integration-slave-trusty-1011 Dec 9 10:47 /mnt/home/jenkins-deploy/tmpfs/jenkins-0

https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/42276/ failed with:

10:48:13 chmod: changing permissions of ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-0’: Operation not permitted

https://integration.wikimedia.org/ci/job/mwext-mw-selenium/2375/ has been canceled at 10:47:31 and has some interesting stuff:

It does invoke mw-install-mysql.sh then mw-apply-settings.sh which reports:

10:41:24 Setting $wgTmpDirectory = '/mnt/home/jenkins-deploy/tmpfs/jenkins-0';

Later export TMPDIR=/mnt/home/jenkins-deploy/tmpfs/jenkins-0

So far all good. Then we invoke the selenium test runner which has export SKIP_TMPFS=1 and thus export TMPDIR=/tmp/jenkins-0 and export MW_TMPDIR=/tmp/jenkins-0.

The browser tests are then run. The trick is that LocalSettings.php still has:

$wgTmpDirectory = '/mnt/home/jenkins-deploy/tmpfs/jenkins-0';

And looking at the mw-debug-www.log I got:

0.1305  11.8M  LocalisationCache::recache: got localisation for en from source
0.1749  15.0M  LCStoreCDB::startWrite: called wfMkdirParents(/mnt/home/jenkins-deploy/tmpfs/jenkins-0)
0.2786  17.2M  IP: 127.0.0.1

So it is definitely the LocalisationCache creating the directory via www-data. What I do not get, is why /mnt/home/jenkins-deploy/tmpfs/jenkins-0 is apparently not created previously :(

Change 258450 had a related patch set uploaded (by Hashar):
Point l10n cache to a subdirectory

https://gerrit.wikimedia.org/r/258450

Been running mwext-mw-selenium while watching for files belonging to www-data with:

while true; do date; salt --show-timeout '*slave-trusty-1013*' cmd.run 'find /mnt/home/jenkins-deploy/tmpfs -user www-data -not -name "*lesscache" '; sleep 1; done;

Only yields:

/mnt/home/jenkins-deploy/tmpfs/jenkins-0/l10n_cache-en.cdb

The .lesscache do belong to www-data as well.

So I have been digging into it more and I am out of idea. I can't even think of a good way to detect a job is actually doing the change :-\

Random notes from evening digging, probably not worth reading

Bunch of console logs having rm: cannot remove

$ find mediawiki* mwext-mw-selenium* -type f -name log -exec grep -l 'rm: cannot remove' {} \;
mediawiki-core-jsduck-publish/builds/5190/log
mediawiki-core-npm/builds/8187/log
mediawiki-core-qunit/builds/55051/log
mediawiki-core-qunit/builds/54941/log
mediawiki-extensions-hhvm/builds/42242/log
mediawiki-extensions-hhvm/builds/42285/log
mediawiki-extensions-hhvm/builds/42477/log
mediawiki-extensions-hhvm/builds/42564/log
mediawiki-extensions-hhvm/builds/42276/log
mediawiki-extensions-qunit/builds/22837/log
mediawiki-extensions-qunit/builds/22838/log
mediawiki-extensions-qunit/builds/22844/log
mediawiki-extensions-qunit/builds/22565/log
mediawiki-extensions-qunit/builds/22567/log
mediawiki-extensions-qunit/builds/22849/log
mediawiki-extensions-qunit/builds/22440/log
mediawiki-extensions-qunit/builds/22834/log
mediawiki-extensions-qunit/builds/22835/log
mediawiki-extensions-qunit/builds/22839/log
mediawiki-phpunit-hhvm/builds/20613/log
mediawiki-phpunit-hhvm/builds/20611/log
mediawiki-phpunit-hhvm/builds/20614/log
mediawiki-phpunit-hhvm-composer/builds/1565/log
mwext-mw-selenium-composer/builds/211/log
mwext-mw-selenium-composer/builds/278/log
mwext-mw-selenium-composer/builds/212/log
mwext-mw-selenium-composer/builds/279/log
mwext-mw-selenium-composer/builds/275/log
mwext-mw-selenium-composer/builds/221/log
mwext-mw-selenium-composer/builds/193/log
mwext-mw-selenium-composer/builds/215/log
mwext-mw-selenium-composer/builds/289/log
mwext-mw-selenium-composer/builds/276/log

The earliest record is https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/22440/ which triggered the creation of this task.

14:15:32 chmod: changing permissions of ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-2’: Operation not permitted


Then looking at Wikibase builds with find mwext-Wikibase-* -type f -name log -exec grep -l 'rm: cannot remove' {} \;

mwext-Wikibase-client-tests-mysql-hhvm/builds/6963/log
mwext-Wikibase-client-tests-mysql-hhvm/builds/7067/log
mwext-Wikibase-client-tests-mysql-hhvm/builds/6960/log
mwext-Wikibase-client-tests-mysql-hhvm/builds/6992/log
mwext-Wikibase-client-tests-mysql-hhvm/builds/6989/log
mwext-Wikibase-client-tests-mysql-hhvm/builds/7004/log
mwext-Wikibase-client-tests-mysql-hhvm/builds/7061/log
mwext-Wikibase-client-tests-mysql-hhvm/builds/7062/log
mwext-Wikibase-client-tests-sqlite-hhvm/builds/6997/log
mwext-Wikibase-client-tests-sqlite-hhvm/builds/6449/log
mwext-Wikibase-client-tests-sqlite-hhvm/builds/7069/log
mwext-Wikibase-client-tests-sqlite-hhvm/builds/6993/log
mwext-Wikibase-client-tests-sqlite-hhvm/builds/7054/log
mwext-Wikibase-qunit/builds/15848/log
mwext-Wikibase-qunit/builds/15777/log
mwext-Wikibase-qunit/builds/15775/log
mwext-Wikibase-qunit/builds/15778/log
mwext-Wikibase-repo-tests-mysql-hhvm/builds/7067/log
mwext-Wikibase-repo-tests-mysql-hhvm/builds/7011/log
mwext-Wikibase-repo-tests-mysql-hhvm/builds/7069/log
mwext-Wikibase-repo-tests-mysql-hhvm/builds/7080/log
mwext-Wikibase-repo-tests-mysql-hhvm/builds/7000/log
mwext-Wikibase-repo-tests-mysql-hhvm/builds/6982/log
mwext-Wikibase-repo-tests-mysql-hhvm/builds/7063/log
mwext-Wikibase-repo-tests-mysql-hhvm/builds/7064/log
mwext-Wikibase-repo-tests-mysql-hhvm/builds/7077/log
mwext-Wikibase-repo-tests-mysql-hhvm/builds/6968/log
mwext-Wikibase-repo-tests-sqlite-hhvm/builds/6997/log
mwext-Wikibase-repo-tests-sqlite-hhvm/builds/7067/log
mwext-Wikibase-repo-tests-sqlite-hhvm/builds/6984/log
mwext-Wikibase-repo-tests-sqlite-hhvm/builds/6988/log
mwext-Wikibase-repo-tests-sqlite-hhvm/builds/6959/log
mwext-Wikibase-repo-tests-sqlite-hhvm/builds/7063/log
mwext-Wikibase-repo-tests-sqlite-hhvm/builds/6986/log
mwext-Wikibase-repo-tests-sqlite-hhvm/builds/6972/log
mwext-Wikibase-repo-tests-sqlite-hhvm/builds/6996/log
mwext-Wikibase-repo-tests-sqlite-hhvm/builds/7062/log

The earliest from Dec 8 ~14:00 UTC are:

rw-rw-r-- 1 jenkins jenkins 226916 Dec  8 14:12 mwext-Wikibase-client-tests-mysql-hhvm/builds/6963/log
-rw-rw-r-- 1 jenkins jenkins 236782 Dec  8 14:11 mwext-Wikibase-repo-tests-mysql-hhvm/builds/6968/log
-rw-rw-r-- 1 jenkins jenkins 236722 Dec  8 14:09 mwext-Wikibase-repo-tests-sqlite-hhvm/builds/6959/log
-rw-rw-r-- 1 jenkins jenkins 226981 Dec  8 14:04 mwext-Wikibase-client-tests-mysql-hhvm/builds/6960/log

The last four Wikibase builds ran on integration-slave-trusty-1012 on Dec 8 around 14:00 UTC and all fail during mw-install-mysql.sh with:

chmod: changing permissions of ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-2’: Operation not permitted

Gotta find a build that ran before that on that slave and executor 2 :(

Found the sequence of three jobs running on slave-trusty-1012 executor #2 by grepping jenkins.log

The npm job did global-teardown.sh in PostBuild step properly which suggest there are no tmpfs files belonging to www-data.

The 3rd job (mwext-Wikibase-repo-tests-sqlite-hhvm) dies during build at mw-install-sqlite.sh

14:09:13 chmod: changing permissions of ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-2’: Operation not permitted

That is an entire mystery to me.

Change 258450 abandoned by Hashar:
Point l10n cache to a subdirectory

Reason:
That is probably NOT going to fix it.

https://gerrit.wikimedia.org/r/258450

So far all good. Then we invoke the selenium test runner which has export SKIP_TMPFS=1 and thus export TMPDIR=/tmp/jenkins-0 and export MW_TMPDIR=/tmp/jenkins-0.
The browser tests are then run. The trick is that LocalSettings.php still has:

$wgTmpDirectory = '/mnt/home/jenkins-deploy/tmpfs/jenkins-0';

And looking at the mw-debug-www.log I got:

0.1305  11.8M  LocalisationCache::recache: got localisation for en from source
0.1749  15.0M  LCStoreCDB::startWrite: called wfMkdirParents(/mnt/home/jenkins-deploy/tmpfs/jenkins-0)
0.2786  17.2M  IP: 127.0.0.1

So it is definitely the LocalisationCache creating the directory via www-data. What I do not get, is why /mnt/home/jenkins-deploy/tmpfs/jenkins-0 is apparently not created previously :(

No idea either, but that means there is a bug that one of the directories does not get deleted.

Change 258634 had a related patch set uploaded (by JanZerebecki):
Don't leave one of the TMPDIRs around when it is switched inbetween

https://gerrit.wikimedia.org/r/258634

FWIW, just happened again on integration-slave-trusty-1016, removing the tmpfs/*.lesscache files.

https://integration.wikimedia.org/ci/job/npm/42014/console

Still happening, https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/42958/console for https://gerrit.wikimedia.org/r/#/c/258729/

+ /srv/deployment/integration/slave-scripts/bin/mw-install-mysql.sh
chmod: changing permissions of ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-3’: Operation not permitted
Build step 'Execute shell' marked build as failure
Krinkle raised the priority of this task from High to Unbreak Now!.Dec 15 2015, 2:25 AM

Change 258634 merged by jenkins-bot:
Create both possible TMPDIR so they belong to jenkins-deploy

https://gerrit.wikimedia.org/r/258634

hashar lowered the priority of this task from Unbreak Now! to High.Dec 15 2015, 4:50 PM

I just marked integration-slave-trusty-1012 and integration-slave-trusty-1014 as offline for now due to this.

This issue is still happening despite https://gerrit.wikimedia.org/r/#/c/258634/ :(

Change 260007 had a related patch set uploaded (by Hashar):
Dump stat() for tmp directory

https://gerrit.wikimedia.org/r/260007

Change 260007 merged by jenkins-bot:
Dump stat() for tmp directory

https://gerrit.wikimedia.org/r/260007

Change 260008 had a related patch set uploaded (by Hashar):
Tie mwext-mw-selenium* jobs to specific slaves

https://gerrit.wikimedia.org/r/260008

Change 260008 merged by jenkins-bot:
Tie mwext-mw-selenium* jobs to specific slaves

https://gerrit.wikimedia.org/r/260008

ArticlePlaceholder has mwext-mw-selenium-composer in the experimental pipeline. I crafted dummy change https://gerrit.wikimedia.org/r/#/c/260009/ to be able to trigger via Zuul easily.

Looks like this or something similar is happening again:

[13:17]	RoanKattouw	Hey folks, looks like there are CI issues?
[13:17]	RoanKattouw	Lost of things are getting V-1ed because one of the jobs hits a permissions error
[13:17]	RoanKattouw	e.g. https://integration.wikimedia.org/ci/job/npm/42776/console
[13:17]	RoanKattouw	21:12:56 chmod: changing permissions of ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-0’: Operation not permitted
[13:17]	RoanKattouw	21:12:56 rm: cannot remove ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-0/lessphp_805a13p17e8skww04swww4s0w48w00w.lesscache’: Permission denied
[13:17]	RoanKattouw	21:12:56 rm: cannot remove ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-0/lessphp_h8b1csvw54gs4gwk0o4kgkww044040w.lesscache’: Permission denied
[13:18]	RoanKattouw	or https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/43610/console
[13:18]	RoanKattouw	What's weird is it's not every job all the time, but at least one job for most commits
[13:18]	RoanKattouw	So maybe some servers are broken and others are OK?

I just executed what is explained in the description, so it should be work again for a while.

Roan mentioned https://integration.wikimedia.org/ci/job/npm/42776/ which ran on executor 0 of integration-slave-trusty-1014. Since I earlier tied the mwext-mw-selenium jobs to different slaves, that means they are innocent.

The stat calls on each tmp directory I added show on https://integration.wikimedia.org/ci/job/npm/42776/

21:12:56 +++ stat /mnt/home/jenkins-deploy/tmpfs/jenkins-0
21:12:56   File: ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-0’
21:12:56   Size: 80        	Blocks: 0          IO Block: 4096   directory
21:12:56 Device: 18h/24d	Inode: 313437908   Links: 2
21:12:56 Access: (0755/drwxr-xr-x)  Uid: (   33/www-data)   Gid: (   33/www-data)
21:12:56 Access: 2015-12-18 20:51:37.791584293 +0000
21:12:56 Modify: 2015-12-18 20:51:37.899584276 +0000
21:12:56 Change: 2015-12-18 20:51:37.899584276 +0000
21:12:56  Birth: -
21:12:56 +++ stat /tmp/jenkins-0
21:12:56   File: ‘/tmp/jenkins-0’
21:12:56   Size: 4096      	Blocks: 8          IO Block: 4096   directory
21:12:56 Device: fd01h/64769d	Inode: 5370        Links: 2
21:12:56 Access: (0755/drwxr-xr-x)  Uid: ( 2947/jenkins-deploy)   Gid: (  500/ wikidev)
21:12:56 Access: 2015-12-18 20:52:46.967573517 +0000
21:12:56 Modify: 2015-12-18 20:52:46.967573517 +0000
21:12:56 Change: 2015-12-18 20:52:46.967573517 +0000
21:12:56  Birth: -
21:12:56 +++ mkdir -p /mnt/home/jenkins-deploy/tmpfs/jenkins-0
21:12:56 +++ mkdir -p /tmp/jenkins-0
21:12:56 +++ chmod 777 /mnt/home/jenkins-deploy/tmpfs/jenkins-0
21:12:56 chmod: changing permissions of ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-0’: Operation not permitted

Gotta hunt the job that ran at 20:51:37 UTC.

hashar added a comment.EditedDec 18 2015, 9:50 PM

From gallium /var/log/jenkins/jenkins.log and some manual filtering, here are the job starts on slave-trusty-1014

Note the exec-# does not seem to correspond to the EXECUTOR_NUMBER Jenkins env variable but is assigned by the Gearman plugin.

Time (UTC)Gearman executorJob / BuildEXECUTOR_NUMBER
Dec 18, 2015 19:40:25integration-slave-trusty-1014_exec-3https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/43598
Dec 18, 2015 19:40:34integration-slave-trusty-1014_exec-1https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/43599
Dec 18, 2015 19:45:23integration-slave-trusty-1014_exec-0https://integration.wikimedia.org/ci/job/npm/42763
Dec 18, 2015 19:45:53integration-slave-trusty-1014_exec-3https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/23877
Dec 18, 2015 19:48:48integration-slave-trusty-1014_exec-3https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/43602
Dec 18, 2015 19:48:48integration-slave-trusty-1014_exec-1https://integration.wikimedia.org/ci/job/npm/42765
Dec 18, 2015 19:59:29integration-slave-trusty-1014_exec-0https://integration.wikimedia.org/ci/job/php-composer-test-hhvm/2677
Dec 18, 2015 20:02:53integration-slave-trusty-1014_exec-0https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/23879
Dec 18, 2015 20:02:53integration-slave-trusty-1014_exec-3https://integration.wikimedia.org/ci/job/mediawiki-core-phpcs/4446
Dec 18, 2015 20:06:55integration-slave-trusty-1014_exec-3https://integration.wikimedia.org/ci/job/mediawiki-core-qunit/55630
Dec 18, 2015 20:08:50integration-slave-trusty-1014_exec-2https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-publish/12952
Dec 18, 2015 20:23:47integration-slave-trusty-1014_exec-3https://integration.wikimedia.org/ci/job/mwext-qunit/9205
Dec 18, 2015 20:31:16integration-slave-trusty-1014_exec-0https://integration.wikimedia.org/ci/job/apps-android-wikipedia-test/1146
Dec 18, 2015 20:44:20integration-slave-trusty-1014_exec-0https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/238820 (ok)
Dec 18, 2015 20:50:03integration-slave-trusty-1014_exec-3https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/238830 (canceled by zuul)
Dec 18, 2015 20:50:03integration-slave-trusty-1014_exec-0https://integration.wikimedia.org/ci/job/mediawiki-phpunit-hhvm/211423 (canceled by zuul)
Dec 18, 2015 20:52:04integration-slave-trusty-1014_exec-0https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/436080 (fail)
Dec 18, 2015 20:52:04integration-slave-trusty-1014_exec-1https://integration.wikimedia.org/ci/job/mediawiki-core-phpcs/4449no tmpfs
Dec 18, 2015 20:58:44integration-slave-trusty-1014_exec-0https://integration.wikimedia.org/ci/job/operations-puppet-puppetlint-strict/35195no tmpfs
Dec 18, 2015 21:00:37integration-slave-trusty-1014_exec-3https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/238853 (ok)
Dec 18, 2015 21:01:00integration-slave-trusty-1014_exec-0https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/436100 (fail)
Dec 18, 2015 21:05:32integration-slave-trusty-1014_exec-3https://integration.wikimedia.org/ci/job/operations-puppet-puppetlint-strict/35196no tmpfs
Dec 18, 2015 21:07:16integration-slave-trusty-1014_exec-3https://integration.wikimedia.org/ci/job/operations-puppet-puppetlint-strict/35197no tmpfs
Dec 18, 2015 21:12:53integration-slave-trusty-1014_exec-2https://integration.wikimedia.org/ci/job/npm/427760 (fail)
Dec 18, 2015 21:17:34integration-slave-trusty-1014_exec-0https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/436140 (fail)

So the build order show:

https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/23883 which got cancelled by Zuul and shows a karma error at 20:51:37 !!

The next build is https://integration.wikimedia.org/ci/job/mediawiki-extensions-hhvm/43608 , at 20:52:46 the stat calls show:

File: ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-0’
Access: (0755/drwxr-xr-x)  Uid: (   33/www-data)   Gid: (   33/www-data)
Access: 2015-12-18 20:51:37.791584293 +0000
stat: cannot stat ‘/tmp/jenkins-0’: No such file or directory
chmod: changing permissions of ‘/mnt/home/jenkins-deploy/tmpfs/jenkins-0’: Operation not permitted

What I don't get is the qunit build before executes mw-teardown-mysql.sh as a post build script and it should delete both temp dir via global-teardown.sh which does:

. global-set-env.sh
rm -rf "${TMPDIR_FS}"
rm -rf "${TMPDIR_REGULAR}"

Change 260098 had a related patch set uploaded (by Hashar):
global-teardown: be more verbose

https://gerrit.wikimedia.org/r/260098

Change 260098 merged by jenkins-bot:
global-teardown: be more verbose

https://gerrit.wikimedia.org/r/260098

dduvall added a subscriber: dduvall.EditedDec 18 2015, 11:12 PM

From IRC with @hashar, timestamps in PST (TL;DR: parallelism killed Kenny):

14:43 marxarelli: hashar: is it possible that some request to an hhvm process wouldn't be entirely complete before a job is finished?
14:43 marxarelli: and result in something being written to a temporary file
14:43 hashar: ohhhhhh
14:44 marxarelli: after the teardown script has run, etc.
14:44 hashar: yup that would explain it
14:44 marxarelli: it seems we should probably set MW's tmpdir to somewhere different
14:44 hashar: the qunit jobs often fails because of an ajax timeout
14:45 hashar: so the karma runner does hit hhvm
14:45 hashar: timeout for some reason or the job get canceled
14:46 hashar: post build run
14:46 marxarelli: it's the only way to explain the www-data ownership
14:46 hashar: and hhvm continue running the system
14:46 marxarelli: right
14:46 hashar: yeah
14:46 hashar: given Zuul cancel jobs
14:46 hashar: and Jenkins probably kill -9 every sub process
14:47 hashar: I am not sure how we can have post build watch / wait till hhvm finished
14:47 marxarelli: right, but not hhvm (for good reason)
14:47 marxarelli: i'm not sure we need to, really
14:47 marxarelli: we just need to make sure that hhvm/MW processes write to their own tmpdir
14:47 hashar: or instead of deleting the directory we can delete the files underneath? This way the dir still belong to jenkins-deploy
14:48 marxarelli: one that's cleaned up perdiocally, but outside the scope of each job
14:48 hashar: oh
14:48 hashar: but that needs to be scoped per job :(
14:48 marxarelli: if we leave the tmpdir as owned by jenkins-deploy, and hhvm tries to write to it, it will bork
14:48 marxarelli: right? what's the mode?
14:49 hashar: it is 777
14:49 marxarelli: ah, ok
14:49 marxarelli: so that might work
14:49 hashar: we did that because MediaWiki under hhvm does write to the cache dir
14:49 marxarelli: i kind of think the cleanup schedule should mirror the process(es) lifespan though
14:49 hashar: and the process has no idea of the job context
14:49 marxarelli: hhvm processes are long running and separate from job processes themselves
14:50 hashar: or we can namespace tmpdirs based on jenkins context again
14:51 hashar: i.e. using the job/build number or some hash of it
14:51 marxarelli: yeah, but apache/hhvm isnt' being run by jenkins
14:51 marxarelli: i don't think it necessarily needs it's tmpdir scoped to the executors that run against it
14:52 hashar: we can inject it via the mediawiki.d which would set  $wgTmpDirectory = getenv('TMPDIR') / getenv('BUILD_TAG')
14:52 marxarelli: we just need to ensure that something is triggered or run periodically to clean up hhvm temp files
14:54 marxarelli: hashar: we _could_ have some hhvm maintenance endpoint that's requested when jobs complete
14:54 marxarelli: and let it clean up its own shite, as www-data
14:55 marxarelli: hashar: it would have to be a sync operation to avoid race conditions
14:56 hashar: marxarelli: so embedded in hhvm ?
14:57 marxarelli: hashar: i.e. assign MW's tmpdir to /whatever/jenkins-0/hhvm, and at the end of the job, request http://127.0.0.1/jenkins/cleanup.php?dir=/whatever/jenkins-0/hhvm
14:57 marxarelli: or some other wacky shit like that :)
14:58 hashar: but you can still have a long running process standing bye
14:58 hashar: by
14:58 marxarelli: hashar: grr...
14:58 marxarelli: you're right :)
14:58 hashar: maybe we could use the hhvm console to list threads  based on some info they might have
14:59 hashar: maybe apache pass the virtual host to it
14:59 marxarelli: `while lsof +D /tmp/jenkins-0; sleep` :)
14:59 hashar: and the vhost has the job name + build #
14:59 hashar: ahaha
15:00 hashar: marxarelli: would you mind summarizing your suggestion at https://phabricator.wikimedia.org/T120824 ?  I don't want to steal your finding :-}
15:00 marxarelli: hashar: sure. mind if i just copy/paste?
15:00 hashar: yeah
15:00 hashar: will be fine

Change 260174 had a related patch set uploaded (by JanZerebecki):
Workaround for tmpdir being created with a different user

https://gerrit.wikimedia.org/r/260174

hashar updated the task description. (Show Details)Dec 19 2015, 9:33 AM

I have adjusted the task description for magic command to delete www-data files:

salt -v '*slave*' cmd.run 'find /mnt/home/jenkins-deploy/tmpfs -user www-data -mmin +10 -delete'

Change 260174 merged by jenkins-bot:
Workaround for tmpdir being created with a different user

https://gerrit.wikimedia.org/r/260174

Change 260335 had a related patch set uploaded (by Hashar):
Revert "Tie mwext-mw-selenium* jobs to specific slaves"

https://gerrit.wikimedia.org/r/260335

Change 260335 merged by jenkins-bot:
Revert "Tie mwext-mw-selenium* jobs to specific slaves"

https://gerrit.wikimedia.org/r/260335

Following the discussion on Friday evening with Dan, the workaround would be to only delete files under the directories and leave the dir behind. This way it is still owned by jenkins-deploy user. That is what Jan patch https://gerrit.wikimedia.org/r/260174 does and its deployed.

Leaving the task open though until we are sure the issue is probably solved.

Change 260407 had a related patch set uploaded (by Dduvall):
Fix globs in global and mw-selenium teardown scripts

https://gerrit.wikimedia.org/r/260407

Change 260407 merged by jenkins-bot:
Fix globs in global and mw-selenium teardown scripts

https://gerrit.wikimedia.org/r/260407

@dduvall Thank you! I noticed some left over before our weekly meeting and checked again a minute or so ago there was nothing left. Looks like it did it. Been verifying with:

salt -v '*slave*' cmd.run 'find /mnt/home/jenkins-deploy/tmpfs -user www-data -mmin +10'

Change 260545 had a related patch set uploaded (by Hashar):
Remove debugging statements related to TMPDIR

https://gerrit.wikimedia.org/r/260545

Change 260545 merged by jenkins-bot:
Remove debugging statements related to TMPDIR

https://gerrit.wikimedia.org/r/260545

hashar closed this task as Resolved.Dec 22 2015, 10:39 AM
hashar claimed this task.

I am claiming this issue is now fixed. The root cause were PHP process still running as the web server which would attempt to write after the temp directory got deleted. Somehow the dir ended up being recreated by MediaWiki but owned by www-data.

By only deleting files instead of the directory, the dir is left behind as being owned by jenkins-deploy user.

Thank you @dduvall for the head up on Friday. @Legoktm for cleaning the slaves and @JanZerebecki for the patch proposed on a saturday!