Page MenuHomePhabricator
Create Task
Maniphest T198517

Quibble docker instance running on CI instance for 6 hours
Closed, ResolvedPublic
Actions

Description

I noticed a CI job took 26 minutes to run: https://integration.wikimedia.org/ci/job/mediawiki-quibble-vendor-mysql-hhvm-docker/1685/console on integration-slave-docker-1008.

I decided to login to the machine and the load was making everything very slow.

$ sudo docker ps --no-trunc
CONTAINER ID                                                       IMAGE                                                         COMMAND                                                                               CREATED             STATUS              PORTS               NAMES
2edf684738e396ed6a00ca343203608ca9a8c4aafd6bc2ceff7543a39e411a39   docker-registry.wikimedia.org/releng/quibble-stretch:0.0.19   "/usr/local/bin/quibble --packages-source composer --db mysql --skip composer-test"   6 hours ago         Up 6 hours                              keen_hawkin

Not sure if this is happening on other instances, i.e., left-over containers running, but we should look into why this happened and if it happens often.

Details

SubjectRepoBranchLines +/-
Ensure Docker containers are properly killed following buildsintegration/configmaster+86 -60
Provide common docker-run builderintegration/configmaster+186 -214
Customize query in gerrit

Related Objects

Event Timeline

thcipriani created this task.Jun 29 2018, 10:36 PM
hashar added a comment.Jul 2 2018, 1:01 PM

I have killed a few docker containers this morning but failed to capture a process list.

I think I noticed a few on Friday that were blocked on [bash] defunct. There is at least one place in which Quibble uses bash:

quibble/cmd.py
165     def ext_skin_submodule_update(self):
166         self.log.info('Updating git submodules of extensions and skins')
167         # From JJB macro ext-skins-submodules-update
168         # jjb/mediawiki-extensions.yaml
169         subprocess.check_call([
170             # Do not add ., or that will process mediawiki/core submodules in
171             # wmf branches which is a mess.
172             'find', 'extensions', 'skins',
173             '-maxdepth', '2',
174             '-name', '.gitmodules',
175             '-print',
176             '-execdir', 'bash', '-xe', '-c',
177             '\n'.join([
178                  'git submodule foreach git clean -xdff -q',
179                  'git submodule update --init --recursive',
180                  'git submodule status',
181                  ]),
182             ';',  # end of -execdir
183              ], cwd=self.mw_install_path)

So if a job get canceled/aborted, Jenkins sends the container a SIGTERM and bash happily ignores it.

So I guess it is a continuation of the infamous T176747: When jenkins kills a build due to max execution time the docker containers stay running . Potentially it matches the Jenkins upstream issue JENKINS-17116

thcipriani mentioned this in T202457: mediawiki-quibble docker jobs fails due to disk full.Sep 7 2018, 4:18 PM
thcipriani added a project: Continuous-Integration-Infrastructure.Sep 14 2018, 7:36 PM
thcipriani added a comment.Oct 4 2018, 6:26 PM

Can we use docker kill and docker run --name to fix this? That is, could we name each container after $JOB_NAME and $BUILD_NUMBER and then create a post-build job for all docker jobs that runs docker kill to stop the container for a particular job?

• dduvall subscribed.Oct 4 2018, 8:24 PM

There are currently a number of long-running Docker processes in CI:

P7632 long running (> 1 hour) docker processes in CI
1$ for h in $(curl -s 'https://integration.wikimedia.org/ci/computer/api/json' | ruby -r json -e 'JSON.load(STDIN)["computer"].each { |c| puts c["displayName"] if c["assignedLabels"].any? { |l| l["name"] == "DebianJessieDocker" } }'); do echo "# $h:"; ssh $h.integration.eqiad.wmflabs 'sudo docker ps --format '"'"'{{json .}}'"'"' | ruby -r json -r date -ne '"'"'JSON.load($_).tap { |c| puts c["ID"] if (DateTime.now - DateTime.parse(c["CreatedAt"])) > Rational(1,24) }'"'"' | while read id; do sudo docker top $id; done'; done
2
3# integration-slave-docker-1017:
4UID PID PPID C STIME TTY TIME CMD
5nobody 20475 20459 0 Sep28 ? 00:00:00 bash /run.sh roundtrip
6nobody 20717 20475 0 Sep28 ? 00:00:00 npm
7nobody 20727 20717 0 Sep28 ? 00:00:00 sh -c node bin/roundtrip-test.js -c --domain en.wikipedia.org 'Barack Obama' && node bin/roundtrip-test.js -c --domain fr.wikipedia.org Chope && node bin/roundtrip-test.js -c --domain en.wikipedia.org --xml Parkour && node bin/roundtrip-test.js -c --domain www.mediawiki.org --oldid 2170316 'User:Legoktm/test_this_is_json'
8nobody 20728 20727 0 Sep28 ? 00:00:57 node bin/roundtrip-test.js -c --domain en.wikipedia.org Barack Obama
9# integration-slave-docker-1021:
10UID PID PPID C STIME TTY TIME CMD
11nobody 18531 18517 0 19:07 ? 00:00:00 bash /run.sh
12nobody 19899 18531 0 19:08 ? 00:00:00 npm
13nobody 19927 19899 0 19:08 ? 00:00:00 sh -c npm run nsp && npm run lint && npm run parserTests && npm run mocha
14nobody 25354 19927 0 19:12 ? 00:00:00 npm
15nobody 25364 25354 0 19:12 ? 00:00:00 sh -c mocha --opts tests/mocha/mocha.opts tests/mocha
16nobody 25365 25364 0 19:12 ? 00:00:00 node /src/node_modules/.bin/mocha --opts tests/mocha/mocha.opts tests/mocha
17nobody 25371 25365 1 19:12 ? 00:00:42 /usr/bin/nodejs /src/node_modules/mocha/bin/_mocha --check-leaks --reporter spec --timeout 10s --recursive --opts tests/mocha/mocha.opts tests/mocha
18nobody 25387 18531 0 19:12 ? 00:00:33 /usr/bin/nodejs /src/node_modules/worker-farm/lib/child/index.js /usr/bin/nodejs /src/node_modules/service-runner/service-runner.js --check-leaks --reporter spec --timeout 10s --recursive --opts tests/mocha/mocha.opts tests/mocha
19nobody 28977 25371 0 19:44 ? 00:00:02 /usr/bin/nodejs /src/node_modules/service-runner/service-runner.js --check-leaks --reporter spec --timeout 10s --recursive --opts tests/mocha/mocha.opts tests/mocha
20# integration-slave-docker-1033:
21UID PID PPID C STIME TTY TIME CMD
22nobody 25478 25462 0 16:27 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --skip composer-test
23nobody 25758 25478 0 16:28 ? 00:00:10 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-dox4gan0 --log-error=/workspace/db/quibble-mysql-dox4gan0/error.log --pid-file=/workspace/db/quibble-mysql-dox4gan0/mysqld.pid --socket=/workspace/db/quibble-mysql-dox4gan0/socket
24nobody 25925 25478 87 16:28 ? 03:22:56 npm
25UID PID PPID C STIME TTY TIME CMD
26nobody 22715 22700 0 Oct03 ? 00:00:00 bash /run.sh
27nobody 22769 22715 99 Oct03 ? 1-01:07:59 npm
28# integration-slave-docker-1034:
29# integration-slave-docker-1037:
30UID PID PPID C STIME TTY TIME CMD
31nobody 11738 11723 0 Sep28 ? 00:00:00 bash /run.sh roundtrip
32nobody 12001 11738 0 Sep28 ? 00:00:00 npm
33nobody 12011 12001 0 Sep28 ? 00:00:00 sh -c node bin/roundtrip-test.js -c --domain en.wikipedia.org 'Barack Obama' && node bin/roundtrip-test.js -c --domain fr.wikipedia.org Chope && node bin/roundtrip-test.js -c --domain en.wikipedia.org --xml Parkour && node bin/roundtrip-test.js -c --domain www.mediawiki.org --oldid 2170316 'User:Legoktm/test_this_is_json'
34nobody 12012 12011 0 Sep28 ? 00:01:09 node bin/roundtrip-test.js -c --domain en.wikipedia.org Barack Obama
35# integration-slave-docker-1038:
36UID PID PPID C STIME TTY TIME CMD
37nobody 18391 18376 0 16:27 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --run composer-test
38nobody 18666 18391 0 16:28 ? 00:00:08 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-k1oi0awv --log-error=/workspace/db/quibble-mysql-k1oi0awv/error.log --pid-file=/workspace/db/quibble-mysql-k1oi0awv/mysqld.pid --socket=/workspace/db/quibble-mysql-k1oi0awv/socket
39nobody 18836 18391 90 16:28 ? 03:29:20 npm
40UID PID PPID C STIME TTY TIME CMD
41nobody 19343 19328 0 Oct03 ? 00:00:00 bash /run.sh roundtrip
42nobody 19400 19343 99 Oct03 ? 1-06:21:28 npm
43# integration-slave-docker-1040:
44UID PID PPID C STIME TTY TIME CMD
45nobody 8599 30028 78 07:03 ? 10:23:09 npm
46nobody 30028 29914 0 06:59 ? 00:00:03 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db
47UID PID PPID C STIME TTY TIME CMD
48nobody 17113 31724 81 Oct03 ? 18:47:12 npm
49nobody 31724 31610 0 Oct03 ? 00:00:04 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --phpunit-testsuite=extensions
50# integration-slave-docker-1041:
51UID PID PPID C STIME TTY TIME CMD
52nobody 27415 27383 0 16:27 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db sqlite --db-dir /workspace/db --skip composer-test
53nobody 28826 27415 99 16:28 ? 04:25:02 npm
54UID PID PPID C STIME TTY TIME CMD
55nobody 24686 24670 0 13:32 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db
56nobody 25232 24686 75 13:35 ? 05:05:46 npm
57nobody 27508 24686 0 13:34 ? 00:01:03 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-2_marox8 --log-error=/workspace/db/quibble-mysql-2_marox8/error.log --pid-file=/workspace/db/quibble-mysql-2_marox8/mysqld.pid --socket=/workspace/db/quibble-mysql-2_marox8/socket
58UID PID PPID C STIME TTY TIME CMD
59nobody 2250 2234 0 04:17 ? 00:00:02 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db
60nobody 9780 2250 84 04:23 ? 13:29:23 npm
61UID PID PPID C STIME TTY TIME CMD
62nobody 11746 11731 0 Oct03 ? 00:00:00 bash /run.sh toolcheck
63nobody 11796 11746 99 Oct03 ? 1-06:10:12 npm
64UID PID PPID C STIME TTY TIME CMD
65nobody 449 31225 0 Oct02 ? 00:00:00 npm
66nobody 470 449 0 Oct02 ? 00:00:00 sh -c node bin/roundtrip-test.js -c --domain en.wikipedia.org 'Barack Obama' && node bin/roundtrip-test.js -c --domain fr.wikipedia.org Chope && node bin/roundtrip-test.js -c --domain en.wikipedia.org --xml Parkour && node bin/roundtrip-test.js -c --domain www.mediawiki.org --oldid 2170316 'User:Legoktm/test_this_is_json'
67nobody 471 470 0 Oct02 ? 00:00:26 node bin/roundtrip-test.js -c --domain en.wikipedia.org Barack Obama
68nobody 31225 31209 0 Oct02 ? 00:00:00 bash /run.sh roundtrip
69# integration-slave-docker-1043:
70UID PID PPID C STIME TTY TIME CMD
71nobody 351 336 0 Oct03 ? 00:00:00 bash /run.sh
72nobody 407 351 99 Oct03 ? 1-03:54:25 npm
73

Stashbot subscribed.Oct 4 2018, 8:31 PM

Mentioned in SAL (#wikimedia-releng) [2018-10-04T20:31:36Z] <marxarelli> terminating long-running (> 3 hours) CI docker containers (T198517)

• dduvall mentioned this in T206134: Nodes taken offline after /var/lib/docker partition fills due to container logging.Oct 5 2018, 9:10 PM
• dduvall added a comment.EditedOct 5 2018, 9:37 PM
In T198517#4643236, @thcipriani wrote:

Can we use docker kill and docker run --name to fix this? That is, could we name each container after $JOB_NAME and $BUILD_NUMBER and then create a post-build job for all docker jobs that runs docker kill to stop the container for a particular job?

I like this approach and think we should combine it the docker run --init option you mentioned in our IRC discussion so that processes are reaped as nicely as possible. And if we're being nice, I suppose we should use docker stop instead of docker kill as the former tries sending SIGTERM first.

I remember there being a 30 character limit on container names though I can't confirm in the source that it's still exists, but regardless we can use labels instead, which might actually work out better for jobs where we run multiple containers.

In other words, do something like docker run --init --rm --label jenkins.job=$JOB_NAME --label jenkins.build=$BUILD_NUMBER ... and then docker ps -q --filter label=jenkins.job=$JOB_NAME --filter label=jenkins.build=$BUILD_NUMBER | xargs -r docker stop in the publisher.

• dduvall claimed this task.Oct 10 2018, 4:09 PM
• dduvall triaged this task as High priority.
• dduvall moved this task from Untriaged to In-progress on the Continuous-Integration-Infrastructure board.
• dduvall moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.
gerritbot subscribed.Oct 10 2018, 5:27 PM

Change 465668 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[integration/config@master] Provide common docker-run builder

https://gerrit.wikimedia.org/r/465668

gerritbot added a project: Patch-For-Review.Oct 10 2018, 5:27 PM
gerritbot added a comment.Oct 10 2018, 5:50 PM

Change 465671 had a related patch set uploaded (by Dduvall; owner: Dduvall):
[integration/config@master] Ensure Docker containers are properly killed following builds

https://gerrit.wikimedia.org/r/465671

• dduvall added a comment.Oct 11 2018, 5:44 PM

More long-running containers seen today:

P7668 long-running (> 3 hours) Docker containers
1# integration-slave-docker-1017:
2UID PID PPID C STIME TTY TIME CMD
3nobody 18079 18063 0 Oct10 ? 00:00:00 bash /run.sh
4nobody 18119 18079 92 Oct10 ? 1-06:09:24 npm
5UID PID PPID C STIME TTY TIME CMD
6nobody 19966 19952 0 Oct09 ? 00:00:00 bash /run.sh
7nobody 20010 19966 99 Oct09 ? 2-03:14:44 npm
8# integration-slave-docker-1021:
9UID PID PPID C STIME TTY TIME CMD
10nobody 18531 18517 0 Oct04 ? 00:00:00 bash /run.sh
11nobody 19899 18531 0 Oct04 ? 00:00:00 npm
12nobody 19927 19899 0 Oct04 ? 00:00:00 sh -c npm run nsp && npm run lint && npm run parserTests && npm run mocha
13nobody 25354 19927 0 Oct04 ? 00:00:00 npm
14nobody 25364 25354 0 Oct04 ? 00:00:00 sh -c mocha --opts tests/mocha/mocha.opts tests/mocha
15nobody 25365 25364 0 Oct04 ? 00:00:00 node /src/node_modules/.bin/mocha --opts tests/mocha/mocha.opts tests/mocha
16nobody 25371 25365 0 Oct04 ? 00:08:40 /usr/bin/nodejs /src/node_modules/mocha/bin/_mocha --check-leaks --reporter spec --timeout 10s --recursive --opts tests/mocha/mocha.opts tests/mocha
17nobody 25387 18531 0 Oct04 ? 00:07:36 /usr/bin/nodejs /src/node_modules/worker-farm/lib/child/index.js /usr/bin/nodejs /src/node_modules/service-runner/service-runner.js --check-leaks --reporter spec --timeout 10s --recursive --opts tests/mocha/mocha.opts tests/mocha
18# integration-slave-docker-1033:
19UID PID PPID C STIME TTY TIME CMD
20nobody 7024 7008 0 09:39 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db sqlite --db-dir /workspace/db --commands phpunit-patch-coverage check --command "php7.0 -d zend_extension=xdebug.so tests/phpunit/phpunit.php" --html /log/coverage.html
21nobody 8102 7024 99 09:39 ? 10:38:24 npm
22UID PID PPID C STIME TTY TIME CMD
23nobody 2548 2533 0 Oct10 ? 00:00:03 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --phpunit-testsuite=extensions
24nobody 4962 2548 99 Oct10 ? 1-12:21:54 npm
25UID PID PPID C STIME TTY TIME CMD
26nobody 22862 22848 0 Oct08 ? 00:00:00 bash /run.sh roundtrip
27nobody 23375 22862 0 Oct08 ? 00:00:00 npm
28nobody 23389 23375 0 Oct08 ? 00:00:00 sh -c node bin/roundtrip-test.js -c --domain en.wikipedia.org 'Barack Obama' && node bin/roundtrip-test.js -c --domain fr.wikipedia.org Chope && node bin/roundtrip-test.js -c --domain en.wikipedia.org --xml Parkour && node bin/roundtrip-test.js -c --domain www.mediawiki.org --oldid 2170316 'User:Legoktm/test_this_is_json'
29nobody 23390 23389 0 Oct08 ? 00:00:29 node bin/roundtrip-test.js -c --domain en.wikipedia.org Barack Obama
30# integration-slave-docker-1034:
31UID PID PPID C STIME TTY TIME CMD
32nobody 8762 28624 0 12:14 ? 00:00:10 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-3nn85mjt --log-error=/workspace/db/quibble-mysql-3nn85mjt/error.log --pid-file=/workspace/db/quibble-mysql-3nn85mjt/mysqld.pid --socket=/workspace/db/quibble-mysql-3nn85mjt/socket
33nobody 22966 28624 55 12:14 ? 03:02:43 npm
34nobody 28624 28563 0 12:13 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db
35UID PID PPID C STIME TTY TIME CMD
36nobody 5121 9574 0 11:21 ? 00:00:11 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-e0p5hxqm --log-error=/workspace/db/quibble-mysql-e0p5hxqm/error.log --pid-file=/workspace/db/quibble-mysql-e0p5hxqm/mysqld.pid --socket=/workspace/db/quibble-mysql-e0p5hxqm/socket
37nobody 9574 9515 0 11:20 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db
38nobody 28362 9574 57 11:22 ? 03:36:11 npm
39UID PID PPID C STIME TTY TIME CMD
40nobody 4476 19271 57 11:03 ? 03:47:37 npm
41nobody 19271 19211 0 11:02 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db
42UID PID PPID C STIME TTY TIME CMD
43nobody 3700 29236 0 08:16 ? 00:00:19 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-ezpk7dq1 --log-error=/workspace/db/quibble-mysql-ezpk7dq1/error.log --pid-file=/workspace/db/quibble-mysql-ezpk7dq1/mysqld.pid --socket=/workspace/db/quibble-mysql-ezpk7dq1/socket
44nobody 4166 29236 74 08:18 ? 06:56:58 php tests/phpunit/phpunit.php --debug-tests --testsuite extensions --exclude-group Broken,ParserFuzz,Stub,Database
45nobody 4175 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 0 8
46nobody 4179 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 1 8
47nobody 4186 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 2 8
48nobody 4191 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 3 8
49nobody 4204 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 4 8
50nobody 4220 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 5 8
51nobody 4491 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 6 8
52nobody 4497 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 7 8
53nobody 4517 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 8 8
54nobody 4528 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 9 8
55nobody 4553 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 10 8
56nobody 4567 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 11 8
57nobody 4682 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 12 8
58nobody 4702 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 13 8
59nobody 4745 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 14 8
60nobody 4770 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 15 8
61nobody 4854 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 16 8
62nobody 4877 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 17 8
63nobody 4934 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 18 8
64nobody 4951 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 19 8
65nobody 5006 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 20 8
66nobody 5024 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 21 8
67nobody 5105 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 22 8
68nobody 5129 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 23 8
69nobody 5228 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 24 8
70nobody 5262 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 25 8
71nobody 5409 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 26 8
72nobody 5446 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 27 8
73nobody 5477 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 28 8
74nobody 5677 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 29 8
75nobody 5729 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 30 8
76nobody 5777 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 31 8
77nobody 6002 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 32 8
78nobody 6064 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 33 8
79nobody 20926 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 34 8
80nobody 21002 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 35 8
81nobody 21183 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 36 8
82nobody 21250 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 37 8
83nobody 21386 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 38 8
84nobody 21456 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 39 8
85nobody 21583 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 40 8
86nobody 21653 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 41 8
87nobody 21827 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 42 8
88nobody 21908 4166 0 08:18 ? 00:00:00 /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/binaries/lua5_1_5_linux_64_generic/lua /workspace/src/extensions/Scribunto/includes/engines/LuaStandalone/mw_main.lua /workspace/src/extensions/Scribunto/includes 43 8
89nobody 29236 29220 0 08:12 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db
90UID PID PPID C STIME TTY TIME CMD
91nobody 11878 11862 0 04:08 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db
92nobody 12959 11878 0 04:09 ? 00:00:26 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-om6nooig --log-error=/workspace/db/quibble-mysql-om6nooig/error.log --pid-file=/workspace/db/quibble-mysql-om6nooig/mysqld.pid --socket=/workspace/db/quibble-mysql-om6nooig/socket
93nobody 13374 11878 95 04:09 ? 12:52:06 npm
94UID PID PPID C STIME TTY TIME CMD
95nobody 14763 14747 0 03:10 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db
96nobody 16544 14763 67 03:11 ? 09:47:00 npm
97UID PID PPID C STIME TTY TIME CMD
98nobody 10824 10809 0 03:09 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db
99nobody 14243 10824 0 03:10 ? 00:00:28 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-5w6_jaig --log-error=/workspace/db/quibble-mysql-5w6_jaig/error.log --pid-file=/workspace/db/quibble-mysql-5w6_jaig/mysqld.pid --socket=/workspace/db/quibble-mysql-5w6_jaig/socket
100nobody 14644 10824 67 03:10 ? 09:45:23 npm
101UID PID PPID C STIME TTY TIME CMD
102nobody 6878 6863 0 00:13 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db --commands=mwext-phpunit-coverage-patch
103nobody 8397 6878 0 00:14 ? 00:00:34 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-07wxq1vx --log-error=/workspace/db/quibble-mysql-07wxq1vx/error.log --pid-file=/workspace/db/quibble-mysql-07wxq1vx/mysqld.pid --socket=/workspace/db/quibble-mysql-07wxq1vx/socket
104nobody 8470 6878 99 00:14 ? 17:35:58 npm
105# integration-slave-docker-1037:
106UID PID PPID C STIME TTY TIME CMD
107nobody 19098 19081 0 03:08 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db
108nobody 20565 19098 0 03:09 ? 00:01:58 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-qya9qxlw --log-error=/workspace/db/quibble-mysql-qya9qxlw/error.log --pid-file=/workspace/db/quibble-mysql-qya9qxlw/mysqld.pid --socket=/workspace/db/quibble-mysql-qya9qxlw/socket
109nobody 21170 19098 99 03:10 ? 16:54:34 npm
110UID PID PPID C STIME TTY TIME CMD
111nobody 8337 25332 0 Oct09 ? 00:07:16 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-obs6d6r0 --log-error=/workspace/db/quibble-mysql-obs6d6r0/error.log --pid-file=/workspace/db/quibble-mysql-obs6d6r0/mysqld.pid --socket=/workspace/db/quibble-mysql-obs6d6r0/socket
112nobody 8922 25332 80 Oct09 ? 1-15:55:30 npm
113nobody 25332 25310 0 Oct09 ? 00:00:04 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --phpunit-testsuite=extensions
114UID PID PPID C STIME TTY TIME CMD
115nobody 1156 29324 0 Oct08 ? 00:09:47 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-rs2drew8 --log-error=/workspace/db/quibble-mysql-rs2drew8/error.log --pid-file=/workspace/db/quibble-mysql-rs2drew8/mysqld.pid --socket=/workspace/db/quibble-mysql-rs2drew8/socket
116nobody 1768 29324 99 Oct08 ? 3-21:12:06 npm
117nobody 29324 29309 0 Oct08 ? 00:00:04 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --phpunit-testsuite=extensions --skip composer-test
118UID PID PPID C STIME TTY TIME CMD
119nobody 12716 12701 0 Oct08 ? 00:00:00 bash /run.sh
120nobody 12872 12716 84 Oct08 ? 2-21:44:21 npm
121# integration-slave-docker-1038:
122UID PID PPID C STIME TTY TIME CMD
123nobody 1962 1946 0 09:38 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --skip composer-test
124nobody 2208 1962 0 09:39 ? 00:00:15 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-68r9_2qg --log-error=/workspace/db/quibble-mysql-68r9_2qg/error.log --pid-file=/workspace/db/quibble-mysql-68r9_2qg/mysqld.pid --socket=/workspace/db/quibble-mysql-68r9_2qg/socket
125nobody 2646 1962 74 09:39 ? 05:57:16 npm
126UID PID PPID C STIME TTY TIME CMD
127nobody 6826 29633 76 07:36 ? 07:41:27 npm
128nobody 29633 29617 0 07:30 ? 00:00:02 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --phpunit-testsuite=extensions
129UID PID PPID C STIME TTY TIME CMD
130nobody 8749 32450 99 Oct10 ? 1-01:47:49 npm
131nobody 32450 32435 0 Oct10 ? 00:00:02 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --phpunit-testsuite=extensions
132UID PID PPID C STIME TTY TIME CMD
133nobody 25981 25966 0 Oct09 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db --skip composer-test
134nobody 29830 25981 0 Oct09 ? 00:01:25 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-obzb1g0i --log-error=/workspace/db/quibble-mysql-obzb1g0i/error.log --pid-file=/workspace/db/quibble-mysql-obzb1g0i/mysqld.pid --socket=/workspace/db/quibble-mysql-obzb1g0i/socket
135nobody 30125 25981 99 Oct09 ? 2-07:27:49 npm
136UID PID PPID C STIME TTY TIME CMD
137nobody 18552 18527 0 Oct09 ? 00:00:02 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --phpunit-testsuite=extensions
138nobody 28840 18552 0 Oct09 ? 00:01:32 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-yl9po_nf --log-error=/workspace/db/quibble-mysql-yl9po_nf/error.log --pid-file=/workspace/db/quibble-mysql-yl9po_nf/mysqld.pid --socket=/workspace/db/quibble-mysql-yl9po_nf/socket
139nobody 29554 18552 87 Oct09 ? 1-17:21:41 npm
140# integration-slave-docker-1040:
141UID PID PPID C STIME TTY TIME CMD
142nobody 20887 20872 0 04:15 ? 00:00:02 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --skip selenium
143nobody 22233 20887 75 04:16 ? 10:04:19 npm
144UID PID PPID C STIME TTY TIME CMD
145nobody 10473 10457 0 04:07 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db
146nobody 13564 10473 0 04:09 ? 00:05:06 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-maj46b4a --log-error=/workspace/db/quibble-mysql-maj46b4a/error.log --pid-file=/workspace/db/quibble-mysql-maj46b4a/mysqld.pid --socket=/workspace/db/quibble-mysql-maj46b4a/socket
147nobody 13642 10473 99 04:09 ? 14:17:18 npm
148UID PID PPID C STIME TTY TIME CMD
149nobody 8617 8598 0 04:07 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db
150nobody 12741 8617 0 04:09 ? 00:05:17 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-kw9eprp8 --log-error=/workspace/db/quibble-mysql-kw9eprp8/error.log --pid-file=/workspace/db/quibble-mysql-kw9eprp8/mysqld.pid --socket=/workspace/db/quibble-mysql-kw9eprp8/socket
151nobody 12831 8617 75 04:09 ? 10:09:34 npm
152UID PID PPID C STIME TTY TIME CMD
153nobody 3922 3907 0 03:15 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db
154nobody 5864 3922 75 03:16 ? 10:54:14 npm
155UID PID PPID C STIME TTY TIME CMD
156nobody 14862 14828 0 Oct10 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --run composer-test
157nobody 16346 14862 0 Oct10 ? 00:10:02 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-2jneyn64 --log-error=/workspace/db/quibble-mysql-2jneyn64/error.log --pid-file=/workspace/db/quibble-mysql-2jneyn64/mysqld.pid --socket=/workspace/db/quibble-mysql-2jneyn64/socket
158nobody 17148 14862 99 Oct10 ? 1-08:14:49 npm
159# integration-slave-docker-1041:
160UID PID PPID C STIME TTY TIME CMD
161nobody 1649 29348 0 07:14 ? 00:01:26 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-dq26fer3 --log-error=/workspace/db/quibble-mysql-dq26fer3/error.log --pid-file=/workspace/db/quibble-mysql-dq26fer3/mysqld.pid --socket=/workspace/db/quibble-mysql-dq26fer3/socket
162nobody 3078 29348 99 07:15 ? 13:00:55 npm
163nobody 29348 29333 0 07:12 ? 00:00:03 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --phpunit-testsuite=extensions --skip composer-test
164UID PID PPID C STIME TTY TIME CMD
165nobody 11276 11261 0 03:41 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db
166nobody 13477 11276 0 03:42 ? 00:02:12 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-0o1rlnjd --log-error=/workspace/db/quibble-mysql-0o1rlnjd/error.log --pid-file=/workspace/db/quibble-mysql-0o1rlnjd/mysqld.pid --socket=/workspace/db/quibble-mysql-0o1rlnjd/socket
167nobody 13648 11276 99 03:42 ? 16:04:26 npm
168UID PID PPID C STIME TTY TIME CMD
169nobody 17080 17065 0 Oct09 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --skip composer-test
170nobody 18326 17080 0 Oct09 ? 00:07:11 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-6du8f1rv --log-error=/workspace/db/quibble-mysql-6du8f1rv/error.log --pid-file=/workspace/db/quibble-mysql-6du8f1rv/mysqld.pid --socket=/workspace/db/quibble-mysql-6du8f1rv/socket
171nobody 20403 17080 99 Oct09 ? 2-03:00:00 npm
172UID PID PPID C STIME TTY TIME CMD
173nobody 14310 14208 0 Oct08 ? 00:00:02 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db
174nobody 27415 14310 99 Oct08 ? 3-21:11:46 npm
175# integration-slave-docker-1043:
176UID PID PPID C STIME TTY TIME CMD
177nobody 1727 1709 0 04:47 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db
178nobody 2706 1727 0 04:48 ? 00:02:08 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-6mashoph --log-error=/workspace/db/quibble-mysql-6mashoph/error.log --pid-file=/workspace/db/quibble-mysql-6mashoph/mysqld.pid --socket=/workspace/db/quibble-mysql-6mashoph/socket
179nobody 2962 1727 74 04:49 ? 09:33:45 npm
180UID PID PPID C STIME TTY TIME CMD
181nobody 347 332 0 04:45 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db
182nobody 1977 347 0 04:47 ? 00:02:01 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-39zbgr1w --log-error=/workspace/db/quibble-mysql-39zbgr1w/error.log --pid-file=/workspace/db/quibble-mysql-39zbgr1w/mysqld.pid --socket=/workspace/db/quibble-mysql-39zbgr1w/socket
183nobody 2623 347 74 04:48 ? 09:33:53 npm
184UID PID PPID C STIME TTY TIME CMD
185nobody 13750 13734 0 03:40 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db
186nobody 16685 13750 0 03:42 ? 00:02:24 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-4i1y75zo --log-error=/workspace/db/quibble-mysql-4i1y75zo/error.log --pid-file=/workspace/db/quibble-mysql-4i1y75zo/mysqld.pid --socket=/workspace/db/quibble-mysql-4i1y75zo/socket
187nobody 16954 13750 74 03:42 ? 10:22:46 npm
188UID PID PPID C STIME TTY TIME CMD
189nobody 24191 24174 0 03:15 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db
190nobody 24706 24191 77 03:16 ? 11:09:28 npm
191UID PID PPID C STIME TTY TIME CMD
192nobody 10926 10911 0 03:08 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db
193nobody 14451 10926 0 03:11 ? 00:02:33 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-eqrriruq --log-error=/workspace/db/quibble-mysql-eqrriruq/error.log --pid-file=/workspace/db/quibble-mysql-eqrriruq/mysqld.pid --socket=/workspace/db/quibble-mysql-eqrriruq/socket
194nobody 15756 10926 54 03:11 ? 07:53:34 npm
195UID PID PPID C STIME TTY TIME CMD
196nobody 22457 22443 0 Oct10 ? 00:00:04 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --phpunit-testsuite=extensions
197nobody 28096 22457 60 Oct10 ? 14:30:26 npm
198UID PID PPID C STIME TTY TIME CMD
199nobody 1381 1366 0 Oct10 ? 00:00:01 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --skip composer-test
200nobody 2227 1381 0 Oct10 ? 00:04:57 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-4hhxfsso --log-error=/workspace/db/quibble-mysql-4hhxfsso/error.log --pid-file=/workspace/db/quibble-mysql-4hhxfsso/mysqld.pid --socket=/workspace/db/quibble-mysql-4hhxfsso/socket
201nobody 3371 1381 82 Oct10 ? 21:58:25 npm
202UID PID PPID C STIME TTY TIME CMD
203nobody 435 420 0 Oct10 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source vendor --db mysql --db-dir /workspace/db --skip composer-test
204nobody 2059 435 0 Oct10 ? 00:04:43 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-kmg8ewli --log-error=/workspace/db/quibble-mysql-kmg8ewli/error.log --pid-file=/workspace/db/quibble-mysql-kmg8ewli/mysqld.pid --socket=/workspace/db/quibble-mysql-kmg8ewli/socket
205nobody 3308 435 92 Oct10 ? 1-00:37:20 npm
206UID PID PPID C STIME TTY TIME CMD
207nobody 4047 4032 0 Oct10 ? 00:00:00 /usr/bin/python3 /usr/local/bin/quibble --packages-source composer --db mysql --db-dir /workspace/db
208nobody 4643 4047 0 Oct10 ? 00:05:33 /usr/sbin/mysqld --skip-networking --datadir=/workspace/db/quibble-mysql-v9v4wv6i --log-error=/workspace/db/quibble-mysql-v9v4wv6i/error.log --pid-file=/workspace/db/quibble-mysql-v9v4wv6i/mysqld.pid --socket=/workspace/db/quibble-mysql-v9v4wv6i/socket
209nobody 4702 4047 97 Oct10 ? 1-04:30:53 npm
210UID PID PPID C STIME TTY TIME CMD
211nobody 12840 12825 89 Oct05 ? 5-03:21:16 npm

Stashbot added a comment.Oct 11 2018, 5:47 PM

Mentioned in SAL (#wikimedia-releng) [2018-10-11T17:47:12Z] <marxarelli> killing long-running containers (> 3 hours) on docker integration nodes (T198517)

Stashbot added a comment.Oct 11 2018, 6:17 PM

Mentioned in SAL (#wikimedia-releng) [2018-10-11T18:17:38Z] <marxarelli> killed 42 long-running (> 3 hours) containers on docker integration nodes (T198517)

Stashbot added a comment.Oct 11 2018, 11:45 PM

Mentioned in SAL (#wikimedia-releng) [2018-10-11T23:45:45Z] <marxarelli> killed 6 long-running containers (> 3 hours) on docker integration nodes (T198517)

Stashbot added a comment.Oct 12 2018, 3:26 PM

Mentioned in SAL (#wikimedia-releng) [2018-10-12T15:26:15Z] <marxarelli> killing 51 long-running containers (> 1 hour) on docker integration nodes (T198517)

gerritbot added a comment.Oct 12 2018, 3:47 PM

Change 465668 merged by jenkins-bot:
[integration/config@master] Provide common docker-run builder

https://gerrit.wikimedia.org/r/465668

Stashbot added a comment.Oct 12 2018, 4:05 PM

Mentioned in SAL (#wikimedia-releng) [2018-10-12T16:05:13Z] <marxarelli> deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465671 for 1 job mediawiki-quibble-vendor-mysql-php70-docker (T198517)

Stashbot added a comment.Oct 12 2018, 4:25 PM

Mentioned in SAL (#wikimedia-releng) [2018-10-12T16:25:45Z] <marxarelli> deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465671 for 321 affected jobs (T198517)

gerritbot added a comment.Oct 12 2018, 4:36 PM

Change 465671 merged by jenkins-bot:
[integration/config@master] Ensure Docker containers are properly killed following builds

https://gerrit.wikimedia.org/r/465671

Stashbot added a comment.Oct 12 2018, 4:42 PM

Mentioned in SAL (#wikimedia-releng) [2018-10-12T16:42:57Z] <marxarelli> killing 9 long-running containers (> 1 hour) on docker integration nodes (T198517)

Stashbot added a comment.Oct 12 2018, 5:15 PM

Mentioned in SAL (#wikimedia-releng) [2018-10-12T17:15:56Z] <marxarelli> killing 7 long-running containers (> 1 hour) on docker integration nodes (T198517)

• dduvall closed this task as Resolved.EditedOct 12 2018, 6:25 PM

Two integration/config patches were deployed to fix the issue. The first refactored all Docker based jobs to invoke docker run via a common builder. The second adds to the common docker-run builder the --init option which ensures a PID 1 within the container that will properly reap child processes and forward signals, and --label options which tag the running containers with the job name and build number; it also implements an additional safety measure, a docker-reap-containers post-build script that kills any running containers that could be errantly running at the end of the build (using the added labels to filter for only the build's containers).

Between the deployed fix and periodically running a manual process to kill off long-running containers that were started prior to the fix being deployed, I think we may be out of the woods for now.

It should be noted that the impact of this issue is not fully known (yet another argument for increased monitoring/metrics for CI) but I did observe a number of zombied containers being due to builds canceled and rescheduled by Zuul. This means that during periods of high load, it's likely that aborted builds (due to subsequent patchsets, manual aborts by frustrated devs waiting on merges, etc.) would have compounded CI load by leaving the already running container processes and having new ones spawned by the freed up Jenkins executors—during a few different periods, I observed upwards of 16 concurrent containers being executed on a single CI instance.

• dduvall mentioned this in Blog Post: Bring in 'da noise, bring in defunct. It's a zombie party!.Nov 15 2018, 6:12 PM
hashar mentioned this in T235680: Failed docker build leaves dangling container.Apr 16 2020, 8:51 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 16 2020, 9:11 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL