Broken quibble jobs fail all mediawiki commits
Closed, ResolvedPublic

Description

The recently introduced quibble-composer-mysql-php70-docker and quibble-vendor-mysql-php70-docker jobs for mediawiki/core commits seems to be consistently failing every commit.

So far, seen two unique errors:

https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php70-docker/30/console

error: copy-fd: write returned: No space left on device
fatal: failed to copy file to '/workspace/src/.git/objects/pack/pack-fcfa4ea3a2a8997a5e1836cc7b4181e57735b1e3.pack': No space left on device
fatal: The remote end hung up unexpectedly

Finished: FAILURE

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php70-docker/31/console

 [Symfony\Component\Process\Exception\ProcessTimedOutException]         
The process "composer phpcs '.'" exceeded the timeout of 300 seconds.  

Finished: FAILURE
Krinkle created this task.Apr 19 2018, 5:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 19 2018, 5:21 PM
Krinkle triaged this task as Unbreak Now! priority.Apr 19 2018, 5:21 PM
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptApr 19 2018, 5:21 PM
error: copy-fd: write returned: No space left on device
fatal: failed to copy file to '/workspace/src/.git/objects/pack/pack-fcfa4ea3a2a8997a5e1836cc7b4181e57735b1e3.pack': No space left on device
fatal: The remote end hung up unexpectedly

Finished: FAILURE

is due to integration-slave-docker-1004 being out of disk space. The host is now offline while cleanup happens.

Change 427717 had a related patch set uploaded (by Krinkle; owner: Krinkle):
[integration/config@master] Revert quibble-related changes

https://gerrit.wikimedia.org/r/427717

Change 427717 merged by jenkins-bot:
[integration/config@master] Revert quibble-related changes

https://gerrit.wikimedia.org/r/427717

thcipriani lowered the priority of this task from Unbreak Now! to Normal.Apr 19 2018, 5:46 PM

Deployed a revert of adding quibble to mediawiki gate-and-submit so lowering priority

 [Symfony\Component\Process\Exception\ProcessTimedOutException]         
The process "composer phpcs '.'" exceeded the timeout of 300 seconds.  

Finished: FAILURE

is happening somewhere outside of quibble afaict. @Krinkle mentioned that it could have something to do with filtering, i.e. it's trying to look at too many files at once.

 [Symfony\Component\Process\Exception\ProcessTimedOutException]         
The process "composer phpcs '.'" exceeded the timeout of 300 seconds.  

Finished: FAILURE

is happening somewhere outside of quibble afaict. @Krinkle mentioned that it could have something to do with filtering, i.e. it's trying to look at too many files at once.

Yes, see my comment on https://gerrit.wikimedia.org/r/#/c/427698/ – MW core's too big for a naïve phpcs run (which looks at all the files), so mediawiki-core-phpcs-docker previously would run differently…

No space left on device

The Docker slaves have their /var/lib/docker on / which is only 20GBytes large. Puppet should provision the docker directory under the extended disk space. And we need slaves with larger disk.


For phpcs, I made it to run on all files whenever composer.json or phpcs.xml is changed. Otherwise composer test is only run against file that got changed in the patch.

The step times out with:

The process "composer phpcs '.'" exceeded the timeout of 300 seconds.

That is my fault really. The Nodepool based jobs invoke composer test with COMPOSER_PROCESS_TIMEOUT=600:

- builder:
    name: composer-test
    builders:
     - shell: |
        cd "{dir}"
        [[ -f "composer.json" ]] || exit 0
        COMPOSER_PROCESS_TIMEOUT=600 composer --ansi test
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Change 427761 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/quibble@master] Bump composer timeout

https://gerrit.wikimedia.org/r/427761

Change 427761 merged by jenkins-bot:
[integration/quibble@master] Bump composer timeout

https://gerrit.wikimedia.org/r/427761

Change 427763 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: quibble 0.0.9

https://gerrit.wikimedia.org/r/427763

Change 427763 merged by Hashar:
[integration/config@master] docker: quibble 0.0.9

https://gerrit.wikimedia.org/r/427763

Change 427767 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Use quibble on mediawiki/core@master [2]

https://gerrit.wikimedia.org/r/427767

Change 427768 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Also use quibble in gate-and-submit

https://gerrit.wikimedia.org/r/427768

I have rebuild the failing job and it passed successfully https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php70-docker/35/console

00:03:43.984 INFO:test.run_composer_test:Running composer test
00:03:44.581 > composer lint '.'
00:03:44.956 > parallel-lint --exclude vendor '.'
00:04:15.751 Checked 2662 files in 30 seconds

00:04:15.770 > composer phpcs '.'
00:04:16.181 > phpcs -p -s '.'

00:12:08.752 Time: 7 mins, 52.49 secs; Memory: 128.01Mb

Change 427767 merged by jenkins-bot:
[integration/config@master] Use quibble on mediawiki/core@master [2]

https://gerrit.wikimedia.org/r/427767

Change 427768 merged by jenkins-bot:
[integration/config@master] Also use quibble in gate-and-submit [2]

https://gerrit.wikimedia.org/r/427768

Mentioned in SAL (#wikimedia-releng) [2018-04-19T21:01:21Z] <hashar> Bringing back quibble (0.0.9), this time with COMPOSER_PROCESS_TIMEOUT=600 | T192576

Krinkle closed this task as Resolved.Apr 20 2018, 9:28 PM
Krinkle claimed this task.
Krinkle reassigned this task from Krinkle to hashar.

Change 450076 had a related patch set uploaded (by Legoktm; owner: Legoktm):
[mediawiki/extensions/Wikibase@master] Bump composer process-timeout to 600 to match quibble

https://gerrit.wikimedia.org/r/450076

Change 450076 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Bump composer process-timeout to 600 to match quibble

https://gerrit.wikimedia.org/r/450076