Reporting that a job failed early is nice, but it would be even better if we could provide a link to the error output for the command, and nothing else.
e.g. compare the output of https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php74-docker/15523/consoleFull with having a page where you could just see:
19:54:33 There was 1 failure: 19:54:33 19:54:33 1) GrowthExperiments\Tests\GrowthExperimentsMultiConfigTest::testGetWithFlagsFromWiki 19:54:33 Failed asserting that 0 is identical to 1. 19:54:33 19:54:33 /workspace/src/extensions/GrowthExperiments/tests/phpunit/unit/Config/GrowthExperimentsMultiConfigTest.php:103 19:54:33 /workspace/src/tests/phpunit/MediaWikiUnitTestCase.php:115 19:54:33 phpvfscomposer:///workspace/src/vendor/phpunit/phpunit/phpunit:97 19:54:33 19:54:33 FAILURES! 19:54:33 Tests: 19298, Assertions: 172188, Failures: 1, Skipped: 14.
earlywarningbot currently posts a comment in the format of:
Failed command: "composer phpunit:unit -- --exclude-group Broken,ParserFuzz,Stub"
Phase: "Run Post-dependency install, pre-database dependent steps in parallel (concurrency=3)
Details: https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php74-docker/15523/consoleFull
Note: If the build failed due to a non-deterministic test, please manually remove the "Verified: -1" vote after submitting a "recheck" comment.
What I'd like to change it to:
Failed command: "composer phpunit:unit -- --exclude-group Broken,ParserFuzz,Stub"
Output from failed command: https://earlywarningbot.toolforge.org/build/{somehash}
Full details: https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php74-docker/15523/consoleFull
Note: If the build failed due to a non-deterministic test, please manually remove the "Verified: -1" vote after submitting a "recheck" comment.
Then developers could save a bit more time by viewing the URL from earlywarningbot which has just the stderr/stdout from the failed command.
In order to do that, we need to use subprocess.run(); in Python 3.7 it can be invoked with check=True and capture_output=True. We'd need to replace our mix of subprocess.check_call/Popen calls and drop support for Python 3.5 and 3.6. (Or maybe there is some other way I am not realizing.)
If we can get the output into the CalledProcessError that is used when calling transmit_error(), then we could add a new field like stderr and then earlywarningbot can store this in a database on Toolforge, and allow users to view its value.