Page MenuHomePhabricator

CI outage for mediawiki/extensions/VisualEditor – job "mwext-node10-rundoc-docker" always fails and we can't merge code due to repo configuration
Closed, ResolvedPublic

Description

Earlier this afternoon I reported a CI issue with the "wmf-quibble-vendor-mysql-hhvm-docker" job on #wikimedia-releng (it was failing seemingly in relation to T218844) and it was fixed (thanks!). This task is about a different problem.

For changes in mediawiki/extensions/VisualEditor repo, the job "mwext-node10-rundoc-docker" always fails.

This is even more annoying for us than any of the common random CI outages because the mediawiki/extensions/VisualEditor repo is configured so that no humans are allowed to vote V+2 and override Jenkins. When CI poops itself, we can't merge any code, we can't deploy UBN patches, etc. I would be happier if that was changed (or even if someone told me why it's configured that way, because I'm certain we never requested it).

Example failure:

23:05:00 + npm install --no-progress
23:05:03 npm WARN deprecated nomnom@1.8.1: Package no longer supported. Contact support@npmjs.com for more info.
23:05:06 npm WARN deprecated circular-json@0.3.3: CircularJSON is in maintenance only, flatted is its successor.
23:05:07 npm WARN deprecated gulp-util@3.0.8: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
23:05:13 npm ERR! code Z_DATA_ERROR
23:05:13 npm ERR! errno -3
23:05:13 npm ERR! invalid distance too far back
23:05:22 
23:05:22 > gifsicle@3.0.4 postinstall /src/node_modules/gifsicle
23:05:22 > node lib/install.js
23:05:22 
23:05:23   ✔ gifsicle pre-build test passed successfully
23:05:23 
23:05:23 > guetzli@1.0.1 postinstall /src/node_modules/guetzli
23:05:23 > node lib/install.js
23:05:23 
23:05:24   ⚠ The `/src/node_modules/guetzli/vendor/guetzli` binary doesn't seem to work correctly
23:05:24   ⚠ guetzli pre-build test failed
23:05:24   ℹ compiling from source
23:05:27   ✖ Error: mkdir -p /src/node_modules/guetzli/vendor && make && mv bin/Release/guetzli /src/node_modules/guetzli/vendor/guetzli
23:05:27 Command failed: make && mv bin/Release/guetzli /src/node_modules/guetzli/vendor/guetzli
23:05:27 /bin/sh: 1: pkg-config: not found
23:05:27 /bin/sh: 1: pkg-config: not found
23:05:27 /bin/sh: 1: pkg-config: not found
23:05:27 /bin/sh: 1: pkg-config: not found
23:05:27 /bin/sh: 1: pkg-config: not found
23:05:27 /bin/sh: 1: pkg-config: not found
23:05:27 /bin/sh: 1: pkg-config: not found
23:05:27 guetzli/guetzli.cc:23:17: fatal error: png.h: No such file or directory
23:05:27  #include "png.h"
23:05:27                  ^
23:05:27 compilation terminated.
23:05:27 make[1]: *** [obj/Release/guetzli.o] Error 1
23:05:27 make: *** [guetzli] Error 2
23:05:27 
23:05:27     at ChildProcess.exithandler (child_process.js:291:12)
23:05:27     at ChildProcess.emit (events.js:182:13)
23:05:27     at maybeClose (internal/child_process.js:961:16)
23:05:27     at Process.ChildProcess._handle.onexit (internal/child_process.js:248:5)
23:05:27 
23:05:27 > jpeg-recompress-bin@3.0.1 postinstall /src/node_modules/jpeg-recompress-bin
23:05:27 > node lib/install.js
23:05:27 
23:05:28   ✔ jpeg-recompress pre-build test passed successfully
23:05:28 
23:05:28 > mozjpeg@5.0.0 postinstall /src/node_modules/mozjpeg
23:05:28 > node lib/install.js
23:05:28 
23:05:29   ⚠ The `/src/node_modules/mozjpeg/vendor/cjpeg` binary doesn't seem to work correctly
23:05:29   ⚠ mozjpeg pre-build test failed
23:05:29   ℹ compiling from source
23:05:30   ✖ Error: autoreconf -fiv && ./configure --disable-shared --disable-dependency-tracking --with-jpeg8  --prefix="/src/node_modules/mozjpeg/vendor" --bindir="/src/node_modules/mozjpeg/vendor" --libdir="/src/node_modules/mozjpeg/vendor" && make -j8 && make install -j8
23:05:30 Command failed: autoreconf -fiv
23:05:30 /bin/sh: 1: autoreconf: not found
23:05:30 
23:05:30     at ChildProcess.exithandler (child_process.js:291:12)
23:05:30     at ChildProcess.emit (events.js:182:13)
23:05:30     at maybeClose (internal/child_process.js:961:16)
23:05:30     at Socket.stream.socket.on (internal/child_process.js:380:11)
23:05:30     at Socket.emit (events.js:182:13)
23:05:30     at Pipe._handle.close [as _onclose] (net.js:595:12)
23:05:30 
23:05:30 > optipng-bin@4.0.0 postinstall /src/node_modules/optipng-bin
23:05:30 > node lib/install.js
23:05:30 
23:05:31   ✔ optipng pre-build test passed successfully
23:05:31 
23:05:31 > pngquant-bin@3.1.1 postinstall /src/node_modules/pngquant-bin
23:05:31 > node lib/install.js
23:05:31 
23:05:32   ⚠ The `/src/node_modules/pngquant-bin/vendor/pngquant` binary doesn't seem to work correctly
23:05:32   ⚠ pngquant pre-build test failed
23:05:32   ℹ compiling from source
23:05:32   ✖ Error: pngquant failed to build, make sure that libpng-dev is installed
23:05:32     at ChildProcess.exithandler (child_process.js:291:12)
23:05:32     at ChildProcess.emit (events.js:182:13)
23:05:32     at maybeClose (internal/child_process.js:961:16)
23:05:32     at Socket.stream.socket.on (internal/child_process.js:380:11)
23:05:32     at Socket.emit (events.js:182:13)
23:05:32     at Pipe._handle.close [as _onclose] (net.js:595:12)
23:05:32 
23:05:32 > zopflipng-bin@4.1.0 postinstall /src/node_modules/zopflipng-bin
23:05:32 > node lib/install.js
23:05:32 
23:05:33   ✔ zopflipng pre-build test passed successfully
23:05:34 npm notice created a lockfile as package-lock.json. You should commit this file.
23:05:44 added 942 packages from 901 contributors in 42.782s
23:05:44 npm ERR! Callback called more than once.
23:05:44 
23:05:44 npm ERR! A complete log of this run can be found in:
23:05:44 npm ERR!     /cache/_logs/2019-03-21T23_05_13_199Z-debug.log
23:05:44 Build step 'Execute shell' marked build as failure
23:05:44 Archiving artifacts

Event Timeline

matmarex triaged this task as Unbreak Now! priority.Mar 21 2019, 11:29 PM
matmarex created this task.
Restricted Application added subscribers: Liuxinyu970226, TerraCodes, Aklapper. · View Herald TranscriptMar 21 2019, 11:29 PM

This is possibly my fault, but it worked before.

Last successful build was:

https://integration.wikimedia.org/ci/job/mwext-node10-rundoc-docker/82/console
21-Mar-2019 20:22
Triggered by change: 498213,1

First failure along these lines was:

https://integration.wikimedia.org/ci/job/mwext-node10-rundoc-docker/83/console
21-Mar-2019 20:43
Triggered by change: 497482,2

No relevant changes in https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL between these two?

(This is technically not a shared build failure, as it's the VE-specific job, but…)

Change 498267 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/extensions/VisualEditor@master] build: Un-commit package-lock

https://gerrit.wikimedia.org/r/498267

Change 498276 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] Follow-up 8c5d25631: Revert Timo's inclusion of changes to VE

https://gerrit.wikimedia.org/r/498276

Change 498276 merged by jenkins-bot:
[integration/config@master] Follow-up 8c5d25631: Revert Timo's inclusion of changes to VE

https://gerrit.wikimedia.org/r/498276

Change 498267 abandoned by Jforrester:
build: Un-commit package-lock

Reason:
Testing artefact, no longer needed.

https://gerrit.wikimedia.org/r/498267

Jdforrester-WMF edited projects, added VisualEditor; removed Patch-For-Review.
Jdforrester-WMF closed this task as Resolved.Mar 22 2019, 12:44 AM

https://integration.wikimedia.org/ci/job/mwext-VisualEditor-jsduck-docker/1975/console

17:42:07 JSDuck 5.3.4 (Ruby 2.3.3)
17:42:07 + jsduck
17:42:29 Finished: SUCCESS
Restricted Application added a project: User-Ryasmeen. · View Herald TranscriptMar 22 2019, 12:44 AM
hashar added a subscriber: hashar.Mar 22 2019, 11:32 PM

I like seeing CI unbreak now tasks being fixed without my involvement :-]