Today I looked at the Jenkins output for a random failing job, the blinders briefly fell from my eyes, and I realized that there are 142 lines of nothing I had to scroll past before I found the line that revealed the problem:
| 1 | Started by user unknown or anonymous |
|---|---|
| 2 | Running as SYSTEM |
| 3 | Building remotely on integration-agent-docker-1053 (pipelinelib Docker blubber) in workspace /srv/jenkins/workspace/mwgate-node20 |
| 4 | [mwgate-node20] $ /bin/bash -xe /tmp/jenkins3228353370275082931.sh |
| 5 | + ls -laF --color=always |
| 6 | total 8 |
| 7 | drwxr-xr-x 2 jenkins-deploy wikidev 4096 May 9 23:43 ./ |
| 8 | drwxrwxr-x 20 jenkins-deploy wikidev 4096 May 9 23:43 ../ |
| 9 | + mkdir -m 2777 -p cache |
| 10 | [mwgate-node20] $ /bin/bash /tmp/jenkins6434104649292833498.sh |
| 11 | + set -o pipefail |
| 12 | ++ pwd |
| 13 | + '[' '!' -d /srv/jenkins/workspace/mwgate-node20/cache ']' |
| 14 | ++ pwd |
| 15 | + exec docker run --volume /srv/jenkins/workspace/mwgate-node20/cache:/cache --security-opt seccomp=unconfined --init --rm --label jenkins.job=mwgate-node20 --label jenkins.build=48125 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/castor:0.4.0 load |
| 16 | ++ /usr/bin/env |
| 17 | ++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)=' |
| 18 | Defined: CASTOR_NAMESPACE="castor-mw-ext-and-skins/master/mwgate-node20" |
| 19 | Syncing... |
| 20 | rsync: [generator] failed to set times on "/cache/.": Operation not permitted (1) |
| 21 | rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1819) [generator=3.2.3] |
| 22 | |
| 23 | Done |
| 24 | [mwgate-node20] $ /bin/bash -xe /tmp/jenkins13607376647352674185.sh |
| 25 | + set -eux |
| 26 | + ls -laF --color=always |
| 27 | total 12 |
| 28 | drwxr-xr-x 3 jenkins-deploy wikidev 4096 May 9 23:43 ./ |
| 29 | drwxrwxr-x 20 jenkins-deploy wikidev 4096 May 9 23:43 ../ |
| 30 | drwxrwsrwx 6 jenkins-deploy wikidev 4096 May 9 23:43 cache/ |
| 31 | + mkdir -m 2777 -p log |
| 32 | [mwgate-node20] $ /bin/bash /tmp/jenkins13481615531491161488.sh |
| 33 | + set -o pipefail |
| 34 | + '[' '!' -d /srv/jenkins/workspace/mwgate-node20 ']' |
| 35 | + exec docker run --entrypoint=/usr/bin/find --user=nobody --volume /srv/jenkins/workspace/mwgate-node20:/workspace --security-opt seccomp=unconfined --init --rm --label jenkins.job=mwgate-node20 --label jenkins.build=48125 --env-file /dev/fd/63 docker-registry.wikimedia.org/buster:latest /workspace/log -mindepth 1 -delete |
| 36 | ++ /usr/bin/env |
| 37 | ++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)=' |
| 38 | [mwgate-node20] $ /bin/bash -xe /tmp/jenkins3007904286006666781.sh |
| 39 | + set -eux |
| 40 | + ls -laF --color=always |
| 41 | total 16 |
| 42 | drwxr-xr-x 4 jenkins-deploy wikidev 4096 May 9 23:43 ./ |
| 43 | drwxrwxr-x 20 jenkins-deploy wikidev 4096 May 9 23:43 ../ |
| 44 | drwxrwsrwx 6 jenkins-deploy wikidev 4096 May 9 23:43 cache/ |
| 45 | drwxrwsrwx 2 jenkins-deploy wikidev 4096 May 9 23:43 log/ |
| 46 | + mkdir -m 2777 -p src |
| 47 | [mwgate-node20] $ /bin/bash /tmp/jenkins6278013415059062771.sh |
| 48 | + set -o pipefail |
| 49 | + '[' '!' -d /srv/jenkins/workspace/mwgate-node20 ']' |
| 50 | + exec docker run --entrypoint=/usr/bin/find --user=nobody --volume /srv/jenkins/workspace/mwgate-node20:/workspace --security-opt seccomp=unconfined --init --rm --label jenkins.job=mwgate-node20 --label jenkins.build=48125 --env-file /dev/fd/63 docker-registry.wikimedia.org/buster:latest /workspace/src -mindepth 1 -delete |
| 51 | ++ /usr/bin/env |
| 52 | ++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)=' |
| 53 | [mwgate-node20] $ /bin/bash -eu /tmp/jenkins11908092165531199388.sh |
| 54 | + mkdir -m 2777 -p cache |
| 55 | [mwgate-node20] $ /bin/bash /tmp/jenkins7698894319303657189.sh |
| 56 | + set -o pipefail |
| 57 | ++ pwd |
| 58 | + '[' '!' -d /srv/jenkins/workspace/mwgate-node20/src ']' |
| 59 | ++ pwd |
| 60 | + '[' '!' -d /srv/jenkins/workspace/mwgate-node20/cache ']' |
| 61 | + '[' '!' -d /srv/git ']' |
| 62 | ++ pwd |
| 63 | ++ pwd |
| 64 | + exec docker run --volume /srv/jenkins/workspace/mwgate-node20/src:/src --volume /srv/jenkins/workspace/mwgate-node20/cache:/cache --volume /srv/git:/srv/git:ro --security-opt seccomp=unconfined --init --rm --label jenkins.job=mwgate-node20 --label jenkins.build=48125 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/ci-src-setup-simple:0.7.0 |
| 65 | ++ /usr/bin/env |
| 66 | ++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)=' |
| 67 | + [[ https://integration.wikimedia.org/ci/ == '' ]] |
| 68 | + git init |
| 69 | hint: Using 'master' as the name for the initial branch. This default branch name |
| 70 | hint: is subject to change. To configure the initial branch name to use in all |
| 71 | hint: of your new repositories, which will suppress this warning, call: |
| 72 | hint: |
| 73 | hint: git config --global init.defaultBranch <name> |
| 74 | hint: |
| 75 | hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and |
| 76 | hint: 'development'. The just-created branch can be renamed via this command: |
| 77 | hint: |
| 78 | hint: git branch -m <name> |
| 79 | Initialized empty Git repository in /src/.git/ |
| 80 | + git remote add origin https://gerrit.wikimedia.org/r/mediawiki/skins/Vector |
| 81 | + git fetch --quiet --update-head-ok --depth 2 git://contint1002.wikimedia.org/mediawiki/skins/Vector +refs/zuul/master/Z93b4319852894dfe9fb35c2cd1a30fe5:refs/zuul/master/Z93b4319852894dfe9fb35c2cd1a30fe5 |
| 82 | + [[ master == '' ]] |
| 83 | + git checkout -B master FETCH_HEAD |
| 84 | Reset branch 'master' |
| 85 | + set +x |
| 86 | + git submodule --quiet update --init --recursive |
| 87 | [mwgate-node20] $ /bin/bash -eu /tmp/jenkins17041439869405202659.sh |
| 88 | + chmod 2777 src |
| 89 | + mkdir -m 2777 -p cache log |
| 90 | [mwgate-node20] $ /bin/bash /tmp/jenkins3039322460920528501.sh |
| 91 | + set -o pipefail |
| 92 | ++ pwd |
| 93 | + '[' '!' -d /srv/jenkins/workspace/mwgate-node20/src ']' |
| 94 | ++ pwd |
| 95 | + '[' '!' -d /srv/jenkins/workspace/mwgate-node20/cache ']' |
| 96 | ++ pwd |
| 97 | + '[' '!' -d /srv/jenkins/workspace/mwgate-node20/log ']' |
| 98 | ++ pwd |
| 99 | ++ pwd |
| 100 | ++ pwd |
| 101 | + exec docker run --volume /srv/jenkins/workspace/mwgate-node20/src:/src --volume /srv/jenkins/workspace/mwgate-node20/cache:/cache --volume /srv/jenkins/workspace/mwgate-node20/log:/log --security-opt seccomp=unconfined --init --rm --label jenkins.job=mwgate-node20 --label jenkins.build=48125 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/node20-test:20.19.1 |
| 102 | ++ /usr/bin/env |
| 103 | ++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)=' |
| 104 | + '[' '!' -f package.json ']' |
| 105 | + node --version |
| 106 | v20.19.1 |
| 107 | + npm --version |
| 108 | 10.8.2 |
| 109 | + '[' -e npm-shrinkwrap.json ']' |
| 110 | + '[' -e package-lock.json ']' |
| 111 | + npm ci |
| 112 | npm warn deprecated inflight@1.0.6: This module is not supported, and leaks memory. Do not use it. Check out lru-cache if you want a good and tested way to coalesce async requests by a key value, which is much more comprehensive and powerful. |
| 113 | npm warn deprecated rimraf@3.0.2: Rimraf versions prior to v4 are no longer supported |
| 114 | npm warn deprecated abab@2.0.6: Use your platform's native atob() and btoa() methods instead |
| 115 | npm warn deprecated glob@7.2.0: Glob versions prior to v9 are no longer supported |
| 116 | npm warn deprecated glob@8.1.0: Glob versions prior to v9 are no longer supported |
| 117 | npm warn deprecated domexception@2.0.1: Use your platform's native DOMException instead |
| 118 | npm warn deprecated w3c-hr-time@1.0.2: Use your platform's native performance.now() and performance.timeOrigin. |
| 119 | npm warn deprecated domexception@4.0.0: Use your platform's native DOMException instead |
| 120 | npm warn deprecated mkdirp@0.3.0: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.) |
| 121 | npm warn deprecated sinon@12.0.1: 16.1.1 |
| 122 | |
| 123 | added 900 packages, and audited 901 packages in 7s |
| 124 | |
| 125 | 115 packages are looking for funding |
| 126 | run `npm fund` for details |
| 127 | |
| 128 | 3 vulnerabilities (1 moderate, 2 high) |
| 129 | |
| 130 | To address issues that do not require attention, run: |
| 131 | npm audit fix |
| 132 | |
| 133 | To address all issues (including breaking changes), run: |
| 134 | npm audit fix --force |
| 135 | |
| 136 | Run `npm audit` for details. |
| 137 | + npm run-script test |
| 138 | |
| 139 | > test |
| 140 | > npm -s run lint && tsc && npm run test:unit |
| 141 | |
| 142 | |
| 143 | /src/resources/skins.vector.js/skin.js |
| 144 | 57:2 error 'echo' is not defined no-undef |
| 145 | |
| 146 | ✖ 1 problem (1 error, 0 warnings) |
| 147 | |
| 148 | Build step 'Execute shell' marked build as failure |
| 149 | Archiving artifacts |
| 150 | [PostBuildScript] - [INFO] Executing post build scripts. |
| 151 | [PostBuildScript] - [INFO] Build does not have any of the results [SUCCESS]. Did not execute build step #0. |
| 152 | [PostBuildScript] - [INFO] Executing post build scripts. |
| 153 | [mwgate-node20] $ /bin/bash -xe /tmp/jenkins6810885314710987536.sh |
| 154 | + echo 'Clearing /srv/jenkins/workspace/mwgate-node20/cache' |
| 155 | Clearing /srv/jenkins/workspace/mwgate-node20/cache |
| 156 | [mwgate-node20] $ /bin/bash /tmp/jenkins9735898547263535922.sh |
| 157 | + set -o pipefail |
| 158 | ++ pwd |
| 159 | + '[' '!' -d /srv/jenkins/workspace/mwgate-node20/cache ']' |
| 160 | ++ pwd |
| 161 | + exec docker run --volume /srv/jenkins/workspace/mwgate-node20/cache:/cache --security-opt seccomp=unconfined --init --rm --label jenkins.job=mwgate-node20 --label jenkins.build=48125 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/castor:0.4.0 clear |
| 162 | ++ /usr/bin/env |
| 163 | ++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)=' |
| 164 | [PostBuildScript] - [INFO] Executing post build scripts. |
| 165 | [mwgate-node20] $ /bin/bash -xe /tmp/jenkins10303143500957086623.sh |
| 166 | + set -euxo pipefail |
| 167 | + docker ps -q --filter label=jenkins.job=mwgate-node20 --filter label=jenkins.build=48125 |
| 168 | + xargs --no-run-if-empty docker stop |
| 169 | [PostBuildScript] - [INFO] Executing post build scripts. |
| 170 | [mwgate-node20] $ /bin/bash /tmp/jenkins5687633097472171454.sh |
| 171 | + set -o pipefail |
| 172 | + '[' '!' -d /srv/jenkins/workspace/mwgate-node20 ']' |
| 173 | + exec docker run --entrypoint=/usr/bin/find --user=root --volume /srv/jenkins/workspace/mwgate-node20:/workspace --security-opt seccomp=unconfined --init --rm --label jenkins.job=mwgate-node20 --label jenkins.build=48125 --env-file /dev/fd/63 docker-registry.wikimedia.org/buster:latest /workspace -mindepth 1 -delete |
| 174 | ++ /usr/bin/env |
| 175 | ++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)=' |
| 176 | [mwgate-node20] $ /bin/bash -xe /tmp/jenkins7757249810609692084.sh |
| 177 | + echo 'Listing potentially remaining files in workspace for T282893' |
| 178 | Listing potentially remaining files in workspace for T282893 |
| 179 | + ls -laF --color=always |
| 180 | total 8 |
| 181 | drwxr-xr-x 2 jenkins-deploy wikidev 4096 May 9 23:44 ./ |
| 182 | drwxrwxr-x 20 jenkins-deploy wikidev 4096 May 9 23:43 ../ |
| 183 | [mwgate-node20] $ /bin/bash -xe /tmp/jenkins6976407823933273425.sh |
| 184 | + set -u |
| 185 | + rmdir /srv/jenkins/workspace/mwgate-node20 |
| 186 | Finished: FAILURE |
Some of these lines are probably valuable debugging information for problems that did not happen in this particular run. Many, however, are useless. I thought I could spend some time trying to trim them down.