Something seems to be wrong with jenkins-bot
Closed, DuplicatePublicBUG REPORT
Actions

Assigned To

None

Authored By

	alex-mashin
	Sep 2 2022, 1:07 PM

Description

https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php73-noselenium-docker/57569/console

Permission errors while merging a change and building a Docker image for php 7.3.

From the build console log:

Building remotely on integration-agent-docker-1039 (pipelinelib Docker blubber) in workspace /srv/jenkins/workspace/quibble-vendor-mysql-php73-noselenium-docker
...
+ mkdir -m 2777 -p cache
...
+ exec docker run --volume /srv/jenkins/workspace/quibble-vendor-mysql-php73-noselenium-docker/cache:/cache --security-opt seccomp=unconfined --init --rm --label jenkins.job=quibble-vendor-mysql-php73-noselenium-docker --label jenkins.build=57569 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/castor:0.2.6 load
...
rsync: recv_generator: mkdir "/cache/composer" failed: Permission denied (13)

And then:

+ mkdir -m 2777 -p log
mkdir: cannot create directory ‘log’: Permission denied
Build step 'Execute shell' marked build as failure

Related Objects

Mentioned Here: T282893: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied"
T299995: Cancelled Jenkins build do not run postbuild script

Event Timeline

alex-mashin created this task.Sep 2 2022, 1:07 PM

Restricted Application added a project: Continuous-Integration-Infrastructure. · View Herald TranscriptSep 2 2022, 1:07 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Daimona awarded a token.Sep 2 2022, 1:27 PM

hashar edited projects, added ci-test-error; removed Jenkins.Sep 5 2022, 7:43 AM

hashar updated the task description. (Show Details)

The jobs run on the WMCS instance as user jenkins-deploy and somehow were not able to create the cache and log directories in the workspace at /srv/jenkins/workspace/quibble-vendor-mysql-php73-noselenium-docker. There is no reason it can happen.

The previous build of this job on the same instance got cancelled which results in some post build scripts to fail cause it is being interrupted by Jenkins

[PostBuildScript] - [INFO] Executing post build scripts.
[quibble-vendor-mysql-php73-noselenium-docker] $ /bin/bash /tmp/jenkins12306452848509465838.sh
+ set -o pipefail
++ pwd
+ exec docker run --entrypoint=/bin/rm --volume /srv/jenkins/workspace/quibble-vendor-mysql-php73-noselenium-docker/log:/log --security-opt seccomp=unconfined --init --rm --label jenkins.job=quibble-vendor-mysql-php73-noselenium-docker --label jenkins.build=57563 --env-file /dev/fd/63 docker-registry.wikimedia.org/buster:latest -fR /log/rawSeleniumVideoGrabs
++ /usr/bin/env
++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)='
[PostBuildScript] - [INFO] Executing post build scripts.
[quibble-vendor-mysql-php73-noselenium-docker] $ /bin/bash -xe /tmp/jenkins16672079246489926259.sh
+ find log/ -name 'mw-debug-*.log' -exec gzip '{}' +
Archiving artifacts
[PostBuildScript] - [INFO] Executing post build scripts.
Waiting for the completion of castor-save-workspace-cache
[PostBuildScript] - [ERROR] An error occured during post-build processing.
org.jenkinsci.plugins.postbuildscript.PostBuildScriptException: java.lang.InterruptedException
	at org.jenkinsci.plugins.postbuildscript.processor.Processor.processBuildSteps(Processor.java:190)
	at org.jenkinsci.plugins.postbuildscript.processor.Processor.processScripts(Processor.java:91)
	at org.jenkinsci.plugins.postbuildscript.processor.Processor.process(Processor.java:79)
	at org.jenkinsci.plugins.postbuildscript.processor.Processor.process(Processor.java:73)
	at org.jenkinsci.plugins.postbuildscript.PostBuildScript.perform(PostBuildScript.java:116)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:816)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:765)
	at hudson.model.Build$BuildExecution.post2(Build.java:179)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:709)
	at hudson.model.Run.execute(Run.java:1922)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
	at hudson.model.ResourceController.execute(ResourceController.java:101)
	at hudson.model.Executor.run(Executor.java:442)
Caused by: java.lang.InterruptedException
	at java.base/java.lang.Object.wait(Native Method)
	at java.base/java.lang.Object.wait(Object.java:328)
	at hudson.remoting.AsyncFutureImpl.get(AsyncFutureImpl.java:78)
	at hudson.model.queue.FutureImpl.waitForStart(FutureImpl.java:70)
	at hudson.plugins.parameterizedtrigger.TriggerBuilder.perform(TriggerBuilder.java:151)
	at org.jenkinsci.plugins.postbuildscript.processor.Processor.processBuildSteps(Processor.java:180)
	... 13 more
Build step 'Execute scripts' changed build result to FAILURE
Build step 'Execute scripts' marked build as failure

And:

[PostBuildScript] - [INFO] Executing post build scripts.
[quibble-vendor-mysql-php73-noselenium-docker] $ /bin/bash -xe /tmp/jenkins17837377703259829850.sh
+ echo 'Clearing /srv/jenkins/workspace/quibble-vendor-mysql-php73-noselenium-docker/cache'
Clearing /srv/jenkins/workspace/quibble-vendor-mysql-php73-noselenium-docker/cache
[quibble-vendor-mysql-php73-noselenium-docker] $ /bin/bash /tmp/jenkins1830596025319281990.sh
+ set -o pipefail
++ pwd
+ exec docker run --volume /srv/jenkins/workspace/quibble-vendor-mysql-php73-noselenium-docker/cache:/cache --security-opt seccomp=unconfined --init --rm --label jenkins.job=quibble-vendor-mysql-php73-noselenium-docker --label jenkins.build=57563 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/castor:0.2.6 clear
++ /usr/bin/env
++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)='
[PostBuildScript] - [INFO] Executing post build scripts.
[quibble-vendor-mysql-php73-noselenium-docker] $ /bin/bash -xe /tmp/jenkins5818867865376316084.sh
+ set -euxo pipefail
+ docker ps -q --filter label=jenkins.job=quibble-vendor-mysql-php73-noselenium-docker --filter label=jenkins.build=57563
+ xargs --no-run-if-empty docker stop
[PostBuildScript] - [INFO] Executing post build scripts.
[quibble-vendor-mysql-php73-noselenium-docker] $ /bin/bash /tmp/jenkins8152298423194808525.sh
+ set -o pipefail
+ exec docker run --entrypoint=/usr/bin/find --user=root --volume /srv/jenkins/workspace/quibble-vendor-mysql-php73-noselenium-docker:/workspace --security-opt seccomp=unconfined --init --rm --label jenkins.job=quibble-vendor-mysql-php73-noselenium-docker --label jenkins.build=57563 --env-file /dev/fd/63 docker-registry.wikimedia.org/buster:latest /workspace -mindepth 1 -delete
++ /usr/bin/env
++ egrep -v '^(HOME|SHELL|PATH|LOGNAME|MAIL)='
[quibble-vendor-mysql-php73-noselenium-docker] $ /bin/bash -xe /tmp/jenkins1574587833133851082.sh
+ set -u
+ rmdir /srv/jenkins/workspace/quibble-vendor-mysql-php73-noselenium-docker
Finished: FAILURE

The post build script being interrupted is filed as T299995, I haven't found a solution for it.

Then the whole directory parent directory got deleted (via rmdir), the next build supposedly has a brand new directory and the jenkins-deploy user should be able to create the cache and log directory. It is an entire mystery, the related task is T282893

hashar closed this task as a duplicate of T282893: Various CI jobs failing after "mkdir: cannot create directory ‘log’: Permission denied".Sep 5 2022, 7:57 AM

Something seems to be wrong with jenkins-botClosed, DuplicatePublicBUG REPORTActions

Description

Related Objects

Event Timeline

Something seems to be wrong with jenkins-bot
Closed, DuplicatePublicBUG REPORT
Actions