Page MenuHomePhabricator

Allow `doc-publish` to be run without zuul dependency
Closed, ResolvedPublic

Description

Currently, there is some difficulty getting https://doc.wikimedia.org/mw-tools-scap/ to publish from differential/phab (T117431).

The main roadblock seems that $ZUUL_UUID is used as a sort of shared random number from the {name}-tox-doc-publish job template.

$ZUUL_UUID is used in the docs-publish job (in publish.yaml):

set -u
echo "Making sure that destination does not contain unset variables (like DOC_SUBPATH, see zuul/doc_functions.py): {docdest}"
[ $ZUUL_UUID ]
rsync --recursive "{docsrc}/" "rsync://10.68.16.255/doc/$ZUUL_UUID"
echo
echo "Publishing to https://doc.wikimedia.org/{docdest}/

The $ZUUL_UUID is then passed and used by the publish-on-gallium job that does the actual publishing of docs:

set -u
echo "Publishing $ZUUL_PROJECT from labs to prod"
echo "Zuul UUID: $ZUUL_UUID"
echo "..."

[ $ZUUL_UUID ]
LOCAL_DOCROOT="/srv/$WMF_CI_PUB_DOCROOT"
if [ ! -d $LOCAL_DOCROOT ]; then
   echo "Error: Invalid docroot. Directory $LOCAL_DOCROOT must exist, aborting."
   exit 1
fi
LOCAL_DEST="$LOCAL_DOCROOT/$WMF_CI_PUB_DEST"
mkdir -p "$LOCAL_DEST"
rsync --recursive --delete-after "rsync://10.68.16.255/doc/$ZUUL_UUID"/ "$LOCAL_DEST"

LOCAL_VHOST=$(echo $WMF_CI_PUB_DOCROOOT | awk -F'/' '{ print $3"."$2"."$1 }')
echo
echo "Published to https://$LOCAL_VHOST/$WMF_CI_PUB_DEST/"

The $ZUUL_UUID (and to a more trivial extent $ZUUL_PROJECT [just an echo]) are the only dependencies that the *-tox-doc-publish jobs have on zuul. By removing the $ZUUL_UUID dependency and replacing it with something ($RANDOM or uuidgen (from uuid-runtime?)) we could trivially generate docs from any scm.

Event Timeline

thcipriani raised the priority of this task from to Needs Triage.
thcipriani updated the task description. (Show Details)
thcipriani added subscribers: thcipriani, mmodell, demon, hashar.
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptNov 4 2015, 6:38 PM

Change 251118 had a related patch set uploaded (by Thcipriani):
Publish scap docs without $ZUUL_UUID

https://gerrit.wikimedia.org/r/251118

Change 251118 merged by jenkins-bot:
Publish scap docs without $ZUUL_UUID

https://gerrit.wikimedia.org/r/251118

hashar added a comment.Nov 5 2015, 5:01 PM

The change above got reverted. It was setting a UUID variable but shells in Jenkins are isolated and come up with a clean env.

The use case for ZUUL_UUID is to namespace the builds when they are pushed on integration-publisher. When the parent job doc-publish triggers the child job publish-on-gallium Jenkins copy the parent job parameters, hence the child has the ZUUL_UUID.

Example: https://integration.wikimedia.org/ci/job/publish-on-gallium/lastSuccessfulBuild/parameters/

Jenkins set a bunch of env variables: https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-JenkinsSetEnvironmentVariables

Maybe we can pass to publish-on-gallium a new variable such as:

predefined-parameters: |
   PUBLISHER_PATH=${JOB_NAME}-${BUILD_NUMBER}

Then in the publish-on-gallium job replace any occurrences of $ZUUL_UUID with the injected $PUBLISHER_PATH. Should work ™

Change 251442 had a related patch set uploaded (by Thcipriani):
Use $JOB_NAME-$BUILD_NUMBER in place of $ZUUL_UUID

https://gerrit.wikimedia.org/r/251442

@hashar implemented your suggestion from T117770#1785791

@dduvall and I tested this by creating two jobs, one of which can be deleted after the patch has been reviewed:

Docs for Scap now seem to be generating on SCM update!

Change 251442 merged by jenkins-bot:
Use $JOB_NAME-$BUILD_NUMBER in place of $ZUUL_UUID

https://gerrit.wikimedia.org/r/251442

thcipriani closed this task as Resolved.Feb 5 2016, 6:26 PM
thcipriani claimed this task.

Been working for a while, cleaning up task