It seems that zuul is not taking BUILD_TIMEOUT into account anymore.
https://gerrit.wikimedia.org/r/#/c/operations/debs/trafficserver/+/493660/ times out after about 3 minutes (see for example https://integration.wikimedia.org/ci/job/debian-glue/1441/console). I would not expect that to happen given that we did set BUILD_TIMEOUT for trafficserver to 60: https://github.com/wikimedia/integration-config/blob/master/zuul/parameter_functions.py#L129
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | hashar | T217403 Jenkins job builder ignores BUILD_TIMEOUT | |||
| Resolved | hashar | T143731 Update jenkins job builder |
The job has a BUILD_TIMEOUT parameter that defaults to 30 (minutes). That is configured in the job itself (as well as injected by zuul). But:
10:01:25 Build timed out (after 3 minutes). Marking the build as failed. 10:01:25 Build was aborted
Looking at the configuration, it turns out that the timeout plugin is hardcoded with 3 (minutes).
JJB generates the following config:
<buildWrappers> <hudson.plugins.build__timeout.BuildTimeoutWrapper> <timeoutMinutes>${BUILD_TIMEOUT}</timeoutMinutes> <failBuild>true</failBuild> <writingDescription>false</writingDescription> <timeoutPercentage>0</timeoutPercentage> <timeoutMinutesElasticDefault>3</timeoutMinutesElasticDefault> <timeoutType>absolute</timeoutType> </hudson.plugins.build__timeout.BuildTimeoutWrapper> <hudson.plugins.timestamper.TimestamperBuildWrapper/> <hudson.plugins.ansicolor.AnsiColorBuildWrapper/> </buildWrappers>
Though eventually the plugin reformat the job with:
<buildWrappers> <hudson.plugins.build__timeout.BuildTimeoutWrapper plugin="build-timeout@1.19"> <strategy class="hudson.plugins.build_timeout.impl.AbsoluteTimeOutStrategy"> <timeoutMinutes>3</timeoutMinutes> </strategy> <operationList class="java.util.Arrays$ArrayList"> <a class="hudson.plugins.build_timeout.BuildTimeOutOperation-array"> <hudson.plugins.build__timeout.operations.FailOperation/> </a> </operationList> </hudson.plugins.build__timeout.BuildTimeoutWrapper> <hudson.plugins.timestamper.TimestamperBuildWrapper plugin="timestamper@1.8.10"/> <hudson.plugins.ansicolor.AnsiColorBuildWrapper plugin="ansicolor@0.5.3"/> </buildWrappers>
I have refreshed the jenkins job just in case but it still comes hardcoded to 3 minutes apparently ;-(
I created a job from scratch in jjb:
- job:
name: build-timeout-jjb
node: contint1001
parameters:
- string:
name: BUILD_TIMEOUT
default: '4'
wrappers:
- timeout:
timeout: '${BUILD_TIMEOUT}'
fail: true
builders:
- shell: |
echo "Sleeping for a while..."
for i in $(seq 0 10 200); do
echo "$i ..."
sleep 10
done
echo "Done"I triggered it with BUILD_TIMEOUT=4 but it fails after 3 minutes.
I then created the exact same job using Jenkins web interface. That results in a different XML configuration. Triggerd it with BUILD_TIMEOUT=4 and it passed.
My conclusion is we need to update JJB in integration/jenkins-job-builder.git.
Though looking at our JJB code it generates a different XML version when the build timeout plugin is version 1.14 or later :)
The root cause is JJB tries to fetch informations for a plugin named Jenkins build timeout plugin when it really is Build Timeout. Since it can find the version, JJB relies on the old legacy XML configuration.
It got fixed by https://review.openstack.org/#/c/387799/ which fix the name and default to the new style configuration introduced by 1.14.
$ git describe --tags --contains ecf3933dc2ce6d0eeca3e73946f62fbec606a36d
2.0.2~3^2
But we run JJB 2.0.0.0b2-236-g11aa5de4 :/
Mentioned in SAL (#wikimedia-operations) [2019-03-01T14:33:39Z] <hashar> Updating all debian-glue Jenkins job to properly take in account the BUILD_TIMEOUT parameter # T217403