Page MenuHomePhabricator

Jenkins job builder ignores BUILD_TIMEOUT
Closed, ResolvedPublic

Description

It seems that zuul is not taking BUILD_TIMEOUT into account anymore.

https://gerrit.wikimedia.org/r/#/c/operations/debs/trafficserver/+/493660/ times out after about 3 minutes (see for example https://integration.wikimedia.org/ci/job/debian-glue/1441/console). I would not expect that to happen given that we did set BUILD_TIMEOUT for trafficserver to 60: https://github.com/wikimedia/integration-config/blob/master/zuul/parameter_functions.py#L129

Event Timeline

ema created this task.Mar 1 2019, 10:43 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
ema triaged this task as Medium priority.Mar 1 2019, 10:43 AM

The job has a BUILD_TIMEOUT parameter that defaults to 30 (minutes). That is configured in the job itself (as well as injected by zuul). But:

10:01:25 Build timed out (after 3 minutes). Marking the build as failed.
10:01:25 Build was aborted

Looking at the configuration, it turns out that the timeout plugin is hardcoded with 3 (minutes).

JJB generates the following config:

<buildWrappers>
  <hudson.plugins.build__timeout.BuildTimeoutWrapper>
    <timeoutMinutes>${BUILD_TIMEOUT}</timeoutMinutes>
    <failBuild>true</failBuild>
    <writingDescription>false</writingDescription>
    <timeoutPercentage>0</timeoutPercentage>
    <timeoutMinutesElasticDefault>3</timeoutMinutesElasticDefault>
    <timeoutType>absolute</timeoutType>
  </hudson.plugins.build__timeout.BuildTimeoutWrapper>
  <hudson.plugins.timestamper.TimestamperBuildWrapper/>
  <hudson.plugins.ansicolor.AnsiColorBuildWrapper/>
</buildWrappers>

Though eventually the plugin reformat the job with:

<buildWrappers>
  <hudson.plugins.build__timeout.BuildTimeoutWrapper plugin="build-timeout@1.19">
    <strategy class="hudson.plugins.build_timeout.impl.AbsoluteTimeOutStrategy">
      <timeoutMinutes>3</timeoutMinutes>
    </strategy>
    <operationList class="java.util.Arrays$ArrayList">
      <a class="hudson.plugins.build_timeout.BuildTimeOutOperation-array">
        <hudson.plugins.build__timeout.operations.FailOperation/>
      </a>
    </operationList>
  </hudson.plugins.build__timeout.BuildTimeoutWrapper>
  <hudson.plugins.timestamper.TimestamperBuildWrapper plugin="timestamper@1.8.10"/>
  <hudson.plugins.ansicolor.AnsiColorBuildWrapper plugin="ansicolor@0.5.3"/>
</buildWrappers>

I have refreshed the jenkins job just in case but it still comes hardcoded to 3 minutes apparently ;-(

hashar added a comment.EditedMar 1 2019, 1:28 PM

I created a job from scratch in jjb:

- job:
    name: build-timeout-jjb
    node: contint1001
    parameters:
        - string:
            name: BUILD_TIMEOUT
            default: '4'
    wrappers:
        - timeout:
            timeout: '${BUILD_TIMEOUT}'
            fail: true
    builders:
        - shell: |
            echo "Sleeping for a while..."
            for i in $(seq 0 10 200); do
              echo "$i ..."
              sleep 10
            done
            echo "Done"

I triggered it with BUILD_TIMEOUT=4 but it fails after 3 minutes.

I then created the exact same job using Jenkins web interface. That results in a different XML configuration. Triggerd it with BUILD_TIMEOUT=4 and it passed.

My conclusion is we need to update JJB in integration/jenkins-job-builder.git.

Though looking at our JJB code it generates a different XML version when the build timeout plugin is version 1.14 or later :)

hashar added a project: Upstream.EditedMar 1 2019, 1:39 PM

The root cause is JJB tries to fetch informations for a plugin named Jenkins build timeout plugin when it really is Build Timeout. Since it can find the version, JJB relies on the old legacy XML configuration.

It got fixed by https://review.openstack.org/#/c/387799/ which fix the name and default to the new style configuration introduced by 1.14.

$ git describe --tags --contains ecf3933dc2ce6d0eeca3e73946f62fbec606a36d
2.0.2~3^2

But we run JJB 2.0.0.0b2-236-g11aa5de4 :/

hashar renamed this task from zuul seemingly ignoring BUILD_TIMEOUT to Jenkins job builder ignores BUILD_TIMEOUT.Mar 1 2019, 1:40 PM

Mentioned in SAL (#wikimedia-operations) [2019-03-01T14:33:39Z] <hashar> Updating all debian-glue Jenkins job to properly take in account the BUILD_TIMEOUT parameter # T217403

hashar closed this task as Resolved.Mar 1 2019, 2:48 PM

It works ! :)