Page MenuHomePhabricator

Make debian-glue job timeout configurable
Closed, ResolvedPublic

Description

The debian-glue Jenkins jobs times out after 30 minutes regardless of the Gerrit project. operations/debs/contenttranslation/giella-sme is an heavy build that takes hours (ex: https://gerrit.wikimedia.org/r/294430 ).

We need a way to set the timeout differently. Probably passed as a Zuul parameter to be used in the Jenkins job. We would need to keep a sane default.

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 22 2016, 12:07 PM

The plugin defaults to 3 minutes and exposes it as BUILD_TIMEOUT. It does support Token Macro and we can add a build parameter that default to 3 or 30 or whatever, then override the value via Zuul parameters function.

Lame Jenkins conf ( https://integration.wikimedia.org/ci/job/hashar-T143546-timeout/ ):

<?xml version='1.0' encoding='UTF-8'?>
<project>
  <properties>
    <hudson.model.ParametersDefinitionProperty>
      <parameterDefinitions>
        <hudson.model.StringParameterDefinition>
          <name>BUILD_TIMEOUT</name>
          <description>Timeout in minutes</description>
          <defaultValue>3</defaultValue>
        </hudson.model.StringParameterDefinition>
      </parameterDefinitions>
    </hudson.model.ParametersDefinitionProperty>
  </properties>
  <builders>
    <hudson.tasks.Shell>
      <command>echo &quot;BUILD_TIMEOUT: $BUILD_TIMEOUT&quot;
sleep 185</command>
    </hudson.tasks.Shell>
  </builders>
  <publishers/>
  <buildWrappers>
    <hudson.plugins.build__timeout.BuildTimeoutWrapper plugin="build-timeout@1.17.1">
      <strategy class="hudson.plugins.build_timeout.impl.AbsoluteTimeOutStrategy">
        <timeoutMinutes>${BUILD_TIMEOUT}</timeoutMinutes>
      </strategy>
      <operationList/>
    </hudson.plugins.build__timeout.BuildTimeoutWrapper>
  </buildWrappers>
</project>

The job just sleep 185. Outcome:

BUILD_TIMEOUT: 3
+ sleep 185
Build timed out (after 3 minutes). Marking the build as aborted.
Build was aborted
Finished: ABORTED

Setting the parameter to 4:

BUILD_TIMEOUT: 4
+ sleep 185
Finished: SUCCESS

That looks promising.

Change 306414 had a related patch set uploaded (by Hashar):
debian-glue: let us override the build timeout

https://gerrit.wikimedia.org/r/306414

hashar triaged this task as Medium priority.

Change 306414 merged by jenkins-bot:
debian-glue: let us override the build timeout

https://gerrit.wikimedia.org/r/306414

Change 306416 had a related patch set uploaded (by Hashar):
Fix BUILD_TIMEOUT token extension in debian-glue

https://gerrit.wikimedia.org/r/306416

I have added a BUILD_TIMEOUT parameter to both debian-glue jobs with a default value of 30 (minutes). Made Zuul to inject 180 (minutes) for operations/debs/contenttranslation/giella-sme

Did a recheck on https://gerrit.wikimedia.org/r/#/c/294430/ and it is currently building on https://integration.wikimedia.org/ci/job/debian-glue/562/console

Change 306416 merged by jenkins-bot:
Fix BUILD_TIMEOUT token extension in debian-glue

https://gerrit.wikimedia.org/r/306416

IF https://integration.wikimedia.org/ci/job/debian-glue/562/console manages to complete, we can mark this task as solved.

Note that the Jenkins slaves have two CPU and run only a single build. I have noticed the operations/debs/contenttranslation/giella-sme package only use a single CPU when building.

There is a make -j1 in the chain:

\_ /usr/bin/make -f debian/rules build
    \_ /usr/bin/perl -w /usr/bin/dh build --parallel --with autoreconf
        \_ /usr/bin/perl -w /usr/bin/dh_auto_build -O--parallel
            \_ make -j1
...

Though that invokes a few other make as sub / sub sub childs. But that would be a different task maybe

KartikMistry closed this task as Resolved.Aug 24 2016, 3:59 PM

https://integration.wikimedia.org/ci/job/debian-glue/564/ is passing tests, so build timeout issue is fixed.