Page MenuHomePhabricator

Build glue from Differential to Nodepool
Closed, ResolvedPublic



Gearman not actually necessary

Turns out that gearman isn't necessary. The only remaining problem I have is with jenkins jobs looking for obscure ZUUL_* parameters.

Configuration Details


Build script

Jenkins Job Builder patch

Event Timeline

greg moved this task from To Triage to Tooling on the Gerrit-Migration board.
greg edited projects, added releng-201617-q1; removed releng-201516-q4.
mmodell raised the priority of this task from Medium to High.May 27 2016, 8:20 PM
This comment was removed by mmodell.
mmodell renamed this task from Build glue from Differential to Gearman to Nodepool to Build glue from Differential to Nodepool.Jun 1 2016, 8:33 PM
mmodell updated the task description. (Show Details)

OFFLINE_NODE_WHEN_COMPLETE is not recognized by the Jenkins Gearman plugin, which was the reason I have asked to have harbormaster to trigger job using the Gearman functions. As @mmodel found out yesterday, there is the Jenkins plugin "single use slave" which given a list of labels, will put the slave offline whenever a build has completed with those labels. Essentially that mean we can trigger the job using Jenkins REST API. Since OFFLINE_NODE_WHEN_COMPLETE has never been recognized, we can drop it.

The ZUUL parameters are really just meant for Zuul and I would rather not reuse them for the Differential integration. It would work, but semantically that is incorrect. I would rather not abuse them.

I haven't looked but buildWithParameters is probably not blocking. So we would get to poll Jenkins to figure out when the build is complete in order to report back to Phabricator. (might be FUD sorry).

We should probably instead use the Jenkins Phabricator Differential Plugin with doc at which instead rely on a couple parameters: DIFF_ID and PHID. It then comes with postbuild actions to report back.

We would want to add support for it in Jenkins Job Builder. Which should be straightforward (maybe we want a sub task).

A harder part is reusing the code logic and crafting Phabricator friendly jobs. But for most of them it is going to be straightforward since we have macros or very simple shell commands. The tox jessie one looks like:

- job-template: &job_tox-jessie
    name: 'tox-jessie'
    node: ci-jessie-wikimedia
    defaults: use-remote-zuul-shallow-clone
    concurrent: true
     - zeromq-event
     - zuul
     - castor-load
     - tox-all-envs
     - archive-log-dir
     - archive-tox-logs
     - castor-save

We would change the defaults (which is a template really), drop the triggers: zuul which inject built in parameters and add a publisher action to report back.

Following up: in our jobs definitions, the git definition is usually provided by the default (in the above example use-remote-zuul-shallow-clone. So we can craft a new default that would add the DIFF_ID and PHPID parameters + the appropriate git plugin configuration.

For the publishing / post build part, turns out it has been implemented already in JJB and one can:

  - phabricator:
      comment-on-success: false
      uberalls-enabled: false
      comment-with-console-link-on-failure: false

When later we switch to use arc lint / arc unit, we would get their output sent to something like /log/arcresult.txt and have it sent back to the Diff as a comment by adding: comment-file: /log/arcresult.txt

@hashar: I've already been using the jenkins phabricator plugin.

And here's one job which handles various test entrypoints:

Currently supports:

  • npm run test
  • arc lint
  • arc unit
  • tox
  • composer test
  • rake -t

(See P3205)

I will attempt to get it templatized in jjb

Build status now shows for individual commits on diffusion commit history lists. Example: /r/p/phabricator/extensions/

Change 295396 had a related patch set uploaded (by Legoktm; owner: 20after4):
[integration/config@master] Phabricator/harbormaster job templates

Change 295396 abandoned by Jforrester:
Phabricator/harbormaster job templates

Alas, poor Yorick! I knew him, Horatio…