Page MenuHomePhabricator

Build glue from Differential to Nodepool
Closed, ResolvedPublic

Description

Subtasks

Gearman not actually necessary

Turns out that gearman isn't necessary. The only remaining problem I have is with jenkins jobs looking for obscure ZUUL_* parameters.

Configuration Details

tokenharbormaster-token
PHID${target.phid}
DIFF_ID${buildable.diff}
CLONE_URI${repository.clone.uri}
OFFLINE_NODE_WHEN_COMPLETE1

Build script

Jenkins Job Builder patch

Event Timeline

greg created this task.Mar 25 2016, 6:37 PM
greg moved this task from To Triage to Tooling on the Gerrit-Migration board.
greg edited projects, added releng-201617-q1; removed releng-201516-q4.
mmodell claimed this task.Apr 7 2016, 9:01 PM
greg moved this task from To Triage to Backlog on the Differential board.May 12 2016, 10:36 PM
mmodell raised the priority of this task from Normal to High.May 27 2016, 8:20 PM
This comment was removed by mmodell.
mmodell renamed this task from Build glue from Differential to Gearman to Nodepool to Build glue from Differential to Nodepool.Jun 1 2016, 8:33 PM
mmodell updated the task description. (Show Details)
mmodell updated the task description. (Show Details)Jun 1 2016, 9:43 PM
hashar added a comment.Jun 2 2016, 8:39 AM

OFFLINE_NODE_WHEN_COMPLETE is not recognized by the Jenkins Gearman plugin, which was the reason I have asked to have harbormaster to trigger job using the Gearman functions. As @mmodel found out yesterday, there is the Jenkins plugin "single use slave" which given a list of labels, will put the slave offline whenever a build has completed with those labels. Essentially that mean we can trigger the job using Jenkins REST API. Since OFFLINE_NODE_WHEN_COMPLETE has never been recognized, we can drop it.

The ZUUL parameters are really just meant for Zuul and I would rather not reuse them for the Differential integration. It would work, but semantically that is incorrect. I would rather not abuse them.

I haven't looked but buildWithParameters is probably not blocking. So we would get to poll Jenkins to figure out when the build is complete in order to report back to Phabricator. (might be FUD sorry).

We should probably instead use the Jenkins Phabricator Differential Plugin with doc at https://github.com/uber/phabricator-jenkins-plugin which instead rely on a couple parameters: DIFF_ID and PHID. It then comes with postbuild actions to report back.

We would want to add support for it in Jenkins Job Builder. Which should be straightforward (maybe we want a sub task).

A harder part is reusing the code logic and crafting Phabricator friendly jobs. But for most of them it is going to be straightforward since we have macros or very simple shell commands. The tox jessie one looks like:

- job-template: &job_tox-jessie
    name: 'tox-jessie'
    node: ci-jessie-wikimedia
    defaults: use-remote-zuul-shallow-clone
    concurrent: true
    properties: 
     - zeromq-event
    triggers:
     - zuul
    builders:
     - castor-load
     - tox-all-envs
    publishers:
     - archive-log-dir
     - archive-tox-logs
     - castor-save

We would change the defaults (which is a template really), drop the triggers: zuul which inject built in parameters and add a publisher action to report back.

hashar added a comment.EditedJun 2 2016, 8:45 AM

Following up: in our jobs definitions, the git definition is usually provided by the default (in the above example use-remote-zuul-shallow-clone. So we can craft a new default that would add the DIFF_ID and PHPID parameters + the appropriate git plugin configuration.

For the publishing / post build part, turns out it has been implemented already in JJB http://docs.openstack.org/infra/jenkins-job-builder/publishers.html?highlight=phabricator#publishers.phabricator and one can:

publishers:
  - phabricator:
      comment-on-success: false
      uberalls-enabled: false
      comment-with-console-link-on-failure: false

When later we switch to use arc lint / arc unit, we would get their output sent to something like /log/arcresult.txt and have it sent back to the Diff as a comment by adding: comment-file: /log/arcresult.txt

mmodell added a comment.EditedJun 2 2016, 4:06 PM

@hashar: I've already been using the jenkins phabricator plugin.

And here's one job which handles various test entrypoints:
https://integration.wikimedia.org/ci/job/phab-ci-jessie/

Currently supports:

  • npm run test
  • arc lint
  • arc unit
  • tox
  • composer test
  • rake -t

(See P3205)

mmodell updated the task description. (Show Details)Jun 2 2016, 4:17 PM

I will attempt to get it templatized in jjb

mmodell updated the task description. (Show Details)Jun 2 2016, 4:20 PM
mmodell updated the task description. (Show Details)
Paladox added a subscriber: Paladox.Jun 2 2016, 4:29 PM
mmodell added a comment.EditedJun 7 2016, 10:14 PM

Build status now shows for individual commits on diffusion commit history lists. Example: /r/p/phabricator/extensions/

mmodell updated the task description. (Show Details)Jun 22 2016, 4:51 AM
mmodell closed this task as Resolved.Jun 22 2016, 8:02 AM
mmodell updated the task description. (Show Details)
mmodell added subscribers: TerraCodes, Mholloway, Luke081515 and 9 others.

Change 295396 had a related patch set uploaded (by Legoktm; owner: 20after4):
[integration/config@master] Phabricator/harbormaster job templates

https://gerrit.wikimedia.org/r/295396

Change 295396 abandoned by Jforrester:
Phabricator/harbormaster job templates

Reason:
Alas, poor Yorick! I knew him, Horatio…

https://gerrit.wikimedia.org/r/295396