Page MenuHomePhabricator

Switch CI from jsduck deb package to a gemfile/bundler system
Closed, ResolvedPublic

Description

The generated documention for Parsoid does not properly display code blocks fenced with triple-backtick even though this is supported in the latest release of RDiscount (which jsduck uses to format markdown). JSDuck is verision 5.3.4, according to the generated documentation, so it is up to date. But RDiscount must be at 2.0.x, since fenced code block support appears to be missing. The latest version of RDiscount is 2.1.8. See: https://github.com/davidfstr/rdiscount/blob/master/CHANGELOG.md

Details

Related Gerrit Patches:
mediawiki/services/parsoid : masterHave npm doc to install jsduck with bundler
integration/config : masterjsduck-jessie no more install from rubygems
integration/config : masterAdd npm-run-doc-jsduck-node-4.3 test
integration/config : masterdib: install jsduck with a .deb package on Trusty
integration/config : masterdib: on Trusty set gem to gem2.0
integration/config : masterdib: add jsduck to Nodepool instances
mediawiki/extensions/MobileFrontend : wmf/1.27.0-wmf.19Have npm doc to install jsduck with bundler
mediawiki/extensions/MobileFrontend : wmf/1.27.0-wmf.20Have npm doc to install jsduck with bundler
mediawiki/extensions/MobileFrontend : masterHave npm doc to install jsduck with bundler
mediawiki/services/parsoid : masterHave npm doc to install jsduck with bundler
integration/config : masterPass BUNDLE_PATH to Nodepool jobs

Related Objects

Event Timeline

cscott created this task.Aug 13 2015, 9:31 PM
cscott raised the priority of this task from to Medium.
cscott updated the task description. (Show Details)
cscott added a subscriber: cscott.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 13 2015, 9:31 PM
cscott updated the task description. (Show Details)Aug 13 2015, 9:37 PM
cscott set Security to None.
cscott renamed this task from Update RDiscount gem on jenkins build servers to Update RDiscount gem on jenkins build servers (UbuntuTrusty).Aug 13 2015, 9:40 PM
cscott added a project: acl*sre-team.
cscott added a subscriber: MoritzMuehlenhoff.
Restricted Application added a subscriber: Matanya. · View Herald TranscriptAug 13 2015, 9:40 PM
cscott renamed this task from Update RDiscount gem on jenkins build servers (UbuntuTrusty) to Update RDiscount gem/package on jenkins build servers (UbuntuTrusty).Aug 13 2015, 9:51 PM

We wanted to get a Debian package for JSDuck on Jessie (T95008). The packaging works involved ends up being rather painful so we gave up.

Instead we should have the Gem list defined in the software repositories. Then we can use bundler to grab them in the Jenkins job and execute whatever JSDuck version we want.

We have most of the logic existing already in JJB such as for yard (another ruby documentation tool):

- job-template:
    name: '{name}-bundle-yard-publish'
    node: contintLabsSlave && UbuntuTrusty
    defaults: use-remoteonly-zuul
    triggers:
     - zuul
    builders:
     - bundle:
         command: 'yard'
     - doc-publish:
         docsrc: 'doc'
         docdest: 'rubygems/{name}'

In short it runs:

bundle install
bundle exec yard

Then assume the doc has been generated under /doc/ and rsync the content to doc.wikimedia.org

So we would need a bundle-jsduck-publish macro, craft Gemfile and then migrate existing jobs definitions in CI. Probably want to repurposes this task.

@cscott mind if we change this task to Continuous-Integration-Config and retitle it to switch CI from jsduck deb package to a gemfile/bundler system?

Go ahead! I was just guesing when I categorized it.

cscott renamed this task from Update RDiscount gem/package on jenkins build servers (UbuntuTrusty) to Switch CI from jsduck deb package to a gemfile/bundler system.Aug 26 2015, 3:29 PM

Change 280906 had a related patch set uploaded (by Hashar):
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/280906

Change 280919 had a related patch set uploaded (by Hashar):
Pass BUNDLE_PATH to Nodepool jobs

https://gerrit.wikimedia.org/r/280919

Change 280919 merged by jenkins-bot:
Pass BUNDLE_PATH to Nodepool jobs

https://gerrit.wikimedia.org/r/280919

This comment was removed by hashar.
hashar added a comment.Apr 1 2016, 3:13 PM

So repositories should get a Gemfile with jsduck and trigger the job rake-jessie

Alternatively, the npm entry point can be abused to run bundle install (still need a Gemfile) example https://gerrit.wikimedia.org/r/#/c/280906/1/package.json,cm

I'd like to offer an alternative to adding a Gemfile everywhere.

Repositories should not have multiple test entry points or make files. Especially when these do not relate to the software in question. This is confusing to new contributors and complicates a lot of things.

First off, I agree we should not preinstall JSDuck on CI slaves. For the same reason that we don't preinstall Grunt or JSHint there. Preinstalling does not scale. It adds maintenance burden to CI administrators, and makes it very difficult to upgrade since it requires all jobs upgrade at once. Even if (which is a big if) all projects are to the upgrade, it's still impossible to orchestrate patches everywhere at the same time and then do the upgrade. Most dependencies belong either in the repository or in the job configuration.

These tools that should self-install as part of the test entry point. Or, if most contributors don't need to interact with it, it can be considered a prerequisite from the repo perspective and instead be installed as part of the job.

For the moment it seems clean and easy for everyone to simply run gem install jsduck as part of the relevant jobs.

In the future we can evaluate a way to store these snippets in the repository to simplify job configuration (e.g. similar to .travis.yml#before_install). But either way, we do not need and should not have a Gemfile just to install jsduck.

Change 281627 had a related patch set uploaded (by Paladox):
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/281627

Change 281627 abandoned by Paladox:
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/281627

Change 280937 had a related patch set uploaded (by Hashar):
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/280937

Change 281937 had a related patch set uploaded (by Hashar):
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/281937

Change 281938 had a related patch set uploaded (by Hashar):
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/281938

Change 280937 merged by jenkins-bot:
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/280937

Change 281938 merged by jenkins-bot:
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/281938

Change 281937 merged by jenkins-bot:
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/281937

@Krinkle +1

Adding bundle install jsduck to node's predoc target is emphatically *not* the right way to do this.

With T136096 I have introduced a Jenkins job template that does roughly:

  • gem install jsduck
  • jsduck

While at it I have switched unicodejs, oojs/core and VisualEditor/VisualEditor. All passes.

Change 290564 had a related patch set uploaded (by Paladox):
Add npm-run-doc-jsduck-node-4.3 test

https://gerrit.wikimedia.org/r/290564

Given how jsduck is present in a wide range of jobs, I am just going to provision jsduck from ruby gems with something like:

package { 'jsduck':
  provider => 'gem',
  ensure   => present,
}

Change 290704 had a related patch set uploaded (by Hashar):
dib: add jsduck to Nodepool instances

https://gerrit.wikimedia.org/r/290704

Change 290704 merged by jenkins-bot:
dib: add jsduck to Nodepool instances

https://gerrit.wikimedia.org/r/290704

Jdforrester-WMF closed this task as Resolved.May 25 2016, 3:59 PM
Jdforrester-WMF claimed this task.
Jdforrester-WMF reassigned this task from Jdforrester-WMF to hashar.
Jdforrester-WMF added a subscriber: Jdforrester-WMF.

Change 290708 had a related patch set uploaded (by Hashar):
dib: on Trusty set gem to gem2.0

https://gerrit.wikimedia.org/r/290708

Change 290708 merged by jenkins-bot:
dib: on Trusty set gem to gem2.0

https://gerrit.wikimedia.org/r/290708

Change 290709 had a related patch set uploaded (by Hashar):
dib: install jsduck with a .deb package on Trusty

https://gerrit.wikimedia.org/r/290709

Change 290709 merged by Hashar:
dib: install jsduck with a .deb package on Trusty

https://gerrit.wikimedia.org/r/290709

Mentioned in SAL [2016-05-25T16:26:35Z] <hashar> 2016-05-25 16:24:35,491 INFO nodepool.image.build.wmflabs-eqiad.ci-trusty-wikimedia: Notice: /Stage[main]/Main/Package[ruby-jsduck]/ensure: ensure changed 'purged' to 'present' T109005

Trusty: JSDuck 5.3.4 (Ruby 1.9.3)

Jessie: JSDuck 5.3.4 (Ruby 2.1.5)

Change 290564 abandoned by Hashar:
Add npm-run-doc-jsduck-node-4.3 test

Reason:
No more needed. jsduck is now installed on the Nodepool instances :-}

https://gerrit.wikimedia.org/r/290564

Change 290928 had a related patch set uploaded (by Hashar):
jsduck-jessie no more install from rubygems

https://gerrit.wikimedia.org/r/290928

Change 290928 merged by jenkins-bot:
jsduck-jessie no more install from rubygems

https://gerrit.wikimedia.org/r/290928

Change 280906 abandoned by Hashar:
Have npm doc to install jsduck with bundler

Reason:
This was really just a short hack. Can revisit later if that is really needed.

https://gerrit.wikimedia.org/r/280906