Page MenuHomePhabricator

Migrate Parsoid CI jobs to Nodepool and add Node 4.3 support
Closed, ResolvedPublic

Description

The parsoid and parsoid/deploy repositories are still running jobs using NodeJS 0.10 on CI slaves using Trusty which are permanent

I would like to have all the jobs migrated to disposable instances provided by Nodepool. They come with either Ubuntu Trusty NodeJS 0.10 or Debian Jessie NodeJS 4.3.

StatusOld jobNew jobDesc.
parsoidsvc-jshintparsoidsvc-npm-run-lint-node-4.3npm run-script lint
parsoidsvc-jsonlintparsoidsvc-npm-run-lint-node-4.3npm run-script lint
parsoidsvc-php-parsertestsparsoidsvc-hhvm-parsertests-jessieNeeded pnmtojpeg
parsoidsvc-source-parse-tool-checkparsoidsvc-source-parse-tool-check-trustyStill Trusty and NodeJS 0.10.x
parsoidsvc-deploy-parse-tool-checkparsoidsvc-deploy-parse-tool-check-trustyStill Trusty and NodeJS 0.10.x
parsoidsvc-source-roundtrip-test-checkparsoidsvc-source-roundtrip-test-check-trustyStill Trusty and NodeJS 0.10.x
parsoidsvc-deploy-roundtrip-test-checkparsoidsvc-deploy-roundtrip-test-checkStill Trusty and NodeJS 0.10.x
parsoidsvc-source-npm-0.10parsoidsvc-source-npm-node-0.10+ a Node 4.3 flavor
parsoidsvc-deploy-npm-0.10parsoidsvc-deploy-npm-node-0.10+ a Node 4.3 flavor

For reference the complete CI workflow for both repository is as of May 25th 2016:

- name: mediawiki/services/parsoid
  test:
    - parsoidsvc-jshint
    - parsoidsvc-jsonlint
    - parsoidsvc-php-parsertests
    - parsoidsvc-source-parse-tool-check
    - parsoidsvc-source-roundtrip-test-check
    - parsoidsvc-source-npm-0.10
  gate-and-submit:
    - parsoidsvc-jshint
    - parsoidsvc-jsonlint
    - parsoidsvc-php-parsertests
    - parsoidsvc-source-parse-tool-check
    - parsoidsvc-source-roundtrip-test-check
    - parsoidsvc-source-npm-0.10
  postmerge:
    - beta-parsoid-update-eqiad
    - parsoidsvc-source-jsduck-publish
  experimental:
    - parsoidsvc-source-npm-node-4.3
- name: mediawiki/services/parsoid/deploy
  test:
    - parsoidsvc-debian-glue
    - parsoidsvc-deploy-parse-tool-check
    - parsoidsvc-deploy-roundtrip-test-check
    - parsoidsvc-deploy-npm-0.10
  gate-and-submit:
    - parsoidsvc-deploy-parse-tool-check
    - parsoidsvc-deploy-roundtrip-test-check
    - parsoidsvc-deploy-npm-0.10
  postmerge:
    - beta-parsoid-update-eqiad
  experimental:
    - parsoidsvc-deploy-npm-node-4.3

Related Objects

Event Timeline

hashar created this task.Feb 15 2016, 4:59 PM
hashar raised the priority of this task from to Medium.
hashar updated the task description. (Show Details)
hashar added subscribers: JanZerebecki, hashar, greg and 8 others.

We are fine dropping 0.8 and 0.10 support in Travis CI.

Travis is covering us there already. https://travis-ci.org/wikimedia/parsoid/builds/107946917 is the most recent run. But, of course, those are post-merge testing, vs. Jenkins CI which is pre-merge. But, we don't expect problems with it. This switch to 4.2 in CI can enable us to switch the production cluster to node 4.2.

cscott added a subscriber: cscott.Feb 17 2016, 6:50 PM

I'm fine with moving CI to node 4.3 as long as we move production to node 4.3 as close as possible to the same time, so that we continue to "test as we deploy".

We have travis running to provide some coverage for node 0.8/0.10, but it's not the same as having pre-commit checks via CI. (Of course, we could run npm-travis to allow us to keep those precommit checks....)

We should do this as part of a general deprecation of running parsoid on node < 4.3, including looking at the dependencies of our debian/ubuntu packaging to ensure we've got a supported configuration there.

I'm fine with moving CI to node 4.3 as long as we move production to node 4.3 as close as possible to the same time, so that we continue to "test as we deploy".

I am not too concerned about this. Our rt-testing has been running node 4.x for the last 3 weeks while production is still node 0.10. But, yes, we want to reduce this lag and do this switch as soon as reasonably can.

@hashar ruthenium which runs jessie has node v4.2.4 specifically. How come CI will get node v4.3 since you mention jessie too?

@hashar ruthenium which runs jessie has node v4.2.4 specifically. How come CI will get node v4.3 since you mention jessie too?

There's a new back-ported release of node v4.3.0. It's already in our repo. We are currently evaluating it in staging and should be ready to move to full prod soon.

@hashar ruthenium which runs jessie has node v4.2.4 specifically. How come CI will get node v4.3 since you mention jessie too?

There's a new back-ported release of node v4.3.0. It's already in our repo. We are currently evaluating it in staging and should be ready to move to full prod soon.

And, will it be applied to ruthenium too then?

@hashar ruthenium which runs jessie has node v4.2.4 specifically. How come CI will get node v4.3 since you mention jessie too?

There's a new back-ported release of node v4.3.0. It's already in our repo. We are currently evaluating it in staging and should be ready to move to full prod soon.

And, will it be applied to ruthenium too then?

You can do it right now if you want: apt-get install nodejs is all it takes.

Change 273271 had a related patch set uploaded (by Paladox):
Add npm-node-4.3 and parsoidsvc-deploy-npm-node-4.3 to experimental:

https://gerrit.wikimedia.org/r/273271

Change 273271 abandoned by Hashar:
Add npm-node-4.3 and parsoidsvc-deploy-npm-node-4.3 to experimental:

Reason:
Has a few more prerequisites before it happens.

https://gerrit.wikimedia.org/r/273271

Change 279529 had a related patch set uploaded (by Hashar):
[parsoidsvc] Update npm tests

https://gerrit.wikimedia.org/r/279529

Change 279529 merged by jenkins-bot:
[parsoidsvc] Update npm tests

https://gerrit.wikimedia.org/r/279529

hashar added a comment.Apr 1 2016, 1:08 PM

https://gerrit.wikimedia.org/r/#/c/279529/ Drops the node 0.8 jobs and add the 4.3 jobs.

The test output is stalled when running:

usr/bin/nodejs /home/jenkins/workspace/parsoidsvc-deploy-npm-4.3/src/tests/../tests/mockAPI.js

Note how the path is off. Anyway that is the output being buffered and after ~5 minutes the output is print.

The job fails when executing npm run doc:

sh: 1: jsduck: not found

On permanent slaves it is provided via a Debian package but that is not available on Debian Jessie. So we would need to have jsduck installed via gem/bundler. That is what T109005: Switch CI from jsduck deb package to a gemfile/bundler system is about.

Change 280906 had a related patch set uploaded (by Hashar):
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/280906

Change 280919 had a related patch set uploaded (by Hashar):
Pass BUNDLE_PATH to Nodepool jobs

https://gerrit.wikimedia.org/r/280919

Change 280919 merged by jenkins-bot:
Pass BUNDLE_PATH to Nodepool jobs

https://gerrit.wikimedia.org/r/280919

The npm install step fails due to npm 1.x with 'cb() has not been called'. Would need npm 2.x to be added to Nodepool instances: T124474: Package npm 2.14

Change 280932 had a related patch set uploaded (by Hashar):
Parsoid: stop triggering Node4.3 job

https://gerrit.wikimedia.org/r/280932

Change 280932 merged by jenkins-bot:
Parsoid: stop triggering Node4.3 job

https://gerrit.wikimedia.org/r/280932

hashar changed the task status from Open to Stalled.Apr 1 2016, 3:26 PM

stalled until npm 2.x is provisioned T124474

hashar changed the task status from Stalled to Open.Apr 1 2016, 9:43 PM
hashar closed subtask T124474: Package npm 2.14 as Resolved.

They have been added to experimental: and are now passing.

Just need to export bundle path to npm 0.10 or else it will fail.

Once that is done we can switch 4.3 test back on.

Change 281627 had a related patch set uploaded (by Paladox):
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/281627

Change 281627 abandoned by Paladox:
Have npm doc to install jsduck with bundler

https://gerrit.wikimedia.org/r/281627

I am not finishing up the Parsoid migration yet. Been switching priority to migrate PHP / Composer based jobs instead.

Krinkle removed a subscriber: Krinkle.Apr 28 2016, 12:30 PM

Change 286510 had a related patch set uploaded (by Hashar):
dib: add nodejs/npm on Trusty images

https://gerrit.wikimedia.org/r/286510

Change 286510 merged by jenkins-bot:
dib: add nodejs/npm on Trusty images

https://gerrit.wikimedia.org/r/286510

@hashar is there anything left here to do? For the transition period, is it possible to have both 0.10 and 4.x tests running on the repo?

@mobrovac yes but one problem though jsduck is not installed on nodepool Jessie which now has to be installed with bundle from ruby.

@mobrovac yes but one problem though jsduck is not installed on nodepool Jessie which now has to be installed with bundle from ruby.

Not sure I understand the implications of that. Is it not possible to install it via bundler install jsduck and save the instance image?

@mobrovac Yes @hashar did that in https://gerrit.wikimedia.org/r/#/c/280906/

But I added a workaround so it dosent fail on npm 0.10.

Other repositories are using npm run-script doc job which really just run jsduck: T128091

@mobrovac I havent actively worked on the parsoid migration to Nodepool. Been dealing with the migration of PHP jobs again.

Some of the blockers (jsduck/node 0.10.x) are easy enough now:

  • We have Trusty instances now, so we can keep the node 0.10.x jobs :)
  • T136096 is about creating a job that runs jsduck

So not so far from completion :-)

hashar renamed this task from Migrate Parsoid CI jobs from node 0.8/0.10 to 4.3 to Migrate Parsoid CI jobs to Nodepool and add Node 4.3 support.May 25 2016, 9:01 AM
hashar updated the task description. (Show Details)

Updated the task description since there are no more jobs using NodeJS 0.8 and we now have Nodepool instances using Trusty and hence NodeJS 0.10.

Change 290651 had a related patch set uploaded (by Hashar):
Migrate parsoidsvc-php-parsertests to Nodepool

https://gerrit.wikimedia.org/r/290651

Change 290651 merged by jenkins-bot:
Migrate parsoidsvc-php-parsertests to Nodepool

https://gerrit.wikimedia.org/r/290651

Mentioned in SAL [2016-05-25T10:58:33Z] <hashar> Updating Nodepool ci-jessie-wikimedia snapshot image to get netpbm package installed into it. T126992 https://gerrit.wikimedia.org/r/290651

Notice: /Stage[main]/Packages::Netpbm/Package[netpbm]/ensure: ensure changed 'purged' to 'present'

Change 290657 had a related patch set uploaded (by Hashar):
zuul: trigger the parsoid parser tests

https://gerrit.wikimedia.org/r/290657

Change 290657 merged by jenkins-bot:
zuul: trigger the parsoid parser tests

https://gerrit.wikimedia.org/r/290657

Change 290659 had a related patch set uploaded (by Hashar):
parsoidsvc-hhvm-parsertests-jessie now archive log dir

https://gerrit.wikimedia.org/r/290659

Change 290659 merged by jenkins-bot:
parsoidsvc-hhvm-parsertests-jessie now archive log dir

https://gerrit.wikimedia.org/r/290659

hashar updated the task description. (Show Details)May 25 2016, 11:29 AM

Change 290663 had a related patch set uploaded (by Hashar):
parsoidsvc: migrate jshint/jsonlint to npm

https://gerrit.wikimedia.org/r/290663

Change 290663 merged by jenkins-bot:
parsoidsvc: migrate jshint/jsonlint to npm

https://gerrit.wikimedia.org/r/290663

hashar updated the task description. (Show Details)May 25 2016, 11:54 AM

Change 290675 had a related patch set uploaded (by Hashar):
Migrate parsoid test jobs to Nodepool

https://gerrit.wikimedia.org/r/290675

hashar updated the task description. (Show Details)May 25 2016, 12:44 PM

Change 290675 merged by jenkins-bot:
Migrate parsoid test jobs to Nodepool

https://gerrit.wikimedia.org/r/290675

parsoidsvc-source-npm-0.10 is blocked pending jsduck.

Change 290704 had a related patch set uploaded (by Hashar):
dib: add jsduck to Nodepool instances

https://gerrit.wikimedia.org/r/290704

Change 290704 merged by jenkins-bot:
dib: add jsduck to Nodepool instances

https://gerrit.wikimedia.org/r/290704

Change 290744 had a related patch set uploaded (by Hashar):
parsoid: switch 'npm test' to Nodepool Node 4.3

https://gerrit.wikimedia.org/r/290744

hashar updated the task description. (Show Details)May 25 2016, 8:02 PM

Change 290744 merged by jenkins-bot:
parsoid: switch 'npm test' to Nodepool Node 4.3

https://gerrit.wikimedia.org/r/290744

hashar closed this task as Resolved.May 25 2016, 8:07 PM
hashar claimed this task.

Complete! With a lot of help from @Paladox :-}

Your welcome. :)

Change 280906 abandoned by Hashar:
Have npm doc to install jsduck with bundler

Reason:
This was really just a short hack. Can revisit later if that is really needed.

https://gerrit.wikimedia.org/r/280906