Page MenuHomePhabricator

Migrate leftover Nodepool jobs to Docker
Closed, ResolvedPublic

Description

As of Feb 20th, jobs that are assigned to ci-jessie-wikimedia Jenkins label, ie: Nodepool

(DONE) MediaWiki services

Done thanks to https://gerrit.wikimedia.org/r/#/c/417260/

JobTest changeStatus
3d2png-deploy-npm-node-6-jessie417276Migrated. PASS if bumping the src submodule (T187904)
change-propagation-deploy-npm-node-6-jessie325935Migrated
citoid-deploy-npm-node-6-jessie417268Migrated, PASS
cxserver-deploy-npm-node-6-jessie184606Migrated, PASS
graphoid-deploy-npm-node-6-jessie417269Migrated, PASS
kartotherian-deploy-npm-node-6-jessie417273Migrated. Experimenal and can't find tests.
mathoid-deploy-npm-node-6-jessie222107Migrated, PASS
mobileapps-deploy-npm-node-6-jessie417271Migrated, PASS
tilerator-deploy-npm-node-6-jessie417274Migrated. Experimenal and can't find tests.
trending-edits-deploy-npm-node-6-jessie417275Migrated. Segmentation fault on both Docker and Nodepool

(DONE) Parsoid specific jobs

Job for ParsoidStatus
parsoidsvc-deploy-npm-node-6-jessiePASS
parsoidsvc-deploy-parse-tool-check-jessiePASS
parsoidsvc-deploy-roundtrip-test-check-jessiePASS

Experimental jobs: https://gerrit.wikimedia.org/r/#/c/417453/
Test change: https://gerrit.wikimedia.org/r/#/c/222106/

Needed a hack so that when npm-shrinkwrap.json is present one has to npm install --only=dev. That is to ensure we use the pinned versions. https://gerrit.wikimedia.org/r/417960

Longtail

Mostly completed.

Job nameAction?
apps-android-java-mwapiDone, obsolete repo T187995
fundraising-tools-tox-jessieDone https://gerrit.wikimedia.org/r/#/c/413374/
integration-composer-check-php55-jessieDone https://gerrit.wikimedia.org/r/413388
integration-config-dib-jessieDone will be deleted when Nodepool is phased out
integration-jjb-config-diffDone Gerrit: 413404 415554
language-screenshots-VisualEditorDone T189122 https://github.com/amire80/commons_upload/issues/10 https://github.com/amire80/commons_upload/issues/11
mediawiki-core-doxygen-publishDone rewritten to use poll T115755
mediawiki-core-tag-doxygen-publishDropped T115755
mediawiki-core-jsduck-publishDone https://gerrit.wikimedia.org/r/413686
mwext-MobileFrontend-npm-run-lint-modulesDone Gerrit: 413692, 413694, 413693
mwext-PoolCounter-build-jessieDone various changes. Note test suite is flappy T178517
mwext-VisualEditor-publishDone https://gerrit.wikimedia.org/r/413705
oojs-ui-coverageDone https://gerrit.wikimedia.org/r/415295
oojs-ui-demos-publishDone https://gerrit.wikimedia.org/r/415295
oojs-ui-jsduck-publishDone https://gerrit.wikimedia.org/r/#/c/415295/
oojs-ui-npm-run-jenkins-node-6-jessieDone Gerrit 415270 made the switch
parsoidsvc-source-jsduck-publishDone https://gerrit.wikimedia.org/r/#/c/413720/
parsoidsvc-source-npm-node-6-jessieDone https://gerrit.wikimedia.org/r/#/c/413727/
parsoidsvc-source-parse-tool-check-jessie Done https://gerrit.wikimedia.org/r/#/c/413727/
parsoidsvc-source-roundtrip-test-check-jessieDone https://gerrit.wikimedia.org/r/#/c/413727/
phpunit-coverage-publishDone https://gerrit.wikimedia.org/r/413737
remexhtml-phpunit-coverage-publishDone https://gerrit.wikimedia.org/r/413737
pywikibot-tests-beta-clusterDone (deleted) T188256
search-xgboost-mavenDone T190032
wikidata-query-rdf-mavenDone T188285
wikidata-query-rdf-maven-site-publishDone T190042
wikimedia-fundraising-civicrm-jessieDone (deleted) https://gerrit.wikimedia.org/r/414703
wikimedia-portals-buildNeeds to be overhauled and use a predefined entrypoint. See T190073

Mediawiki related

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
hashar updated the task description. (Show Details)Mar 1 2018, 11:24 AM

Change 415270 merged by jenkins-bot:
[integration/config@master] Migrate oojs npm job to Docker

https://gerrit.wikimedia.org/r/415270

Change 415295 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Merge oojs/ui publish job and move them to Docker

https://gerrit.wikimedia.org/r/415295

hashar updated the task description. (Show Details)Mar 1 2018, 11:52 AM

Change 415295 merged by jenkins-bot:
[integration/config@master] Merge oojs/ui publish job and move them to Docker

https://gerrit.wikimedia.org/r/415295

hashar updated the task description. (Show Details)Mar 1 2018, 12:25 PM

Change 415554 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Promote integration-jjb-config-diff-docker

https://gerrit.wikimedia.org/r/415554

hashar updated the task description. (Show Details)Mar 1 2018, 12:29 PM

Change 415554 merged by jenkins-bot:
[integration/config@master] Promote integration-jjb-config-diff-docker

https://gerrit.wikimedia.org/r/415554

hashar added a comment.Mar 6 2018, 5:20 PM

Tyler and I had a chat about migrating the /deploy repositories to Blubber. That is entirely doable and quite simple to do.

A culpirt is the test variant does not get the dev dependencies installed and thus the tests lack commands (eg: mocha command not found).
We would need to port /srv/deployment/integration/slave-scripts/bin/npm-install-dev.py to Blubber. A subject for another task.

Change 416904 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] [PoolCounter] tweak rake file filter

https://gerrit.wikimedia.org/r/416904

Change 416907 had a related patch set uploaded (by Hashar; owner: Hashar):
[mediawiki/extensions/PoolCounter@master] build: add daemon compilation and test to rake

https://gerrit.wikimedia.org/r/416907

Change 416904 merged by jenkins-bot:
[integration/config@master] Add more specific rake job for PoolCounter

https://gerrit.wikimedia.org/r/416904

Change 416913 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Drop mwext-PoolCounter-build-jessie

https://gerrit.wikimedia.org/r/416913

hashar updated the task description. (Show Details)Mar 7 2018, 9:12 AM
hashar updated the task description. (Show Details)Mar 7 2018, 2:09 PM
hashar updated the task description. (Show Details)Mar 7 2018, 2:42 PM
hashar updated the task description. (Show Details)Mar 8 2018, 8:04 AM

Change 416907 merged by Hashar:
[mediawiki/extensions/PoolCounter@master] build: add daemon compilation and test to rake

https://gerrit.wikimedia.org/r/416907

hashar updated the task description. (Show Details)Mar 8 2018, 8:14 AM

Change 416913 merged by jenkins-bot:
[integration/config@master] Drop mwext-PoolCounter-build-jessie

https://gerrit.wikimedia.org/r/416913

Change 417217 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: add /deploy repo support to npm-test

https://gerrit.wikimedia.org/r/417217

Change 417217 merged by jenkins-bot:
[integration/config@master] docker: add /deploy repo support to npm-test

https://gerrit.wikimedia.org/r/417217

hashar updated the task description. (Show Details)Mar 8 2018, 1:57 PM

Change 417260 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Experimental Docker jobs for /deploy repositories

https://gerrit.wikimedia.org/r/417260

hashar updated the task description. (Show Details)Mar 8 2018, 2:01 PM

Change 417260 merged by jenkins-bot:
[integration/config@master] Experimental Docker jobs for /deploy repositories

https://gerrit.wikimedia.org/r/417260

hashar updated the task description. (Show Details)Mar 8 2018, 2:29 PM
hashar updated the task description. (Show Details)Mar 8 2018, 2:38 PM
hashar updated the task description. (Show Details)Mar 8 2018, 2:44 PM

Change 417279 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Use Docker for change-propagation-deploy

https://gerrit.wikimedia.org/r/417279

Change 417279 merged by jenkins-bot:
[integration/config@master] Use Docker for change-propagation-deploy

https://gerrit.wikimedia.org/r/417279

Change 417281 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate kartotherian-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417281

Change 417281 merged by jenkins-bot:
[integration/config@master] Migrate trending-edits-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417281

Change 417282 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate 3d2png-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417282

Change 417283 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate citoid-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417283

Change 417284 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate cxserver-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417284

Change 417285 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate mathoid-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417285

Change 417286 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate graphoid-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417286

Change 417288 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Drop kartotherian-deploy-npm-node-6-jessie

https://gerrit.wikimedia.org/r/417288

Change 417289 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate mobileapps-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417289

hashar updated the task description. (Show Details)Mar 8 2018, 3:05 PM

Change 417282 merged by Hashar:
[integration/config@master] Migrate 3d2png-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417282

Change 417283 merged by Hashar:
[integration/config@master] Migrate citoid-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417283

Change 417284 merged by Hashar:
[integration/config@master] Migrate cxserver-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417284

Change 417285 merged by Hashar:
[integration/config@master] Migrate mathoid-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417285

Change 417286 merged by Hashar:
[integration/config@master] Migrate graphoid-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417286

Change 417288 merged by Hashar:
[integration/config@master] Drop kartotherian-deploy-npm-node-6-jessie

https://gerrit.wikimedia.org/r/417288

Change 417289 merged by Hashar:
[integration/config@master] Migrate mobileapps-deploy-npm-node-6 to Docker

https://gerrit.wikimedia.org/r/417289

hashar updated the task description. (Show Details)Mar 8 2018, 9:35 PM

Change 417390 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove tilerator-deploy-npm-node-6-jessie

https://gerrit.wikimedia.org/r/417390

hashar updated the task description. (Show Details)Mar 8 2018, 9:39 PM

Change 417390 merged by jenkins-bot:
[integration/config@master] Remove tilerator-deploy-npm-node-6-jessie

https://gerrit.wikimedia.org/r/417390

Change 417453 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Experimental docker jobs for parsoid/deploy

https://gerrit.wikimedia.org/r/417453

Change 417453 merged by jenkins-bot:
[integration/config@master] Experimental docker jobs for parsoid/deploy

https://gerrit.wikimedia.org/r/417453

hashar updated the task description. (Show Details)Mar 8 2018, 10:36 PM

parsoidsvc-deploy-npm-node-6-jessie fails with:

00:04:10.685 > parsoid@0.8.0 eslint /src/src
00:04:10.685 > eslint bin lib tests tools core-upgrade.js

00:04:40.032 /src/src/lib/html2wt/WikitextSerializer.js
00:04:40.033   1005:1  error  Expected indentation of 1 tab but found 4  indent
00:04:40.033   1006:1  error  Expected indentation of 1 tab but found 4  indent
00:04:40.034   1007:1  error  Expected indentation of 1 tab but found 4  indent
00:04:40.034   1008:1  error  Expected indentation of 1 tab but found 4  indent

Change 417932 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Migrate some parsoid/deploy jobs to Docker

https://gerrit.wikimedia.org/r/417932

Change 417960 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: npm-test support for shrinkwrap

https://gerrit.wikimedia.org/r/417960

Change 417960 merged by jenkins-bot:
[integration/config@master] docker: npm-test support for shrinkwrap

https://gerrit.wikimedia.org/r/417960

hashar updated the task description. (Show Details)Mar 9 2018, 5:56 PM

parsoidsvc-deploy-npm-node-6-jessie fails with:

00:04:10.685 > parsoid@0.8.0 eslint /src/src
00:04:10.685 > eslint bin lib tests tools core-upgrade.js
00:04:40.032 /src/src/lib/html2wt/WikitextSerializer.js
00:04:40.033   1005:1  error  Expected indentation of 1 tab but found 4  indent
00:04:40.033   1006:1  error  Expected indentation of 1 tab but found 4  indent
00:04:40.034   1007:1  error  Expected indentation of 1 tab but found 4  indent
00:04:40.034   1008:1  error  Expected indentation of 1 tab but found 4  indent

It uses a script that installed a different eslint version. https://gerrit.wikimedia.org/r/417960 uses npm install --only=dev whenever there is a npm-shrinkwrap.json file and that fixed it.

Change 417932 merged by jenkins-bot:
[integration/config@master] Migrate parsoid/deploy jobs to Docker

https://gerrit.wikimedia.org/r/417932

Change 418025 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Clean up npm-oid stuff and /deploy job template

https://gerrit.wikimedia.org/r/418025

Change 418025 merged by jenkins-bot:
[integration/config@master] Clean up npm-oid stuff and /deploy job template

https://gerrit.wikimedia.org/r/418025

Left to move excluding mediawiki stuff:

integration-config-dib-jessie
language-screenshots-VisualEditor
search-xgboost-maven
selenium-CirrusSearch-jessie
selenium-Popups-jessie
selenium-RelatedArticles-jessie
wikidata-query-rdf-maven-site-publish
wikimedia-fundraising-civicrm-jessie
wikimedia-portals-build

hashar updated the task description. (Show Details)Mar 19 2018, 10:19 AM
hashar updated the task description. (Show Details)Mar 19 2018, 5:01 PM
greg updated the task description. (Show Details)Mar 19 2018, 6:36 PM

Change 414703 merged by jenkins-bot:
[integration/config@master] Delete wikimedia-fundraising-civicrm-jessie

https://gerrit.wikimedia.org/r/414703

Change 420662 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove maven templates based on Nodepool

https://gerrit.wikimedia.org/r/420662

Change 420663 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove now unused '{name}-npm-run-{script}-node-6-jessie'

https://gerrit.wikimedia.org/r/420663

Change 420662 merged by jenkins-bot:
[integration/config@master] Remove maven templates based on Nodepool

https://gerrit.wikimedia.org/r/420662

Change 420663 merged by jenkins-bot:
[integration/config@master] Remove now unused '{name}-npm-run-{script}-node-6-jessie'

https://gerrit.wikimedia.org/r/420663

hashar closed this task as Resolved.Mar 20 2018, 9:40 AM

Almost all of the long tail got migrated. What is left will be dealt with via the super task T190097 and MediaWiki related jobs via T183512