Page MenuHomePhabricator

Migrate leftover tox jobs to CI Nodepool
Closed, ResolvedPublic

Description

Yesterday I noticed we still have some tox jobs running on the permanent slave. An example is tox-flake8, we also have jobs suffixed with -trusty which was to use python 3.4.

This task is about:

  • identify all tox jobs still running on permanent slaves
  • update the JJB config to use Nodepool (node: ci-jessie-wikimedia and parameters: zeromq)
  • update jobs
  • verify it works on the various repo
  • celebrate

Status as of 2bb918dd220ace0f3bc802c69c1c3c6ff949b15d April 5th 09:07 UTC

$ grep -l tox output/*|xargs grep -l assignedNode.*Ubuntu|cut -d/  -f2
mw-tools-releng-tox-doc-publish
mw-tools-scap-tox-doc-publish
pywikibot-core-tox-doc-publish
pywikibot-tests-beta-cluster
tox-flake8

The publish jobs will be annoying they push to integration-publisher then trigger a job that grab the material from gallium. https://www.mediawiki.org/wiki/Continuous_integration/Documentation_generation Can be migrated later.

Event Timeline

hashar created this task.Feb 11 2016, 9:55 AM
hashar assigned this task to zeljkofilipin.
hashar raised the priority of this task from to Medium.
hashar updated the task description. (Show Details)
hashar added subscribers: Aklapper, greg, hashar, JanZerebecki.
hashar updated the task description. (Show Details)Feb 12 2016, 10:54 AM
hashar set Security to None.

Will pair this one with @zeljkofilipin for training purposes whenever he is done with browser tests related work.

greg added a subscriber: zeljkofilipin.
zeljkofilipin removed zeljkofilipin as the assignee of this task.Mar 10 2016, 2:10 PM
hashar lowered the priority of this task from Medium to Low.Mar 22 2016, 2:09 PM

Change 278896 had a related patch set uploaded (by Hashar):
[operations/software/conftool] migrate to tox-jessie

https://gerrit.wikimedia.org/r/278896

Change 278896 merged by jenkins-bot:
[operations/software/conftool] migrate to tox-jessie

https://gerrit.wikimedia.org/r/278896

Change 281429 had a related patch set uploaded (by Hashar):
Clear out some legacy tox* jobs

https://gerrit.wikimedia.org/r/281429

Change 281429 merged by jenkins-bot:
Clear out some legacy tox* jobs

https://gerrit.wikimedia.org/r/281429

hashar updated the task description. (Show Details)Apr 4 2016, 12:43 PM

Change 281431 had a related patch set uploaded (by Hashar):
Make Jenkins to run flake8/flake8-py3

https://gerrit.wikimedia.org/r/281431

Change 281433 had a related patch set uploaded (by Hashar):
[pywikibot/wikibase] migrate to tox-jessie

https://gerrit.wikimedia.org/r/281433

Change 281434 had a related patch set uploaded (by Hashar):
mediawiki/tools/releng migrate to tox-jessie

https://gerrit.wikimedia.org/r/281434

Change 281434 merged by jenkins-bot:
mediawiki/tools/releng migrate to tox-jessie

https://gerrit.wikimedia.org/r/281434

Change 281433 merged by jenkins-bot:
[pywikibot/wikibase] migrate to tox-jessie

https://gerrit.wikimedia.org/r/281433

Change 281431 merged by jenkins-bot:
Make Jenkins to run flake8/flake8-py3

https://gerrit.wikimedia.org/r/281431

hashar claimed this task.Apr 5 2016, 8:52 AM
hashar moved this task from Backlog to In progress on the Continuous-Integration-Config board.

Change 281608 had a related patch set uploaded (by Hashar):
[pywikibot/core] move some tox jobs to Nodepool

https://gerrit.wikimedia.org/r/281608

Change 281608 merged by jenkins-bot:
[pywikibot/core] move some tox jobs to Nodepool

https://gerrit.wikimedia.org/r/281608

Change 281613 had a related patch set uploaded (by Hashar):
[pywikibot/wikibase] move some tox jobs to Nodepool

https://gerrit.wikimedia.org/r/281613

Change 281613 merged by jenkins-bot:
[pywikibot/wikibase] move some tox jobs to Nodepool

https://gerrit.wikimedia.org/r/281613

hashar added a comment.Apr 5 2016, 9:20 AM

pywikibot repositories got migrated :-)

hashar updated the task description. (Show Details)Apr 5 2016, 9:21 AM

Change 281616 had a related patch set uploaded (by Hashar):
[analytics/limn-language-data] drop tox jobs

https://gerrit.wikimedia.org/r/281616

Change 281616 merged by jenkins-bot:
[analytics/limn-language-data] drop tox jobs

https://gerrit.wikimedia.org/r/281616

Change 281617 had a related patch set uploaded (by Hashar):
Have Jenkins solely run flake8

https://gerrit.wikimedia.org/r/281617

Change 281618 had a related patch set uploaded (by Hashar):
[analytics/wikimetrics] Migrate tox job to Nodepool

https://gerrit.wikimedia.org/r/281618

Change 281618 merged by jenkins-bot:
[analytics/wikimetrics] Migrate tox job to Nodepool

https://gerrit.wikimedia.org/r/281618

Change 281617 merged by jenkins-bot:
Have Jenkins solely run flake8

https://gerrit.wikimedia.org/r/281617

Change 281619 had a related patch set uploaded (by Hashar):
Remove tox templates that ran on permanent slaves

https://gerrit.wikimedia.org/r/281619

Change 281619 merged by jenkins-bot:
Remove tox templates that ran on permanent slaves

https://gerrit.wikimedia.org/r/281619

hashar closed this task as Resolved.Apr 5 2016, 10:02 AM

The publish jobs need a proper infra/system to be able to grab artifacts from the job running on Nodepool instances and have them exposed publicly.

pywikibot-tests-beta-cluster is triggered by Jenkins via a midnight cron. As such it can not run on Nodepool since the job can't instruct the Gearman plugin to disconnect the slave.

Those would be migrated later. Overall the bulk of tox jobs have been switched to Nodepool.