Page MenuHomePhabricator
Create Task
Maniphest T114315

[keyresult] Migrate majority of CI jobs to Nodepool (part 1)
Closed, ResolvedPublic
Actions

Description

Subtasks (where work/discussion will happen) will be created and/or marked as blockers of this one.

Follow-on is T119138: [keyresult] Migrate majority of CI jobs to Nodepool (part 2).

Related Objects
Search...

StatusSubtypeAssignedTask
 DuplicateNoneT125017 Switch Parsoid to Jessie and Node 4.2
 ResolvedhasharT119138 [keyresult] Migrate majority of CI jobs to Nodepool (part 2)
 DuplicateNoneT136249 Track what tests we have left to convert to nodepool
 ResolvedhasharT119140 [keyresult] Migrate as many misc CI jobs as possible to Nodepool
 ResolvedhasharT114315 [keyresult] Migrate majority of CI jobs to Nodepool (part 1)
 ResolvedgregT119141 Migrate python tox CI jobs to Nodepool
 ResolvedhasharT103251 Create generic "tox" job that runs multiple environments
ResolvedhasharT126588 Migrate leftover tox jobs to CI Nodepool
 ResolvedLegoktmT126532 Wikipedia Android CI tests are failing
 ResolvedhasharT119143 Migrate javascript npm CI jobs to Nodepool
 ResolvedhasharT113903 Puppetize npm/grunt manual setup
 ResolvedhasharT95003 Use systemd for xvfb service on Debian/Jessie
 ResolvedhasharT112560 [tracking] Disposable VMs need a cache for package managers
 ResolvedhasharT112561 Evaluate angry-caching-proxy as a package managers cache
 ResolvedhasharT114871 Evaluate devpi for caching Pypi python packages
 DeclinedhasharT116015 Investigate using a Squid based man in the middle proxy to cache package manager SSL connections
 ResolvedhasharT116017 Investigate using a cache store/restore system for package managers
 DeclinedhasharT116038 Investigate using Drydock for CI
 ResolvedhasharT117207 Puppetize / play out with devpi
 DeclinedhasharT119196 Investigate using gemstash
 ResolvedhasharT126991 Migrate services CI jobs to node 4.3
 ResolvedhasharT106668 Enable jenkins test & submit for graphoid repo
 Resolved• mobrovacT119693 Extract Graphoid Debian packages dependencies to a standalone puppet class so they can be installed on CI slaves
 ResolvedhasharT129092 Hiera is not properly configured on Nodepool instances
 ResolvedhasharT126230 Provision pkg-config on Nodepool instances
 Resolved• mobrovacT124447 Enable node4 for mathoid tests (mathoid/deploy tests fail)
 ResolvedhasharT126246 Provision openjdk-7-jre-headless on Nodepool slaves
 DuplicateNoneT129880 Give @mobrovac access to CI instances
 ResolvedhasharT126992 Migrate Parsoid CI jobs to Nodepool and add Node 4.3 support
 ResolvedhasharT124474 Package npm 2.14
 ResolvedhasharT126995 Migrate mediawiki/services/mobileapps CI jobs from node 0.10 to 4.3
 ResolvedhasharT127447 Migrate cxserver CI jobs to nodeJs 4.3
 ResolvedPaladoxT128089 Migrate OOjs npm CI job to Nodepool
 ResolvedhasharT128091 Migrate OOjs UI npm, npm-run-doc and npm-run-demos CI jobs to Nodepool
 ResolvedhasharT128092 Provide composer on the nodepool servers so OOjs UI can use it in the npm job
 ResolvedhasharT128093 Migrate VisualEditor core npm CI job to Nodepool
 ResolvedhasharT128090 Provide the Chrome binary on the nodepool servers so OOjs, VE core can use it in testing
 ResolvedhasharT129320 Diskimage fail on xvfb/systemd
 ResolvedhasharT129345 Xvfb service does not start on Nodepool instances
 InvalidNoneT128135 Jessie should have the ability to run several tests at once like UbuntuTrusty can
 ResolvedhasharT130207 Migrate pywikibot/i18n npm job to Nodepool
 DeclinedNoneT130208 Jobs should checkout source in a subdirectory of the workspace to avoid polluting it with CI related files
 ResolvedhasharT130587 Migrate MobileFrontend npm job to Nodepool
 ResolvedhasharT109005 Switch CI from jsduck deb package to a gemfile/bundler system
 ResolvedhasharT136096 Create jsduck job that runs on Nodepool instances
 ResolvedhasharT132144 Migrate mediawiki/extensions/Collection/OfflineContentGenerator npm jobs to Nodepool instances / Node 4.3
 ResolvedzeljkofilipinT114860 Move Bundler Jenkins jobs to Nodepool instances
 ResolveddduvallT104024 Define a CI entry point for Ruby code
 DuplicatezeljkofilipinT114262 Delete ruby2.0lint job and only run bundle-rubocop job for repositories with Ruby code

Event Timeline

greg created this task.Sep 30 2015, 9:35 PM
greg raised the priority of this task from to Medium.
greg updated the task description. (Show Details)
greg added a project: releng-201516-q2.
greg moved this task to Backlog on the releng-201516-q2 board.
greg added subscribers: greg, hashar.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 30 2015, 9:35 PM
greg updated the task description. (Show Details)Sep 30 2015, 10:11 PM
greg set Security to None.
JanZerebecki subscribed.Nov 4 2015, 2:38 PM
greg renamed this task from [keyresult] Migrate majority of CI jobs to Nodepool to [keyresult] Migrate majority of CI jobs to Nodepool (part 1).Nov 19 2015, 10:49 PM

I changed the title to include "part 1" because this won't be completed completed by Dec 31st.

Jobs that we hope to have migrated by then, however, are:

  • python tox
  • ruby rake
  • javascript npm
greg updated the task description. (Show Details)Nov 19 2015, 10:53 PM
hashar added a subtask: T114860: Move Bundler Jenkins jobs to Nodepool instances.Nov 20 2015, 2:48 PM
Jdforrester-WMF mentioned this in T119138: [keyresult] Migrate majority of CI jobs to Nodepool (part 2).Nov 20 2015, 5:48 PM
Jdforrester-WMF updated the task description. (Show Details)
Jdforrester-WMF updated the task description. (Show Details)
greg removed a subtask: T119142: Migrate ruby rake CI jobs to Nodepool.Nov 25 2015, 3:53 PM
greg updated the task description. (Show Details)Nov 30 2015, 5:15 PM
greg closed subtask T119141: Migrate python tox CI jobs to Nodepool as Resolved.Dec 1 2015, 4:47 PM
greg updated the task description. (Show Details)
greg updated the task description. (Show Details)Dec 1 2015, 4:49 PM
hashar added a parent task: T119140: [keyresult] Migrate as many misc CI jobs as possible to Nodepool.Dec 3 2015, 8:39 AM
greg closed subtask T114860: Move Bundler Jenkins jobs to Nodepool instances as Resolved.Dec 3 2015, 6:56 PM
greg updated the task description. (Show Details)
Ricordisamoa subscribed.Mar 8 2016, 7:12 PM
Paladox subscribed.Mar 16 2016, 12:55 PM
hashar mentioned this in T136249: Track what tests we have left to convert to nodepool.May 25 2016, 9:23 PM
hashar closed subtask T119143: Migrate javascript npm CI jobs to Nodepool as Resolved.May 25 2016, 9:38 PM
hashar updated the task description. (Show Details)
hashar closed this task as Resolved.May 25 2016, 9:42 PM
hashar claimed this task.
hashar added subscribers: • mobrovac, jayvdb, Jdforrester-WMF.

Python/Ruby/Javascript jobs have been almost all migrated. There are a few left over but they need some architecture work or are jobs that need to be heavily refactored.

That took a while, I have largely underestimated the amount of side maintenance work I have to conduct as well as all the puppet provisioning and crazy double check before migrate. We also had some weird cases such as requiring Trusty or npm jobs relying on PHP composer or Ruby JSDuck.

Overall it went fine with a tremendous help from a lot of people, specially:

And more!!!!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits