Page MenuHomePhabricator
Create Task
Maniphest T190097

Migrate all CI jobs from Nodepool, deprecate its use
Closed, ResolvedPublic
Actions

Description

We want to move away from Nodepool for many reasons. We will instead be using a docker-based setup. In the longer term this setup will match what is being used in production (blubber, helm, k8s).

This task is to track all needs for deprecating Nodepool use.

Details

SubjectRepoBranchLines +/-
Archive operations/debs/nodepoolintegration/configmaster+3 -1
Drop integration-config-dib-jessieintegration/configmaster+0 -42
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT220199 Quibble: deprecate and phase out EXT_DEPENDENCIES / SKIN_DEPENDENCIES
 Resolved• hasharT190097 Migrate all CI jobs from Nodepool, deprecate its use
 Resolved• hasharT183512 [EPIC] Migrate Mediawiki jobs from Nodepool to Docker
 Resolved• hasharT190503 MediaWiki PHP based built-in server does not output log requests for index.php queries
 Resolved• hasharT189567 Phaseout CI mediawiki config / extensions_load.txt to load extensions
 Resolved• hasharT188717 Move Wikibase to using the normal mediawiki extension (quibble) jobs
 Resolved• hasharT193767 Scribunto: Scribunto_LuaStandaloneInterpreterTest::testGetStatus Initial vsize Failed asserting that 0 is greater than 0
 ResolvedMaxSemT193855 [GeoData] extension.json config is not loaded by install.php --with-extensions
 ResolvedCommunityTechBotT195020 AntiSpoof extension does not pass quibble-vendor-mysql-php70-docker
 DuplicateNoneT195585 TimezoneSelector isLoggedIn() on null ($wgUser)
 Resolved• hasharT195634 quibble: with hhvm, "could not load extension asset expand.svg" due to lack of proper Content-Type header
 ResolvedLegoktmT195766 [AccessControl] Special:Version causes Invalid or virtual namespace -1 given.
 Resolved• hasharT180171 ArticlePlaceholder should use MediaWiki qunit runner
 Resolved• hasharT195783 [PropertySuggester] MWException from line 167 of /workspace/src/includes/Hooks.php: Invalid callback PropertySuggester\Hooks::onCreateSchema in hooks for LoadExtensionSchemaUpdates
 ResolvedEjeggT195814 [ContributionTracking] default DB settings differs when using extension registration
 Resolved• hasharT195831 [AkismetKlik] MWException Set $wgAKkey in LocalSettings.php or relevant configuration file.
 DeclinedJayprakash12345T190268 Getting Call to a member function isLoggedIn() on null Error in Extension:TimezoneSelector
 Resolved• hasharT196013 Get Wikibase + dependencies to run with Quibble
 Resolved• hasharT196062 EXT_DEPENDENCIES no more being injected on quibble jobs
 ResolvedAnomieT191537 MediaWiki core's selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job
 ResolvedOsnardT196084 [BlueSpiceFoundation] onMakeGlobalVariablesScript calls deprecated BsConfig::loadSettings
 ResolvedRequestMarcoAurelioT196442 Archive the InterwikiIntegration extension
 ResolvedRequestMarcoAurelioT196433 Archive the EditOwn extension
 Resolved• hasharT196431 [EditNotify] Invalid argument supplied for foreach() in extensions/EditNotify/EditNotify.hooks.php on line 917
 ResolvedRequestMarcoAurelioT196430 Archive the DidYouMean extension
 ResolvedRequestMarcoAurelioT196429 Archive the CustomPage skin
 DeclinedNoneT196436 EditSubpages extension fails mediawiki/core webdriver.io tests
 DeclinedNoneT196444 [Html2Wiki] PHP Notice: Undefined offset: 0 in extensions/Html2Wiki/specials/SpecialHtml2Wiki.php on line 336
 Resolved• hasharT181898 Special:Preferences "Global default '' is invalid for field theme" MWException on 1.30 RC
Resolved• hasharT151871 SpellingDictionary: Message 'uls-select-language' required by 'ext.SpellingDictionary.viewByLanguage' must exist
 ResolvedPastakhovT188523 Mpdf extension fails PHP 7.0 lint
 Resolved• hasharT196960 Quibble should have a way for extensions to opt out of core selenium browser tests
 Resolved• hasharT197469 Migrate gated extensions jobs to Quibble/Docker
 ResolvedthiemowmdeT197721 [Translate] Error: 1283 Column 'tmf_text' cannot be part of FULLTEXT index
 Resolved• hasharT197933 CI no more inject extensions dependencies on REL branches
 Resolved• hasharT200017 Quibble should be able to be forced to run extensions testsuite when triggered from mediawiki/core
 Resolved• hasharT197687 Wikispeech QUnit tests fail with Chrome 66: play() failed because the user didn't interact with the document first.
 ResolvedLucas_Werkmeister_WMDET198077 [Wikidata.org] AutoLoaderStructureTest::testPSR4Completeness fails
 Resolved• hasharT198120 [TrustedXFF] on fresh install trusted-xff.cdb is missing and User::getDefaultOptions() cause a fatal
 Resolved• hasharT198171 [RevisionSlider] QUnit test ext.RevisionSlider.DiffPage "Push state" fails due to Chrome rate limiting history.pushState
 ResolvedPRODUCTION ERRORLegoktmT198173 [LabeledSectionTransclusion] Use of LanguageGetMagic hook (used in LabeledSectionTransclusion::setupMagic) was deprecated in MediaWiki 1.16
 Resolved• hasharT198229 [Graph] extension fails QUnit tests due to timeout
 Resolved• hasharT198331 [Cognate] install fails with BadMethodCallException from line 83 of includes/libs/rdbms/lbfactory/LBFactorySingle.php: Method is not supported.
 Resolved• hasharT198330 [GlobalPreferences] install fails with MultiConfig::get: undefined option: 'GlobalPreferencesDB'
 ResolvedKarimaT198874 LinkedWiki extension requires php 7.1 due to bordercloud/sparql
 DeclinedNoneT199032 [MOOC] MoocContentHandler::makeEmptyContent() did not return a valid content (MoocContent::isValid())
 Resolved• hasharT199047 Add "gmp" php extension to Quibble images
 Resolved• hasharT195918 Move MediaWiki extension PHPUnit coverage jobs to docker + quibble
 Resolved• hasharT115436 ResourcesTest::testFileExistence does not ignore anchor in url
 Resolved• hasharT200063 [intersection] parser tests fail and CI ignore tests
 ResolvedAddshoreT201336 Use Quibble for WikibaseLexeme CI
 ResolvedAddshoreT197771 Couple errors with Wikibase\Lexeme\Tests\MediaWiki\Diff\LexemeDiffVisualizerIntegrationTest
 ResolvedAddshoreT201356 Fix QUnit failures blocking using quibble for WikibaseLexeme in CI
 ResolvedAddshoreT200891 Warnings in qunit tests on quibble: "Cannot read property 'errors' of undefined'"
 ResolvedAddshoreT200890 Warnings in qunit tests on quibble: "Cannot read property 'id' of undefined'"
 ResolvedAddshoreT201337 Quibble & WikibaseLexeme: selenium, TypeError: Cannot read property 'value' of undefined
 Resolved• hasharT202523 Migrate parsoidsvc-hhvm-parsertests-jessie off of Nodepool
 Resolved• hasharT203084 Migrate DonationInterface CI job to Docker
 Resolved• hasharT203178 Migrate Selenium tests based on the ruby library mediawiki_selenium
 Resolved• hasharT187797 Migrate leftover Nodepool jobs to Docker
 Declined• hasharT188256 Migrate pywikibot-tests-beta-cluster to a tox env in pywikibot/core
 Resolved• hasharT188285 Running mvn on wikidata/query/rdf fails with: /bin/sh: 1: npm: not found
 Resolved• hasharT115755 Rewrite mediawiki-core-doxygen-publish Jenkins job to poll scm instead of being triggered by Zuul
 Resolved• hasharT189122 Migrate language-screenshots-VisualEditor off of Nodepool to Docker containers
 Resolved• hasharT190032 Migrate CI job search-xgboost-maven to use a Docker container
 Resolved• hasharT190042 Migrate CI job wikidata-query-rdf-maven-site-publish to use a Docker container

Event Timeline

greg triaged this task as High priority.Mar 19 2018, 10:23 PM
greg created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 19 2018, 10:23 PM
greg added subtasks: T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker, T187797: Migrate leftover Nodepool jobs to Docker.Mar 19 2018, 10:25 PM
greg moved this task from INBOX to Epics (ARCHIVED) on the Release-Engineering-Team board.
• hashar added a subtask: T190073: Migrate Jenkins jobs wikimedia-portals-build and wikidata-query-gui-build to Docker and to use an entry point (eg: npm builddeploy).Mar 20 2018, 9:40 AM
• hashar mentioned this in T187797: Migrate leftover Nodepool jobs to Docker.
• hashar closed subtask T187797: Migrate leftover Nodepool jobs to Docker as Resolved.
Jdforrester-WMF subscribed.Mar 20 2018, 4:28 PM
Jdforrester-WMF mentioned this in T184102: Provide the same version of Chrome/Chromium in Docker and Nodepool (or migrate qunit for extensions to Docker).Mar 22 2018, 7:09 PM
fgiunchedi subscribed.Mar 27 2018, 10:54 AM
EddieGP mentioned this in T192217: Remove the "check" pipeline and Zuul's user-filter.Apr 14 2018, 8:09 PM
Ebe123 renamed this task from Migrate all CI jobs from Nodepool, deprecate it's use to Migrate all CI jobs from Nodepool, deprecate its use.Apr 30 2018, 4:44 PM
• hashar removed a subtask: T190073: Migrate Jenkins jobs wikimedia-portals-build and wikidata-query-gui-build to Docker and to use an entry point (eg: npm builddeploy).Aug 21 2018, 9:59 AM
gerritbot subscribed.Nov 13 2018, 1:23 PM

Change 473199 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Archive operations/debs/nodepool

https://gerrit.wikimedia.org/r/473199

gerritbot added a project: Patch-For-Review.Nov 13 2018, 1:23 PM
gerritbot added a comment.Nov 13 2018, 1:27 PM

Change 473201 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Stop running rake test for integration/config

https://gerrit.wikimedia.org/r/473201

gerritbot added a comment.Nov 13 2018, 1:32 PM

Change 473202 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Remove Nodepool / diskimage-builder material

https://gerrit.wikimedia.org/r/473202

gerritbot added a comment.Nov 13 2018, 1:55 PM

Change 473201 merged by jenkins-bot:
[integration/config@master] Drop integration-config-dib-jessie

https://gerrit.wikimedia.org/r/473201

gerritbot added a comment.Nov 13 2018, 2:01 PM

Change 473199 merged by jenkins-bot:
[integration/config@master] Archive operations/debs/nodepool

https://gerrit.wikimedia.org/r/473199

• hashar added a parent task: T209361: Phase out Nodepool from production.Nov 13 2018, 2:23 PM
• hashar closed subtask T183512: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker as Resolved.Nov 15 2018, 7:25 PM
• hashar closed this task as Resolved.Nov 15 2018, 7:27 PM
• hashar claimed this task.
• hashar subscribed.

All jobs have been migrated. There are still some sub tasks opened but they are no more blockers for phasing out Nodepool.

• hashar removed a parent task: T209361: Phase out Nodepool from production.Nov 15 2018, 8:38 PM
• hashar mentioned this in T209361: Phase out Nodepool from production.
greg added a comment.Nov 16 2018, 5:23 PM
In T190097#4751523, @hashar wrote:

All jobs have been migrated. There are still some sub tasks opened but they are no more blockers for phasing out Nodepool.

\o/ CELEBRATION! Well done Antoine. I know this was a long and grueling and sometimes sad project. I really appreciated your effort here to do things as completely as possible.

Why did we think this was going to only take 6 months? :P I guess the normal "multiple by 3 any time estimates" is mostly still true :)

• mmodell awarded a token.Nov 16 2018, 5:48 PM
zeljkofilipin awarded a token.Nov 19 2018, 9:37 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits