Page MenuHomePhabricator

[keyresult] Migrate as many misc CI jobs as possible to Nodepool
Closed, ResolvedPublic

Description

Nov 4th 201653 jobs
Jan 10th 201746 jobs
Mar 13th 201726 jobs
Mar 14th 2017 23 jobs
Mar 30th 201721 jobs
May 1st 201720 jobs
$ grep -l UbuntuTrusty *
analytics-refinery-release
analytics-refinery-update-jars
analytics-wikistats
composer-package-validate
composer-validate
jsduck
mediawiki-core-jsduck
mediawiki-vendor-composer-security
mwext-CirrusSearch-whitespaces
mwext-VisualEditor-jsduck
mwgate-composer-validate
mwgate-jsduck
operations-dns-tabs
operations-mw-config-typos
performance-webpagetest-wmf
performance-webpagetest-wpt-org
php-compile-hhvm
php-compile-hhvm-test
php-compile-php55
wikimedia-fundraising-civicrm

Details

Related Gerrit Patches:

Related Objects

StatusAssignedTask
DuplicateNone
Resolvedhashar
DuplicateNone
Resolvedhashar
Resolvedhashar
Resolvedgreg
Resolvedhashar
Resolvedhashar
ResolvedLegoktm
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Declinedhashar
Resolvedhashar
Declinedhashar
Resolvedhashar
Declinedhashar
Resolvedhashar
Resolvedhashar
Resolvedmobrovac
Resolvedhashar
Resolvedhashar
Resolvedmobrovac
Resolvedhashar
DuplicateNone
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
ResolvedPaladox
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
InvalidNone
Resolvedhashar
DeclinedNone
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedzeljkofilipin
Resolveddduvall
Duplicatezeljkofilipin
Resolvedhashar
Resolvedhashar
Resolvedhashar
Resolvedhashar
ResolvedEjegg

Event Timeline

greg created this task.Nov 19 2015, 10:53 PM
greg raised the priority of this task from to Medium.
greg updated the task description. (Show Details)
greg added subscribers: hashar, Aklapper, greg.
greg set Security to None.
greg renamed this task from Migrate as many misc CI jobs as possible to Nodepool to [keyresult] Migrate as many misc CI jobs as possible to Nodepool.Jan 20 2016, 12:14 AM
hashar added a comment.Jun 6 2016, 9:40 PM

More or less, at least part 1 is complete. Part 2 is T119138 and is made of:

  • T119139 Migrate PHP based jobs, a lot of that is accomplished
  • T119140 (this task): leftover / misc

I am willing to complete as much as what is remaining from PHP and will reuse this misc task as a parent task for the huge remaining.

Currently 58 jobs are on Nodepool and 195 on permanent slaves (the ratio does not reflect the actual usage, some jobs are way more used and are already on Nodepool). Random examples:

So potentially that bullet list would end up as sub tasks here :-)

Change 314688 had a related patch set uploaded (by Hashar):
Let castor save from publish pipeline

https://gerrit.wikimedia.org/r/314688

Change 314690 had a related patch set uploaded (by Hashar):
Migrate unicodejs-publish to Nodepool

https://gerrit.wikimedia.org/r/314690

Change 314688 merged by jenkins-bot:
Let castor save from postmerge pipeline

https://gerrit.wikimedia.org/r/314688

Change 314690 merged by jenkins-bot:
Migrate unicodejs-publish to Nodepool

https://gerrit.wikimedia.org/r/314690

Change 315919 had a related patch set uploaded (by Hashar):
tox doc publishing jobs to Nodepool/Jessie

https://gerrit.wikimedia.org/r/315919

Change 315919 merged by jenkins-bot:
tox doc publishing jobs to Nodepool/Jessie

https://gerrit.wikimedia.org/r/315919

Change 317830 had a related patch set uploaded (by Hashar):
phpunit-coverage-publish to Nodepool

https://gerrit.wikimedia.org/r/317830

Change 317830 merged by jenkins-bot:
phpunit-coverage-publish to Nodepool

https://gerrit.wikimedia.org/r/317830

Change 319292 had a related patch set uploaded (by Hashar):
dib: doxygen on Jessie images

https://gerrit.wikimedia.org/r/319292

Change 319292 merged by jenkins-bot:
dib: doxygen on Jessie images

https://gerrit.wikimedia.org/r/319292

Mentioned in SAL (#wikimedia-releng) [2016-11-02T10:47:39Z] <hashar> Force refresh Nodepool snapshot for Jessie so it get doxygen included T119140

Change 319296 had a related patch set uploaded (by Hashar):
Migrate doxygen jobs to Nodepool/Jessie

https://gerrit.wikimedia.org/r/319296

Change 319296 merged by jenkins-bot:
Migrate doxygen jobs to Nodepool/Jessie

https://gerrit.wikimedia.org/r/319296

Change 319305 had a related patch set uploaded (by Hashar):
mediawiki-core-doxygen-publish to Jessie/Nodepool

https://gerrit.wikimedia.org/r/319305

Change 319305 merged by jenkins-bot:
mediawiki-core-doxygen-publish to Jessie/Nodepool

https://gerrit.wikimedia.org/r/319305

Change 319324 had a related patch set uploaded (by Hashar):
php-compile jobs to Nodepool

https://gerrit.wikimedia.org/r/319324

Change 319332 had a related patch set uploaded (by Hashar):
mediawiki-core-jsduck-publish migrate to Nodepool

https://gerrit.wikimedia.org/r/319332

Change 319332 merged by jenkins-bot:
mediawiki-core-jsduck-publish migrate to Nodepool

https://gerrit.wikimedia.org/r/319332

Change 319844 had a related patch set uploaded (by Hashar):
Migrate maven jobs to Jessie/Nodepool

https://gerrit.wikimedia.org/r/319844

Change 319844 merged by jenkins-bot:
Migrate maven jobs to Jessie/Nodepool

https://gerrit.wikimedia.org/r/319844

hashar updated the task description. (Show Details)Nov 4 2016, 2:01 PM

Change 319324 merged by jenkins-bot:
php-compile experimental jobs on Nodepool

https://gerrit.wikimedia.org/r/319324

hashar updated the task description. (Show Details)Jan 10 2017, 11:33 AM
hashar updated the task description. (Show Details)Mar 13 2017, 4:57 PM

selenium-* jobs are now on their own label / dedicated slaves.

Change 342508 had a related patch set uploaded (by Hashar):
[integration/config] Migrate mw-tools-scap-tox-doc-publish to Nodepool

https://gerrit.wikimedia.org/r/342508

Change 342508 merged by jenkins-bot:
[integration/config] Migrate mw-tools-scap-tox-doc-publish to Nodepool

https://gerrit.wikimedia.org/r/342508

Change 342515 had a related patch set uploaded (by Hashar):
[integration/config] Migrate apps-android-java-mwapi to nodepool

https://gerrit.wikimedia.org/r/342515

Change 342515 merged by jenkins-bot:
[integration/config] Migrate apps-android-java-mwapi to nodepool

https://gerrit.wikimedia.org/r/342515

hashar updated the task description. (Show Details)Mar 14 2017, 10:21 AM

Change 342604 had a related patch set uploaded (by Hashar):
[operations/puppet] Migrate typos check to a rake task

https://gerrit.wikimedia.org/r/342604

Change 342604 merged by Filippo Giunchedi:
[operations/puppet@production] Migrate typos check to a rake task

https://gerrit.wikimedia.org/r/342604

Change 344444 had a related patch set uploaded (by Hashar):
[integration/config@master] Delete operations-puppet-typos

https://gerrit.wikimedia.org/r/344444

Change 344444 merged by jenkins-bot:
[integration/config@master] Delete operations-puppet-typos

https://gerrit.wikimedia.org/r/344444

EddieGP updated the task description. (Show Details)Mar 28 2017, 11:54 AM
hashar updated the task description. (Show Details)Mar 31 2017, 9:45 AM
EddieGP updated the task description. (Show Details)Apr 7 2017, 7:56 PM
hashar updated the task description. (Show Details)May 1 2017, 8:09 PM
greg moved this task from INBOX to Epics on the Release-Engineering-Team board.May 20 2017, 3:53 PM

Change 319331 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Move wikimedia-fundraising-civicrm to Nodepool

https://gerrit.wikimedia.org/r/319331

Change 319331 merged by jenkins-bot:
[integration/config@master] Move wikimedia-fundraising-civicrm to Nodepool

https://gerrit.wikimedia.org/r/319331

Change 368145 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Revert "Move wikimedia-fundraising-civicrm to Nodepool"

https://gerrit.wikimedia.org/r/368145

Change 368148 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] wikimedia-fundraising-civicrm-jessie as experimental job

https://gerrit.wikimedia.org/r/368148

Change 368145 merged by jenkins-bot:
[integration/config@master] Revert "Move wikimedia-fundraising-civicrm to Nodepool"

https://gerrit.wikimedia.org/r/368145

Change 368148 merged by jenkins-bot:
[integration/config@master] wikimedia-fundraising-civicrm-jessie as experimental job

https://gerrit.wikimedia.org/r/368148

I blindly switched the wikimedia-fundraising-civicrm job to the Nodepool disposable instances however it is failing due to a missing org.wikimedia.omnimail

++++ cat sites/default/enabled_modules
+++ drush -y en advanced_help adyen_audit amazon_audit astropay_audit banner_history civicrm contribution_tracking ctools environment_indicator exchange_rates ganglia_reporter globalcollect_audit ingenico_audit large_donation libraries oauth_common oauth_common_providerui offline2civicrm oneoffs queue2civicrm recurring recurring_globalcollect rest_server services services_oauth thank_you wmf_audit wmf_campaigns wmf_civicrm wmf_common wmf_communication wmf_fredge_qc wmf_logging wmf_refund_qc wmf_reports wmf_unsubscribe_qc
PHP Deprecated:  Comments starting with '#' are deprecated in /etc/php5/cli/conf.d/20-xhprof.ini on line 2 in Unknown on line 0
The following extensions will be enabled: advanced_help, adyen_audit, amazon_audit, astropay_audit, banner_history, civicrm, contribution_tracking, ctools, environment_indicator, exchange_rates, ganglia_reporter, globalcollect_audit, ingenico_audit, large_donation, libraries, oauth_common, oauth_common_providerui, offline2civicrm, oneoffs, queue2civicrm, recurring, recurring_globalcollect, rest_server, services, services_oauth, thank_you, wmf_audit, wmf_campaigns, wmf_civicrm, wmf_common, wmf_communication, wmf_fredge_qc, wmf_logging, wmf_refund_qc, wmf_reports, wmf_unsubscribe_qc
Do you really want to continue? (y/n): y
WD php: CiviCRM_API3_Exception: Unknown extension:                       [error]
org.wikimedia.omnimail in civicrm_api3() (line 45 of
/home/jenkins/workspace/wikimedia-fundraising-civicrm-jessie/src/wikimedia/fundraising/crm/civicrm/api/api.php).

Full console log captured as P5807

I have restored the previous job and the nodepool job (suffixed with -jessie) is now in the experimental pipeline and thus can be triggered manually by commenting in Gerrit check experimental.

hashar closed this task as Resolved.Feb 8 2018, 7:57 AM
hashar claimed this task.

This has been mostly done. The long tail of jobs still on permanent slaves will end up being migrated to Docker containers since Nodepool is now legacy.