Page MenuHomePhabricator
Create Task
Maniphest T349796

Move MediaWiki jobs to mw-on-k8s
Closed, ResolvedPublic
Actions

Description

We want to move all jobrunning to mediawiki on k8s, eventually.

We need to do the following:

  • Configure the mw-jobrunner deployment to use a similar php configuration to the production jobrunners
  • Move jobs, first one by one, in the configuration of changeprop to be pointed to the new deployment.

We will probably need to repurpose some servers to be k8s nodes when we move a fraction near to 100% of jobs to mw on k8s.

Use https://grafana.wikimedia.org/goto/TSO0UxHIz?orgId=1 to estimate the impact of moving a particular job.

Details

SubjectRepoBranchLines +/-
CommonSettings.php: Fix jobrunner hostnameoperations/mediawiki-configmaster+1 -1
service catalog: disable paging on jobrunner and videoscaler servicesoperations/puppetproduction+2 -0
mw-jobrunner: reduce replicas furtheroperations/deployment-chartsmaster+1 -1
changeprop: clean up k8s jobrunner referencesoperations/deployment-chartsmaster+5 -40
mw-jobrunner: begin to scale down replicasoperations/deployment-chartsmaster+1 -1
changeprop-jobqueue: migrate refreshLinks to k8soperations/deployment-chartsmaster+1 -1
mw-jobrunner: bump replicas in order to migrate refreshLinksoperations/deployment-chartsmaster+2 -2
mw-jobrunner: bump replicasoperations/deployment-chartsmaster+2 -2
jobqueue: migrate cirrusSearchLinksUpdate to k8soperations/deployment-chartsmaster+1 -1
mw-jobrunner: bump replicas for cirrusSearchLinksUpdateoperations/deployment-chartsmaster+2 -2
changeprop-jobqueue: disable ThumbnailRender on k8soperations/deployment-chartsmaster+1 -1
changeprop-jobqueue: migrate parsoidCachePrewarm to k8soperations/deployment-chartsmaster+1 -1
mw-jobrunner: increase replicas for parsoidCachePrewarmoperations/deployment-chartsmaster+1 -1
changeprop-jobqueue: move cirrusSearchElasticaWriteoperations/deployment-chartsmaster+1 -1
mw-jobrunner: increase replicasoperations/deployment-chartsmaster+1 -1
changeprop-jobqueue: move PublishStashedFile back to non-k8s jobrunneroperations/deployment-chartsmaster+5 -0
kubernetes::worker: add mw-jobrunner to poolsoperations/puppetproduction+0 -0
mw-jobrunner: increase replicasoperations/deployment-chartsmaster+1 -1
changeprop-jobqueue: migrate all low-traffic jobsoperations/deployment-chartsmaster+1 -0
changeprop-jobqueue: migrate all remaining small jobs, also cdnPurgeoperations/deployment-chartsmaster+10 -0
changeprop-jobqueue: restore cirrussearchlinksupdate to metaloperations/deployment-chartsmaster+0 -1
changeprop-jobqueue: migrate one large and a few small jobsoperations/deployment-chartsmaster+4 -0
jobqueue: migrate a larger job (and one smaller one)operations/deployment-chartsmaster+2 -0
jobqueue: migrate a heavyweight job to jobqueueoperations/deployment-chartsmaster+1 -0
mw-jobrunner: bump replicas furtheroperations/deployment-chartsmaster+1 -1
jobqueue: migrate a moderately weighty joboperations/deployment-chartsmaster+1 -0
mw-jobrunner: increase replicasoperations/deployment-chartsmaster+1 -1
jobqueue: switch a medium weight joboperations/deployment-chartsmaster+1 -0
jobqueue: migrate another medium-weight joboperations/deployment-chartsmaster+1 -0
changeprop-jobqueue: toggle another joboperations/deployment-chartsmaster+1 -0
changeprop-jobqueue: migrate more lower-impact jobsoperations/deployment-chartsmaster+4 -0
jobqueue: move move jobs to k8s jobrunneroperations/deployment-chartsmaster+5 -0
changejob-jobqueue: move two more jobs to k8soperations/deployment-chartsmaster+2 -0
jobqueue: migrate thumbnailrender to k8soperations/deployment-chartsmaster+1 -5
mw-jobrunner: increase replicasoperations/deployment-chartsmaster+1 -1
jobqueue: migrate first job to Kubernetes jobrunneroperations/deployment-chartsmaster+5 -0
changeprop: add config support for migration to k8s jobrunnersoperations/deployment-chartsmaster+15 -4
jobrunner: remove php version related checks from httpbboperations/puppetproduction+1 -7
mw-jobrunner: add vhost for jobrunner.discovery.wmnetoperations/deployment-chartsmaster+1 -1
wmnet: add mw-jobrunner discovery recordoperations/dnsmaster+2 -0
service, kubernetes: mw-jobrunner fixesoperations/puppetproduction+2 -0
mw-jobrunner: use proxy in socket definitionoperations/deployment-chartsmaster+2 -3
service: move mw-jobrunner to prodoperations/puppetproduction+2 -2
service: move mw-jobrunner to lvs_setupoperations/puppetproduction+1 -1
wmnet: fix typooperations/dnsmaster+1 -1
service, conftool: add mw-jobrunner configoperations/puppetproduction+36 -0
wmnet: add records for mw-jobrunneroperations/dnsmaster+4 -0
mw-jobrunner: add virtualhost explicitly for jobrunningoperations/deployment-chartsmaster+30 -0
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 StalledNoneT255792 Quibble runs core:unit tests twice!
 OpenNoneT328919 Upgrade to PHPUnit 10
 OpenNoneT338103 Micro-optimize ApiResult::isMetadataKey with str_starts_with once we support PHP8+
 OpenNoneT328921 Drop PHP 7.4 support from MediaWiki
 StalledNoneT334726 Use return type `never` in Wikibase
 OpenNoneT328922 Drop PHP 8.0 support from MediaWiki
 StalledNoneT319055 Upgrade to psr/container 2.x
 StalledKrinkleT319432 Migrate WMF production from PHP 7.4 to PHP 8.1
 OpenNoneT291916 Tracking task for Bullseye migrations in production
 StalledNoneT356293 Migrate MW appservers' base images to bullseye
 OpenNoneT290536 Serve production traffic via Kubernetes
 ResolvedhnowlanT349796 Move MediaWiki jobs to mw-on-k8s
 ResolvedakosiarisT352906 mediawiki k8s jobrunner fails connecting to cloudelastic with a TLS error
 OpenhnowlanT354791 Reclaim jobrunner hardware for k8s
 ResolvedClement_GoubertT355622 scap not installed on mw1486.eqiad.wmnet which breaks deployment: /usr/bin/scap: No such file or directory
 ResolvedNoneT355243 MediaModeration maintenance script scanFilesInScanTable.php indirectly calls $wgImageMagickConvertCommand
 ResolvedDreamy_JazzT355309 Don't use RENDER_NOW when generating thumbnails for PhotoDNA scans
 ResolvedPRODUCTION ERRORDreamy_JazzT355177 Error creating thumbnail: mkdir: cannot create directory '/sys/fs/cgroup/memory/mediawiki/job/XXXXX':
 ResolvedkostajhT356047 Attempt to generate thumbnail by requesting URL
 ResolvedjijikiT359154 Update DC switchover cookbooks to handle mw-jobrunners

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Nov 24 2023, 2:05 PM

Change 977209 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: increase replicas

https://gerrit.wikimedia.org/r/977209

gerritbot added a comment.Nov 24 2023, 2:07 PM

Change 977210 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] jobqueue: migrate thumbnailrender to k8s

https://gerrit.wikimedia.org/r/977210

gerritbot added a comment.Nov 27 2023, 10:11 AM

Change 977209 merged by jenkins-bot:

[operations/deployment-charts@master] mw-jobrunner: increase replicas

https://gerrit.wikimedia.org/r/977209

gerritbot added a comment.Nov 27 2023, 10:44 AM

Change 977210 merged by jenkins-bot:

[operations/deployment-charts@master] jobqueue: migrate thumbnailrender to k8s

https://gerrit.wikimedia.org/r/977210

gerritbot added a comment.Nov 28 2023, 11:31 AM

Change 978032 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changejob-jobqueue: move two more jobs to k8s

https://gerrit.wikimedia.org/r/978032

gerritbot added a comment.Nov 29 2023, 10:46 AM

Change 978032 merged by jenkins-bot:

[operations/deployment-charts@master] changejob-jobqueue: move two more jobs to k8s

https://gerrit.wikimedia.org/r/978032

gerritbot added a comment.Nov 29 2023, 11:57 AM

Change 978500 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: increase replicas

https://gerrit.wikimedia.org/r/978500

gerritbot added a comment.Nov 29 2023, 4:18 PM

Change 978630 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] jobqueue: move move jobs to k8s jobrunner

https://gerrit.wikimedia.org/r/978630

gerritbot added a comment.Nov 29 2023, 4:23 PM

Change 978630 merged by jenkins-bot:

[operations/deployment-charts@master] jobqueue: move move jobs to k8s jobrunner

https://gerrit.wikimedia.org/r/978630

gerritbot added a comment.Nov 30 2023, 11:28 AM

Change 979051 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: migrate more lower-impact jobs

https://gerrit.wikimedia.org/r/979051

Clement_Goubert updated the task description. (Show Details)Nov 30 2023, 11:38 AM
gerritbot added a comment.Nov 30 2023, 11:55 AM

Change 979051 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: migrate more lower-impact jobs

https://gerrit.wikimedia.org/r/979051

gerritbot added a comment.Nov 30 2023, 4:00 PM

Change 979122 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: toggle another job

https://gerrit.wikimedia.org/r/979122

gerritbot added a comment.Nov 30 2023, 5:19 PM

Change 979122 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: toggle another job

https://gerrit.wikimedia.org/r/979122

gerritbot added a comment.Dec 1 2023, 2:43 PM

Change 979365 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] jobqueue: migrate another medium-weight job

https://gerrit.wikimedia.org/r/979365

gerritbot added a comment.Dec 1 2023, 3:25 PM

Change 979365 merged by jenkins-bot:

[operations/deployment-charts@master] jobqueue: migrate another medium-weight job

https://gerrit.wikimedia.org/r/979365

gerritbot added a comment.Dec 1 2023, 4:55 PM

Change 979395 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] jobqueue: switch a medium weight job

https://gerrit.wikimedia.org/r/979395

gerritbot added a comment.Dec 1 2023, 4:58 PM

Change 979396 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] jobqueue: migrate a moderately weighty job

https://gerrit.wikimedia.org/r/979396

gerritbot added a comment.Dec 1 2023, 4:59 PM

Change 979397 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] jobqueue: migrate a heavyweight job to jobqueue

https://gerrit.wikimedia.org/r/979397

gerritbot added a comment.Dec 4 2023, 3:22 PM

Change 979395 merged by jenkins-bot:

[operations/deployment-charts@master] jobqueue: switch a medium weight job

https://gerrit.wikimedia.org/r/979395

gerritbot added a comment.Dec 4 2023, 5:21 PM

Change 980011 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: increase replicas

https://gerrit.wikimedia.org/r/980011

gerritbot added a comment.Dec 5 2023, 11:05 AM

Change 980011 merged by jenkins-bot:

[operations/deployment-charts@master] mw-jobrunner: increase replicas

https://gerrit.wikimedia.org/r/980011

gerritbot added a comment.Dec 5 2023, 11:11 AM

Change 979396 merged by jenkins-bot:

[operations/deployment-charts@master] jobqueue: migrate a moderately weighty job

https://gerrit.wikimedia.org/r/979396

gerritbot added a comment.Dec 5 2023, 12:28 PM

Change 980369 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: bump replicas further

https://gerrit.wikimedia.org/r/980369

gerritbot added a comment.Dec 5 2023, 3:51 PM

Change 980369 merged by jenkins-bot:

[operations/deployment-charts@master] mw-jobrunner: bump replicas further

https://gerrit.wikimedia.org/r/980369

gerritbot added a comment.Dec 5 2023, 4:14 PM

Change 979397 merged by jenkins-bot:

[operations/deployment-charts@master] jobqueue: migrate a heavyweight job to jobqueue

https://gerrit.wikimedia.org/r/979397

gerritbot added a comment.Dec 5 2023, 4:39 PM

Change 980433 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] jobqueue: migrate a larger job (and one smaller one)

https://gerrit.wikimedia.org/r/980433

gerritbot added a comment.Dec 5 2023, 5:26 PM

Change 980433 merged by jenkins-bot:

[operations/deployment-charts@master] jobqueue: migrate a larger job (and one smaller one)

https://gerrit.wikimedia.org/r/980433

gerritbot added a comment.Dec 6 2023, 11:23 AM

Change 980849 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: migrate one large and a few small jobs

https://gerrit.wikimedia.org/r/980849

gerritbot added a comment.Dec 6 2023, 11:38 AM

Change 980849 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: migrate one large and a few small jobs

https://gerrit.wikimedia.org/r/980849

gerritbot added a comment.Dec 6 2023, 11:50 AM

Change 980853 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: restore cirrussearchlinksupdate to metal

https://gerrit.wikimedia.org/r/980853

gerritbot added a comment.Dec 6 2023, 12:01 PM

Change 980853 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: restore cirrussearchlinksupdate to metal

https://gerrit.wikimedia.org/r/980853

gerritbot added a comment.Dec 6 2023, 12:29 PM

Change 980855 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: migrate all remaining small jobs, also cdnPurge

https://gerrit.wikimedia.org/r/980855

gerritbot added a comment.Dec 6 2023, 12:32 PM

Change 980856 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: migrate all low-traffic jobs

https://gerrit.wikimedia.org/r/980856

gerritbot added a comment.Dec 6 2023, 1:44 PM

Change 980855 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: migrate all remaining small jobs, also cdnPurge

https://gerrit.wikimedia.org/r/980855

gerritbot added a comment.Dec 6 2023, 3:44 PM

Change 980856 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: migrate all low-traffic jobs

https://gerrit.wikimedia.org/r/980856

hnowlan changed the task status from Open to In Progress.Dec 6 2023, 4:27 PM
hnowlan claimed this task.
hnowlan subscribed.
gerritbot added a comment.Dec 6 2023, 4:37 PM

Change 978500 abandoned by Hnowlan:

[operations/deployment-charts@master] mw-jobrunner: increase replicas

Reason:

Scaled up elsewhere, waiting for next jobs and more hw

https://gerrit.wikimedia.org/r/978500

gerritbot added a comment.Dec 6 2023, 4:42 PM

Change 973824 abandoned by Hnowlan:

[operations/puppet@production] kubernetes::worker: add mw-jobrunner to pools

Reason:

Done already

https://gerrit.wikimedia.org/r/973824

Maintenance_bot removed a project: Patch-For-Review.Dec 6 2023, 5:10 PM
Quiddity unsubscribed.Dec 6 2023, 6:32 PM
JMeybohm added a subtask: T352906: mediawiki k8s jobrunner fails connecting to cloudelastic with a TLS error.Dec 7 2023, 9:20 AM
Kappakayala closed subtask T352906: mediawiki k8s jobrunner fails connecting to cloudelastic with a TLS error as Resolved.Dec 9 2023, 1:36 AM
hnowlan reopened subtask T352906: mediawiki k8s jobrunner fails connecting to cloudelastic with a TLS error as Open.Dec 11 2023, 10:32 AM
gerritbot added a comment.Dec 14 2023, 4:40 PM

Change 983216 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: move PublishStashedFile back to non-k8s jobrunner

https://gerrit.wikimedia.org/r/983216

gerritbot added a project: Patch-For-Review.Dec 14 2023, 4:40 PM
akosiaris closed subtask T352906: mediawiki k8s jobrunner fails connecting to cloudelastic with a TLS error as Resolved.Dec 14 2023, 5:24 PM
Urbanecm_WMF subscribed.Jan 2 2024, 10:40 PM

I think the k8s migration work as part of this ticket caused T354229: Moving jobs to MW-on-k8s decreased their timeout from 1200s to 200s.

gerritbot added a comment.Jan 8 2024, 4:10 PM

Change 983216 abandoned by Hnowlan:

[operations/deployment-charts@master] changeprop-jobqueue: move PublishStashedFile back to non-k8s jobrunner

Reason:

Already implemented

https://gerrit.wikimedia.org/r/983216

Maintenance_bot removed a project: Patch-For-Review.Jan 8 2024, 4:32 PM
Urbanecm_WMF unsubscribed.Jan 8 2024, 6:08 PM
gerritbot added a comment.Jan 9 2024, 12:14 PM

Change 989131 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: increase replicas

https://gerrit.wikimedia.org/r/989131

gerritbot added a project: Patch-For-Review.Jan 9 2024, 12:14 PM
gerritbot added a comment.Jan 9 2024, 12:31 PM

Change 989133 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: move cirrusSearchElasticaWrite

https://gerrit.wikimedia.org/r/989133

gerritbot added a comment.Jan 9 2024, 12:50 PM

Change 989131 merged by jenkins-bot:

[operations/deployment-charts@master] mw-jobrunner: increase replicas

https://gerrit.wikimedia.org/r/989131

gerritbot added a comment.Jan 9 2024, 3:12 PM

Change 989133 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: move cirrusSearchElasticaWrite

https://gerrit.wikimedia.org/r/989133

gerritbot added a comment.Jan 10 2024, 12:10 PM

Change 989488 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: increase replicas for parsoidCachePrewarm

https://gerrit.wikimedia.org/r/989488

gerritbot added a comment.Jan 10 2024, 12:11 PM

Change 989489 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: migrate parsoidCachePrewarm to k8s

https://gerrit.wikimedia.org/r/989489

gerritbot added a comment.Jan 10 2024, 12:35 PM

Change 989488 merged by jenkins-bot:

[operations/deployment-charts@master] mw-jobrunner: increase replicas for parsoidCachePrewarm

https://gerrit.wikimedia.org/r/989488

Clement_Goubert mentioned this in T339867: RESTbase: Turn off pre-generation and caching for parsoid endpoints.Jan 10 2024, 12:36 PM
gerritbot added a comment.Jan 10 2024, 1:43 PM

Change 989489 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: migrate parsoidCachePrewarm to k8s

https://gerrit.wikimedia.org/r/989489

gerritbot added a comment.Jan 17 2024, 4:06 PM

Change 991377 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: disable ThumbnailRender on k8s

https://gerrit.wikimedia.org/r/991377

Clement_Goubert changed the status of subtask T355243: MediaModeration maintenance script scanFilesInScanTable.php indirectly calls $wgImageMagickConvertCommand from Open to In Progress.Jan 17 2024, 4:08 PM
gerritbot added a comment.Jan 17 2024, 4:12 PM

Change 991377 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: disable ThumbnailRender on k8s

https://gerrit.wikimedia.org/r/991377

Dreamy_Jazz closed subtask T355243: MediaModeration maintenance script scanFilesInScanTable.php indirectly calls $wgImageMagickConvertCommand as Resolved.Feb 2 2024, 12:30 PM
gerritbot added a comment.Feb 14 2024, 6:27 PM

Change 1003499 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: bump replicas for cirrusSearchLinksUpdate

https://gerrit.wikimedia.org/r/1003499

gerritbot added a comment.Feb 15 2024, 11:51 AM

Change 1003499 merged by jenkins-bot:

[operations/deployment-charts@master] mw-jobrunner: bump replicas for cirrusSearchLinksUpdate

https://gerrit.wikimedia.org/r/1003499

Stashbot mentioned this in T357507: Move 50% of mediawiki external requests to mw on k8s.Feb 15 2024, 11:56 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T11:56:22Z] <cgoubert@deploy2002> Started scap: Deploying mw-on-k8s 1003499 1003393 - T349796 T357507

Stashbot added a comment.Feb 15 2024, 11:57 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-15T11:57:13Z] <cgoubert@deploy2002> Finished scap: Deploying mw-on-k8s 1003499 1003393 - T349796 T357507 (duration: 00m 50s)

gerritbot added a comment.Feb 15 2024, 12:01 PM

Change 1003617 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] jobqueue: migrate cirrusSearchLinksUpdate to k8s

https://gerrit.wikimedia.org/r/1003617

gerritbot added a comment.Feb 15 2024, 2:34 PM

Change 1003617 merged by jenkins-bot:

[operations/deployment-charts@master] jobqueue: migrate cirrusSearchLinksUpdate to k8s

https://gerrit.wikimedia.org/r/1003617

gerritbot added a comment.Feb 15 2024, 2:59 PM

Change 1003772 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: bump replicas

https://gerrit.wikimedia.org/r/1003772

gerritbot added a comment.Feb 15 2024, 3:54 PM

Change 1003772 merged by jenkins-bot:

[operations/deployment-charts@master] mw-jobrunner: bump replicas

https://gerrit.wikimedia.org/r/1003772

brennen mentioned this in T357713: RuntimeException: Received cirrusSearchElasticaWrite job for an unwritable cluster cloudelastic..Feb 15 2024, 8:25 PM
gerritbot added a comment.Feb 16 2024, 11:03 AM

Change 1004062 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: bump replicas in order to migrate refreshLinks

https://gerrit.wikimedia.org/r/1004062

gerritbot added a comment.Feb 16 2024, 11:06 AM

Change 1004063 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: migrate refreshLinks to k8s

https://gerrit.wikimedia.org/r/1004063

gerritbot added a comment.Feb 16 2024, 11:19 AM

Change 1004066 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop: clean up k8s jobrunner references

https://gerrit.wikimedia.org/r/1004066

gerritbot added a comment.Feb 19 2024, 12:32 PM

Change 1004062 merged by jenkins-bot:

[operations/deployment-charts@master] mw-jobrunner: bump replicas in order to migrate refreshLinks

https://gerrit.wikimedia.org/r/1004062

gerritbot added a comment.Feb 19 2024, 12:39 PM

Change 1004063 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: migrate refreshLinks to k8s

https://gerrit.wikimedia.org/r/1004063

gerritbot added a comment.Feb 19 2024, 12:53 PM

Change 1004662 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: begin to scale down replicas

https://gerrit.wikimedia.org/r/1004662

gerritbot added a comment.Feb 19 2024, 4:28 PM

Change 1004662 merged by jenkins-bot:

[operations/deployment-charts@master] mw-jobrunner: begin to scale down replicas

https://gerrit.wikimedia.org/r/1004662

hnowlan updated the task description. (Show Details)Feb 19 2024, 5:21 PM
hnowlan updated the task description. (Show Details)
gerritbot added a comment.Feb 20 2024, 4:54 PM

Change 1005121 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] mw-jobrunner: reduce replicas further

https://gerrit.wikimedia.org/r/1005121

gerritbot added a comment.Feb 21 2024, 10:32 AM

Change 1004066 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: clean up k8s jobrunner references

https://gerrit.wikimedia.org/r/1004066

hnowlan closed this task as Resolved.Feb 21 2024, 10:38 AM

All (non-videoscaler) jobs migrated to Kubernetes jobrunners. Videoscaler work tracked in T355292

Lucas_Werkmeister_WMDE awarded a token.Feb 21 2024, 10:56 AM
jijiki added a subtask: T359154: Update DC switchover cookbooks to handle mw-jobrunners.Tue, Apr 2, 12:35 PM
gerritbot added a comment.Tue, Apr 2, 2:30 PM

Change #1005121 abandoned by Hnowlan:

[operations/deployment-charts@master] mw-jobrunner: reduce replicas further

Reason:

Decided against

https://gerrit.wikimedia.org/r/1005121

gerritbot added a comment.Thu, Apr 11, 7:46 PM

Change #1018420 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] service catalog: disable paging on jobrunner and videoscaler services

https://gerrit.wikimedia.org/r/1018420

gerritbot added a comment.Fri, Apr 12, 2:02 PM

Change #1018420 merged by Cwhite:

[operations/puppet@production] service catalog: disable paging on jobrunner and videoscaler services

https://gerrit.wikimedia.org/r/1018420

gerritbot added a comment.Fri, Apr 19, 10:29 AM

Change #1021886 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/mediawiki-config@master] CommonSettings.php: Fix jobrunner hostname

https://gerrit.wikimedia.org/r/1021886

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL