Page MenuHomePhabricator

Migrate updatequerypages/update_special_pages/initsitestats jobs to mw-cron
Closed, ResolvedPublic

Description

Migrate MediaWiki-Special-pages periodic mediawiki jobs from mwmaint to mw-cron on kubernetes.

Job nameCriticalityDone?
mediawiki_job_updatequerypages_ancientpages_s1@11.timerLY
mediawiki_job_updatequerypages_ancientpages_s2@12.timerLY
mediawiki_job_updatequerypages_ancientpages_s3@13.timerLY
mediawiki_job_updatequerypages_ancientpages_s4@14.timerLY
mediawiki_job_updatequerypages_ancientpages_s5@15.timerLY
mediawiki_job_updatequerypages_ancientpages_s6@16.timerLY
mediawiki_job_updatequerypages_ancientpages_s7@17.timerLY
mediawiki_job_updatequerypages_ancientpages_s8@18.timerLY
mediawiki_job_updatequerypages_deadendpages_s1@11.timerLY
mediawiki_job_updatequerypages_deadendpages_s2@12.timerLY
mediawiki_job_updatequerypages_deadendpages_s3@13.timerLY
mediawiki_job_updatequerypages_deadendpages_s4@14.timerLY
mediawiki_job_updatequerypages_deadendpages_s5@15.timerLY
mediawiki_job_updatequerypages_deadendpages_s6@16.timerLY
mediawiki_job_updatequerypages_deadendpages_s7@17.timerLY
mediawiki_job_updatequerypages_deadendpages_s8@18.timerLY
mediawiki_job_updatequerypages_fewestrevisions_s1@11.timerHY
mediawiki_job_updatequerypages_fewestrevisions_s2@12.timerHY
mediawiki_job_updatequerypages_fewestrevisions_s3@13.timerHY
mediawiki_job_updatequerypages_fewestrevisions_s4@14.timerHY
mediawiki_job_updatequerypages_fewestrevisions_s5@15.timerHY
mediawiki_job_updatequerypages_fewestrevisions_s6@16.timerHY
mediawiki_job_updatequerypages_fewestrevisions_s7@17.timerHY
mediawiki_job_updatequerypages_lonelypages_s1.timerHY
mediawiki_job_updatequerypages_mostcategories_s1.timerMY
mediawiki_job_updatequerypages_mostlinked_s1@11.timerMY
mediawiki_job_updatequerypages_mostlinked_s2@12.timerMY
mediawiki_job_updatequerypages_mostlinked_s3@13.timerMY
mediawiki_job_updatequerypages_mostlinked_s4@14.timerMY
mediawiki_job_updatequerypages_mostlinked_s5@15.timerMY
mediawiki_job_updatequerypages_mostlinked_s6@16.timerMY
mediawiki_job_updatequerypages_mostlinked_s7@17.timerMY
mediawiki_job_updatequerypages_mostlinkedtemplates_s1.timerMY
mediawiki_job_updatequerypages_mostrevisions_s1@11.timerLY
mediawiki_job_updatequerypages_mostrevisions_s2@12.timerLY
mediawiki_job_updatequerypages_mostrevisions_s3@13.timerLY
mediawiki_job_updatequerypages_mostrevisions_s4@14.timerLY
mediawiki_job_updatequerypages_mostrevisions_s5@15.timerLY
mediawiki_job_updatequerypages_mostrevisions_s6@16.timerLY
mediawiki_job_updatequerypages_mostrevisions_s7@17.timerLY
mediawiki_job_updatequerypages_uncategorizedcategories_s1.timerHY
mediawiki_job_updatequerypages_uncatpages_s4@14.timerHY
mediawiki_job_updatequerypages_wantedpages_s1@11.timerHY
mediawiki_job_updatequerypages_wantedpages_s2@12.timerHY
mediawiki_job_updatequerypages_wantedpages_s3@13.timerHY
mediawiki_job_updatequerypages_wantedpages_s4@14.timerHY
mediawiki_job_updatequerypages_wantedpages_s5@15.timerHY
mediawiki_job_updatequerypages_wantedpages_s6@16.timerHY
mediawiki_job_updatequerypages_wantedpages_s7@17.timerHY
mediawiki_job_updatequerypages_wantedpages_s8@18.timerHY
mediawiki_job_updatequerypages_wantedtemplates_s1.timerHY
mediawiki_job_update_special_pages_s1.timerHY
mediawiki_job_update_special_pages_s2.timerHY
mediawiki_job_update_special_pages_s3.timerHY
mediawiki_job_update_special_pages_s4.timerHY
mediawiki_job_update_special_pages_s5.timerHY
mediawiki_job_update_special_pages_s6.timerHY
mediawiki_job_update_special_pages_s7.timerHY
mediawiki_job_update_special_pages_s8.timerHY
mediawiki_job_initsitestats.timerLY

Doc on the new platform

ServiceOps will handle migrating the jobs, but would appreciate input from MediaWiki-Special-pages on:

  • jobs that should be watched more
  • jobs that are low criticality and could be migrated first
  • outdated jobs that can be removed
  • any potential gotchas in the way these jobs use MediaWiki

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Scott Frenchoperations/puppetproduction+9 -8
Hnowlanoperations/puppetproduction+9 -2
Hnowlanoperations/puppetproduction+18 -4
Scott Frenchoperations/puppetproduction+8 -2
Scott Frenchoperations/puppetproduction+0 -7
Hnowlanoperations/puppetproduction+10 -2
Kamila Součkováoperations/puppetproduction+18 -4
Hnowlanoperations/puppetproduction+3 -0
Hnowlanoperations/puppetproduction+1 -0
Hnowlanoperations/puppetproduction+1 -0
Hnowlanoperations/puppetproduction+1 -1
Hnowlanoperations/puppetproduction+1 -1
Hnowlanoperations/puppetproduction+1 -0
Hnowlanoperations/puppetproduction+1 -1
Clément Goubertoperations/puppetproduction+71 -86
Clément Goubertoperations/puppetproduction+9 -4
Clément Goubertoperations/puppetproduction+36 -0
Clément Goubertoperations/puppetproduction+37 -0
Show related patches Customize query in gerrit

Event Timeline

Pppery renamed this task from Migrate mediawiki-special-pages jobs to mw-cron to Migrate updatequerypages/update_special_pages jobs to mw-cron.Mar 11 2025, 5:18 PM
Pppery renamed this task from Migrate updatequerypages/update_special_pages jobs to mw-cron to Migrate updatequerypages/update_special_pages/init_site_stats jobs to mw-cron.Mar 11 2025, 5:32 PM
Pppery renamed this task from Migrate updatequerypages/update_special_pages/init_site_stats jobs to mw-cron to Migrate updatequerypages/update_special_pages/initsitestats jobs to mw-cron.

Change #1135753 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] alertmanager: Add team/project receivers for Phab

https://gerrit.wikimedia.org/r/1135753

Change #1135754 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] alertmanager: Add routing for task alerts

https://gerrit.wikimedia.org/r/1135754

Change #1135753 merged by Clément Goubert:

[operations/puppet@production] alertmanager: Add team/project receivers for Phab

https://gerrit.wikimedia.org/r/1135753

Change #1135754 merged by Clément Goubert:

[operations/puppet@production] alertmanager: Add routing for task alerts

https://gerrit.wikimedia.org/r/1135754

Change #1137261 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] updatequerypages: Move deadendpages-s3 to kubernetes

https://gerrit.wikimedia.org/r/1137261

Change #1137306 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] updatequerypages: Move all to sharded_periodic_job

https://gerrit.wikimedia.org/r/1137306

Change #1137261 merged by Clément Goubert:

[operations/puppet@production] updatequerypages: Move deadendpages-s3 to kubernetes

https://gerrit.wikimedia.org/r/1137261

Clement_Goubert changed the task status from Open to In Progress.Apr 17 2025, 2:42 PM
Clement_Goubert triaged this task as Medium priority.
Clement_Goubert updated the task description. (Show Details)

Change #1137306 merged by Clément Goubert:

[operations/puppet@production] updatequerypages: Move all to sharded_periodic_job

https://gerrit.wikimedia.org/r/1137306

Cross-referencing T392441: MediaWikiCronJobFailed in case it’s e.g. caused by anything specific relating to the move to Kubernetes :)

Change #1139432 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw:maintenance:updatequerypages: move all ancientpages jobs to k8s

https://gerrit.wikimedia.org/r/1139432

Change #1139437 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw::maintenance: migrate a single updatequerypages_ancientpages shard

https://gerrit.wikimedia.org/r/1139437

Change #1139438 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw:maintenance: migrate all updatequerypages_ancientpages jobs to k8s

https://gerrit.wikimedia.org/r/1139438

Change #1139432 merged by Hnowlan:

[operations/puppet@production] mw:maintenance:updatequerypages: move all deadendpages jobs to k8s

https://gerrit.wikimedia.org/r/1139432

Change #1140151 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw::maintenance: migrate ancientpages jobs to Kubernetes

https://gerrit.wikimedia.org/r/1140151

Change #1139437 merged by Hnowlan:

[operations/puppet@production] mw::maintenance: migrate a single updatequerypages_ancientpages shard

https://gerrit.wikimedia.org/r/1139437

Change #1139438 abandoned by Hnowlan:

[operations/puppet@production] mw:maintenance: migrate all updatequerypages_ancientpages jobs to k8s

Reason:

dupe

https://gerrit.wikimedia.org/r/1139438

Thank you for the criticality updates! With the resolution of T392441 it seems like the deadendpages jobs are looking healthy. Given that all other jobs use the same mechanism, we'd like to move on some of the more sensitive jobs - just to be clear given that they all run similarly infrequently, the criticality is a reflection of the impact if they don't run rather than overall risk or complexity of their operation, right?

Change #1140151 merged by Hnowlan:

[operations/puppet@production] mw::maintenance: migrate ancientpages jobs to Kubernetes

https://gerrit.wikimedia.org/r/1140151

Change #1140188 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw::maintenance: move mostrevisions jobs to k8s

https://gerrit.wikimedia.org/r/1140188

Change #1140188 merged by Hnowlan:

[operations/puppet@production] mw::maintenance: move mostrevisions jobs to k8s

https://gerrit.wikimedia.org/r/1140188

Change #1140214 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw::maintenance: migrate mostlinked job to kubernetes

https://gerrit.wikimedia.org/r/1140214

Change #1140216 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw::maintenance: migrate all remaining general updatequerypages jobs

https://gerrit.wikimedia.org/r/1140216

Change #1140214 merged by Hnowlan:

[operations/puppet@production] mw::maintenance: migrate mostlinked job to kubernetes

https://gerrit.wikimedia.org/r/1140214

Thank you for the criticality updates! With the resolution of T392441 it seems like the deadendpages jobs are looking healthy. Given that all other jobs use the same mechanism, we'd like to move on some of the more sensitive jobs - just to be clear given that they all run similarly infrequently, the criticality is a reflection of the impact if they don't run rather than overall risk or complexity of their operation, right?

Given that the migrations have gone successfully up to this point, we plan on migrating the higher criticality jobs to Kubernetes today. To be sure that everything is working okay, is it safe to run these jobs out of step or should they stick to their configured schedule?

hnowlan updated the task description. (Show Details)

Change #1140216 merged by Hnowlan:

[operations/puppet@production] mw::maintenance: migrate all remaining general updatequerypages jobs

https://gerrit.wikimedia.org/r/1140216

Change #1143803 had a related patch set uploaded (by Kamila Součková; author: Kamila Součková):

[operations/puppet@production] mw-cron/updatequerypages: Migrate Mostcategories,Mostlinkedtemplates

https://gerrit.wikimedia.org/r/1143803

Change #1144517 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw::maintenance: migrate initSiteStats cron to k8s

https://gerrit.wikimedia.org/r/1144517

Change #1143803 merged by Kamila Součková:

[operations/puppet@production] mw-cron/updatequerypages: Migrate Mostcategories,Mostlinkedtemplates

https://gerrit.wikimedia.org/r/1143803

Change #1144517 merged by Hnowlan:

[operations/puppet@production] mw::maintenance: migrate initSiteStats cron to k8s

https://gerrit.wikimedia.org/r/1144517

Change #1146787 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] P:mw:maint:update_special_pages: remove absented non-sharded job

https://gerrit.wikimedia.org/r/1146787

Change #1146788 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] P:mw:maint:update_special_pages: updateSpecialPages in s6 to mw-cron

https://gerrit.wikimedia.org/r/1146788

Change #1146789 had a related patch set uploaded (by Scott French; author: Scott French):

[operations/puppet@production] P:mw:maint:update_special_pages: all updateSpecialPages shards to mw-cron

https://gerrit.wikimedia.org/r/1146789

Change #1146787 merged by Scott French:

[operations/puppet@production] P:mw:maint:update_special_pages: remove absented non-sharded job

https://gerrit.wikimedia.org/r/1146787

Change #1146788 merged by Scott French:

[operations/puppet@production] P:mw:maint:update_special_pages: updateSpecialPages in s6 to mw-cron

https://gerrit.wikimedia.org/r/1146788

Clement_Goubert raised the priority of this task from Medium to High.May 20 2025, 3:40 PM

Change #1148384 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw::maintenance: move uncategorizedtemplates and wantedtemplates to k8s

https://gerrit.wikimedia.org/r/1148384

Change #1148384 merged by Hnowlan:

[operations/puppet@production] mw::maintenance: move uncategorizedtemplates and wantedtemplates to k8s

https://gerrit.wikimedia.org/r/1148384

Change #1148391 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/puppet@production] mw::maintenance: migrate lonelypages job to k8s

https://gerrit.wikimedia.org/r/1148391

Change #1148391 merged by Hnowlan:

[operations/puppet@production] mw::maintenance: migrate lonelypages job to k8s

https://gerrit.wikimedia.org/r/1148391

Change #1146789 merged by Scott French:

[operations/puppet@production] P:mw:maint:update_special_pages: all updateSpecialPages shards to mw-cron

https://gerrit.wikimedia.org/r/1146789

Remaining update-special-pages jobs have been migrated, following a successful manual test of the s6 shard earlier today. First scheduled run for all will happen at 5:00 UTC on the 22nd (i.e., soon).

Clement_Goubert updated the task description. (Show Details)