Page MenuHomePhabricator

Create new maintenance script to resubmit stuck jobs
Closed, ResolvedPublic

Description

Apparently, some 0.2% of our DispatchChanges jobs are submitted but just never run (see T292609). While the exact cause is still unclear, we need some way to deal with it.

We need to achieve the following:

  • the subscribed page at the client wikis should be purged at some point
  • the change should be injected in the recent changes and watch lists at the client wikis
  • the row should be removed at wb_changes
  • things should not explode in case the jobs lag a bit
  • the number of job rescheduled that way should be tracked

Acceptance criteria:

  • write a maintenance script that resubmits jobs for entries in wb_changes that are older than 24 hours
  • run this maintenance script with a cron job regularly (specific interval TBD)

Event Timeline

Michael triaged this task as High priority.Oct 7 2021, 12:55 PM

Change 727343 had a related patch set uploaded (by Michael Große; author: Michael Große):

[mediawiki/extensions/Wikibase@master] Add maintenance script to resubmit lost DispatchChanges jobs

https://gerrit.wikimedia.org/r/727343

Change 730444 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[mediawiki/extensions/Wikibase@master] Pass IDatabase into EntityChangeLookup::loadChanges()

https://gerrit.wikimedia.org/r/730444

Change 730444 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Pass IDatabase into EntityChangeLookup::loadChanges()

https://gerrit.wikimedia.org/r/730444

Change 727343 merged by jenkins-bot:

[mediawiki/extensions/Wikibase@master] Add maintenance script to resubmit lost DispatchChanges jobs

https://gerrit.wikimedia.org/r/727343

Mentioned in SAL (#wikimedia-releng) [2021-10-21T09:49:07Z] <Lucas_WMDE> lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/Wikibase/repo/maintenance/ResubmitChanges.php wikidatawiki # test T292728

Looks like the script works fine – before:

Screenshot 2021-10-21 at 11-45-15 Change dispatch statistics - Wikidata.png (269×1 px, 38 KB)

After (including a bit of wait time for the job queue):
Screenshot 2021-10-21 at 11-49-56 Change dispatch statistics - Wikidata.png (181×1 px, 25 KB)

Output:

09:48:44 Resubmitting 2 changes older than 86400 seconds.
09:48:44 Resubmitting changes is done.