Page MenuHomePhabricator
Create Task
Maniphest T355169

Run scanFilesInScanTable.php automatically on WMF wikis
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

We have settled on the following command to scan the backlog of files on commonswiki:

mwscript extensions/MediaModeration/maintenance/scanFilesInScanTable.php --wiki=commonswiki --use-jobqueue --sleep 30 --verbose

This script is currently being run "manually" via a tmux session on maint2002. However, this means that the script:

  1. Will in a weeks time be running on a no longer released version of mediawiki
  2. Is not resistant to interruptions, such as restarts of the maint2002 hosts or changes of the active DC.

Instead this script should be run through a puppet configuration such that the job is:

  1. Restarted semi-frequently to ensure updates to the script are applied
  2. Resistant to being randomly interrupted, as it would automatically restart

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Update MediaModeration module to run scans automaticallyoperations/puppetproduction+12 -0
Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.phpmediawiki/extensions/MediaModerationwmf/1.44.0-wmf.6+87 -17
Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.phpmediawiki/extensions/MediaModerationwmf/1.44.0-wmf.5+87 -17
Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.phpmediawiki/extensions/MediaModerationmaster+87 -17
Create a DB list for wikis with continuous MediaModeration scansoperations/mediawiki-configmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Dreamy_Jazz created this task.Jan 16 2024, 5:30 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 16 2024, 5:30 PM
kostajh moved this task from MediaModeration 2.0 to MediaModeration 2.1 on the MediaModeration board.Jan 31 2024, 9:42 AM
kostajh edited projects, added MediaModeration (MediaModeration 2.1); removed MediaModeration (MediaModeration 2.0).
kostajh moved this task from MediaModeration 2.1 to MediaModeration 2.0 on the MediaModeration board.
kostajh edited projects, added MediaModeration (MediaModeration 2.0); removed MediaModeration (MediaModeration 2.1).
kostajh edited projects, added MediaModeration (MediaModeration 2.1); removed MediaModeration (MediaModeration 2.0).Feb 6 2024, 12:11 PM

Discussed in a meeting just now. We'll see if this breaks in practice. If so, we'll revisit automating the script restart.

kostajh added a project: Trust and Safety Product Sprint (Sprint Gong (November 18 - December 6)).Dec 3 2024, 11:36 AM
Dreamy_Jazz claimed this task.Dec 4 2024, 10:37 AM
Dreamy_Jazz set the point value for this task to 2.
Dreamy_Jazz moved this task from Priority Backlog to In Progress on the Trust and Safety Product Sprint (Sprint Gong (November 18 - December 6)) board.
gerritbot added a comment.Dec 4 2024, 11:22 AM

Change #1100421 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php

https://gerrit.wikimedia.org/r/1100421

gerritbot added a project: Patch-For-Review.Dec 4 2024, 11:22 AM
Dreamy_Jazz merged a task: T351400: Run the maintenance script scanning images in mediamoderation_scan on WMF wikis.Dec 4 2024, 11:41 AM
Dreamy_Jazz added a parent task: T351200: [EPIC] Start automated scanning.
gerritbot added a comment.Dec 4 2024, 11:47 AM

Change #1100426 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[operations/mediawiki-config@master] Create a DB list for wikis with continuous MediaModeration scans

https://gerrit.wikimedia.org/r/1100426

gerritbot added a comment.Dec 4 2024, 11:50 AM

Change #1100427 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[operations/puppet@production] [WIP] Update MediaModeration module to run scans automatically

https://gerrit.wikimedia.org/r/1100427

gerritbot added a comment.Dec 4 2024, 11:52 AM

Change #1100426 merged by jenkins-bot:

[operations/mediawiki-config@master] Create a DB list for wikis with continuous MediaModeration scans

https://gerrit.wikimedia.org/r/1100426

Stashbot added a comment.Dec 4 2024, 11:53 AM

Mentioned in SAL (#wikimedia-operations) [2024-12-04T11:53:04Z] <dreamyjazz@deploy2002> Started scap sync-world: Backport for [[gerrit:1100426|Create a DB list for wikis with continuous MediaModeration scans (T355169)]]

gerritbot added a comment.Dec 4 2024, 11:56 AM

Change #1100430 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@wmf/1.44.0-wmf.6] Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php

https://gerrit.wikimedia.org/r/1100430

Stashbot added a comment.Dec 4 2024, 11:59 AM

Mentioned in SAL (#wikimedia-operations) [2024-12-04T11:59:12Z] <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:1100426|Create a DB list for wikis with continuous MediaModeration scans (T355169)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

gerritbot added a comment.Dec 4 2024, 12:03 PM

Change #1100434 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@wmf/1.44.0-wmf.5] Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php

https://gerrit.wikimedia.org/r/1100434

Stashbot added a comment.Dec 4 2024, 12:06 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-04T12:06:06Z] <dreamyjazz@deploy2002> Finished scap sync-world: Backport for [[gerrit:1100426|Create a DB list for wikis with continuous MediaModeration scans (T355169)]] (duration: 13m 02s)

gerritbot added a comment.Dec 4 2024, 12:12 PM

Change #1100421 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php

https://gerrit.wikimedia.org/r/1100421

gerritbot added a comment.Dec 4 2024, 12:57 PM

Change #1100434 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@wmf/1.44.0-wmf.5] Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php

https://gerrit.wikimedia.org/r/1100434

gerritbot added a comment.Dec 4 2024, 12:57 PM

Change #1100430 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@wmf/1.44.0-wmf.6] Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php

https://gerrit.wikimedia.org/r/1100430

Stashbot added a comment.Dec 4 2024, 12:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-04T12:59:12Z] <dreamyjazz@deploy2002> Started scap sync-world: Backport for [[gerrit:1100442|Stats: Move StatsFactory flush into emitBufferedStats (T380609)]], [[gerrit:1100434|Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)]], [[gerrit:1100430|Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)]]

Stashbot mentioned this in T380609: Maintenance scripts do not emit StatsLib metrics.Dec 4 2024, 12:59 PM
ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.6; 2024-12-03).Dec 4 2024, 1:00 PM
Stashbot added a comment.Dec 4 2024, 1:00 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-04T13:00:51Z] <dreamyjazz@deploy2002> Started scap sync-world: Backport for [[gerrit:1100442|Stats: Move StatsFactory flush into emitBufferedStats (T380609)]], [[gerrit:1100434|Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)]], [[gerrit:1100430|Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)]]

Stashbot added a comment.Dec 4 2024, 1:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-04T13:28:15Z] <dreamyjazz@deploy2002> Started scap sync-world: Backport for [[gerrit:1100442|Stats: Move StatsFactory flush into emitBufferedStats (T380609)]], [[gerrit:1100434|Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)]], [[gerrit:1100430|Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)]], [[gerrit:1100449|Revert "Stats: Move StatsFactory flush into emitBufferedSta

Stashbot added a comment.Dec 4 2024, 1:33 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-04T13:33:49Z] <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:1100442|Stats: Move StatsFactory flush into emitBufferedStats (T380609)]], [[gerrit:1100434|Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)]], [[gerrit:1100430|Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)]], [[gerrit:1100449|Revert "Stats: Move StatsFactory flush into emitBufferedStats"]] synced

Stashbot added a comment.Dec 4 2024, 1:42 PM

Mentioned in SAL (#wikimedia-operations) [2024-12-04T13:42:53Z] <dreamyjazz@deploy2002> Finished scap sync-world: Backport for [[gerrit:1100442|Stats: Move StatsFactory flush into emitBufferedStats (T380609)]], [[gerrit:1100434|Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)]], [[gerrit:1100430|Fix handling of 'last-checked' as 'never' in scanFilesInScanTable.php (T355169)]], [[gerrit:1100449|Revert "Stats: Move StatsFactory flush into emitBufferedSt

gerritbot added a comment.Dec 5 2024, 5:15 PM

Change #1100427 merged by RLazarus:

[operations/puppet@production] Update MediaModeration module to run scans automatically

https://gerrit.wikimedia.org/r/1100427

Maintenance_bot removed a project: Patch-For-Review.Dec 5 2024, 5:30 PM
Dreamy_Jazz closed this task as Resolved.Dec 5 2024, 6:20 PM
Dreamy_Jazz moved this task from In Progress to Done on the Trust and Safety Product Sprint (Sprint Gong (November 18 - December 6)) board.

This will be hard to QA, given that the logs are private and that the only other thing to check would be that the script is scanning (which I have verified).

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits