Create a maintenance script to re-send emails for files determined to be a match through PhotoDNA
Closed, ResolvedPublic2 Estimated Story Points
Actions

Assigned To

Authored By

	Dreamy_Jazz
	Nov 16 2023, 1:52 PM

Description

A maintenance script should be created that allows re-sending arbitrary scan results or scan results after a specific timestamp.

This is done so that if the emailing service fails to send an email, the emails can be re-attempted once the problem is fixed. This script would be used manually to re-send the emails which were not received.

Acceptance criteria

This maintenance script is created
The maintenance script is well tested

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Add maintenance script to re-send emails for matches	mediawiki/extensions/MediaModeration	master	+453 -0

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Dreamy_Jazz	T351406 Create a maintenance script to re-send emails for files determined to be a match through PhotoDNA
Resolved	Dreamy_Jazz	T351407 Create a service that sends an email when a file matches
Resolved	Dreamy_Jazz	T351417 Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1
Resolved	Dreamy_Jazz	T351416 [S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis
Resolved	Dreamy_Jazz	T351409 Expand the database lookup service to get rows from the mediamoderation_scan table
Resolved	Dreamy_Jazz	T351112 [S] Create an initial service to lookup scan rows from the database
Resolved	Dreamy_Jazz	T350321 [M] Create database table to store status of scans

Event Timeline

Dreamy_Jazz created this task.Nov 16 2023, 1:52 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2023, 1:52 PM

Dreamy_Jazz mentioned this in T351407: Create a service that sends an email when a file matches.Nov 16 2023, 1:54 PM

Dreamy_Jazz added a subtask: T351407: Create a service that sends an email when a file matches.

Dreamy_Jazz claimed this task.Jan 15 2024, 6:20 PM

Dreamy_Jazz added a project: Trust and Safety Product Sprint (Sprint Northumbrian smallpipes (8th Jan.‘24 - 19th Jan.'24)).

Dreamy_Jazz moved this task from Priority Backlog to Ready on the Trust and Safety Product Sprint (Sprint Northumbrian smallpipes (8th Jan.‘24 - 19th Jan.'24)) board.

Dreamy_Jazz moved this task from Ready to In Progress on the Trust and Safety Product Sprint (Sprint Northumbrian smallpipes (8th Jan.‘24 - 19th Jan.'24)) board.

Dreamy_Jazz set the point value for this task to 2.Jan 15 2024, 7:29 PM

Change 990739 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] [Very WIP] Add maintenance script to re-send emails for matches

https://gerrit.wikimedia.org/r/990739

gerritbot added a project: Patch-For-Review.Jan 15 2024, 7:56 PM

Dreamy_Jazz moved this task from In Progress to Needs review on the Trust and Safety Product Sprint (Sprint Northumbrian smallpipes (8th Jan.‘24 - 19th Jan.'24)) board.Jan 18 2024, 7:41 PM

Change 990739 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Add maintenance script to re-send emails for matches

https://gerrit.wikimedia.org/r/990739

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.15; 2024-01-23).Jan 22 2024, 10:01 PM

Maintenance_bot removed a project: Patch-For-Review.Jan 22 2024, 10:30 PM

Dreamy_Jazz moved this task from Needs review to Needs QA on the Trust and Safety Product Sprint (Sprint Northumbrian smallpipes (8th Jan.‘24 - 19th Jan.'24)) board.Jan 24 2024, 1:23 PM

Suggested QA steps which need a local wiki:

Enable email support on your wiki so that you can view emails, if not already enabled.
1. To set-up email support for MediaWiki-Docker, you can follow the steps at https://www.mediawiki.org/wiki/MediaWiki-Docker/Configuration_recipes/Mail_handling to add Mail hog to your MediaWiki-Docker instance. You should add the code listed in that page to the listed files and then down and up your MediaWiki-Docker instance.
If you have recently installed the MediaModeration extension, follow the steps in T354408: Support parallelizing scans to PhotoDNA to add some test positive matches. Otherwise skip this step.
Define the following in your LocalSettings.php, if you don't already have it there:

$wgMediaModerationRecipientList = [ 'admin@wikimedia.org' ];
$wgMediaModerationFrom = 'no-reply@wikimedia.org';

Run ./maintenance/run MediaModeration:resendMatchEmails.php 20200304 --verbose
Verify that the script outputted multiple lines which match the format Sent email for SHA-1 X. where X is replaced with a long string of numbers and characters (which is the SHA-1).
Open the email server for your wiki (if you use mail hog / did step 1A then open http://localhost:8025 ).
Verify emails reporting matches now appear in the inbox.

QA'ing the contents of the email will be done in T351407: Create a service that sends an email when a file matches which is not yet ready for QA due to a follow-up change. As such, you don't need to verify the contents of the email for this ticket and can instead just verify that the emails appeared when you ran the maintenance script.

I was able to successfully execute your maintenance script at QA Step 4...
I verified the email format produced by the maintenance script was matching the email format listed in the QA Step 5...
I also verified the emails produced/sent by the maintenance script was also appearing in the Inbox at MailHog (QA Steps 6 & 7).

As always thank you @Dreamy_Jazz!!!

Sent email for SHA-1 3fkfxskkihcp9v4inui6pj6rmaxyy7v.
Sent email for SHA-1 4105gxdmakqz8lw51jwkvqgazhy64oi.
Sent email for SHA-1 5eo34e0cor8jtjm9ez6ao1c1axrn6up.
Sent email for SHA-1 5nqk7zb6m0raz5duyqkt9xipn4kf030.
Sent email for SHA-1 7qgxpadss3faemqicnwucnyjsen58ry.
Sent email for SHA-1 85kq9xh8pns0gup332ffzppku3v4q4h.
Sent email for SHA-1 am0q63x644kw07vim5snur4crcd5gt2.
Sent email for SHA-1 cz2f6pux67tfjth7nqp5xl3s2zooi9y.
Sent email for SHA-1 evm1rwp519o9o7btjx33nsd5l9y9x0e.
Sent email for SHA-1 eyd0lt2no4famgt1rvhnm3vd03dkb69.
Sent email for SHA-1 jddldk1kwlto2cahou7wyt7l7dq714i.
Sent email for SHA-1 jqqahqge0vbvvcq4cxvl5c60ntcmmnj.
Sent email for SHA-1 o4q2ucvji1nwyhxm3na3ucwt39ruo1t.
Sent email for SHA-1 q529eyzica68ynoz91rywjt3wddf8en.
Sent email for SHA-1 qqm2ru006idqbi68cemyszkoxov2cqw.
Sent email for SHA-1 rkb1kje4zrdnl753ol9tqas19pm3j9c.
Sent email for SHA-1 tllzov5mf3lmkk703oxdti5rn29eujj.
Sent email for SHA-1 twc6covf7x1ea11j91x642t030opmc2.

kostajh closed this task as Resolved.Jan 26 2024, 5:44 AM

kostajh closed subtask T351407: Create a service that sends an email when a file matches as Resolved.Jan 30 2024, 5:28 AM

	F41718341: image.png
	Jan 26 2024, 3:42 AM

	F41713826: image.png
	Jan 24 2024, 1:44 PM

Create a maintenance script to re-send emails for files determined to be a match through PhotoDNAClosed, ResolvedPublic2 Estimated Story PointsActions