Create a Grafana chart to plot the number of PhotoDNA requests per day per wiki
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

A Grafana chart should be created that details the number of requests to PhotoDNA per day per wiki.

Acceptance critiera

The Grafana chart shows how requests are made per day per wiki
The chart shows how many of these requests fail, including for what reason

Details

Related Changes in Gerrit:

	Subject	Author	Repo	Branch	Lines +/-
	Add more statsd counters and add logstash logging	Dreamy Jazz	mediawiki/extensions/MediaModeration	wmf/1.42.0-wmf.13	+174 -7
	Add more statsd counters and add logstash logging	Dreamy Jazz	mediawiki/extensions/MediaModeration	master	+174 -7

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	• Dreamy_Jazz	T351200 [EPIC] Start automated scanning
Resolved	None	T351420 [Sub EPIC] Create Grafana charts, and Logstash filters
Resolved	• Dreamy_Jazz	T351419 Create a Grafana chart to plot the number of PhotoDNA requests per day per wiki
Resolved	kostajh	T353441 Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA
Resolved	kostajh	T351401 Create service(s) to send an image to PhotoDNA for a scan
Resolved	• Dreamy_Jazz	T351417 Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1
Resolved	• Dreamy_Jazz	T351416 [S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis
Resolved	• Dreamy_Jazz	T351409 Expand the database lookup service to get rows from the mediamoderation_scan table
Resolved	• Dreamy_Jazz	T351112 [S] Create an initial service to lookup scan rows from the database
Resolved	• Dreamy_Jazz	T350321 [M] Create database table to store status of scans

Event Timeline

• Dreamy_Jazz created this task.Nov 16 2023, 2:46 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2023, 2:46 PM

• Dreamy_Jazz renamed this task from Create a Grafana chart to plot the number of PhotoDNA requests per day to Create a Grafana chart to plot the number of PhotoDNA requests per day per wiki.Nov 16 2023, 2:46 PM

• Dreamy_Jazz added a subtask: T351401: Create service(s) to send an image to PhotoDNA for a scan.

• Dreamy_Jazz added a parent task: T351200: [EPIC] Start automated scanning.

• Dreamy_Jazz added a parent task: T351420: [Sub EPIC] Create Grafana charts, and Logstash filters.Nov 16 2023, 2:53 PM

• Dreamy_Jazz removed a parent task: T351200: [EPIC] Start automated scanning.

• Dreamy_Jazz added a subtask: T353441: Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA.Dec 14 2023, 12:31 PM

• Dreamy_Jazz mentioned this in T353441: Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA.Dec 19 2023, 12:34 AM

• Dreamy_Jazz closed subtask T353441: Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA as Resolved.Dec 22 2023, 12:16 AM

• Dreamy_Jazz closed subtask T351401: Create service(s) to send an image to PhotoDNA for a scan as Resolved.Jan 3 2024, 3:35 PM

• Dreamy_Jazz updated the task description. (Show Details)Jan 8 2024, 2:43 PM

The second acceptance criteria is mostly done, but I will finish this as part of T354408: Support parallelizing scans to PhotoDNA.

• Dreamy_Jazz moved this task from Priority Backlog to In Progress on the Trust and Safety Product Sprint (Sprint Northumbrian smallpipes (8th Jan.‘24 - 19th Jan.'24)) board.Jan 9 2024, 2:16 PM

To reduce the overall size of the patch in T354408: Support parallelizing scans to PhotoDNA, I will add the extra logging in a separate patch.

• Dreamy_Jazz moved this task from Ready to In Progress on the Trust and Safety Product Sprint (Sprint Northumbrian smallpipes (8th Jan.‘24 - 19th Jan.'24)) board.Jan 12 2024, 12:09 AM

Change 989975 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] [WIP] Add more statsd counters and add logstash logging

https://gerrit.wikimedia.org/r/989975

gerritbot added a project: Patch-For-Review.Jan 12 2024, 12:32 AM

• Dreamy_Jazz moved this task from In Progress to Needs review on the Trust and Safety Product Sprint (Sprint Northumbrian smallpipes (8th Jan.‘24 - 19th Jan.'24)) board.Jan 12 2024, 6:05 PM

Change 989975 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Add more statsd counters and add logstash logging

https://gerrit.wikimedia.org/r/989975

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2024, 9:30 PM

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.14; 2024-01-16).Jan 15 2024, 11:00 PM

Suggested QA steps to be performed on or after the 17th:

Load https://grafana.wikimedia.org/d/STSXVVdSk/mediamoderation-photodna-stats?orgId=1&refresh=5m
Scroll down to the bottom of the page
Verify that the Transform errors/warnings graph has data
Open logstash
Go to the Trust and Safety Product logstash board
Verify that logstash entries appear related to not being able to scan SHA-1 values.

• Dreamy_Jazz updated the task description. (Show Details)Jan 16 2024, 12:15 AM

Change 990760 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.13] Add more statsd counters and add logstash logging

https://gerrit.wikimedia.org/r/990760

gerritbot added a project: Patch-For-Review.Jan 16 2024, 2:11 PM

Change 990760 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.13] Add more statsd counters and add logstash logging

https://gerrit.wikimedia.org/r/990760

Mentioned in SAL (#wikimedia-operations) [2024-01-16T14:16:55Z] <dreamyjazz@deploy2002> Started scap: Backport for [[gerrit:990760|Add more statsd counters and add logstash logging (T351419)]]

Mentioned in SAL (#wikimedia-operations) [2024-01-16T14:18:33Z] <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:990760|Add more statsd counters and add logstash logging (T351419)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-01-16T14:24:10Z] <dreamyjazz@deploy2002> Finished scap: Backport for [[gerrit:990760|Add more statsd counters and add logstash logging (T351419)]] (duration: 07m 15s)

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2024, 2:31 PM

ReleaseTaggerBot edited projects, added: MW-1.42-notes (1.42.0-wmf.13; 2024-01-09 ); removed: MW-1.42-notes (1.42.0-wmf.14; 2024-01-16).Jan 16 2024, 3:00 PM

I have verified that the new code has been implemented and is functioning and displaying as expected... Thank you for the QA Steps @Dreamy_Jazz.

• Dreamy_Jazz closed this task as Resolved.Jan 17 2024, 6:54 PM

Create a Grafana chart to plot the number of PhotoDNA requests per day per wikiClosed, ResolvedPublic2 Estimated Story PointsActions