Create a Grafana chart to plot the number of PhotoDNA requests per day per wiki
Closed, ResolvedPublic2 Estimated Story Points
Actions

Description

A Grafana chart should be created that details the number of requests to PhotoDNA per day per wiki.

Acceptance critiera

The Grafana chart shows how requests are made per day per wiki
The chart shows how many of these requests fail, including for what reason

Details

	Subject	Repo	Branch	Lines +/-
	Add more statsd counters and add logstash logging	mediawiki/extensions/MediaModeration	wmf/1.42.0-wmf.13	+174 -7
	Add more statsd counters and add logstash logging	mediawiki/extensions/MediaModeration	master	+174 -7

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Open	None	T351200 [EPIC] Start automated scanning
Resolved	None	T351420 [Sub EPIC] Create Grafana charts, and Logstash filters
Resolved	Dreamy_Jazz	T351419 Create a Grafana chart to plot the number of PhotoDNA requests per day per wiki
Resolved	kostajh	T353441 Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA
Resolved	kostajh	T351401 Create service(s) to send an image to PhotoDNA for a scan
Resolved	Dreamy_Jazz	T351417 Create a service that creates LocalFile and ArchivedFile objects for files that have a given SHA-1
Resolved	Dreamy_Jazz	T351416 [S] Explore how many rows in the file tables have duplicated SHA-1 values on WMF wikis
Resolved	Dreamy_Jazz	T351409 Expand the database lookup service to get rows from the mediamoderation_scan table
Resolved	Dreamy_Jazz	T351112 [S] Create an initial service to lookup scan rows from the database
Resolved	Dreamy_Jazz	T350321 [M] Create database table to store status of scans

Event Timeline

Dreamy_Jazz created this task.Nov 16 2023, 2:46 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2023, 2:46 PM

Dreamy_Jazz renamed this task from Create a Grafana chart to plot the number of PhotoDNA requests per day to Create a Grafana chart to plot the number of PhotoDNA requests per day per wiki.Nov 16 2023, 2:46 PM

Dreamy_Jazz added a subtask: T351401: Create service(s) to send an image to PhotoDNA for a scan.

Dreamy_Jazz added a parent task: T351200: [EPIC] Start automated scanning.

Dreamy_Jazz added a parent task: T351420: [Sub EPIC] Create Grafana charts, and Logstash filters.Nov 16 2023, 2:53 PM

Dreamy_Jazz removed a parent task: T351200: [EPIC] Start automated scanning.

Dreamy_Jazz added a subtask: T353441: Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA.Dec 14 2023, 12:31 PM

Dreamy_Jazz mentioned this in T353441: Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA.Dec 19 2023, 12:34 AM

Dreamy_Jazz closed subtask T353441: Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA as Resolved.Dec 22 2023, 12:16 AM

Dreamy_Jazz closed subtask T351401: Create service(s) to send an image to PhotoDNA for a scan as Resolved.Jan 3 2024, 3:35 PM

Dreamy_Jazz updated the task description. (Show Details)Jan 8 2024, 2:43 PM

The second acceptance criteria is mostly done, but I will finish this as part of T354408: Support parallelizing scans to PhotoDNA.

Dreamy_Jazz moved this task from Priority Backlog to In Progress on the Trust and Safety Product Sprint (Sprint Northumbrian smallpipes (8th Jan.‘24 - 19th Jan.'24)) board.Jan 9 2024, 2:16 PM

To reduce the overall size of the patch in T354408: Support parallelizing scans to PhotoDNA, I will add the extra logging in a separate patch.

Dreamy_Jazz moved this task from Ready to In Progress on the Trust and Safety Product Sprint (Sprint Northumbrian smallpipes (8th Jan.‘24 - 19th Jan.'24)) board.Jan 12 2024, 12:09 AM

Change 989975 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] [WIP] Add more statsd counters and add logstash logging

https://gerrit.wikimedia.org/r/989975

gerritbot added a project: Patch-For-Review.Jan 12 2024, 12:32 AM

Dreamy_Jazz moved this task from In Progress to Needs review on the Trust and Safety Product Sprint (Sprint Northumbrian smallpipes (8th Jan.‘24 - 19th Jan.'24)) board.Jan 12 2024, 6:05 PM

Change 989975 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Add more statsd counters and add logstash logging

https://gerrit.wikimedia.org/r/989975

Maintenance_bot removed a project: Patch-For-Review.Jan 15 2024, 9:30 PM

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.14; 2024-01-16).Jan 15 2024, 11:00 PM

Suggested QA steps to be performed on or after the 17th:

Load https://grafana.wikimedia.org/d/STSXVVdSk/mediamoderation-photodna-stats?orgId=1&refresh=5m
Scroll down to the bottom of the page
Verify that the Transform errors/warnings graph has data
Open logstash
Go to the Trust and Safety Product logstash board
Verify that logstash entries appear related to not being able to scan SHA-1 values.

Dreamy_Jazz updated the task description. (Show Details)Jan 16 2024, 12:15 AM

Change 990760 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.13] Add more statsd counters and add logstash logging

https://gerrit.wikimedia.org/r/990760

gerritbot added a project: Patch-For-Review.Jan 16 2024, 2:11 PM

Change 990760 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.13] Add more statsd counters and add logstash logging

https://gerrit.wikimedia.org/r/990760

Mentioned in SAL (#wikimedia-operations) [2024-01-16T14:16:55Z] <dreamyjazz@deploy2002> Started scap: Backport for [[gerrit:990760|Add more statsd counters and add logstash logging (T351419)]]

Mentioned in SAL (#wikimedia-operations) [2024-01-16T14:18:33Z] <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:990760|Add more statsd counters and add logstash logging (T351419)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-01-16T14:24:10Z] <dreamyjazz@deploy2002> Finished scap: Backport for [[gerrit:990760|Add more statsd counters and add logstash logging (T351419)]] (duration: 07m 15s)

Maintenance_bot removed a project: Patch-For-Review.Jan 16 2024, 2:31 PM

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.13; 2024-01-09 ); removed MW-1.42-notes (1.42.0-wmf.14; 2024-01-16).Jan 16 2024, 3:00 PM