Page MenuHomePhabricator

Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA
Closed, ResolvedPublic

Description

In T351401: Create service(s) to send an image to PhotoDNA for a scan, services are created to interface with the PhotoDNA API in a method that is compatible with scanning files from the mediamoderation_scan table. This service should be expanded to collect and publish to statsd information about how long it takes to receive a response from the PhotoDNA API as well as the number of requests sent to the API.

The existing implementation in RequestModerationCheck::requestModeration collects data on how long it takes to receive a response and publishes it to statsd. We can use this as a basis for the new services.

This data collection will allow monitoring of the response times along with setting up alerting to indicate when the response times exceed a threshold and/or if requests to the API drop below a threshold. This is needed so that we can investigate if the API becomes too slow or less requests are being made, as this is likely to make it harder or impossible to scan files faster than they are uploaded.

Acceptance criteria
  • Collect and publish to statsd metrics about how long it takes for the PhotoDNA API to respond to a API request
  • Collect and publish to statsd the average number of requests to PhotoDNA per day

Event Timeline

Dreamy_Jazz renamed this task from Collect response times from the PhotoDNA API as a metric and publish these to statsd to Collect response times from the PhotoDNA API and publish these to statsd.Dec 14 2023, 12:10 PM
Dreamy_Jazz renamed this task from Collect response times from the PhotoDNA API and publish these to statsd to Collect statsd metrics of requests per day to PhotoDNA and response times from the PhotoDNA.Dec 14 2023, 12:34 PM
Dreamy_Jazz updated the task description. (Show Details)
Dreamy_Jazz updated the task description. (Show Details)

Change 983180 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/MediaModeration@master] PhotoDNAServiceProvider: Add statsd instrumentation

https://gerrit.wikimedia.org/r/983180

Change 983180 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] PhotoDNAServiceProvider: Add statsd instrumentation

https://gerrit.wikimedia.org/r/983180

I propose that no QA is needed for this ticket, as this should be QA'able in T351419.

Djackson-ctr subscribed.

I agree with Dreamy_Jazz's comment regarding QA, therefore I am pushing this ticket to Done.

Change 984156 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/MediaModeration@master] statsd: Log check attempt and HTTP failures

https://gerrit.wikimedia.org/r/984156

Change 984168 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.10] statsd: Log check attempt failures

https://gerrit.wikimedia.org/r/984168

Change 984156 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] statsd: Log check attempt failures

https://gerrit.wikimedia.org/r/984156

Change 984168 abandoned by Kosta Harlan:

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.10] statsd: Log check attempt failures

Reason:

https://gerrit.wikimedia.org/r/984168

Going to mark this as resolved. Adding the Grafana graphs for this data will be done in the parent task.