Page MenuHomePhabricator
Create Task
Maniphest T385273

MediaModeration: Investigate error increase for image transformation
Closed, ResolvedPublic
Actions

Description

Summary

There have been two days in a row with an increase in the number of errors and warnings in the MediaModeration PhotoDNA dashboard related to image Transformarion (x4) and lookups (x3), although today (Friday 31st) it looks like the numbers are lower.

Background

image.png (835×1 px, 145 KB)

Technical details

It appears that the success rate for getting thumbnails from the Thumbor service has decreased. It represents about 4% of images failing to get a thumbnail up from 1%. The increase in the number of source files being too large is a result of this, as if we don't have a thumbnail we try to use the source file.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
jobqueue: bump ThumbnailRender concurrencyoperations/deployment-chartsmaster+1 -1
Customize query in gerrit

Related Objects

Event Timeline

hector.arroyo created this task.Jan 31 2025, 12:52 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 31 2025, 12:52 PM
hector.arroyo updated the task description. (Show Details)Jan 31 2025, 12:55 PM
hector.arroyo renamed this task from {component}: {use imperative mood to describe desired outcome} to MediaModeration: Investigate increase in errors for iage transformation.Jan 31 2025, 1:01 PM
hector.arroyo renamed this task from MediaModeration: Investigate increase in errors for iage transformation to MediaModeration: Investigate error increase for image transformation.
Dreamy_Jazz added a project: Thumbor.Jan 31 2025, 2:17 PM
Dreamy_Jazz updated the task description. (Show Details)Jan 31 2025, 2:21 PM
Dreamy_Jazz updated the task description. (Show Details)
Dreamy_Jazz updated the task description. (Show Details)Jan 31 2025, 2:24 PM
Joe subscribed.Jan 31 2025, 2:53 PM

Errors from thumbor are stable:

image.png (899×1 px, 364 KB)

image.png (899×1 px, 238 KB)

hnowlan subscribed.Jan 31 2025, 4:25 PM
hnowlan mentioned this in T385292: Add file size metric to Thumbor metrics.Jan 31 2025, 4:55 PM
gerritbot added a comment.Jan 31 2025, 5:40 PM

Change #1115899 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] jobqueue: bump ThumbnailRender concurrency

https://gerrit.wikimedia.org/r/1115899

gerritbot added a project: Patch-For-Review.Jan 31 2025, 5:40 PM
Dreamy_Jazz subscribed.EditedFeb 3 2025, 12:02 PM

Work for the TSP team that we can do towards this task:

  1. Log the HTTP response codes from PhotoDNA - T385448
  2. Increase the request timeout for requests to Thumbor - T385450
  3. Re-attempt images that we failed to scan 1 day after their upload, as thumbnails are likely to have generated by then - T385478
Dreamy_Jazz added a project: Trust and Safety Product Sprint (Sprint Tuba (Jan 20 - Feb 7)).Feb 3 2025, 12:05 PM
Dreamy_Jazz moved this task from Priority Backlog to Ready on the Trust and Safety Product Sprint (Sprint Tuba (Jan 20 - Feb 7)) board.
Dreamy_Jazz mentioned this in T385448: MediaModeration: Graph HTTP response codes from Thumbor in Grafana dashboard.Feb 3 2025, 12:10 PM
Dreamy_Jazz mentioned this in T385450: MediaModeration: Increase request timeout for requests to the Thumbor service.Feb 3 2025, 12:25 PM
Dreamy_Jazz mentioned this in T385478: MediaModeration: Re-attempt scans of images scanned close to upload one day later.Feb 3 2025, 2:02 PM
gerritbot added a comment.Feb 4 2025, 11:13 AM

Change #1115899 merged by jenkins-bot:

[operations/deployment-charts@master] jobqueue: bump ThumbnailRender concurrency

https://gerrit.wikimedia.org/r/1115899

Maintenance_bot removed a project: Patch-For-Review.Feb 4 2025, 11:30 AM
hnowlan mentioned this in T385707: Add prometheus metrics for Commons file uploads.Feb 5 2025, 12:30 PM
kostajh edited projects, added Trust and Safety Product Sprint (Sprint Hurdy-Gurdy (Feb 10 - Feb 28)); removed Trust and Safety Product Sprint (Sprint Tuba (Jan 20 - Feb 7)).Feb 10 2025, 10:01 AM
kostajh moved this task from Priority Backlog to Ready on the Trust and Safety Product Sprint (Sprint Hurdy-Gurdy (Feb 10 - Feb 28)) board.
Dreamy_Jazz closed this task as Resolved.Feb 24 2025, 11:45 AM
Dreamy_Jazz moved this task from Ready to Done on the Trust and Safety Product Sprint (Sprint Hurdy-Gurdy (Feb 10 - Feb 28)) board.

The error rate has dropped back to it's normal levels, so I think we can close this. T385478: MediaModeration: Re-attempt scans of images scanned close to upload one day later should help increase the overall success rate further, but can be done later.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits