Attempt to generate thumbnail by requesting URL
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	kostajh
	Jan 29 2024, 12:56 PM

Description

As a follow-up to the parent task, one thing we can try is an HTTP GET request to the the URL associated with the thumbnail. That should result in the thumbnail getting created, which we can then make use of in jobs that send thumbnail contents to PhotoDNA.

In T355243#9470445, @Tgr wrote:

In T355243#9470364, @Tgr wrote:

Although when I try this, there are a bunch of Thumbor-* headers on the response so it doesn't seem like it was created by MediaWiki shelling out. So I don't really understand what's going on - thumb.php also uses RENDER_NOW internally.

I think what happens (but not at all sure about it) is that the existence check in thumb.php makes Swift/Thumbor generate that file on demand. So I think you just need to copy that logic (we should probably encapsulate it in some File method for convenience).

Details

Subject	Repo	Branch	Lines +/-
Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file	mediawiki/extensions/MediaModeration	master	+1 -1
Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file	mediawiki/extensions/MediaModeration	wmf/1.42.0-wmf.16	+1 -1
Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file	mediawiki/extensions/MediaModeration	wmf/1.42.0-wmf.17	+1 -1
MediaModerationImageContentsLookup: use proxied HTTP request to generate file	mediawiki/extensions/MediaModeration	wmf/1.42.0-wmf.16	+162 -7
MediaModerationImageContentsLookup: use proxied HTTP request to generate file	mediawiki/extensions/MediaModeration	wmf/1.42.0-wmf.17	+162 -7
MediaModerationImageContentsLookup: use proxied HTTP request to generate file	mediawiki/extensions/MediaModeration	master	+162 -7

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Stalled	None	T255792 Quibble runs core:unit tests twice!
Open	None	T328919 Upgrade to PHPUnit 10
Open	None	T338103 Micro-optimize ApiResult::isMetadataKey with str_starts_with once we support PHP8+
Open	None	T328921 Drop PHP 7.4 support from MediaWiki
Stalled	None	T334726 Use return type `never` in Wikibase
Open	None	T328922 Drop PHP 8.0 support from MediaWiki
Stalled	None	T319055 Upgrade to psr/container 2.x
Stalled	Krinkle	T319432 Migrate WMF production from PHP 7.4 to PHP 8.1
Open	None	T291916 Tracking task for Bullseye migrations in production
Stalled	None	T356293 Migrate MW appservers' base images to bullseye
Open	None	T290536 Serve production traffic via Kubernetes
Resolved	hnowlan	T349796 Move MediaWiki jobs to mw-on-k8s
Open	None	T260504 Get rid of remaining non-Thumbor MediaWiki image scaling in WMF production
Resolved	None	T355243 MediaModeration maintenance script scanFilesInScanTable.php indirectly calls $wgImageMagickConvertCommand
Resolved	kostajh	T356047 Attempt to generate thumbnail by requesting URL

Event Timeline

kostajh created this task.Jan 29 2024, 12:56 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 29 2024, 12:56 PM

kostajh updated the task description. (Show Details)Jan 29 2024, 12:57 PM

kostajh added a subscriber: Tgr.

kostajh added a subscriber: Dreamy_Jazz.

We'll try to get to this if we have time.

Per the comment you quoted, I think the simpler and better option is to just call fileExists. Not sure about it but I think on Wikimedia servers that will generate the file. On non-Wikimedia servers you can keep shelling out with RENDER_NOW.

kostajh claimed this task.Jan 31 2024, 8:40 PM

Change 994829 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/extensions/MediaModeration@master] [WIP] MediaModerationImageContentsLookup: use fileExists to generate file

https://gerrit.wikimedia.org/r/994829

gerritbot added a project: Patch-For-Review.Jan 31 2024, 8:42 PM

kostajh moved this task from Priority Backlog to In Progress on the Trust and Safety Product Sprint (Sprint Kazoo (Jan 29 - Feb 9 2024)) board.Jan 31 2024, 8:43 PM

kostajh moved this task from In Progress to Needs Review on the Trust and Safety Product Sprint (Sprint Kazoo (Jan 29 - Feb 9 2024)) board.Jan 31 2024, 8:59 PM

Dreamy_Jazz mentioned this in T355243: MediaModeration maintenance script scanFilesInScanTable.php indirectly calls $wgImageMagickConvertCommand.Feb 2 2024, 12:30 PM

Change 998703 had a related patch set uploaded (by Dreamy Jazz; author: Kosta Harlan):

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.16] MediaModerationImageContentsLookup: use proxied HTTP request to generate file

https://gerrit.wikimedia.org/r/998703

Change 994829 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] MediaModerationImageContentsLookup: use proxied HTTP request to generate file

https://gerrit.wikimedia.org/r/994829

Change 998704 had a related patch set uploaded (by Dreamy Jazz; author: Kosta Harlan):

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.17] MediaModerationImageContentsLookup: use proxied HTTP request to generate file

https://gerrit.wikimedia.org/r/998704

Dreamy_Jazz moved this task from Needs Review to Needs QA on the Trust and Safety Product Sprint (Sprint Kazoo (Jan 29 - Feb 9 2024)) board.Feb 8 2024, 1:15 PM

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.18; 2024-02-13).Feb 8 2024, 2:01 PM

Change 998704 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.17] MediaModerationImageContentsLookup: use proxied HTTP request to generate file

https://gerrit.wikimedia.org/r/998704

Mentioned in SAL (#wikimedia-operations) [2024-02-08T14:46:54Z] <dreamyjazz@deploy2002> Started scap: Backport for [[gerrit:998704|MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]]

Mentioned in SAL (#wikimedia-operations) [2024-02-08T14:48:27Z] <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:998704|MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-02-08T14:54:43Z] <dreamyjazz@deploy2002> Finished scap: Backport for [[gerrit:998704|MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]] (duration: 07m 49s)

Change 998703 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.16] MediaModerationImageContentsLookup: use proxied HTTP request to generate file

https://gerrit.wikimedia.org/r/998703

Mentioned in SAL (#wikimedia-operations) [2024-02-08T15:08:18Z] <dreamyjazz@deploy2002> Started scap: Backport for [[gerrit:998703|MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]]

Mentioned in SAL (#wikimedia-operations) [2024-02-08T15:09:48Z] <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:998703|MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-02-08T15:17:01Z] <dreamyjazz@deploy2002> Finished scap: Backport for [[gerrit:998703|MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]] (duration: 08m 42s)

Maintenance_bot removed a project: Patch-For-Review.Feb 8 2024, 3:30 PM

Change 998954 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@master] Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file

https://gerrit.wikimedia.org/r/998954

gerritbot added a project: Patch-For-Review.Feb 8 2024, 3:33 PM

Dreamy_Jazz moved this task from Needs QA to Needs Review on the Trust and Safety Product Sprint (Sprint Kazoo (Jan 29 - Feb 9 2024)) board.Feb 8 2024, 3:34 PM

Change 998966 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.17] Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file

https://gerrit.wikimedia.org/r/998966

Change 998967 had a related patch set uploaded (by Dreamy Jazz; author: Dreamy Jazz):

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.16] Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file

https://gerrit.wikimedia.org/r/998967

Change 998966 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.17] Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file

https://gerrit.wikimedia.org/r/998966

Change 998967 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@wmf/1.42.0-wmf.16] Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file

https://gerrit.wikimedia.org/r/998967

Mentioned in SAL (#wikimedia-operations) [2024-02-08T15:46:09Z] <dreamyjazz@deploy2002> Started scap: Backport for [[gerrit:998967|Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]], [[gerrit:998966|Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]]

Dreamy_Jazz moved this task from Needs Review to Needs QA on the Trust and Safety Product Sprint (Sprint Kazoo (Jan 29 - Feb 9 2024)) board.Feb 8 2024, 3:46 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-08T15:47:39Z] <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:998967|Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]], [[gerrit:998966|Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-02-08T15:54:13Z] <dreamyjazz@deploy2002> Finished scap: Backport for [[gerrit:998967|Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]], [[gerrit:998966|Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file (T356047)]] (duration: 08m 03s)

Change 998954 merged by jenkins-bot:

[mediawiki/extensions/MediaModeration@master] Follow-up: MediaModerationImageContentsLookup: use proxied HTTP request to generate file

https://gerrit.wikimedia.org/r/998954

Maintenance_bot removed a project: Patch-For-Review.Feb 8 2024, 4:30 PM

ReleaseTaggerBot edited projects, added MW-1.42-notes (1.42.0-wmf.17; 2024-02-06); removed MW-1.42-notes (1.42.0-wmf.18; 2024-02-13).Feb 8 2024, 5:00 PM

QA has been executed and no issues were found during testing.

kostajh closed this task as Resolved.Feb 8 2024, 10:00 PM

Impact of this change (from Grafana)

Drop in request time	Drop in "source file used instead of thumbnail" instances	Drop in "Thumbnail and source image lookup failed"	Drop in transform errors/warnings

	F41912391: image.png
	Feb 15 2024, 1:47 PM

	F41912387: image.png
	Feb 15 2024, 1:47 PM

	F41912379: image.png
	Feb 15 2024, 1:47 PM

	F41912370: image.png
	Feb 15 2024, 1:47 PM

Attempt to generate thumbnail by requesting URLClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Attempt to generate thumbnail by requesting URL
Closed, ResolvedPublic
Actions

Related Objects
Search...