Audit and update data pipelines that extract thumbnail requests from webrequest_text
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	GGoncalves-WMF
	Jul 21 2026, 2:26 PM

Project Tags

Referenced Files

None

Subscribers

Description

In the parent ticket, information about thumbnail requests is being moved from the upload cluster to text. This means queries like:

SELECT ...
FROM wmf.webrequest
WHERE webrequest_source = 'upload'
AND ... // Logic that expects to include thumbnails

needs to change, temporarily, to remove the webrequest_source line, as thumbnail information will be split across clusters during the rollout of the change.

Then, once that is done, we can reintroduce the line with webrequest_source = 'text'.

The most obvious example is the mediarequest pipeline, which is tracked in T431475.

In this ticket, we should look for the above pattern in other pipelines under analytics-refinery and migrate them, if any.

Details

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	mediacounts/mediarequest: wait on both webrequest sources for thumbnail move	repos/data-engineering/airflow-dags!2526	ebysans	thumbnail	main
	Update media metrics models to align with cluster changes	repos/data-engineering/dbt-jobs!163	kcvelaga	update-media-models-for-thumbnails	main

Customize query in GitLab

Related Objects
Search...

Status	Assigned	Task
Open	JTweed-WMF	T431141 FY26-27 PP3.6: Image thumbnailing
Open	Ladsgroup	T431719 PP3.6.2: Thumbnail delivery
In Progress	None	T427465 Move thumbnail caching from upload cluster to text
Resolved	Snwachukwu	T432735 Audit and update data pipelines that extract thumbnail requests from webrequest_text
Resolved	JAllemandou	T431475 Data Lake - mediarequests - adapt HQL query to include counting mediarequests from webrequest_text

Event Timeline

GGoncalves-WMF created this task.Jul 21 2026, 2:26 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2026, 2:26 PM

GGoncalves-WMF added a subtask: T431475: Data Lake - mediarequests - adapt HQL query to include counting mediarequests from webrequest_text.Jul 21 2026, 2:27 PM

GGoncalves-WMF added a parent task: T427465: Move thumbnail caching from upload cluster to text.

GGoncalves-WMF added subscribers: Ottomata, KHernandez-WMF.

KCVelaga_WMF subscribed.Jul 21 2026, 7:27 PM

Ottomata assigned this task to Ahoelzl.Jul 22 2026, 1:38 PM

Ottomata moved this task from Incoming (new tickets) to Q4 FS25/26 April 1st - June 30st on the Data-Engineering board.

Ottomata edited projects, added: Data-Engineering (Q4 FS25/26 April 1st - June 30st); removed: Data-Engineering.

Ottomata moved this task from Q4 FS25/26 April 1st - June 30st to Q1 FS26/27 July 1st - September 30th on the Data-Engineering board.

Ottomata edited projects, added: Data-Engineering (Q1 FS26/27 July 1st - September 30th); removed: Data-Engineering (Q4 FS25/26 April 1st - June 30st).

Ottomata added a subscriber: Ahoelzl.

Ottomata reassigned this task from Ahoelzl to Snwachukwu.Aug 3 2026, 4:38 PM

Did the audit across analytics-refinery, dbt-jobs and airflow-dags. Apart from the mediacounts + mediarequest HQLs already handled by Joseph under {T431475: Data Lake - mediarequests - adapt HQL query to include counting mediarequests from webrequest_text}, there is no other HQL script that needs top be changed in analytics-refinery.

However I found the following:

dbt-jobs: stg_webrequest_upload_bandwidth and stg_webrequest_media_clients were both pinned to the upload cluster .
airflow-dags: the mediacounts/mediarequest HQLs now read both sources, but the DAGs were only waiting on the upload partition, I would update both to wait on both partitions.

Left MediaFileUrlParser job as-is. it just won't validate thumb URLs but I believe that's okay for since it's temporarily.