NEW BUG REPORT : Referer Name field is Null on Druid pageviews_daily
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	Mayakp.wiki
	Oct 6 2025, 10:01 PM

Description

Data Platform Engineering Bug Report or Data Problem Form.

Please fill out the following

Please ensure you set priority

What kind of problem are you reporting?

Access related problem
Service related problem
Data related problem

For a data related problem:

Is this a data quality issue?

Yes

What datasets and/or dashboards are affected?

Druid - pageviews_daily

What are the observed vs expected results? Please include information such as location of data, any initial assessments, sql statements, screenshots

pageviews_hourly has referer_name field which is not populated in pageviews_daily. We are interested in looking at referal data in detail and would like to visualize it on Turnilo and Superset.

Current result:
All referer names are null - https://w.wiki/FZv3

Expected result:
pageviews_daily should have valid values for referer name, similar to pageviews_hourly - https://w.wiki/FZv4

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Add referer_name to druid pageview_daily	analytics/refinery	master	+10 -0

Customize query in gerrit

Related Changes in GitLab:

	Title	Reference	Author	Source Branch	Dest Branch
	Add referer_name to pageviews_daily druid loading	repos/data-engineering/airflow-dags!1734	joal	update_pageviews_daily_druid_load	main

Customize query in GitLab

Related Objects

Mentioned Here: T334120: Update Hive pageview_hourly with referer_name
T334104: [Airflow] Migrate pageview-related Druid loading Oozie jobs

Event Timeline

Mayakp.wiki created this task.Oct 6 2025, 10:01 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 6 2025, 10:01 PM

hmm, apparently this was done as a part of T334104#8759879, but I wonder if we encountered a bug later and the data stopped loading from hive to druid?

Mayakp.wiki triaged this task as High priority.Oct 6 2025, 10:08 PM

Mayakp.wiki added a project: Movement-Insights.

The referer_name field is loaded in the pageviews_hourly datasource containing 90 days of data.
We don't load it in the pageviews_daily, to reduce data size as this datasource contains multiple years of data. We have the referer_class but not the name.
Is this enough for you to have the latest 90 days in turnilo knowing that longer term data is available in the wmf_readership.referer_daily table (this table is optimized for superset, response times should be fast :).

@JAllemandou - We would prefer to have the full history of referer_name in pageviews_daily data sources. We have 'Turnilo-only' users who are interested in viewing the full history of individual referrers and Turnilo offers a much better experience compared to Superset. Will loading this additional field cause any additional challenges beyond the extra storage cost?

GGoncalves-WMF subscribed.Oct 8 2025, 4:41 PM

Ottomata assigned this task to JAllemandou.Oct 8 2025, 5:07 PM

Ahoelzl moved this task from Incoming (new tickets) to Q1 FY25/26 July 1st - September 30th on the Data-Engineering board.Oct 8 2025, 5:07 PM

Ahoelzl edited projects, added Data-Engineering (Q1 FY25/26 July 1st - September 30th); removed Data-Engineering.

@OSefu-WMF - I think druid can cope with the field addition. The datasource currently weights 250Gb, and I assume that adding the new field will not make it grow more than 50Gb (20/30Gb is probably closer to reality). The druid cluster can handle that. Adding the field will require a full reload of the datasource, which will take time.
Also, I'd be interested to have a broader discussion on turnilo vs superset, which is actually more about Druid vs Presto. I'd like for us to remove druid from our analytics use-cases, replacing it by dedicated datasets to be used with presto, providing a similar experience (probably a bit slower) in Superset. The concern is that this implies removing Turnilo as it's only available for Druid, which does't seem to be an option.
I'd be happy to have your views on this.