Page MenuHomePhabricator

Batch index webrequest_sampled data in Druid
Closed, ResolvedPublic

Description

This batch indexation job allows for realtime tasks to fail without dataloss (lambda architecture :).

  • Create a gobblin job for the webrequest_sampled topic
  • Create a table in hive wmf_rawfor the gobbled data
  • Create a spark HQL file to deduplicate rows from the wmf_raw table into a temporary table
  • Create an airflow job that
    • waits for gobblin data to be present (hour+2)
    • creates the hive partition in the wmf_raw table
    • Run the spark HQL job to deduplicate the raw data
    • Run the HiveToDruid job on the temporary table
    • Cleans up the temporary table

Details

Related Changes in Gerrit:
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Fix druid webrequest_sampled load jobrepos/data-engineering/airflow-dags!1972joalfix_druid_webrequest_sampledmain
Update druid webrequest_sampled load jobrepos/data-engineering/airflow-dags!1971joalupdate_druid_webrequest_sampledmain
Load webrequest_sampled in druid hourlyrepos/data-engineering/airflow-dags!1967joalwebrequest_sampled_druidmain
Add Gobblin webrequest_sampled jobrepos/data-engineering/airflow-dags!1960joaladd_webrequest_sampledmain
Customize query in GitLab

Event Timeline

Change #1235740 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery@master] HQL for druid webrequest_sampled ingestion

https://gerrit.wikimedia.org/r/1235740

Change #1235740 merged by Joal:

[analytics/refinery@master] HQL for druid webrequest_sampled ingestion

https://gerrit.wikimedia.org/r/1235740

All code has been merged but I paused the job as the newly indexed data shows a discrepancy on the time_firstbyte metric:

image.png (1,600×1,151 px, 240 KB)

I will investigate that tomorrow.
It is interesting to note that my previous manual batch-ingestion job after cluster-upgrade has not generated this problem.
What is different between the new job and the manual one:

  • Data is deduplicated and stored in parquet as an intermediary format before being ingested
  • Ingestion is made through a dedicated copy field in hive instead of base field (the loading job we have needs to use the same hive/druid field names).

Change #1236351 had a related patch set uploaded (by Joal; author: Joal):

[analytics/refinery@master] Fix druid webrequest_sampled temporary data job

https://gerrit.wikimedia.org/r/1236351

Change #1236351 merged by Joal:

[analytics/refinery@master] Fix druid webrequest_sampled temporary data job

https://gerrit.wikimedia.org/r/1236351

Bug found, I had inverted typed for the two dedicated fields, making time_firstbyte a bigint instead of double.
Hot fix deployed, we can call this done.