Broken data in edits_hourly Druid data cube
Closed, ResolvedPublic
Actions

Description

The edits_hourly data cube in Druid contains data that when queried results in errors in both Superset and Turnilo.

In Superset, querying edits_hourly for data that spans 2021-06-12 to 2021-06-30 (inclusive on both ends) will throw this error (as reported by @MNeisler in T346636#9179412):

java.lang.RuntimeException: net.jpountz.lz4.LZ4Exception: Error decoding offset 53533 of input buffer

In Turnilo, querying edits_hourly for data at any point within June 2021 results in an error:

Query error
Request failed with status
code 500

At this point, we don't know if other date ranges will also results in errors.

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		KCVelaga_WMF	T346636 Content Translation analytics dashboard shows errors for some graphs that were working before
		Resolved		JAllemandou	T346844 Broken data in edits_hourly Druid data cube

Event Timeline

nettrom_WMF created this task.Sep 19 2023, 10:08 PM

nettrom_WMF moved this task from Triage to Tracking on the Product-Analytics board.Sep 19 2023, 10:10 PM

Thanks @nettrom_WMF, this is underview as part of our ops week.

xcollazo added a subscriber: xcollazo.Sep 20 2023, 1:35 PM

VirginiaPoundstone moved this task from Incoming to To be discussed/To be estimated on the Data Products board.Sep 20 2023, 4:01 PM

Initial checks:

Error is reproducible in turnilo for month 2021-06.
This month is the only generating error in turnilo (other time-periods show data).
The data can be queried in the Hive edit_hourly table.

Interesting counter-intuitive result:

SQL queries on the druid dataset from SQLlab work!

Continuing investigations

JAllemandou moved this task from Next Up to In progress on the Data Engineering and Event Platform Team (Sprint 2) board.Sep 21 2023, 8:35 AM

The issue is with the __time druid field, but the error doesn't always show up depending on how it is accessed.
When accessed for filtering only at month level, no error, while it errors at day level for days between 2021-06-11 and 2021-06-30 (both included).
Given the error from druid (decompression error), this feels like a corrupted file from indexation.

I will launch a re-indexation of the dataset, which hopefully should fix the issue.

Issue fixed!

JAllemandou mentioned this in T346636: Content Translation analytics dashboard shows errors for some graphs that were working before.Sep 21 2023, 9:34 AM

JAllemandou moved this task from In progress to Done on the Data Engineering and Event Platform Team (Sprint 2) board.Sep 21 2023, 9:39 AM

lbowmaker closed this task as Resolved.Sep 29 2023, 1:11 PM

Nikerabbit moved this task from Quarter Backlog to Done on the Language-Team (Language-2023-July-September) board.Oct 2 2023, 9:58 AM