Page MenuHomePhabricator

[EL2Druid] Make RefineTarget compatible with Druid and use it from EventLoggingToDruid
Open, MediumPublic


The problem:
EventLoggingToDruid does not use RefineTarget to determine which data pieces
are available at a given moment and are to be loaded to Druid, because
currently RefineTarget does not support Druid. Instead, EventLoggingToDruid
just assumes that the passed date/time interval is correct and loads it
without any check or filter. The interval checking needs to be done then by
puppet (cron), passing a relative number of hours ago as since and until.

Potential issues:

  1. If the data pipeline is late for any reason (high load, outage, restarts, etc.) EventLoggingToDruid might not find the input data, or find it incomplete, thus loading corrupt data to Druid for that hour.
  2. If the cluster is busy and the EventLoggingToDruid job takes more than 1 hour to launch (waiting), then 'since 6 hours ago' will skip 1 hour (or more) and there will be a hole in the corresponding Druid datasource.

This would cause user confusion, frustration and give the maintainers lots
of work to manually backfill datasources.

The right solution:
Allow RefineTarget to deal with Druid segments

Event Timeline

fdans triaged this task as Medium priority.Oct 18 2018, 4:49 PM
fdans moved this task from Incoming to Smart Tools for Better Data on the Analytics board.

ping @mforns I think with recent changes this ticket might need updating.

Are you referencing the "double ingestion" (first hourly, then after a couple days daily), that is supposed to reduce the backfilling ingestion problems?
If so, I believe we agreed that was an interim solution that would buy us time to develop this task, no?