Problem / Context:
When adding a new stream to Refine, we currently need to coordinate three jobs:
1/ Canary events creation
2/ Gobblin ingestion to HDFS
3/ Refinement of the partition

The problematic interaction is:

• ESC makes it immediately accessible from Canary & Refine.
• Canary-events dag generates events without delay.
• Gobblin, on a new topic, starts ingestion from the last event onward.
• Refine fetches the config from ESC with a 1h25min delay.
• Refine then "senses" for data for 2 consecutive hours.

Observed Failure:
When a new job is declared, Refine times out while sensing for data in a partition because:

• Canary events may not yet exist for the stream.
• Gobblin may not have imported the first canary events to HDFS.

This leads to unnecessary job failures on first runs of new streams.

Proposed Solution:
When building the list of streams to refine at the start of the Refine DAG run, fetch only the list of streams for which canary events were produced at least 3 hours ago.

• This ensures Canary has a head start over Refine.
• Keeps the pipeline consistent by avoiding streams that aren’t ready.
• Cost is minimal since the filtering is lightweight.

Drawback: we may miss the Refinement of an hour or two. But those are likely to be empty. And this will be documented.