Gobblin hiccups are currently generating late events, which in turn trigger unnecessary alerts. As we have alerts for webrequest, this is where we see the problem. But the same applies to events.
Here is a proposal for an easy fix:
1/ Increase the sensing window from 2 consecutive hours to 3.
2/ Delay the jobs by 1 hour using the DelayedTimeTable mechanism.
For those 3 dags:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Make webrequest and refine wait 2 gobblin hours | repos/data-engineering/airflow-dags!1729 | joal | update_gobblin_sensors | main |
I think we should go for option 1, it allows the delay to be based on data rather than calendar time, making it reliable even in case of reruns.
This will mean that event and pageview data will show up one hour later than they currently do. Is it ok from a product perspective?
Pinging my usual suspects @nshahquinn-wmf, @Mayakp.wiki and @Hghani on this by ignorance about whom else to tag :)
joal opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1729
Make webrequest and refine wait 2 gobblin hours
joal merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1729
Make webrequest and refine wait 2 gobblin hours