This batch indexation job allows for realtime tasks to fail without dataloss (lambda architecture :).
- Create a gobblin job for the webrequest_sampled topic
- Create a table in hive wmf_rawfor the gobbled data
- Create a spark HQL file to deduplicate rows from the wmf_raw table into a temporary table
- Create an airflow job that
- waits for gobblin data to be present (hour+2)
- creates the hive partition in the wmf_raw table
- Run the spark HQL job to deduplicate the raw data
- Run the HiveToDruid job on the temporary table
- Cleans up the temporary table
