Change Details

This batch indexation job allows for realtime tasks to fail without dataloss (lambda architecture :). [x] Create a gobblin job for the `webrequest_sampled` topic [] Create a table in hive `wmf_raw`for the gobbled data [] Create a spark HQL file to deduplicate rows from the `wmf_raw` table into a temporary table [] Create an airflow job that * waits for gobblin data to be present (hour+2) * creates the hive partition in the `wmf_raw` table * Run the spark HQL job to deduplicate the raw data * Run the HiveToDruid job on the temporary table * Cleans up the temporary table

This batch indexation job allows for realtime tasks to fail without dataloss (lambda architecture :). [x] Create a gobblin job for the `webrequest_sampled` topic [x] Create a table in hive `wmf_raw`for the gobbled data [x] Create a spark HQL file to deduplicate rows from the `wmf_raw` table into a temporary table [] Create an airflow job that * waits for gobblin data to be present (hour+2) * creates the hive partition in the `wmf_raw` table * Run the spark HQL job to deduplicate the raw data * Run the HiveToDruid job on the temporary table * Cleans up the temporary table

This batch indexation job allows for realtime tasks to fail without dataloss (lambda architecture :). [x] Create a gobblin job for the `webrequest_sampled` topic [x] Create a table in hive `wmf_raw`for the gobbled data [x] Create a spark HQL file to deduplicate rows from the `wmf_raw` table into a temporary table [] Create an airflow job that * waits for gobblin data to be present (hour+2) * creates the hive partition in the `wmf_raw` table * Run the spark HQL job to deduplicate the raw data * Run the HiveToDruid job on the temporary table * Cleans up the temporary table