Page MenuHomePhabricator

webrequest / webrequest raw quality check
Closed, DeclinedPublic

Description

Following the migration of webrequest load from Oozie to Airflow, we are not generating SUCCESS files anymore.

The systemd data quality check check_webrequest_partitions relied on those files. It was forgotten. So we have removed the alert by eliminating the job.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/908529

Maybe we do need a data quality job for those datasets. Something that checks the hive partition exists and that it is populated with a minimum number (TBD) of rows.