Change Details

When all varnishkafka instances are restarted during the same calendar-hour, the `webrequest_sequence_stats_hourly` table is empty as we filter out rows having their `sequence_min` equals to 0. This makes the dataloss checks fail (both ERROR and WARNING) because no file is generated by Hive when the query has no input data. I suggest mitigating this by checking for the file existence before checking for its size [[ https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/check_sequence_statistics_workflow.xml#L158 | here ]] and [[ https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/check_sequence_statistics_workflow.xml#L207 | here ]].

When all varnishkafka instances are restarted during the same calendar-hour, the `webrequest_sequence_stats_hourly` table is empty as we filter out rows having their `sequence_min` equals to 0. This makes the dataloss checks fail (both ERROR and WARNING) because no file is generated by Hive when the query has no input data. I suggest mitigating this by checking for the file existence before checking for its size [[ https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/check_sequence_statistics_workflow.xml#L158 | here ]] and [[ https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/check_sequence_statistics_workflow.xml#L207 | here ]]. When we move the job to airflow and spark we wish to avoid that corner case.

When all varnishkafka instances are restarted during the same calendar-hour, the `webrequest_sequence_stats_hourly` table is empty as we filter out rows having their `sequence_min` equals to 0. This makes the dataloss checks fail (both ERROR and WARNING) because no file is generated by Hive when the query has no input data. I suggest mitigating this by checking for the file existence before checking for its size [[ https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/check_sequence_statistics_workflow.xml#L158 | here ]] and [[ https://github.com/wikimedia/analytics-refinery/blob/master/oozie/webrequest/load/check_sequence_statistics_workflow.xml#L207 | here ]]. When we move the job to airflow and spark we wish to avoid that corner case.