In T376882: 2024-10-10 Data Loss Incident - webrequest Hive table , we encountered a Hadoop bug where concurrent writes from Spark to a Hive table can result in hidden data loss. In many cases, it is possible to detect this bug after the Spark job finishes. If the number of Spark partitions is explicitly defined (via e.g. coalesce or repartition), the number of expected output files in a Hive partition is known.

We'd like to fail the Spark job, or perhaps a subsequent Airflow task (a data quality check?) if the count of output files does not match the expected count.

Possible Solutions

TODO: Expand these as we work on this

SparkSubmitOperator and bash script to check

Spark library function to call at end of Spark job

If we did this, we'd likely need a UDF that is callable from Spark SQL as well?

Subsequent Airflow task

This would allow the failure of a dag run by a task after the writing Spark job finishes. The subsequent task would have to be passed the parameterized coelesce_partitions used.

This solution may allow us to use and/or expand work done for Data Quality checks

Chosen Solution

TBD (Could become a custom Airflow operator)