Change Details

In {T338065} we developed a mechanism that can do typical table maintenance for Iceberg table. However, this mechanism currently works off of the production Iceberg version which is 1.2.1. This version suffers for an issue now known as the Iceberg "dangling delete" problem that, effectively, does not allow us to use `merge-on-read`, since we accumulate delete files that we can never delete. This issue has been fixed on Iceberg 1.6.1 via a new iceberg table maintenance procedure called `rewrite_position_delete_files()` ([[ https://iceberg.apache.org/docs/1.6.1/spark-procedures/#rewrite_position_delete_files | docs ]]). Iceberg 1.8.1 and on deal with this in a [[ https://iceberg.apache.org/docs/1.8.1/spark-procedures/#rewrite_data_files | more elegant way ]], but that version does not support Spark 3.3.2 anymore, and we want to keep the scope of this fix manageable. In this task we should: [] Modify our table maintenance to launch pypsark jobs instead of vanilla `SparkSqlOperator` jobs, so that we can control the Spark and Iceberg version that they bundle. [] Test to see that we can use `rewrite_position_delete_files()` effectively.

In {T338065} we developed a mechanism that can do typical table maintenance for Iceberg table. However, this mechanism currently works off of the production Iceberg version which is 1.2.1. This version suffers for an issue now known as the Iceberg "dangling delete" problem that, effectively, does not allow us to use `merge-on-read`, since we accumulate delete files that we can never delete. This issue has been fixed on Iceberg 1.6.1 via a new iceberg table maintenance procedure called `rewrite_position_delete_files()` ([[ https://iceberg.apache.org/docs/1.6.1/spark-procedures/#rewrite_position_delete_files | docs ]]). Iceberg 1.8.1 and on deal with this in a [[ https://iceberg.apache.org/docs/1.8.1/spark-procedures/#rewrite_data_files | more elegant way ]], but that version does not support Spark 3.3.2 anymore, and we want to keep the scope of this fix manageable. In this task we should: [x] Modify our table maintenance to launch Spark vis conda env jobs instead of vanilla `SparkSqlOperator` jobs, so that we can control the Spark and Iceberg version that they bundle. [] ~~Test to see that we can use `rewrite_position_delete_files()` effectively.~~ Will test this later as we are not ready to turn `merge-on-read` on yet. But the current set of Iceberg table maintenance tasks is running nicely on Spark 3.3.2.

In {T338065} we developed a mechanism that can do typical table maintenance for Iceberg table. However, this mechanism currently works off of the production Iceberg version which is 1.2.1. This version suffers for an issue now known as the Iceberg "dangling delete" problem that, effectively, does not allow us to use `merge-on-read`, since we accumulate delete files that we can never delete. This issue has been fixed on Iceberg 1.6.1 via a new iceberg table maintenance procedure called `rewrite_position_delete_files()` ([[ https://iceberg.apache.org/docs/1.6.1/spark-procedures/#rewrite_position_delete_files | docs ]]). Iceberg 1.8.1 and on deal with this in a [[ https://iceberg.apache.org/docs/1.8.1/spark-procedures/#rewrite_data_files | more elegant way ]], but that version does not support Spark 3.3.2 anymore, and we want to keep the scope of this fix manageable. In this task we should: [x] Modify our table maintenance to launch pypsarkSpark vis conda env jobs instead of vanilla `SparkSqlOperator` jobs, so that we can control the Spark and Iceberg version that they bundle. [] ~~Test to see that we can use `rewrite_position_delete_files()` effectively.~~ Will test this later as we are not ready to turn `merge-on-read` on yet. But the current set of Iceberg table maintenance tasks is running nicely on Spark 3.3.2.