>>! In T376713#10284256, @xcollazo wrote:
> ...
>
> On a related note, even though we have disabled `merge-on-read`, it was necessary to run `rewrite_position_delete_files()` on this table, and in the future, we may need to do it again to remove [[ https://iceberg.apache.org/docs/1.5.1/spark-procedures/#rewrite_position_delete_files | what Iceberg folks call 'dangling deletes', ]] which do affect performance. Unfortunately, this procedure is only available on Iceberg 1.3.0+, and we have 1.2.1 in production. I was able to run it manually since we have the ability to run Iceberg 1.6.1 in PyPsark jointly with Spark 3.3.2, but this is not available in our table maintenance mechanism due to the version. Will open a ticket to potentially pursue this.
[[ https://iceberg.apache.org/multi-engine-support/#current-engine-version-lifecycle-status | As per Iceberg support matrix ]], the last version to support our production Spark 3.1.2 was **Iceberg 1.3.1**.
As per procedures documentation, [[ https://web.archive.org/web/20231128193807/https://iceberg.apache.org/docs/1.3.1/spark-procedures/#rewrite_position_delete_files | Iceberg 1.3.1 does support ]] the `rewrite_position_delete_files()` procedure.
Since upgrading Iceberg is significantly less work than upgrading Spark, in this task we want to bump the production Iceberg library to 1.3.1 to help the work being doing in {T358877}