Adding support for Iceberg in Spark 3.1.x is as simple as:
spark3-sql --packages org.apache.iceberg:iceberg-spark-runtime-3.1_2.12:1.2.1 \ --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \ --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \ --conf spark.sql.catalog.spark_catalog.type=hive
The above jar, iceberg-spark-runtime-3.1_2.12:1.2.1, provides the runtime dependencies for Iceberg 1.2.1 to be used in the Spark 3.1.x series, compiled against Scala 2.12.
However, we'd like the runtime dependency and configuration to be available automatically so that users don't need to think about it.
In this task we should make it so.