As per Iceberg support matrix, the last version to support our production Spark 3.1.2 was Iceberg 1.3.1.
As per procedures documentation, Iceberg 1.3.1 does support the rewrite_position_delete_files() procedure.
Since upgrading Iceberg is significantly less work than upgrading Spark (see T335721), in this task we want to bump the production Iceberg library to 1.3.1 to help the work being doing in T358877: Dumps 2.0 Phase II: Production intermediate table milestone
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Update iceberg to version 1.3.1 | repos/data-engineering/conda-analytics!54 | btullis | bump_iceberg | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | xcollazo | T358877 Dumps 2.0 Phase II: Production intermediate table milestone | |||
| Declined | BTullis | T378819 Bump Iceberg to 1.3.1; the most recent version that still supports Spark 3.1.2 |
@xcollazo - This is ready for testing on the hadoop-test cluster.
btullis@cumin1002:~$ sudo cumin 'an-test-w* or an-test-m* or an-test-c*' "find /opt/conda-analytics -name '*iceberg*'" 7 hosts will be targeted: an-test-client1002.eqiad.wmnet,an-test-coord1001.eqiad.wmnet,an-test-master[1001-1002].eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet OK to proceed on 7 hosts? Enter the number of affected hosts to confirm or "q" to quit: 7 ===== NODE GROUP ===== (7) an-test-client1002.eqiad.wmnet,an-test-coord1001.eqiad.wmnet,an-test-master[1001-1002].eqiad.wmnet,an-test-worker[1001-1003].eqiad.wmnet ----- OUTPUT of 'find /opt/conda-...name '*iceberg*'' ----- /opt/conda-analytics/lib/python3.10/site-packages/pyspark/jars/iceberg-spark-runtime-3.1_2.12-1.3.1.jar ================ PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100% (7/7) [00:01<00:00, 4.57hosts/s] FAIL | | 0% (0/7) [00:01<?, ?hosts/s] 100.0% (7/7) success ratio (>= 100.0% threshold) for command: 'find /opt/conda-...name '*iceberg*''. 100.0% (7/7) success ratio (>= 100.0% threshold) of nodes successfully executed all commands. btullis@cumin1002:~$
The merge request is here: https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/merge_requests/54
TL;DR: I was not able to verify this, as I had missed that this procedure is simply not supported on Spark 3.1. Let's abandon this change. Sorry for the wasted cycles folks.
Longer:
Following similar test steps as we did on T344910#9277895:
Create a new conda env:
conda-analytics-clone conda-analytics-list | grep 11-08 2024-11-08T20.44.22_xcollazo /home/xcollazo/.conda/envs/2024-11-08T20.44.22_xcollazo
Check it has the expected Iceberg version:
$ ls /home/xcollazo/.conda/envs/2024-11-08T20.44.22_xcollazo/lib/python3.10/site-packages/pyspark/jars/ | grep iceberg iceberg-spark-runtime-3.1_2.12-1.3.1.jar
SSH tunnel to have some jupyter fun:
ssh -N an-test-client1002.eqiad.wmnet -L 8880:127.0.0.1:8880
Created an Iceberg table (Jupyter output attached), and attempted to write, and it failed with:
24/11/08 20:59:57 WARN TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1) (an-test-worker1001.eqiad.wmnet executor 1): java.io.InvalidClassException: org.apache.iceberg.spark.source.SparkWrite$WriterFactory; local class incompatible: stream classdesc serialVersionUID = 9204784105757286036, local class serialVersionUID = 7473974864488477747 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:2005) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1852) ...
This is the usual tell tale of class mismatches. IIRC from previous tests, we are likely missing the assembly file. So, let's generate it with a paraphrase of generate_spark_assembly.sh (https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/blob/7cca17c6b0fe8826c5c3fb6122a8919d9b3bc572/generate_spark_assembly.sh):
cd ~/artifacts/ export SPARK_VERSION=3.1.2 export ICEBERG_VERSION=1.3.1 export ASSEMBLY_FILE=spark-$SPARK_VERSION-assembly-with-iceberg-$ICEBERG_VERSION.jar # `jar 0` stands for no compression. Thus the `.jar` extension; not `.zip`. # (The compression gain is minimal) jar cv0f \ $ASSEMBLY_FILE \ -C /home/xcollazo/.conda/envs/2024-11-08T20.44.22_xcollazo/lib/python3.10/site-packages/pyspark/jars/ .
Verify:
jar tvf spark-3.1.2-assembly-with-iceberg-1.3.1.jar | grep iceberg Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 27967291 Tue Nov 05 12:29:40 UTC 2024 iceberg-spark-runtime-3.1_2.12-1.3.1.jar
Upload to HDFS:
hdfs dfs -copyFromLocal spark-3.1.2-assembly-with-iceberg-1.3.1.jar /user/xcollazo/artifacts
Modified the Jupyter notebook to pick this up. Now rerun my test.
Now I hit:
spark.sql("""
CALL spark_catalog.system.rewrite_position_delete_files(
table => 'xcollazo.test_iceberg_1_3_1'
)
""").show(100)
---------------------------------------------------------------------------
AnalysisException Traceback (most recent call last)
....
AnalysisException: Procedure system.rewrite_position_delete_files not foundAt fisrt it seemed like the puppet configuration for the test cluster didn't have @iceberg_enabled = true, so I tried the following:
spark = wmfdata.spark.create_custom_session(
master='yarn',
spark_config={
"spark.yarn.archive": "hdfs:///user/xcollazo/artifacts/spark-3.1.2-assembly-with-iceberg-1.3.1.jar",
"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions", <<<<<<<<
}
)Reran test and failed again.
The real issue is though... that this procedure simply is not there at all. What I missed from my perusal for this rewrite_position_delete_files procedure is that it was first implemented for Spark 3.4 here, then backported to Spark 3.3 here, but that's it. There is no support for Spark 3.1 nor Spark 3.2.
My bad @BTullis. We will not be able to use this until we do a Spark upgrade.
xcollazo closed https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/merge_requests/54
Update iceberg to version 1.3.1