In this task we want to:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T333013 [Iceberg Migration] Apache Iceberg Migration | |||
| Open | None | T370710 Upgrade to Numpy ≥ 1.24 in Conda-Analytics | |||
| Open | Antoine_Quhen | T338057 Upgrade to Spark 3.5 | |||
| Resolved | BTullis | T380417 Test if an existing conda environment with Spark 3.1.2 clients works fine with Spark 3.5.3 |
Create new conda environment with Spark 3.1.2, verify it is indeed the right Spark:
ssh an-test-client1002.eqiad.wmnet conda-analytics-clone test-env-for-spark-3.1.2-on-spark-3.5.3 source conda-analytics-activate test-env-for-spark-3.1.2-on-spark-3.5.3 conda list | grep spark # packages in environment at /home/xcollazo/.conda/envs/test-env-for-spark-3.1.2-on-spark-3.5.3: findspark 2.0.1 pypi_0 pypi pyspark 3.1.2 pyh6c4a22f_0 conda-forge
Now run a simple notebook to verify it works:
ssh -N an-test-client1002.eqiad.wmnet -L 8880:127.0.0.1:8880 Make sure we stop and start server with the test-env-for-spark-3.1.2-on-spark-3.5.3 environment ....
I was not able to setup a successful test due to an Iceberg library mismatch in the test cluster. On a notebook with the new environment:
spark.sparkContext.getConf().get("spark.yarn.archive")
'hdfs:///user/spark/share/lib/spark-3.1.2-assembly.jar'
!jar tf /mnt/hdfs/user/spark/share/lib/spark-3.1.2-assembly.jar | grep iceberg
iceberg-spark-runtime-3.1_2.12-1.2.1.jarBut perusing the environment we find:
!ls ~/.conda/envs/test-env-for-spark-3.1.2-on-spark-3.5.3/lib/python3.10/site-packages/pyspark/jars | grep iceberg iceberg-spark-runtime-3.1_2.12-1.3.1.jar
Note one is Iceberg 1.2.1, as expected, while the other is 1.3.1, which was not expected. I think this is a vestige of T378819, were we attempted to upgrade Iceberg but failed.
We need to fix this before moving forward by upgrading/downgrading conda analytics to a stable version without Iceberg 1.3.1.
Over at conda-analytics, I don't see anything in the changelog that suggest we officially upgraded Iceberg to 1.3.1, so perhaps this test cluster behavir was a manual change?
CC @BTullis
I added a commit to the airflow-dags pyspark upgrade MR here: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/commit/a30add30d519697c59431d940985f135a8586f3c
This is to allow overriding the selection of the spark 3.1.2 shuffler and assembly file and iceberg jar, when version 3.5.3 is made the default.
It will also depend on this change to the shuffler version port numbers: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1093394
...plaus the same change on the prod cluster.
Moving back to the longer-term backlog, since we are not yet able to proceed with the Spark upgrade.
brouberol closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/926
Upgrade pyspark to 3.5.3
We have decided to use a conda-analytics-next environment as a way to allow us to have a production and staging versions of spark available.
As such, these environments are quite isolated from each other and I think that we can close this ticket.
I have verified that the following works:
btullis@stat1008:~$ conda-analytics-next-clone mycoolsparkenv
btullis@stat1008:~$ source conda-analytics-next-activate mycoolsparkenv
mycoolsparkenv) btullis@stat1008:~$ spark35-pyspark
SPARK_HOME: /opt/conda-analytics-next/lib/python3.10/site-packages/pyspark
Using Hadoop client lib jars at hadoop-client-api-3.3.4.jar
hadoop-client-runtime-3.3.4.jar, provided by Spark.
PYSPARK_DRIVER_PYTHON=/srv/home/btullis/.conda/envs/mycoolsparkenv/bin/ipython3
PYSPARK_PYTHON=/srv/home/btullis/.conda/envs/mycoolsparkenv/bin/python3
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
Python 3.10.20 | packaged by conda-forge | (main, Mar 5 2026, 16:42:22) [GCC 14.3.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.37.0 -- An enhanced Interactive Python. Type '?' for help.
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
26/05/18 11:04:40 WARN SparkConf: Note that spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone/kubernetes and LOCAL_DIRS in YARN).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 3.5.8
/_/
Using Python version 3.10.20 (main, Mar 5 2026 16:42:22)
Spark context Web UI available at http://stat1008.eqiad.wmnet:4044
Spark context available as 'sc' (master = yarn, app id = application_1778834239566_49167).
SparkSession available as 'spark'.
In [2]: import wmfdata as wmf
In [3]: countries = wmf.spark.run("""
...: SELECT *
...: FROM canonical_data.countries
...: """)
In [4]: countries
Out[4]:
name iso_code iso_alpha3_code wikidata_id is_protected ... is_un_data_entity is_imf_data_entity is_world_bank_data_entity is_penn_world_table_data_entity market_research_classification
0 Afghanistan AF AFG Q889 True ... True True True False Build
1 Åland AX ALA Q5689 False ... True False False False
2 Albania AL ALB Q222 False ... True True True True Create
3 Algeria DZ DZA Q262 False ... True True True True Expand
4 American Samoa AS ASM Q16641 False ... True False True False
.. ... ... ... ... ... ... ... ... ... ... ...
245 Wallis and Futuna WF WLF Q35555 False ... True False False False
246 Western Sahara EH ESH Q6250 False ... True False False False
247 Yemen YE YEM Q805 True ... True True True True Expand
248 Zambia ZM ZMB Q953 False ... True True True True Expand
249 Zimbabwe ZW ZWE Q954 False ... True True True True Expand
[250 rows x 23 columns]