We currently use conda-analytics to ship our pyspark distribution.
As part of the upgrade to spark in production, we will need to update this environment with pyspark version 3.5.3.
Description
Description
Details
Details
Related Changes in GitLab:
| Title | Reference | Author | Source Branch | Dest Branch | |
|---|---|---|---|---|---|
| Upgrade spark and iceberg | repos/data-engineering/conda-analytics!56 | btullis | upgrade_spark | main |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T333013 [Iceberg Migration] Apache Iceberg Migration | |||
| Open | None | T370710 Upgrade to Numpy ≥ 1.24 in Conda-Analytics | |||
| Open | Antoine_Quhen | T338057 Upgrade to Spark 3.5 | |||
| Resolved | BTullis | T380035 Create Spark docker images for version 3.5.3 | |||
| Resolved | BTullis | T354733 Create Conda Analytics environment including spark version 3.5.3 |
Event Timeline
Comment Actions
btullis updated https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/merge_requests/56
Upgrade spark and iceberg
Comment Actions
We have a debian package of conda-analytics 0.0.37 available here: https://gitlab.wikimedia.org/repos/data-engineering/conda-analytics/-/package_files/5821/download
This includes Spark 3.5.3 and Iceberg 1.7.0 so it's ready for testing.
Comment Actions
Upgrading to Spark 3.5 should allow us to remove the version specs and pins for:
- Pandas (T370705, T370707)
- Numpy (T370710)
- PyArrow (since I believe that its pin is for the current Pandas version)
It's possible that new specs and pins for later versions of these packages will be needed, but the best way to figure that out is to have an analyst test out a pre-release version of the environment.
Comment Actions
Noted. Thanks @nshahquinn-wmf
I'll resolve this ticket, since we have now created a conda-analytics-next environment and deployed this to production.