Page MenuHomePhabricator

Upgrade to Pyspark ≥ 3.5
Closed, DuplicatePublic

Event Timeline

BTullis triaged this task as Medium priority.Aug 8 2024, 2:25 PM
BTullis moved this task from Incoming to Software Upgrades on the Data-Platform-SRE board.

Another feature I'd be excited to get in Spark 3.4+ is support for bound parameters: https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.SparkSession.sql.html

Our current practice is to use string format interpolation which is awkward but good enough for trusted data, and totally unsafe for untrusted data.