While running a couple example queries for @JEbe-WMF, we found that with the following resources:
spark3-sql \
--master yarn \
--executor-memory 16G \
--executor-cores 4 \
--driver-memory 32G \
--driver-cores 4 \
--conf spark.dynamicAllocation.maxExecutors=256 \
--conf spark.sql.shuffle.partitions=10048 \
--conf spark.driver.maxResultSize=8G \It takes 2.6 hours to compute a count! Only 1.2 hours are spent computing the result. Thus, 1.4 hours are being spent in query planning! (It generated 298056 Spark tasks):
... spark-sql (default)> select count(1) as count from wmf_dumps.wikitext_raw_rc1; count 6324909273 Time taken: 9369.366 seconds, Fetched 1 row(s)
Other simpler counts that do not touch all the metadata finish successfully in a reasonable time (25min processing, few seconds of query planning, 7717 tasks):
spark-sql (default)> select count(1) as count from wmf_dumps.wikitext_raw_rc1 where wiki_db = 'eswiki'; count 142466014 Time taken: 1553.839 seconds, Fetched 1 row(s)
I speculate this is a limitation of doing query planning in the driver, as Iceberg does.
In this task we should:
- Figure if the speculation is correct, perhaps by doing some profiling. Or perhaps there is some compaction needed?
-
Figure if the recent Iceberg 1.4.0 release solves this issue, as they now allow distributed query planning.No need, the issue is not Iceberg.