Upgrade Presto to access UDF library improvements
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	awight
	Nov 12 2021, 12:35 PM

Project Tags

Referenced Files

None

Subscribers

Aklapper

awight

BTullis

Description

We ran into an unnecessary obstacle when analyzing some MediaWiki API request logs, trying to use the array_frequency function documented in the latest revision. Discovering this function was not available yet, I tried the historgram function and got an error that the maximum memory usage of 40GB had been exceeded. This is surprising because the columns I was trying to aggregate were on a table of 3.5M rows but with only a small, finite (either 2 or c. 900) number of distinct elements, so this built-in histogram must be doing something unscalable behind the scenes.

This isn't a blocker for our work, but I'm filing a task to put one finger on the scale the next time an upgrade is discussed.

Thank you for maintaining Superset and Presto, my team has been loving the experience so far!

Related Objects
Search...

Status	Assigned	Task
Open	None	T333013 [Iceberg Migration] Apache Iceberg Migration
Resolved	BTullis	T337335 Upgrade Presto to release that aligns with Iceberg 1.2.1
Resolved	BTullis	T295589 Upgrade Presto to access UDF library improvements

Event Timeline

awight created this task.Nov 12 2021, 12:35 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 12 2021, 12:35 PM

Now I've also tried approx_most_frequent and it also fails with an out-of-memory error, even when the capacity parameter is dialed down to a 100k rows: P17737

Full disclosure: I was experiencing OOM even after removing the CTE and switching to the simplest aggregation, the crashing can't be blamed on histogram or approx_most_frequent. By process of elimination, I found that histogram is fine for my needs and the problem is actually with this column:

select
  count_if(
    element_at(http.request_headers, 'user-agent') = 'kartotherian (yurik @ wikimedia)'
  ) as is_kartotherian,
  count_if(is_wmf_domain) as is_wmf_domain
from event.mediawiki_api_request
  where year=2021
  and month=11
  and day=1
;

BTullis subscribed.Nov 22 2021, 2:30 PM

• odimitrijevic moved this task from Incoming (new tickets) to Analyze on the Data-Engineering board.Jan 5 2022, 11:39 PM

• odimitrijevic removed a parent task: T294259: Presto/Superset User Experience Improvement.Apr 7 2022, 6:57 PM

JArguello-WMF added a parent task: T337335: Upgrade Presto to release that aligns with Iceberg 1.2.1.May 24 2023, 4:24 PM

Hi @awight - Apologies for the delay in getting around to this. Just FYI we have upgraded the version of presto on the test cluster to 0.281.

If all goes well with our testing, we should have this version rolled out to the prod cluster within the next week or so.

JArguello-WMF moved this task from Analyze to Event Platform Backlog on the Data-Engineering board.Jun 29 2023, 11:24 PM

Those is now done, since we have presto version 0.281 deployed in production.

BTullis closed this task as Resolved.Jul 14 2023, 11:51 PM

BTullis claimed this task.