Page MenuHomePhabricator

Quarry results serialization (web) should have its memory usage upper-bounded and streamed
Closed, ResolvedPublic

Description

Right now the a few serialization methods generates the entire file in-memory as a string, then sends the file to the client. For very large query results (discouraged, but we still don't have a solution), this is highly inefficient (memory allocation / copying), and may even cause the server to run out of memory. Ideally the file should be sent on-the-fly (streamed) during serialization.

Event Timeline

Change 460443 had a related patch set uploaded (by Zhuyifei1999; owner: Zhuyifei1999):
[analytics/quarry/web@master] results: Fixes for memory consumption and bytes support

https://gerrit.wikimedia.org/r/460443

Change 460443 merged by jenkins-bot:
[analytics/quarry/web@master] results: Fixes for memory consumption and bytes support

https://gerrit.wikimedia.org/r/460443

Framawiki assigned this task to zhuyifei1999.