Right now the a few serialization methods generates the entire file in-memory as a string, then sends the file to the client. For very large query results (discouraged, but we still don't have a solution), this is highly inefficient (memory allocation / copying), and may even cause the server to run out of memory. Ideally the file should be sent on-the-fly (streamed) during serialization.
Description
Description
Details
Details
Subject | Repo | Branch | Lines +/- | |
---|---|---|---|---|
results: Fixes for memory consumption and bytes support | analytics/quarry/web | master | +146 -90 |
Status | Subtype | Assigned | Task | ||
---|---|---|---|---|---|
Resolved | zhuyifei1999 | T192696 Quarry results serialization (web) should have its memory usage upper-bounded and streamed | |||
Resolved | Framawiki | T192698 Upgrade Quarry to python 3 |
Event Timeline
Comment Actions
Change 460443 had a related patch set uploaded (by Zhuyifei1999; owner: Zhuyifei1999):
[analytics/quarry/web@master] results: Fixes for memory consumption and bytes support
Comment Actions
Change 460443 merged by jenkins-bot:
[analytics/quarry/web@master] results: Fixes for memory consumption and bytes support