Page MenuHomePhabricator

Add support to compress arclamp logs before downloading
Open, LowPublic

Description

Downloading logs from Arclamp can take a significant amount of time due to the size of the files, in addition to this, users can get rate limited and be unable to download logs.

Relevant IRC logs:

07:14 <Amir1> Hi, I want to download a log from arclamp but maybe I got to ratelimit and it's doesn't progress after 74% (4.88GB out of 6.5GB). One thing is that maybe the logs should be compressed so the download wouldn't be this painful? Even gzip can compress a lot here
07:14 <Amir1> https://performance.wikimedia.org/arclamp/logs/daily/
07:21 <Amir1> https://performance.wikimedia.org/arclamp/logs/daily/2024-05-26.excimer-k8s-wall.all.log <- it's this
07:21 <Amir1> I still can't download it past 74%, after 20 tries

Event Timeline

I suggest just compressing anything that's above some certain threshold (like 1GB)

lmata triaged this task as Low priority.Wed, Jun 5, 2:04 PM
lmata edited projects, added Observability-Logging; removed SRE Observability.
lmata moved this task from Inbox to Backlog on the Observability-Logging board.