Page MenuHomePhabricator

Let Arc-Lamp store its trace "log" files in compressed format
Closed, ResolvedPublic

Description

This would fit nicely with the upcoming refactor of Arc Lamp (see T200109, T200108).

This task is a follow-up to the webperf*002 hardware alert from T235425.

During the alert response we (accidentally) found that compressing the oldest 30 days of the 45 days of data in arclamp/logs/daily, reduced it from 75G to 45G. (We normally store 90 days worth there, but I already deleted the older half before measuring).

Supporting this natively (in a way that doesn't break our SVG generator/garbage-collector) would buy us some time and might allow us to keep the current retention span at least until traffic grows and/or util/if we migrate XHGui's db to the same server (T180761).

See also T235455: Resolve arclamp disk exhaustion problem (Oct 2019).

Sub tasks:

  • Support compressed log files in internal housekeeping and file readers readers (arclamp-grep, arclamp-svg, and log gargage collection).
  • Serve compresssed files on the web server in a way that browsers and most programmatic clients will still naturally decode. Or only compress fiiles that are more than a few days old
  • Regularly compress log files (perhaps only for "finished", older, files).

Event Timeline

aaron removed aaron as the assignee of this task.Feb 18 2020, 8:42 PM
aaron added a subscriber: dpifke.
aaron added a subscriber: aaron.

@dpifke Do you want to take this on since you're working in this area?

Yup, makes sense to do in concert with T227026 and T244776.

Change 613740 had a related patch set uploaded (by Dave Pifke; owner: Dave Pifke):
[performance/arc-lamp@master] Support reading compressed logs

https://gerrit.wikimedia.org/r/613740

Change 615904 had a related patch set uploaded (by Dave Pifke; owner: Dave Pifke):
[performance/arc-lamp@master] Add script to compress logs more than 7 days old

https://gerrit.wikimedia.org/r/615904

Change 613740 merged by jenkins-bot:
[performance/arc-lamp@master] Support reading gzip'd logs

https://gerrit.wikimedia.org/r/613740

Change 616179 had a related patch set uploaded (by Dave Pifke; owner: Dave Pifke):
[operations/puppet@production] [WIP] arclamp: run arclamp-compress-logs

https://gerrit.wikimedia.org/r/616179

Change 615904 merged by jenkins-bot:
[performance/arc-lamp@master] Add script to compress older data

https://gerrit.wikimedia.org/r/615904

Mentioned in SAL (#wikimedia-operations) [2020-07-27T17:14:07Z] <dpifke@deploy1001> Started deploy [performance/arc-lamp@f14888b]: Deploying arclamp-compress-logs (T235456)

Mentioned in SAL (#wikimedia-operations) [2020-07-27T17:14:13Z] <dpifke@deploy1001> Finished deploy [performance/arc-lamp@f14888b]: Deploying arclamp-compress-logs (T235456) (duration: 00m 05s)

Change 616179 merged by Dzahn:
[operations/puppet@production] arclamp: run arclamp-compress-logs from cron

https://gerrit.wikimedia.org/r/616179

Change 616634 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] arclamp: send stderr of new arclamp compress cron job to a logfile

https://gerrit.wikimedia.org/r/616634

Change 616635 had a related patch set uploaded (by Dave Pifke; owner: Dave Pifke):
[operations/puppet@production] arclamp: fix cron specification

https://gerrit.wikimedia.org/r/616635

Change 616635 merged by Dzahn:
[operations/puppet@production] arclamp: fix cron specification

https://gerrit.wikimedia.org/r/616635

Change 616634 abandoned by Dzahn:
[operations/puppet@production] arclamp: send stderr of new arclamp compress cron job to a logfile

Reason:
https://gerrit.wikimedia.org/r/c/operations/puppet/ /616635

https://gerrit.wikimedia.org/r/616634