Page MenuHomePhabricator

Increase retention of ArcLamp to 2 years by moving to Swift
Open, MediumPublic

Description

[..] Xenon daily data is currently rotated after 90 days, and Xenon hourly data rotated after 14 days. [..] See also T166624.

I'd like to increase retention of profiles, and keep hourly data for 90 days, and daily data for 1-2 years. [..] that would require:

  • daily: 22 GB per 90d = 179 GB per 2 years.
  • hourly: 4 GB per 14d = 26 GB per 90 days.

Due to limited storage on the webperf-2 VMs, this will be blocked on adding an upload mechanism to ArcLamp to support Swift.

Details at T199853#4437177.

Event Timeline

Krinkle renamed this task from Increase retention of ArcLamp to 2 years to Increase retention of ArcLamp to 2 years by moving to Swift.Jun 22 2020, 8:08 PM

Removing T227026 as strict dependency. Probably makes sense to first deploy the non-puppet ArcLamp that uses Swift to the existing webperf1002 first, and then move to k8s from there later as part of T227026.

Main reason being to unlock the 90 day retention sooner (T235455), and indeed the 2y retention (this ticket).

Krinkle triaged this task as Medium priority.Jul 17 2020, 12:09 AM