A few months ago I was wondering why all our Graphite metrics from before July 2016 are missing. Investigation got me nowhere.
Now I'm wondering why our Graphite metrics don't go back beyond November 2016.
Long boring investigation later: Because that's what our Graphite retention rules (used to) specify:
1m:7d,5m:14d,15m:30d,1h:1y
This means 1 minute for the last 7 days, etc. and 1 hour for the last year. And beyond that, nada! For some reason I assumed the configuration somehow specified retention for the previous period and therefore the beyond the last one there is some kind of default, but this means there is actually no unlimited retention by default.
Fortunately, over a year ago this was increased to 5 years in 01d26c2c16e9cbab7c6de1625b705d4ab7ec7c33.
1m:7d,5m:14d,15m:30d,1h:1y,1d:5y
Unfortunately, Graphite hardcodes retention configuration in the Whisper files for individual metrics and there is no logic by default to update retention rules for existing metrics so we need to run some whisper command on each of the metrics we care about to make sure we stop deleting data from last year. We're losing stuff every day now.
Checklist
Using the find sillypipe from T179622#4058076 to verify:
| Path | Status regarding retention config | Comment |
|---|---|---|
| /var/lib/carbon/whisper/ResourceLoader | OK | Fixed. |
| /var/lib/carbon/whisper/frontend | OK | Fixed. |
| /var/lib/carbon/whisper/webpagetest | OK | Fixed, and deleted some stuff. – T179622#4058076 |
| /var/lib/carbon/whisper/browsertime | OK | (Was already fine.) |
| /var/lib/carbon/whisper/mw/ | OK | Fixed and deleted some stuff – T179622#4077863 |
| /var/lib/carbon/whisper/performance/ | OK | (Was already fine.) |
| /var/lib/carbon/whisper/ve/ | OK | Fixed. |
| /var/lib/carbon/whisper/VisualEditor/ | OK | Fixed. |