We are looking at tracking lots of metrics about Wikidata at a daily scale.
We would also like to be able to track them for longer periods of time.
Graphite has customizable retention, (currently for a year at most)
As we are tracking less data we can retain it for longer without having a big impact on disk usage!
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| Retain daily.* graphite metrics for longer (25y) | operations/puppet | production | +6 -0 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Addshore | T117201 [WD] Implement usage KPI | |||
| Resolved | Addshore | T117405 Add entity usage graphs to wikidata dashboard | |||
| Resolved | Addshore | T117403 Track daily entity usage data in graphite for all wikidata clients | |||
| Resolved | Addshore | T117402 Enable retention of daily metrics for longer periods of time in Graphite |
Change 247866 had a related patch set uploaded (by Addshore):
Retain daily.* graphite metrics for longer
for long term data warehousing or analytics type of workflows using ourhadoop/analytics infrastructure will be more appropriate I think. graphite is more focused on operational metrics from applications, services and so on
After a short discussion I'm going to look at creating a small wrapper for storing the metrics in sql tables / hadoop / flat files insead of graphite.
I'll leave this ticket open until we have a concrete path forward / solution! :)
After many discussions in many places we have decided to try and push forward with this config change and start storing data in this graphite instance.
Per discussions revealing there isn't really any guarantee of persistence in this graphite instance we will simply backup all data we add here.
We will also look at a better solution on the analytics cluster in the longer term
Change 247866 merged by Filippo Giunchedi:
Retain daily.* graphite metrics for longer (25y)