The new Grafana 4.6 gives us built in support for annotations: A Grafana API that we want to use to send annotations from our WebPageTest so we easily can correlate graphs and HTML result pages.
4.6 also has another feature we want to use: Create annotations directly through the GUI (super helpful to add links to Phabricator etc when we have a regression).
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Peter | T175708 Add annotations per URL tested in WebPageTest | |||
| Resolved | fgiunchedi | T180428 Upgrade to latest Grafana 4.6 |
I personally don't think that using the native annotations available in Grafana 4.6 from an automated system (WebPageTest in this case) is a good idea in our setup, given that it's currently using SQLite as its own database. @fgiunchedi thoughts?
Grafana already supports to query annotations to add to the graphs from many different backends that scale much better than sqlite for a potentially large amount of annotations. I've added them to ElasticSearch in previous setups (not in WMF) for example.
If the concern is sqlite scalability I don't foresee that being a problem even for automatic annotations unless grafana does really inefficient queries (to be verified of course).
I'm fine with annotations stored in grafana's sqlite, with the understanding that retrieval of annotations is best effort :) Meaning that e.g. we do only daily backups of the sqlite database, grafana is hosted on a single machine, etc. I think for our purposes that's going to be Good Enough (tm).
Mentioned in SAL (#wikimedia-operations) [2017-11-14T12:57:37Z] <godog> upgrade grafana to 4.6.1 on https://grafana-labs.wikimedia.org/ - T180428
@fgiunchedi yes I'm worried about scalability in terms of transactions write rate (if the automated tools will add many annotations in a short period of time and growth of the SQLite database in size, given that is the same that contains all the dashboards metadata.
The automatic annotations from WebPageTest we could purge them like every two weeks if it's an issue, we very very rarely need to go back longer in time. The annotations that we create manually we need to keep.
I've been running the same setup for 7 months (sending an annotation for each tested URL every hour) and that generated 69 mb of data for the sqllite Graphite database, but I haven't looked how different Graphite/Grafana stores the data (only that the share the same API).
Mentioned in SAL (#wikimedia-operations) [2017-11-16T09:19:28Z] <godog> upgrade grafana to 4.6.1 on https://grafana.wikimedia.org/ - T180428
The scope of this task is done, but let's followup on T175708: Add annotations per URL tested in WebPageTest on @Volans' concerns re: sqlite scalability and automatic annotations
Thanks @fgiunchedi . We will not start with the WebPageTest task yet before we have solution that we all agree on.