Page MenuHomePhabricator

Document Graphite annotation (string events) usage
Open, MediumPublic

Description

For some Interactive team graphs, we need to add grafana annotations. Apparently it supports graphite events, but I couldn't find how to do it.

Event Timeline

Something like this is also needed, with the proper field values:

curl -X POST "http://graphite1001.eqiad.wmnet/events/" -d '{"what": "Some server started.", "tags": "server-start", "data":"Some server started.", "when":1436932353}'

Following up from IRC, I asked @Yurik to open a task re: graphite events because I wasn't aware they were being used. Anyways a little bit of context/background, the graphite events are stored in a local sqlite database which would need to be at least backed up and replicated to codfw. Note this is different from what we're currently doing for annotations with e.g. scap deploys where a statsd metric is pushed instead and grafana used to plot it as lines, that's already replicated to codfw being a simple metric (with its limitations).

Another alternative we were talking about with @EBernhardson is using elasticsearch/logstash for the annotations, which would also require some work wrt setting up/maintaining the index.

You would likely also be interested in T147328 and the patch at https://gerrit.wikimedia.org/r/#/c/322220/ which would allow adding annotations on wiki!