IIRC what scap does for deploy events is push a "1" to particular metrics under deploy. in graphite and that's it, grafana picks it up via "Graphite target expression", would that suit?

In T106351#1765756, @fgiunchedi wrote:

IIRC what scap does for deploy events is push a "1" to particular metrics under deploy. in graphite and that's it, grafana picks it up via "Graphite target expression", would that suit?

The docs suggest that an event-based annotation can enrich the hover with event data ("When you hover over an annotation you can get title, tags, and text information for the event."). This didn't seem to work for the one Graphite-based event I created, but if it could be made to work, it would be awesome.

Even for something as simple as a deploy, you could imagine including the change ID, and/or a simple comment.

And, I'd really like to go beyond just deploys, creating annotations for matching SAL entries, Icinga alerts, etc.

TL;DR a deploy counter would be better than nothing, but ideally we'd be able to use events as well

To summarize a discussion with @bd808 on IRC:

• The data exists in an elasticsearch instance in labs
• timestamp, nick, message, and project (prod, releng, etc) are indexed (annotations would search against terms matching message)
• The labs ES instances however, are not reachable outside of labs (Grafana cannot query them)
• Requests proxied by the SAL webapp (https://tools.wmflabs.org/sal/production), are one option

To expand a bit on that last item, Grafana supports ES and Graphite metrics or event queries. The latter is a fairly simple REST interface:

curl -s "http://graphite.wikimedia.org/events/get_data?tags=restbase" | jq
[
  {
    "data": "",
    "what": "rolling deploy of restbase-deploy 3b1f6488f2 to restbase cluster",
    "when": 1446060510,
    "id": 1,
    "tags": "restbase restbase-deploy"
  }
]

So one option that might be possible is to implement an /events/get_data endpoint for SAL, that returns the expected JSON formatted results, and then configure SAL in Grafana as a Graphite datasource.

untagging observability due to staleness