Page MenuHomePhabricator
Create Task
Maniphest T222826

Leverage Grafana annotations to show events in graphs
Closed, ResolvedPublic
Actions

Description

Investigate possibility of using logging cluster events as grafana annotations (https://grafana.com/docs/reference/annotations/)

Some ideas for possibly useful events:

  • Puppet merges and runs
  • Icinga alerts
  • SAL entries
  • Deploys
  • Downtimes

Details

SubjectRepoBranchLines +/-
provision loki on grafana-nextoperations/puppetproduction+110 -0
hiera: specify tlsproxy configuration for grafanaoperations/puppetproduction+25 -3
service::docker: enhance volume supportoperations/puppetproduction+24 -11
profile: add loki output support to the logstash pipelineoperations/puppetproduction+207 -0
Add logging/sal/1.0.0 schemaschemas/event/secondarymaster+222 -0
Add support for posting events to eventgatelabs/tools/stashbotmaster+21 -0
logstash: duplicate alert logs for loki targetoperations/puppetproduction+119 -1
logstash: enable loki public output on productionoperations/puppetproduction+14 -2
hiera: deploy and enable loki on grafana hostsoperations/puppetproduction+54 -0
profile: make loki data directory configurableoperations/puppetproduction+4 -1
loki-beta: increase grpc message sizeoperations/puppetproduction+6 -0
beta-logs: set loki retention to 3doperations/puppetproduction+4 -0
logstash: add loki output supportoperations/puppetproduction+62 -2
loki: add ferm service to control api accessoperations/puppetproduction+15 -3
beta-logs: add minimal grafana configoperations/puppetproduction+5 -0
loki: add loki as an optional grafana componentoperations/puppetproduction+103 -27
logstash: duplicate scap.announce logs for loki targetoperations/puppetproduction+88 -0
logstash: alertmanager use logsource as source for host.name fieldoperations/puppetproduction+6 -1
P:rsyslog: ship puppetmaster logs to kafkaoperations/puppetproduction+2 -1
puppetmaster: enable logstash reportsoperations/puppetproduction+6 -0
P:puppetmaster::common: Add back logstash supportoperations/puppetproduction+12 -37
puppetmaster: drop log messages from logstash reporteroperations/puppetproduction+81 -50
schemas - metrics: Add puppet keys to the metrics name spaceoperations/software/ecsmaster+56 -0
git - schema: Add new schema for adding git informationoperations/software/ecsmaster+51 -1
puppet_agent_stats: add catalog version to prom metricsoperations/puppetproduction+9 -1
debianizationoperations/debs/prometheus-es-exporterdebian/sid+376 -0
jjb: volume mount for logstash must be absolute pathintegration/configmaster+1 -1
Add filter_scripts volume mount to logstash-filter-verifier jobintegration/configmaster+7 -0
profile: add loki_event filter scriptoperations/puppetproduction+32 -0
add loki 1.5.0operations/docker-images/production-imagesmaster+98 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 8 2019, 5:11 PM
colewhite triaged this task as Medium priority.May 8 2019, 5:12 PM
colewhite lowered the priority of this task from Medium to Low.
colewhite added a subtask: T174172: unused grafana-dashboard indices on elasticsearch / logstash.
Volans mentioned this in T223934: Add annotations from ops vendor maintenance calendar to Grafana.Jan 29 2020, 2:59 PM
fgiunchedi moved this task from Inbox to Backlog on the observability board.Apr 6 2020, 12:41 PM
colewhite added a subtask: T223934: Add annotations from ops vendor maintenance calendar to Grafana.May 15 2020, 4:03 PM
colewhite added a comment.May 15 2020, 4:11 PM

Loki looks like a feasible option to try given the resource constraints on the Grafana VM. It appears there is headroom on the host long as we keep events reasonably low-traffic.

colewhite claimed this task.May 15 2020, 4:11 PM
gerritbot added a comment.May 19 2020, 5:45 PM

Change 597317 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/docker-images/production-images@master] add loki 1.4.1

https://gerrit.wikimedia.org/r/597317

gerritbot added a project: Patch-For-Review.May 19 2020, 5:45 PM
akosiaris subscribed.May 20 2020, 1:57 PM
akosiaris added a comment.May 20 2020, 2:09 PM

For what is worth and for kubernetes deploys specifically, we have in grafana an annotation that is working most of the times, but can easily fail us. It's a simple

resets((sum(service_runner_request_duration_seconds_count{service="$service"}))[1m:]) > bool 0

It has at least 2 drawbacks I 've identified in the short time I 've been using it:

  • It is service-runner specific (and in a very specific configuration)
  • Problems with counters being reset multiple times, e.g. when a deploy contains multiple pods that are restarted in timeframes > 1m then 1 deploy == multiple annotations. Which is not ideal.
  • The inverse of the above where if a counter has been reset multiple times in 1m (which is the polling interval of prometheus), we don't catch the deploys either.

An approach I have been thinking about was, since helmfile supports hooks, to have a hook emit a statsd line to a local prometheus-statsd-exporter and then scrape that from prometheus and use it as an annotation. The fact that it is statsd is just an implementation detail of course, it's just the easy to try and do and already tried thing. Since we can run arbitrary commands in that hook [1] (albeit not always having in the environment all the info we would like) we can use other methods of sending that information

[1] https://github.com/roboll/helmfile#hooks

JMeybohm subscribed.May 20 2020, 2:40 PM

Downside of using helmfile hooks would be that we catch the trigger, not the actual event. So deploys triggered by rollbacks for example would not be recognized. We should maybe ask the kubernetes API, it should know best. :-)

There is a "kube-state-metrics" which exposes a lot of metrics about the state of various API objects. See https://github.com/kubernetes/kube-state-metrics/blob/master/docs/deployment-metrics.md for example.

gerritbot added a comment.Jun 1 2020, 11:03 PM

Change 597317 merged by Cwhite:
[operations/docker-images/production-images@master] add loki 1.5.0

https://gerrit.wikimedia.org/r/597317

gerritbot added a comment.Jun 4 2020, 8:43 PM

Change 602490 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add loki output support to the logstash pipeline

https://gerrit.wikimedia.org/r/602490

gerritbot added a comment.Jun 5 2020, 4:13 PM

Change 602729 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] profile: add loki_event filter script

https://gerrit.wikimedia.org/r/602729

gerritbot added a comment.Jun 5 2020, 4:15 PM

Change 602730 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[integration/config@master] add filter_scripts volume mount to logstash-filter-verifier job

https://gerrit.wikimedia.org/r/602730

fgiunchedi moved this task from Backlog to In progress on the observability board.Jun 8 2020, 2:16 PM
gerritbot added a comment.Jun 8 2020, 5:13 PM

Change 602729 merged by Cwhite:
[operations/puppet@production] profile: add loki_event filter script

https://gerrit.wikimedia.org/r/602729

gerritbot added a comment.Jun 12 2020, 11:34 PM

Change 605343 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] service::docker: enhance volume support

https://gerrit.wikimedia.org/r/605343

colewhite added a subtask: T257226: Please create operations/debs/grafana-loki gerrit repository.Jul 6 2020, 4:40 PM
gerritbot added a comment.Jul 7 2020, 5:21 PM

Change 602730 merged by jenkins-bot:
[integration/config@master] Add filter_scripts volume mount to logstash-filter-verifier job

https://gerrit.wikimedia.org/r/602730

gerritbot added a comment.Jul 7 2020, 5:29 PM

Change 610119 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] jjb: volume mount for logstash must be absolute path

https://gerrit.wikimedia.org/r/610119

gerritbot added a comment.Jul 7 2020, 5:32 PM

Change 610119 merged by jenkins-bot:
[integration/config@master] jjb: volume mount for logstash must be absolute path

https://gerrit.wikimedia.org/r/610119

akosiaris closed subtask T257226: Please create operations/debs/grafana-loki gerrit repository as Resolved.Jul 8 2020, 10:44 AM
colewhite mentioned this in T257861: Pipe SAL entries into Logstash.Jul 13 2020, 5:18 PM
colewhite added a subtask: T257861: Pipe SAL entries into Logstash.
gerritbot added a comment.Jul 28 2020, 1:59 PM

Change 616811 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] hiera: specify tlsproxy configuration for grafana

https://gerrit.wikimedia.org/r/616811

gerritbot added a comment.Jul 28 2020, 3:21 PM

Change 616851 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/puppet@production] provision loki on grafana-next

https://gerrit.wikimedia.org/r/616851

gerritbot added a comment.Jul 29 2020, 7:33 PM

Change 617250 had a related patch set uploaded (by Cwhite; owner: Cwhite):
[operations/debs/prometheus-es-exporter@debian/sid] debianization

https://gerrit.wikimedia.org/r/617250

gerritbot added a comment.Aug 3 2020, 10:25 PM

Change 617250 merged by Cwhite:
[operations/debs/prometheus-es-exporter@debian/sid] debianization

https://gerrit.wikimedia.org/r/617250

JMeybohm mentioned this in T264625: Deploy kube-state-metrics.Oct 5 2020, 2:39 PM
lmata moved this task from In progress to Epics In Progress on the observability board.Jun 14 2021, 3:44 PM
lmata edited projects, added SRE Observability (FY2021/2022-Q1); removed observability.Jul 12 2021, 2:41 AM
lmata moved this task from Inbox to Epics In Progress on the SRE Observability (FY2021/2022-Q1) board.
gerritbot added a comment.Sep 6 2021, 10:19 AM

Change 719056 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] puppet_agent_stats: add catalog version to prom metricts

https://gerrit.wikimedia.org/r/719056

jbond subscribed.Sep 7 2021, 11:08 AM

In relation to puppet i think we could look again at creating a puppet logstash report. This was never pushed to production do to concerns about sending the full puppet catalogue diff to logstash. however i think we should be able to ensure we only send meta data and not the actual diffs

gerritbot added a comment.Sep 7 2021, 1:10 PM

Change 719056 merged by Jbond:

[operations/puppet@production] puppet_agent_stats: add catalog version to prom metrics

https://gerrit.wikimedia.org/r/719056

gerritbot added a comment.Sep 8 2021, 9:55 AM

Change 719372 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] P:puppetmaster::common: Add back logstash support

https://gerrit.wikimedia.org/r/719372

gerritbot added a comment.Sep 8 2021, 12:37 PM

Change 719368 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] puppetmaster: drop log messages from logstash reporter

https://gerrit.wikimedia.org/r/719368

gerritbot added a comment.Sep 21 2021, 10:37 AM

Change 722580 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/software/ecs@master] git - schema: Add new schema for adding git information

https://gerrit.wikimedia.org/r/722580

gerritbot added a comment.Sep 22 2021, 2:29 PM

Change 722873 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/software/ecs@master] schemas - metrics: Add puppet keys to the metrics name space

https://gerrit.wikimedia.org/r/722873

colewhite edited projects, added Observability-Logging; removed SRE Observability (FY2021/2022-Q1).Oct 1 2021, 12:19 AM
gerritbot added a comment.Oct 5 2021, 2:09 AM

Change 722580 merged by jenkins-bot:

[operations/software/ecs@master] git - schema: Add new schema for adding git information

https://gerrit.wikimedia.org/r/722580

jbond mentioned this in rOSECb548ed08163d: git - schema: Add new schema for adding git information.Oct 5 2021, 3:15 AM
gerritbot added a comment.Oct 12 2021, 11:00 PM

Change 722873 merged by jenkins-bot:

[operations/software/ecs@master] schemas - metrics: Add puppet keys to the metrics name space

https://gerrit.wikimedia.org/r/722873

jbond mentioned this in rOSEC2dde2a50528e: schemas - metrics: Add puppet keys to the metrics name space.Oct 12 2021, 11:02 PM
gerritbot added a comment.Oct 27 2021, 11:40 AM

Change 719368 merged by Jbond:

[operations/puppet@production] puppetmaster: drop log messages from logstash reporter

https://gerrit.wikimedia.org/r/719368

gerritbot added a comment.Oct 27 2021, 12:03 PM

Change 719372 merged by Jbond:

[operations/puppet@production] P:puppetmaster::common: Add back logstash support

https://gerrit.wikimedia.org/r/719372

gerritbot added a comment.Oct 27 2021, 1:07 PM

Change 734961 had a related patch set uploaded (by Jbond; author: John Bond):

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

gerritbot added a comment.Oct 27 2021, 1:17 PM

Change 734961 merged by Jbond:

[operations/puppet@production] puppetmaster: enable logstash reports

https://gerrit.wikimedia.org/r/734961

gerritbot added a comment.Nov 2 2021, 2:15 PM

Change 736233 had a related patch set uploaded (by Jbond; author: jbond):

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

gerritbot added a comment.Nov 4 2021, 5:00 PM

Change 736233 merged by Jbond:

[operations/puppet@production] P:rsyslog: ship puppetmaster logs to kafka

https://gerrit.wikimedia.org/r/736233

colewhite closed subtask T174172: unused grafana-dashboard indices on elasticsearch / logstash as Resolved.Feb 16 2022, 6:33 PM
gerritbot added a comment.Jun 10 2022, 12:01 AM

Change 804484 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: ship scap.announce channel to loki

https://gerrit.wikimedia.org/r/804484

gerritbot added a comment.Jun 16 2022, 10:33 PM

Change 806349 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: duplicate alert logs for loki target

https://gerrit.wikimedia.org/r/806349

gerritbot added a comment.Jun 17 2022, 2:47 PM

Change 806430 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: alertmanager use logsource as source for host.name field

https://gerrit.wikimedia.org/r/806430

gerritbot added a comment.Jun 21 2022, 8:28 PM

Change 806430 merged by Cwhite:

[operations/puppet@production] logstash: alertmanager use logsource as source for host.name field

https://gerrit.wikimedia.org/r/806430

gerritbot added a comment.Jun 28 2022, 10:01 PM

Change 809302 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] loki: add loki as an optional grafana component

https://gerrit.wikimedia.org/r/809302

gerritbot added a comment.Jun 28 2022, 10:13 PM

Change 804484 merged by Cwhite:

[operations/puppet@production] logstash: duplicate scap.announce logs for loki target

https://gerrit.wikimedia.org/r/804484

gerritbot added a comment.Jun 29 2022, 8:42 PM

Change 809302 merged by Cwhite:

[operations/puppet@production] loki: add loki as an optional grafana component

https://gerrit.wikimedia.org/r/809302

gerritbot added a comment.Jun 29 2022, 9:30 PM

Change 809706 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] beta-logs: add minimal grafana config

https://gerrit.wikimedia.org/r/809706

gerritbot added a comment.Jun 29 2022, 9:58 PM

Change 809706 merged by Cwhite:

[operations/puppet@production] beta-logs: add minimal grafana config

https://gerrit.wikimedia.org/r/809706

gerritbot added a comment.Jun 29 2022, 10:02 PM

Change 809709 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] loki: add ferm rule to control api access

https://gerrit.wikimedia.org/r/809709

gerritbot added a comment.Jun 30 2022, 12:02 AM

Change 809722 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add loki output support

https://gerrit.wikimedia.org/r/809722

gerritbot added a comment.Jun 30 2022, 3:09 PM

Change 809709 merged by Cwhite:

[operations/puppet@production] loki: add ferm service to control api access

https://gerrit.wikimedia.org/r/809709

gerritbot added a comment.Jun 30 2022, 4:36 PM

Change 809722 merged by Cwhite:

[operations/puppet@production] logstash: add loki output support

https://gerrit.wikimedia.org/r/809722

gerritbot added a comment.Jun 30 2022, 5:50 PM

Change 810064 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] beta-logs: set loki retention to 3d

https://gerrit.wikimedia.org/r/810064

gerritbot added a comment.Jun 30 2022, 5:52 PM

Change 810064 merged by Cwhite:

[operations/puppet@production] beta-logs: set loki retention to 3d

https://gerrit.wikimedia.org/r/810064

gerritbot added a comment.Jun 30 2022, 9:57 PM

Change 810110 had a related patch set uploaded (by Cwhite; author: Cwhite):

[labs/tools/stashbot@master] Add support for posting events to eventgate

https://gerrit.wikimedia.org/r/810110

gerritbot added a comment.Jun 30 2022, 10:10 PM

Change 810115 had a related patch set uploaded (by Cwhite; author: Cwhite):

[schemas/event/secondary@master] Add logging/sal/1.0.0 schema

https://gerrit.wikimedia.org/r/810115

gerritbot added a comment.Jul 13 2022, 11:01 PM

Change 813715 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: make loki data directory configurable

https://gerrit.wikimedia.org/r/813715

gerritbot added a comment.Jul 13 2022, 11:51 PM

Change 813724 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] hiera: deploy and enable loki on grafana hosts

https://gerrit.wikimedia.org/r/813724

gerritbot added a comment.Jul 14 2022, 10:55 PM

Change 813985 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] loki-beta: increase grpc message size

https://gerrit.wikimedia.org/r/813985

gerritbot added a comment.Jul 15 2022, 4:37 PM

Change 813985 merged by Cwhite:

[operations/puppet@production] loki-beta: increase grpc message size

https://gerrit.wikimedia.org/r/813985

gerritbot added a comment.Jul 18 2022, 8:00 PM

Change 813715 merged by Cwhite:

[operations/puppet@production] profile: make loki data directory configurable

https://gerrit.wikimedia.org/r/813715

gerritbot added a comment.Jul 18 2022, 10:32 PM

Change 813724 merged by Cwhite:

[operations/puppet@production] hiera: deploy and enable loki on grafana hosts

https://gerrit.wikimedia.org/r/813724

gerritbot added a comment.Jul 18 2022, 11:18 PM

Change 814915 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: enable loki public output on production

https://gerrit.wikimedia.org/r/814915

gerritbot added a comment.Jul 19 2022, 4:49 PM

Change 814915 merged by Cwhite:

[operations/puppet@production] logstash: enable loki public output on production

https://gerrit.wikimedia.org/r/814915

colewhite moved this task from Inbox to Prioritized on the Observability-Logging board.Jul 20 2022, 9:11 PM
colewhite changed the task status from Open to In Progress.Aug 5 2022, 10:38 PM
colewhite added a subscriber: thcipriani.

We've enabled the Public Logs datasource in Grafana and forwarded scap.announce logs to it.

colewhite changed the status of subtask T257861: Pipe SAL entries into Logstash from Open to In Progress.Aug 11 2022, 4:17 PM
gerritbot added a comment.Aug 16 2022, 4:31 PM

Change 806349 merged by Cwhite:

[operations/puppet@production] logstash: duplicate alert logs for loki target

https://gerrit.wikimedia.org/r/806349

colewhite closed subtask T257861: Pipe SAL entries into Logstash as Resolved.Aug 24 2022, 4:27 PM
gerritbot added a comment.Aug 24 2022, 4:33 PM

Change 810110 abandoned by Cwhite:

[labs/tools/stashbot@master] Add support for posting events to eventgate

Reason:

https://gerrit.wikimedia.org/r/810110

gerritbot added a comment.Aug 24 2022, 4:33 PM

Change 810115 abandoned by Cwhite:

[schemas/event/secondary@master] Add logging/sal/1.0.0 schema

Reason:

https://gerrit.wikimedia.org/r/810115

colewhite removed a subtask: T223934: Add annotations from ops vendor maintenance calendar to Grafana.Sep 21 2022, 10:42 AM
colewhite closed this task as Resolved.Sep 21 2022, 10:49 AM

MVP achieved. Further iterations and features should come in separately.

gerritbot added a comment.Feb 17 2023, 12:00 AM

Change 602490 abandoned by Cwhite:

[operations/puppet@production] profile: add loki output support to the logstash pipeline

Reason:

in favor of using the loki output plugin

https://gerrit.wikimedia.org/r/602490

gerritbot added a comment.Feb 17 2023, 12:03 AM

Change 605343 abandoned by Cwhite:

[operations/puppet@production] service::docker: enhance volume support

Reason:

we packaged loki in a deb package instead

https://gerrit.wikimedia.org/r/605343

gerritbot added a comment.Apr 14 2023, 9:37 PM

Change 616811 abandoned by Cwhite:

[operations/puppet@production] hiera: specify tlsproxy configuration for grafana

Reason:

https://gerrit.wikimedia.org/r/616811

gerritbot added a comment.Apr 14 2023, 9:39 PM

Change 616851 abandoned by Cwhite:

[operations/puppet@production] provision loki on grafana-next

Reason:

https://gerrit.wikimedia.org/r/616851

Maintenance_bot removed a project: Patch-For-Review.Apr 14 2023, 10:10 PM
herron added a subtask: T350825: Loki: add a channel(s) for git commits.Nov 8 2023, 7:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL