Maniphest T345712

Make sure we have observability for otel-coll and jaeger
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Sep 6 2023, 9:58 AM

Tags

Referenced Files

None

Subscribers

Description

As per title, we should make sure we have metrics from otel-coll and jaeger collected by Prometheus. And the respective grafana dashboards.

jaeger

dashboard https://grafana.wikimedia.org/d/zLOi95xmk/jaeger adapted from https://grafana.com/grafana/dashboards/10001-jaeger/
alerts https://gerrit.wikimedia.org/r/c/operations/alerts/+/959950
- Found an issue with alerts, reported upstream at https://github.com/jaegertracing/jaeger/issues/4771 (fixed)

otel-coll

metrics https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/960056
- Some metrics collected, not all expected by upstream dashboard below
dashboard https://grafana.wikimedia.org/d/SPebYW7Iz/opentelemetry-collector
alerts https://gerrit.wikimedia.org/r/c/operations/alerts/+/967143

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	sre: ignore pint promql/series checks for otel-coll	operations/alerts	master	+4 -0
	sre: first iteration for otel-coll alerts	operations/alerts	master	+94 -0
	otel-coll: bump resource limits	operations/deployment-charts	master	+4 -0
	otel-coll: enable prometheus scraping	operations/deployment-charts	master	+8 -0
	sre: add jaeger query/collector alerts	operations/alerts	master	+87 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Open		None	T340551 distributed tracing epic
		Open		None	T320549 distributed tracing v0 [minimum viable]
		Resolved		fgiunchedi	T345712 Make sure we have observability for otel-coll and jaeger

Event Timeline

fgiunchedi created this task.Sep 6 2023, 9:58 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 6 2023, 9:58 AM

fgiunchedi updated the task description. (Show Details)Sep 6 2023, 9:58 AM

fgiunchedi updated the task description. (Show Details)Sep 21 2023, 8:53 AM

Change 959950 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] sre: add jaeger query/collector alerts

https://gerrit.wikimedia.org/r/959950

gerritbot added a project: Patch-For-Review.Sep 22 2023, 8:32 AM

fgiunchedi updated the task description. (Show Details)Sep 22 2023, 8:40 AM

Change 960056 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/deployment-charts@master] otel-coll: enable prometheus scraping

https://gerrit.wikimedia.org/r/960056

fgiunchedi updated the task description. (Show Details)Sep 22 2023, 1:15 PM

fgiunchedi added a project: User-fgiunchedi.Sep 25 2023, 9:25 AM

Change 959950 merged by Filippo Giunchedi:

[operations/alerts@master] sre: add jaeger query/collector alerts

https://gerrit.wikimedia.org/r/959950

Change 960056 merged by Filippo Giunchedi:

[operations/deployment-charts@master] otel-coll: enable prometheus scraping

https://gerrit.wikimedia.org/r/960056

fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.Sep 25 2023, 2:52 PM

Maintenance_bot removed a project: Patch-For-Review.Sep 25 2023, 3:11 PM

• lmata moved this task from Inbox to Backlog on the Observability-Tracing board.Sep 25 2023, 5:28 PM

fgiunchedi updated the task description. (Show Details)Oct 13 2023, 1:32 PM

Change 966514 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/deployment-charts@master] otel-coll: bump resource limits

https://gerrit.wikimedia.org/r/966514

gerritbot added a project: Patch-For-Review.Oct 17 2023, 11:22 AM

Change 966514 merged by Filippo Giunchedi:

[operations/deployment-charts@master] otel-coll: bump resource limits

https://gerrit.wikimedia.org/r/966514

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2023, 7:30 AM

fgiunchedi updated the task description. (Show Details)Oct 18 2023, 8:03 AM

fgiunchedi updated the task description. (Show Details)Oct 18 2023, 8:10 AM

fgiunchedi updated the task description. (Show Details)

After bumping the memory limits the collector is able to stay running and metrics/dashboard are working as expected!

Change 967143 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] sre: first iteration for otel-coll alerts

https://gerrit.wikimedia.org/r/967143

gerritbot added a project: Patch-For-Review.Oct 19 2023, 9:09 AM

fgiunchedi updated the task description. (Show Details)Oct 19 2023, 11:25 AM

Change 967143 merged by Filippo Giunchedi:

[operations/alerts@master] sre: first iteration for otel-coll alerts

https://gerrit.wikimedia.org/r/967143

Maintenance_bot removed a project: Patch-For-Review.Oct 30 2023, 8:10 AM

With otel collector alerts in place I'm calling this done!

Change 969703 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/alerts@master] sre: ignore pint promql/series checks for otel-coll

https://gerrit.wikimedia.org/r/969703

gerritbot added a project: Patch-For-Review.Oct 30 2023, 11:33 AM

Change 969703 merged by Filippo Giunchedi:

[operations/alerts@master] sre: ignore pint promql/series checks for otel-coll

https://gerrit.wikimedia.org/r/969703

Maintenance_bot removed a project: Patch-For-Review.Oct 30 2023, 12:10 PM