Page MenuHomePhabricator
Create Task
Maniphest T272714

EventGate should use recent service-runner (^2.8.1) with Prometheus support
Closed, ResolvedPublic
Actions

Description

Currently, EventGate is using service-runner ^2.7.7, emitting metrics via statsd, which in production eventgate-wikimedia is translated to prometheus via promtheus-statsd-exporter. We should do the following:

  • Update to a recent service-runner (^2.8.1) with Prometheus metrics support.
  • Configure eventgate-wikimedia to use Prometheus metrics.
  • Remove prometheus-statsd-exporter usaeg from eventgate helm chart.
  • Make sure Grafana dashboards continue to work, or make new ones if metrics have changed.

This will allow us to do T257237: eventgate-wikimedia should emit metrics about validation errors

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata created this task.Jan 22 2021, 3:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 22 2021, 3:05 PM
Ottomata added a subtask: T257237: eventgate-wikimedia should emit metrics about validation errors.Jan 22 2021, 3:05 PM
• fdans moved this task from Incoming to Event Platform on the Analytics board.Jan 28 2021, 5:36 PM
• fdans triaged this task as Medium priority.Jan 28 2021, 5:38 PM
• fdans closed subtask T257237: eventgate-wikimedia should emit metrics about validation errors as Resolved.Mar 18 2021, 4:02 PM
Ottomata added a subscriber: • Mholloway.Jun 11 2021, 3:36 PM

cc Michael: I know you were looking for some event platform tasks..this one would be really helpful!

Jdforrester-WMF added a subscriber: Jdforrester-WMF.Jun 11 2021, 3:37 PM
Ottomata claimed this task.Jun 21 2021, 3:02 PM
Ottomata added a project: Analytics-Kanban.
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.
gerritbot added a comment.Jun 22 2021, 3:52 PM

Change 700945 had a related patch set uploaded (by Ottomata; author: Ottomata):

[eventgate-wikimedia@master] [WIP] Prometheus support with service-runner 2.8.3

https://gerrit.wikimedia.org/r/700945

gerritbot added a project: Patch-For-Review.Jun 22 2021, 3:52 PM
Ottomata added a subscriber: • Pchelolo.Jun 22 2021, 9:06 PM

Alright, https://gerrit.wikimedia.org/r/c/eventgate-wikimedia/+/700945 is ready for review. @Pchelolo maybe plz? :)

Ottomata added a subscriber: colewhite.Jun 22 2021, 9:07 PM
gerritbot added a comment.Jul 6 2021, 3:30 PM

Change 700945 merged by Ottomata:

[eventgate-wikimedia@master] Prometheus support with service-runner 2.8.3 metrics

https://gerrit.wikimedia.org/r/700945

Maintenance_bot removed a project: Patch-For-Review.Jul 6 2021, 4:10 PM
gerritbot added a comment.Jul 6 2021, 4:23 PM

Change 703463 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate now uses prometheus directly instead of statsd bridge

https://gerrit.wikimedia.org/r/703463

gerritbot added a project: Patch-For-Review.Jul 6 2021, 4:23 PM
gerritbot added a comment.Jul 6 2021, 5:56 PM

Change 703463 merged by Ottomata:

[operations/deployment-charts@master] eventgate now uses prometheus directly instead of statsd bridge

https://gerrit.wikimedia.org/r/703463

Maintenance_bot removed a project: Patch-For-Review.Jul 6 2021, 6:10 PM
gerritbot added a comment.Jul 6 2021, 6:18 PM

Change 703477 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate-analytics/values-staging.yaml - set num_workers: 0

https://gerrit.wikimedia.org/r/703477

gerritbot added a project: Patch-For-Review.Jul 6 2021, 6:18 PM

Change 703477 merged by Ottomata:

[operations/deployment-charts@master] eventgate-analytics/values-staging.yaml - set num_workers: 0

https://gerrit.wikimedia.org/r/703477

Maintenance_bot removed a project: Patch-For-Review.Jul 6 2021, 7:10 PM
gerritbot added a comment.Jul 6 2021, 8:09 PM

Change 703484 had a related patch set uploaded (by Ottomata; author: Ottomata):

[eventgate-wikimedia@master] Manually normalize rdkafka prometheus labels

https://gerrit.wikimedia.org/r/703484

gerritbot added a project: Patch-For-Review.Jul 6 2021, 8:09 PM
gerritbot added a comment.Jul 6 2021, 8:14 PM

Change 703484 merged by Ottomata:

[eventgate-wikimedia@master] Manually normalize rdkafka prometheus labels

https://gerrit.wikimedia.org/r/703484

Maintenance_bot removed a project: Patch-For-Review.Jul 6 2021, 9:10 PM
gerritbot added a comment.Jul 6 2021, 9:29 PM

Change 703487 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] Bump eventgate image version to get normalized prometheus metric labels

https://gerrit.wikimedia.org/r/703487

gerritbot added a project: Patch-For-Review.Jul 6 2021, 9:29 PM
gerritbot added a comment.Jul 7 2021, 1:11 PM

Change 703487 merged by Ottomata:

[operations/deployment-charts@master] Bump eventgate image version to get normalized prometheus metric labels

https://gerrit.wikimedia.org/r/703487

Ottomata added a comment.Jul 12 2021, 2:39 PM

@Pchelolo, @colewhite Q: I'm getting close to getting this working, but I seem to be missing a service-runner metric and I don't know where it came from before. service_runner_request_duration_seconds_count is mapped by promethues-statsd-exporter, but after upgrading to service-runner 2.8.3 with Prometheus I don't see any equivalent http request metrics. I also am having trouble finding any service-runner code where that metric was emitted to statsd previously.

I'd like to keep this metric if possible. What am I missing? Thanks!

Ottomata moved this task from In Progress to In Code Review on the Analytics-Kanban board.Jul 12 2021, 3:05 PM
colewhite added a comment.Jul 12 2021, 3:38 PM

@Ottomata this metric is part of service-template node, but is not yet merged: https://github.com/wikimedia/service-template-node/pull/142

Ottomata added a comment.Jul 12 2021, 5:05 PM

Oh great, thanks Cole. What do we need to do to get that merged and released?

Ottomata added a comment.Jul 12 2021, 5:23 PM

NM, petr answered on PR. Working on it.

gerritbot added a comment.Jul 13 2021, 3:05 PM

Change 704350 had a related patch set uploaded (by Ottomata; author: Ottomata):

[eventgate-wikimedia@master] Prometheus metric label value fixes

https://gerrit.wikimedia.org/r/704350

gerritbot added a comment.Jul 13 2021, 3:10 PM

Change 704350 merged by Ottomata:

[eventgate-wikimedia@master] Prometheus metric label value fixes

https://gerrit.wikimedia.org/r/704350

gerritbot added a comment.Jul 13 2021, 3:14 PM

Change 704353 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate-analytics - bump to 2021-07-13-151027-production

https://gerrit.wikimedia.org/r/704353

gerritbot added a comment.Jul 13 2021, 3:17 PM

Change 704353 merged by Ottomata:

[operations/deployment-charts@master] eventgate-analytics - bump to 2021-07-13-151027-production

https://gerrit.wikimedia.org/r/704353

Ottomata moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Jul 13 2021, 4:07 PM
gerritbot added a comment.Jul 14 2021, 2:44 PM

Change 704548 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Add prometheus precomputed query for express_router_request_duration_seconds

https://gerrit.wikimedia.org/r/704548

gerritbot added a comment.Jul 14 2021, 2:46 PM

Change 704548 merged by Ottomata:

[operations/puppet@production] Add prometheus precomputed query for express_router_request_duration_seconds

https://gerrit.wikimedia.org/r/704548

gerritbot added a comment.Jul 14 2021, 3:18 PM

Change 704554 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Fix express_router_request_duration_seconds precomputed promtheus query

https://gerrit.wikimedia.org/r/704554

gerritbot added a comment.Jul 14 2021, 3:20 PM

Change 704554 merged by Ottomata:

[operations/puppet@production] Fix express_router_request_duration_seconds precomputed promtheus query

https://gerrit.wikimedia.org/r/704554

Ottomata added a project: Sustainability (Incident Followup).Jul 14 2021, 4:51 PM
RhinosF1 added a subscriber: RhinosF1.Jul 14 2021, 5:36 PM
gerritbot added a comment.Jul 14 2021, 6:29 PM

Change 704588 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate-analytics - set num_workers: 0

https://gerrit.wikimedia.org/r/704588

Ottomata added a comment.Jul 14 2021, 8:52 PM

FYI a deploy earlier today caused an MW API outage. Incident report here: https://wikitech.wikimedia.org/wiki/Incident_documentation/2021-07-14_eventgate-analytics_latency_spike_caused_MW_app_server_overload

Ottomata added a comment.Jul 14 2021, 8:55 PM

@colewhite Ok, I need to fork node-rdkafka-prometheus to fix this. Tomorrow I'll make a @wikimedia/node-rdkafka-prometheus package and github repo and apply my patches. I'll ask you to review the PRs if you don't mind!

I'll then adapt eventgate-wikimedia to use @wikimedia/node-rdkafka-prometheus instead of the main one.

gerritbot added a comment.Jul 15 2021, 6:48 PM

Change 704851 had a related patch set uploaded (by Ottomata; author: Ottomata):

[eventgate-wikimedia@master] Use @wikimedia/node-rdkafka-prometheus 1.1.1

https://gerrit.wikimedia.org/r/704851

gerritbot added a comment.Jul 15 2021, 6:52 PM

Change 704851 merged by Ottomata:

[eventgate-wikimedia@master] Use @wikimedia/node-rdkafka-prometheus 1.1.1

https://gerrit.wikimedia.org/r/704851

gerritbot added a comment.Jul 15 2021, 6:52 PM

Change 704588 abandoned by Ottomata:

[operations/deployment-charts@master] eventgate-analytics - set num_workers: 0

Reason:

https://gerrit.wikimedia.org/r/704588

gerritbot added a comment.Jul 15 2021, 7:01 PM

Change 704853 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate-analytics bump to 2021-07-15-185242-production with prom-client fixes

https://gerrit.wikimedia.org/r/704853

gerritbot added a comment.Jul 15 2021, 7:02 PM

Change 704853 merged by Ottomata:

[operations/deployment-charts@master] eventgate-analytics bump to 2021-07-15-185242-production with prom-client fixes

https://gerrit.wikimedia.org/r/704853

Ottomata added a comment.Jul 15 2021, 8:12 PM

^ Done and deployed to eventgate-analytics staging. Looks good.

Will deploy to production the week of July 26th.

gerritbot added a comment.Jul 27 2021, 1:16 PM

Change 708289 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate-analytics - Use canary releases for eqiad and codfw

https://gerrit.wikimedia.org/r/708289

gerritbot added a comment.Jul 27 2021, 1:19 PM

Change 708289 merged by Ottomata:

[operations/deployment-charts@master] eventgate-analytics - Use canary releases for eqiad and codfw

https://gerrit.wikimedia.org/r/708289

gerritbot added a comment.Jul 27 2021, 1:49 PM

Change 708292 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate - use native prometheus in all services

https://gerrit.wikimedia.org/r/708292

gerritbot added a comment.Jul 27 2021, 1:50 PM

Change 708292 merged by Ottomata:

[operations/deployment-charts@master] eventgate - use native prometheus in all services

https://gerrit.wikimedia.org/r/708292

Ottomata added a comment.Jul 27 2021, 2:23 PM

All eventgate clusters deployed, woohoo! This is the new EventGate dashboard.

I'm going to wait a day, the final step will be to remove the old EventGate dashboard.

Ottomata moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Jul 28 2021, 2:10 PM
Ottomata closed this task as Resolved.Jul 29 2021, 4:04 PM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL