Page MenuHomePhabricator
Create Task
Maniphest T277857

Proton metrics broken
Closed, ResolvedPublic
Actions

Description

I wanted to deploy a small change to the chromium-render helm chart today and stumbled upon a service dashboard which looks pretty broken: https://grafana-rw.wikimedia.org/d/llIEd7MMz/proton

This seems to correlate with the deployment at https://sal.toolforge.org/log/IpEeZHcBpU87LSFJhPIC which probably is https://gerrit.wikimedia.org/r/c/mediawiki/services/chromium-render/+/558213 (and that's why this is a subtask of T205870).

I'm not completely sure what the change does/should do, but I guess it broke quite a bunch of metrics. Or those are no longer exposed via statds but natively. Which would be nice, but in that case statsd should probably be disabled completely and we should start scraping the native metrics port.

Details

SubjectRepoBranchLines +/-
Convert gauges to histograms for compatibilitymediawiki/services/chromium-rendermaster+9 -6
Migrate missing metrics to prometheus formatmediawiki/services/chromium-rendermaster+55 -5
Bump helm chart version for chromium-renderoperations/deployment-chartsmaster+1 -1
Fix broken prometheus metricsmediawiki/services/chromium-rendermaster+5 -5
Configure prometheus metrics for chromium-rendereroperations/deployment-chartsmaster+2 -85
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT228380 Tech debt: sunsetting of Graphite
 OpenNoneT205870 Fully migrate producers off statsd
 ResolvedJgiannelosT277857 Proton metrics broken

Event Timeline

JMeybohm created this task.Mar 19 2021, 10:28 AM
Restricted Application added a project: Product-Infrastructure-Team-Backlog-Deprecated. · View Herald TranscriptMar 19 2021, 10:28 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Jgiannelos subscribed.Mar 19 2021, 10:36 AM

I think this is the patch that introduced the change from statsd metrics to native prometheus:
https://gerrit.wikimedia.org/r/c/mediawiki/services/chromium-render/+/558213

gerritbot added a comment.Mar 19 2021, 10:47 AM

Change 673454 had a related patch set uploaded (by Jgiannelos; owner: Jgiannelos):
[operations/deployment-charts@master] Configure prometheus metrics for chromium-renderer

https://gerrit.wikimedia.org/r/673454

gerritbot added a project: Patch-For-Review.Mar 19 2021, 10:47 AM
gerritbot added a comment.Mar 22 2021, 10:31 AM

Change 673454 merged by jenkins-bot:
[operations/deployment-charts@master] Configure prometheus metrics for chromium-renderer

https://gerrit.wikimedia.org/r/673454

Maintenance_bot removed a project: Patch-For-Review.Mar 22 2021, 11:10 AM
MSantos assigned this task to Jgiannelos.Mar 23 2021, 11:54 AM
MSantos triaged this task as High priority.
MSantos moved this task from Needs triage to Kanban on the Product-Infrastructure-Team-Backlog-Deprecated board.
MSantos edited projects, added Product-Infrastructure-Team-Backlog-Deprecated (Kanban); removed Product-Infrastructure-Team-Backlog-Deprecated.
MSantos moved this task from To Do to To Deploy on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.
akosiaris mentioned this in T278141: cxserver missing important metrics after service-runner 2.8.1 upgrade.Mar 24 2021, 12:17 PM
MSantos moved this task from To Deploy to Sign off on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Mar 25 2021, 5:17 PM
MSantos moved this task from Sign off to To Do on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Mar 25 2021, 5:31 PM

@Jgiannelos I'm moving this back to the to-do column because the changes didn't seem to have any effect.

Jgiannelos added a comment.Mar 25 2021, 5:52 PM

Maybe that's relevant:
(from the logs)

endTiming() unsupported for metric type Gauge
gerritbot added a comment.Mar 25 2021, 6:04 PM

Change 674941 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):
[mediawiki/services/chromium-render@master] Fix broken prometheus metrics

https://gerrit.wikimedia.org/r/674941

gerritbot added a project: Patch-For-Review.Mar 25 2021, 6:04 PM
gerritbot added a comment.Mar 26 2021, 3:28 PM

Change 674941 merged by jenkins-bot:
[mediawiki/services/chromium-render@master] Fix broken prometheus metrics

https://gerrit.wikimedia.org/r/674941

Maintenance_bot removed a project: Patch-For-Review.Mar 26 2021, 4:10 PM
Jgiannelos mentioned this in rMSCRdd2bb44514d1: Fix broken prometheus metrics.Mar 26 2021, 4:12 PM
MSantos moved this task from To Do to Sign off on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Mar 30 2021, 5:08 PM
Jgiannelos added a comment.Apr 5 2021, 9:55 AM

Metrics still not working. Things i discovered while debugging:

  • Patch that removes statsd exporter is already landed and shows up in deployment node
  • Manually cURLing the pod on prometheus port returns statsd exporter related stuff rather than native prometheus metrics
curl POD_IP:9102
...
<head><title>StatsD Exporter</title></head>
...
  • Manually describing the pod still shows that production-metrics-exporter container is deployed and running
  • Even though pods are freshly deployed (~last time helm was deployed) the k8s deployment object is old (older than last time helm was deployed)
  • Helmfile diff doesn't show anything to be deployed

I guess somehow the old metrics exporter is still running (without any statsd metrics to export) and prometheus cannot scrape the native prometheus metrics from proton service.
Can we somehow force a new deployment?

akosiaris added a comment.Apr 5 2021, 10:08 AM
In T277857#6971837, @Jgiannelos wrote:

Metrics still not working. Things i discovered while debugging:

  • Patch that removes statsd exporter is already landed and shows up in deployment node
  • Manually cURLing the pod on prometheus port returns statsd exporter related stuff rather than native prometheus metrics
curl POD_IP:9102
...
<head><title>StatsD Exporter</title></head>
...
  • Manually describing the pod still shows that production-metrics-exporter container is deployed and running
  • Even though pods are freshly deployed (~last time helm was deployed) the k8s deployment object is old (older than last time helm was deployed)

That's expected, the deployment objects isn't being recreated on every deploy, it only gets updated, so the CreationTimestamp stays the same.

  • Helmfile diff doesn't show anything to be deployed

I guess somehow the old metrics exporter is still running (without any statsd metrics to export) and prometheus cannot scrape the native prometheus metrics from proton service.
Can we somehow force a new deployment?

The helm chart version wasn't bumped after 5ce71288eb656dc606700a5e75f782919bda47e8 was merged. Bump the version in Chart.yaml, merge it, and try to deploy again.

gerritbot added a comment.Apr 5 2021, 10:28 AM

Change 676869 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[operations/deployment-charts@master] Bump helm chart version for chromium-render

https://gerrit.wikimedia.org/r/676869

gerritbot added a project: Patch-For-Review.Apr 5 2021, 10:28 AM
gerritbot added a comment.Apr 5 2021, 3:13 PM

Change 676869 merged by jenkins-bot:

[operations/deployment-charts@master] Bump helm chart version for chromium-render

https://gerrit.wikimedia.org/r/676869

Maintenance_bot removed a project: Patch-For-Review.Apr 5 2021, 4:10 PM
Jgiannelos moved this task from Sign off to Code Review on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Apr 6 2021, 2:40 PM
Jgiannelos moved this task from Code Review to To Deploy on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.
Jgiannelos added a comment.Apr 8 2021, 10:24 AM

It looks like native prometheus metrics are now exposed in the service. That said we may still need to adapt the grafana dashboard because the metrics names might have changed.

Jgiannelos moved this task from To Deploy to Sign off on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Apr 8 2021, 10:25 AM
gerritbot added a comment.Apr 8 2021, 2:31 PM

Change 677919 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[mediawiki/services/chromium-render@master] Migrate missing metrics to prometheus format

https://gerrit.wikimedia.org/r/677919

gerritbot added a project: Patch-For-Review.Apr 8 2021, 2:31 PM
Jgiannelos moved this task from Sign off to Code Review on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Apr 8 2021, 2:45 PM
gerritbot added a comment.Apr 8 2021, 3:08 PM

Change 677919 merged by jenkins-bot:

[mediawiki/services/chromium-render@master] Migrate missing metrics to prometheus format

https://gerrit.wikimedia.org/r/677919

Jgiannelos mentioned this in rMSCR3920b24b44e6: Migrate missing metrics to prometheus format.Apr 8 2021, 3:10 PM
Jgiannelos moved this task from Code Review to To Deploy on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Apr 8 2021, 3:22 PM
MSantos moved this task from To Deploy to Sign off on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Apr 27 2021, 5:29 PM
gerritbot added a comment.May 4 2021, 1:00 PM

Change 684896 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[mediawiki/services/chromium-render@master] Convert gauges to histograms for compatibility

https://gerrit.wikimedia.org/r/684896

gerritbot added a comment.May 4 2021, 1:58 PM

Change 684896 merged by jenkins-bot:

[mediawiki/services/chromium-render@master] Convert gauges to histograms for compatibility

https://gerrit.wikimedia.org/r/684896

Jgiannelos mentioned this in rMSCRc42dd9dc3642: Convert gauges to histograms for compatibility.May 4 2021, 2:10 PM
Jgiannelos added a comment.Jul 2 2021, 1:41 PM

Chromium-render is now completely moved on native prometheus metrics using service runner.
There were many incompatibilities any many broken dashboards because of metrics/labels/queries that changed over time.

I tried to fix the dashboards but in order to not break the format I started from a copy.
Here is a version with working dashboards: https://grafana-rw.wikimedia.org/d/U4TuF-lMk/proton-tmp-jgiannelos

Maybe we can just review and then export and re-import to the original proton grafana dashboard

MSantos awarded a token.Jul 2 2021, 3:02 PM
MSantos added a comment.Aug 17 2021, 2:51 PM

@Jgiannelos the board looks good for me. I think we should change the primary one with these changes.

akosiaris added a comment.Aug 23 2021, 2:53 PM
In T277857#7288616, @MSantos wrote:

@Jgiannelos the board looks good for me. I think we should change the primary one with these changes.

I just did so since the old one was entirely devoid of data by now. I also fixed a couple of minor issues like the queue depth but otherwise this looks pretty good. Thanks @Jgiannelos. I 'll boldly resolve this, feel free to reopen though.

akosiaris closed this task as Resolved.Aug 23 2021, 2:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits