Page MenuHomePhabricator
Create Task
Maniphest T281243

Expose CPU benchmark data to Prometheus/Grafana
Closed, ResolvedPublic
Actions

Description

This requires:

  • Adding request context to the CPUBenchmark schema
  • Populating that context in the NavigationTiming extension
  • Consuming that context in the navtiming daemon
  • Exposing facetted data in the navtiming daemon
  • Creating a Grafana dashboard consuming the new data

Details

SubjectRepoBranchLines +/-
Reduce cardinality of CpuBenchmark metricsperformance/navtimingmaster+28 -16
Fix CpuBenchmark KeyError on missing batteryLevelperformance/navtimingmaster+10 -1
Fix CpuBenchmark unhandled exceptionperformance/navtimingmaster+6 -0
Send CpuBenchmark data to Prometheusperformance/navtimingmaster+51 -9
Add request context to CpuBenchmark eventmediawiki/extensions/NavigationTimingmaster+103 -62
Add request context fields to CpuBenchmark schemaschemas/event/secondarymaster+417 -4
Customize query in gerrit

Related Objects

Event Timeline

• Gilles created this task.Apr 27 2021, 12:25 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 27 2021, 12:25 PM
Krinkle added a project: NavigationTiming.Apr 27 2021, 10:03 PM
• Gilles moved this task from Inbox, needs triage to To-do: Goals, prioritized next 4 Quarters on the Performance-Team board.Apr 29 2021, 7:04 AM
• Gilles updated the task description. (Show Details)May 20 2021, 1:54 PM
gerritbot added a comment.May 20 2021, 2:06 PM

Change 693163 had a related patch set uploaded (by Gilles; author: Gilles):

[schemas/event/secondary@master] Add request context fields to CpuBenchmark schema

https://gerrit.wikimedia.org/r/693163

gerritbot added a project: Patch-For-Review.May 20 2021, 2:06 PM
• Gilles triaged this task as Medium priority.May 20 2021, 2:06 PM
• Gilles moved this task from To-do: Goals, prioritized next 4 Quarters to Doing: Goals on the Performance-Team board.
gerritbot added a comment.May 21 2021, 7:20 AM

Change 693163 merged by jenkins-bot:

[schemas/event/secondary@master] Add request context fields to CpuBenchmark schema

https://gerrit.wikimedia.org/r/693163

Maintenance_bot removed a project: Patch-For-Review.May 21 2021, 8:10 AM
gerritbot added a comment.May 21 2021, 12:38 PM

Change 693415 had a related patch set uploaded (by Gilles; author: Gilles):

[mediawiki/extensions/NavigationTiming@master] Add request context to CpuBenchmark event

https://gerrit.wikimedia.org/r/693415

gerritbot added a project: Patch-For-Review.May 21 2021, 12:38 PM
gerritbot added a comment.May 21 2021, 1:37 PM

Change 693423 had a related patch set uploaded (by Gilles; author: Gilles):

[performance/navtiming@master] Send CpuBenchmark data to Prometheus

https://gerrit.wikimedia.org/r/693423

gerritbot added a comment.Aug 4 2021, 6:35 PM

Change 693415 merged by jenkins-bot:

[mediawiki/extensions/NavigationTiming@master] Add request context to CpuBenchmark event

https://gerrit.wikimedia.org/r/693415

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.18; 2021-08-09).Aug 4 2021, 7:00 PM
gerritbot added a comment.Aug 4 2021, 7:25 PM

Change 693423 merged by jenkins-bot:

[performance/navtiming@master] Send CpuBenchmark data to Prometheus

https://gerrit.wikimedia.org/r/693423

Krinkle removed • Gilles as the assignee of this task.Aug 9 2021, 6:57 PM
Krinkle moved this task from Doing: Goals to Inbox, needs triage on the Performance-Team board.
Krinkle removed projects: MW-1.37-notes (1.37.0-wmf.18; 2021-08-09), Patch-For-Review.Aug 10 2021, 6:57 PM
• dpifke claimed this task.Aug 10 2021, 7:07 PM
• dpifke moved this task from Inbox, needs triage to Doing: Prio Interrupt on the Performance-Team board.
• dpifke subscribed.

This should go out tomorrow (Tuesday). I'll verify all the dependencies are deployed, and keep an eye on the rollout of this.

• dpifke added a comment.Aug 11 2021, 1:40 AM

There was an open question (at least in my mind) as to whether or not https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/693163 had been deployed, or if we needed to ask the Analytics folks to do something.

I've verified it's live at https://schema.wikimedia.org/#!//secondary/jsonschema/analytics/legacy/cpubenchmark.

Useful background reading: https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate#EventGate_clusters

• dpifke added a comment.Aug 11 2021, 1:54 AM

Looks like a followup patch to https://gerrit.wikimedia.org/r/c/performance/navtiming/+/693423 is needed; I've reverted it for now.

gerritbot added a comment.Aug 12 2021, 11:18 PM

Change 712725 had a related patch set uploaded (by Dave Pifke; author: Dave Pifke):

[performance/navtiming@master] Fix CpuBenchmark unhandled exception

https://gerrit.wikimedia.org/r/712725

gerritbot added a project: Patch-For-Review.Aug 12 2021, 11:18 PM
gerritbot added a comment.Aug 18 2021, 12:08 AM

Change 712725 merged by jenkins-bot:

[performance/navtiming@master] Fix CpuBenchmark unhandled exception

https://gerrit.wikimedia.org/r/712725

Stashbot added a comment.Aug 18 2021, 12:38 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-18T00:38:06Z] <dpifke@deploy1002> Started deploy [performance/navtiming@88f12a0]: Re-deploy fixed CpuBenchmark (T281243)

Stashbot added a comment.Aug 18 2021, 12:38 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-18T00:38:15Z] <dpifke@deploy1002> Finished deploy [performance/navtiming@88f12a0]: Re-deploy fixed CpuBenchmark (T281243) (duration: 00m 06s)

Stashbot added a comment.Aug 18 2021, 12:39 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-18T00:39:13Z] <dpifke@deploy1002> Started deploy [performance/navtiming@88f12a0]: Revert CpuBenchmark again (T281243)

Stashbot added a comment.Aug 18 2021, 12:39 AM

Mentioned in SAL (#wikimedia-operations) [2021-08-18T00:39:21Z] <dpifke@deploy1002> Finished deploy [performance/navtiming@88f12a0]: Revert CpuBenchmark again (T281243) (duration: 00m 05s)

• dpifke added a comment.Aug 18 2021, 12:40 AM

Sigh. New issue:

Aug 18 00:39:15 webperf2001 navtiming[12027]: 2021-08-18 00:39:15,462 [ERROR] (run:899) Unhandled exception in main loop, restarting consumer
Aug 18 00:39:15 webperf2001 navtiming[12027]: Traceback (most recent call last):
Aug 18 00:39:15 webperf2001 navtiming[12027]:   File "/srv/deployment/performance/navtiming-cache/revs/88f12a07957a223a2d6805c290ce97a6471cbd6b/navtiming/__init__.py", line 891, in run
Aug 18 00:39:15 webperf2001 navtiming[12027]:     for stat in f(meta):
Aug 18 00:39:15 webperf2001 navtiming[12027]:   File "/srv/deployment/performance/navtiming-cache/revs/88f12a07957a223a2d6805c290ce97a6471cbd6b/navtiming/__init__.py", line 599, in handle_cpu_benchmark
Aug 18 00:39:15 webperf2001 navtiming[12027]:     bucketed_battery_level = str(int(round(event['batteryLevel'] * 10) * 10))
Aug 18 00:39:15 webperf2001 navtiming[12027]: KeyError: 'batteryLevel'
gerritbot added a comment.Aug 19 2021, 12:28 AM

Change 713723 had a related patch set uploaded (by Dave Pifke; author: Dave Pifke):

[performance/navtiming@master] Fix CpuBenchmark KeyError

https://gerrit.wikimedia.org/r/713723

gerritbot added a comment.Aug 19 2021, 3:34 PM

Change 713723 merged by jenkins-bot:

[performance/navtiming@master] Fix CpuBenchmark KeyError on missing batteryLevel

https://gerrit.wikimedia.org/r/713723

Stashbot added a comment.Aug 19 2021, 3:52 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-19T15:52:56Z] <dpifke@deploy1002> Started deploy [performance/navtiming@f8bf39f]: Deploy CpuBenchmark processor again T281243

Stashbot added a comment.Aug 19 2021, 3:53 PM

Mentioned in SAL (#wikimedia-operations) [2021-08-19T15:53:05Z] <dpifke@deploy1002> Finished deploy [performance/navtiming@f8bf39f]: Deploy CpuBenchmark processor again T281243 (duration: 00m 06s)

• dpifke closed this task as Resolved.Aug 19 2021, 9:33 PM

Quick and dirty dashboard I threw together to verify the data is coming in: https://grafana.wikimedia.org/d/cFMjrb7nz/cpu-benchmark?orgId=1

gerritbot added a comment.Sep 1 2021, 9:41 PM

Change 716045 had a related patch set uploaded (by Dave Pifke; author: Dave Pifke):

[performance/navtiming@master] Reduce cardinality of CpuBenchmark metrics

https://gerrit.wikimedia.org/r/716045

gerritbot added a comment.Sep 1 2021, 10:54 PM

Change 716045 merged by jenkins-bot:

[performance/navtiming@master] Reduce cardinality of CpuBenchmark metrics

https://gerrit.wikimedia.org/r/716045

Stashbot added a comment.Sep 1 2021, 11:04 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-01T23:04:13Z] <dpifke@deploy1002> Started deploy [performance/navtiming@63c9d31]: Deploy fix for CpuBenchmark-related Prometheus timeouts T281243

Stashbot added a comment.Sep 1 2021, 11:04 PM

Mentioned in SAL (#wikimedia-operations) [2021-09-01T23:04:19Z] <dpifke@deploy1002> Finished deploy [performance/navtiming@63c9d31]: Deploy fix for CpuBenchmark-related Prometheus timeouts T281243 (duration: 00m 06s)

Krinkle mentioned this in T290497: Publish CPU benchmark data set with buckets batteryLevel.Sep 7 2021, 4:26 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL