Page MenuHomePhabricator
Create Task
Maniphest T244260

Investigate seemingly broken https://grafana.wikimedia.org/d/u5wAugyik/wikibase-formattercachedashboard
Closed, ResolvedPublic
Actions

Description

https://grafana.wikimedia.org/d/u5wAugyik/wikibase-formattercache
seems to report the same number of misses and hits?
Which might be unexpected?

We want to make sure that this cache is being correctly monitored.
We want to be able to look at this when production has issues with memcached.

Observation: dashboards started to look odd around mid December 2019.

Expected outcome:
Dashboard shows the correct data in three existing panels.

Possible things to check:

  • check how metrics (data points for the dashboard) are generated
  • check the dashboard setting (e.g. if the correct metrics - data points - are plotted) and report here/fix the obvious faults.

To define further:

  • how the data used by Grafana is being collected (i.e. which components populate the data)

Technical Notes:

Details

SubjectRepoBranchLines +/-
Add metric for recording cache hits in StatsdRecordingSimpleCachemediawiki/extensions/Wikibasewmf/1.35.0-wmf.20+102 -49
Add metric for recording cache hits in StatsdRecordingSimpleCachemediawiki/extensions/Wikibasemaster+112 -53
Customize query in gerrit

Event Timeline

Addshore created this task.Feb 4 2020, 5:11 PM
Restricted Application added a project: Wikidata. · View Herald TranscriptFeb 4 2020, 5:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Addshore triaged this task as Medium priority.Feb 4 2020, 5:12 PM
Addshore moved this task from Incoming to Prioritized Wikidata Tech Backlog (prioritised from top to bottom) on the Wikidata-Campsite board.
Addshore updated the task description. (Show Details)
WMDE-leszek updated the task description. (Show Details)Feb 18 2020, 1:27 PM
WMDE-leszek updated the task description. (Show Details)Feb 18 2020, 1:29 PM
WMDE-leszek updated the task description. (Show Details)
WMDE-leszek updated the task description. (Show Details)Feb 18 2020, 1:32 PM
WMDE-leszek moved this task from Prioritized Wikidata Tech Backlog (prioritised from top to bottom) to Wikidata-Campsite-Iteration-∞ (On Hold) on the Wikidata-Campsite board.
WMDE-leszek edited projects, added Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)); removed Wikidata-Campsite.
Jakob_WMDE updated the task description. (Show Details)Feb 18 2020, 1:34 PM
Maintenance_bot moved this task from incoming to in progress on the Wikidata board.Feb 18 2020, 2:15 PM
Rosalie_WMDE moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 18 2020, 5:46 PM
Addshore assigned this task to rosalieper.Feb 19 2020, 10:27 AM
Addshore reassigned this task from rosalieper to Rosalie_WMDE.
Addshore added a subscriber: rosalieper.
Rosalie_WMDE added a comment.Feb 19 2020, 4:18 PM

After a few hours looking at it, this is what I could gather.

From grafana:

  • The possible break on the graph happens on the 18th/12/2019
  • between 12:00 and 14:00 on 18/12/2019, the hit counts are negative
  • Also, I checked and It seems to me the dashboard settings are fine.

Form code:

Rosalie_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 20 2020, 9:00 AM
Rosalie_WMDE moved this task from Peer Review to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 20 2020, 11:33 AM
gerritbot added a comment.Feb 20 2020, 3:05 PM

Change 573582 had a related patch set uploaded (by Rosalie Perside (WMDE); owner: Rosalie Perside (WMDE)):
[mediawiki/extensions/Wikibase@master] [WIP]Fix brocken formatterCache dashboard

https://gerrit.wikimedia.org/r/573582

gerritbot added a project: Patch-For-Review.Feb 20 2020, 3:05 PM
Rosalie_WMDE moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 20 2020, 4:13 PM
Addshore moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Feb 21 2020, 7:42 AM
gerritbot added a comment.Feb 21 2020, 8:01 AM

Change 573582 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add metric for recording cache hits in StatsdRecordingSimpleCache

https://gerrit.wikimedia.org/r/573582

Maintenance_bot removed a project: Patch-For-Review.Feb 21 2020, 8:10 AM
ReleaseTaggerBot added a project: MW-1.35-notes (1.35.0-wmf.21; 2020-02-25).Feb 21 2020, 9:00 AM
gerritbot added a comment.Feb 24 2020, 9:12 AM

Change 574386 had a related patch set uploaded (by Ladsgroup; owner: Rosalie Perside (WMDE)):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.20] Add metric for recording cache hits in StatsdRecordingSimpleCache

https://gerrit.wikimedia.org/r/574386

gerritbot added a project: Patch-For-Review.Feb 24 2020, 9:12 AM
gerritbot added a comment.Feb 24 2020, 10:30 AM

Change 574386 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.20] Add metric for recording cache hits in StatsdRecordingSimpleCache

https://gerrit.wikimedia.org/r/574386

Stashbot added a comment.Feb 24 2020, 10:41 AM

Mentioned in SAL (#wikimedia-operations) [2020-02-24T10:41:14Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.20/extensions/Wikibase: [[gerrit:574386|Add metric for recording cache hits in StatsdRecordingSimpleCache]] (T244260) (duration: 01m 04s)

ReleaseTaggerBot edited projects, added MW-1.35-notes (1.35.0-wmf.20; 2020-02-18); removed MW-1.35-notes (1.35.0-wmf.21; 2020-02-25).Feb 24 2020, 11:00 AM
Maintenance_bot removed a project: Patch-For-Review.Feb 24 2020, 11:11 AM
Addshore closed this task as Resolved.Mar 4 2020, 4:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL