Page MenuHomePhabricator

Investigate seemingly broken https://grafana.wikimedia.org/d/u5wAugyik/wikibase-formattercachedashboard
Closed, ResolvedPublic

Description

https://grafana.wikimedia.org/d/u5wAugyik/wikibase-formattercache
seems to report the same number of misses and hits?
Which might be unexpected?

We want to make sure that this cache is being correctly monitored.
We want to be able to look at this when production has issues with memcached.

Observation: dashboards started to look odd around mid December 2019.

Expected outcome:
Dashboard shows the correct data in three existing panels.

Possible things to check:

  • check how metrics (data points for the dashboard) are generated
  • check the dashboard setting (e.g. if the correct metrics - data points - are plotted) and report here/fix the obvious faults.

To define further:

  • how the data used by Grafana is being collected (i.e. which components populate the data)

Technical Notes:

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Addshore triaged this task as Medium priority.Feb 4 2020, 5:12 PM
Addshore updated the task description. (Show Details)
Addshore reassigned this task from rosalieper to Rosalie_WMDE.
Addshore added a subscriber: rosalieper.

After a few hours looking at it, this is what I could gather.

From grafana:

  • The possible break on the graph happens on the 18th/12/2019
  • between 12:00 and 14:00 on 18/12/2019, the hit counts are negative
  • Also, I checked and It seems to me the dashboard settings are fine.

Form code:

Change 573582 had a related patch set uploaded (by Rosalie Perside (WMDE); owner: Rosalie Perside (WMDE)):
[mediawiki/extensions/Wikibase@master] [WIP]Fix brocken formatterCache dashboard

https://gerrit.wikimedia.org/r/573582

Change 573582 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Add metric for recording cache hits in StatsdRecordingSimpleCache

https://gerrit.wikimedia.org/r/573582

Change 574386 had a related patch set uploaded (by Ladsgroup; owner: Rosalie Perside (WMDE)):
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.20] Add metric for recording cache hits in StatsdRecordingSimpleCache

https://gerrit.wikimedia.org/r/574386

Change 574386 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@wmf/1.35.0-wmf.20] Add metric for recording cache hits in StatsdRecordingSimpleCache

https://gerrit.wikimedia.org/r/574386

Mentioned in SAL (#wikimedia-operations) [2020-02-24T10:41:14Z] <ladsgroup@deploy1001> Synchronized php-1.35.0-wmf.20/extensions/Wikibase: [[gerrit:574386|Add metric for recording cache hits in StatsdRecordingSimpleCache]] (T244260) (duration: 01m 04s)