Move wikidata lag checks off Icinga
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	fgiunchedi
	Aug 31 2021, 2:53 PM

Description

This is a followup in the context of migrating wikidata alerts to AlertManager (T287741). The remaining Icinga checks are mostly about wikidata lag, namely fetching https://www.wikidata.org//w/api.php?action=query&meta=siteinfo&format=json&siprop=statistics to compare the lag value to a threshold (in a shell script)

It was decided to move these checks to be grafana based and ditch the current icinga checks

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	Clean up absented files and unused configs	operations/puppet	production	+1 -54
	Drop wikidata alerts	operations/puppet	production	+2 -46

Customize query in gerrit

Related Objects

Mentioned In: T287741: Convert wikidata-alerts grafana dashboard to AlertManager
Mentioned Here: T240685: MediaWiki Prometheus support
T249164: RFC: Better interface for generating metrics in MediaWiki
T287741: Convert wikidata-alerts grafana dashboard to AlertManager

Event Timeline

fgiunchedi created this task.Aug 31 2021, 2:53 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 31 2021, 2:53 PM

• lmata added a project: Observability-Metrics.Aug 31 2021, 4:17 PM

Addshore subscribed.Aug 31 2021, 5:04 PM

Addshore mentioned this in T287741: Convert wikidata-alerts grafana dashboard to AlertManager.Aug 31 2021, 5:09 PM

After talking to @Addshore it seems all those metrics we have for wikidata are actually can be accessed in grafana and we simply can migrated those to grafana. It seems better, less scattered and much easier to do.

I do it.

I would be happy to help with T240685 regardless.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptAug 31 2021, 5:22 PM

Ladsgroup moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 31 2021, 5:22 PM

Change 715772 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):

[operations/puppet@production] Absent wikidata alerts

https://gerrit.wikimedia.org/r/715772

gerritbot added a project: Patch-For-Review.Aug 31 2021, 6:18 PM

We have three alerts there:

Checking the median lag being below 600s
- It was flaky and we set it to mute (or reduced its severity) and we never looked at it since, let's just nuke it
Checking the median lag being below 4000s
- After the previous alert being flaky we decided on an hour, but (I kid you not), due to the nature of these kind of alerts, we had to go with 4000s
- You want to know more? Look at the regex here (in ./modules/icinga/files/check_wikidata_crit: --ereg '"median":[^}]*"lag":([1-3]?[0-9]?[0-9]?[0-9]),')
Checking the median lag being below 4000s for test wikidata
- I actually don't know about this one, we don't collect that info for statsd, I think it would be easy to do though. I nuked it for now and will add the dispatch to the script.