Move wikitech-static monitoring off Icinga
Open, LowPublicGoal
Actions

Assigned To

None

Authored By

	taavi
	Apr 12 2024, 12:17 PM

Project Tags

Referenced Files

	F89126385: 2026-06-17-161349_1682x918_scrot.png
	Jun 17 2026, 2:26 PM

Subscribers

Description

We have a few wikitech-static monitoring checks in Icinga that need to be ported:

Basic HTTPS reachability/certificate checks for wikitech-static.wikimedia.org and status.wikimedia.org. These should be relatively easy to port to Blackbox probes.
Wikitech and wikitech-static content in sync
Wikitech-static MW version up to date

! MIGRATION TABLE !

Migrated? (Y/N)	Title	Resource Type	Command	File	Profiles
N	wikitech-static-version	Monitoring::Service	check_wikitech_static_version	modules/icinga/manifests/monitor/wikitech_static.pp:21	profile::icinga
N	wikitech-static-sync	Monitoring::Service	check_wikitech_static	modules/icinga/manifests/monitor/wikitech_static.pp:11	profile::icinga
N	https_wikitech-static	Monitoring::Service	check_ssl_http_letsencrypt	modules/icinga/manifests/monitor/wikitech_static.pp:39	profile::icinga
N	https_status-wikimedia	Monitoring::Service	check_ssl_http_letsencrypt	modules/icinga/manifests/monitor/wikitech_static.pp:48	profile::icinga
N	wikitech-static-main-page	Monitoring::Service	check_https_url_at_address_for_string	modules/icinga/manifests/monitor/wikitech_static.pp:30	profile::icinga

Details

Related Changes in Gerrit:

Subject	Author	Repo	Branch	Lines +/-
prometheus: add wikitech-static metrics	Filippo Giunchedi	operations/puppet	production	+26 -3
team-wmcs: add wikitech alerts	Filippo Giunchedi	operations/alerts	master	+80 -0
icinga: remove absented wikitech-static checks	Filippo Giunchedi	operations/puppet	production	+0 -19

Customize query in gerrit

Related Changes in GitLab:

Title	Reference	Author	Source Branch	Dest Branch
nginx: make /wts-metrics display in a browser	repos/sre/wikitech-static-docker!7	filippo	bug/T362397	main
replacesearchbar: use html.parser in all BeautifulSoup calls	repos/sre/wikitech-static-docker!6	filippo	bug/T362397	main
Write Prometheus metrics timestamp file	repos/sre/wikitech-static-docker!5	filippo	bug/T362397	main

Customize query in GitLab

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		tappof	T395441 Port all Icinga checks to Prometheus/Alertmanager: preparation
Open		tappof	T321808 Port all Icinga checks to Prometheus/Alertmanager
Open	Goal	None	T328502 Move WMCS off of Icinga and introduce alertmanager
Open	Goal	None	T362397 Move wikitech-static monitoring off Icinga

Event Timeline

taavi created this task.Apr 12 2024, 12:17 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 12 2024, 12:17 PM

• joanna_borun triaged this task as Low priority.Aug 21 2024, 2:28 PM

• lmata moved this task from Inbox to Radar on the Observability-Alerting board.Oct 31 2024, 9:08 PM

andrea.denisse subscribed.Jan 13 2025, 9:55 PM

Change #1155612 had a related patch set uploaded (by Tiziano Fogli; author: Tiziano Fogli):

[operations/puppet@production] monitoring services: add migration task T362397 to instances

https://gerrit.wikimedia.org/r/1155612

gerritbot added a project: Patch-For-Review.Jun 11 2025, 8:44 AM

Change #1155612 merged by Tiziano Fogli:

[operations/puppet@production] monitoring services: add migration task T362397 to instances

https://gerrit.wikimedia.org/r/1155612

Maintenance_bot removed a project: Patch-For-Review.Jun 23 2025, 10:33 AM

tappof changed the subtype of this task from "Task" to "Goal".Sep 2 2025, 1:33 PM

tappof updated the task description. (Show Details)Sep 2 2025, 1:55 PM

I ran a quick audit on the wikitech-static icinga monitoring today:

wikitech-static.wikimedia.org no longer answers pings, thus icinga considers the host down. That's been the case for the last five months
Since the host is down from icinga's POV, its service alerts notifications are not sent
The "content in sync" alert (/usr/lib/nagios/plugins/check_wikitech_static) has been firing for the last two months. The wikitech-static url it checks no longer exists ('https://wikitech-static.wikimedia.org/w/api.php?action=query&titles=Special:RecentChanges&list=recentchanges&format=xml')

2026-06-17-161349_1682x918_scrot.png (1,682×918 px, 212 KB)

In terms of Prometheus monitoring, the pingthing facility in puppet has been monitoring https://wikitech-static.wikimedia.org/wiki/Main_Page for non 2xx or 3xx responses and issues a warning alert to SRE if that's not the case. e.g. https://wikitech.wikimedia.org/wiki/Prometheus#Pingthing_Non-23xx_HTTP_response

Nowadays the "content in sync" signal can also be read as "how long ago did the docker container CI pipeline last succeed?", specifically this pipeline: https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker/-/pipelines

I inquired if we had such signal already as a metric, and unfortunately we don't, though the feature request is tracked at T347038: Add additional exporter for GitLab Runner metrics

Other ideas that were floated to keep an eye on the pipeline:

use the pushgateway at the end of ci to push a metric, works only on trusted runners
write a "static-metrics" file inside the container, with the timestamp of last update. Then it can be scraped by prometheus at wikitech-static.w.o/static-metrics as any other endpoint. Would work, with some custom prometheus job config to make sure the outbound proxies are used to reach wikitech-static.

filippo opened https://gitlab.wikimedia.org/repos/sre/wikitech-static-docker/-/merge_requests/5

Write Prometheus metrics timestamp file

On balance I like option 2, which ensures truly end-to-end monitoring of container build + deploy. In other words the wikitech_static_update_timestamp_seconds metric going stale (time() - wikitech_static_update_timestamp_seconds) means something went wrong, either the container failed to build or failed to deploy. Either way we can take a look when/if that happens, @Andrew I sent the MR your way, please let me know what you think