The o11y team has discussed this internally and has decided to sunset Watchmouse as we do not seem to be obtaining great value from Watchmouse as external monitoring. Recent examples in T292603.
Historically the tool was initially used for the public status page and to expose some KPIs for external availability and an external uptime checker. Additional background: T81454, T85829, T89877, T79416
StatusPage effectively replaced Watchmouse recently (See T202061 and T285769), and the new stack is ready for production.
We also have enough redundancy with our existing Icinga and external testing infrastructure to sunset this tool without losing core functionality while reducing the scope of technology to support. In addition, we aim to improve external monitoring as part of plans and roadmap for alerting. However, we will not wait for the new solution's implementation as a dependency for moving forward with this decommission this quarter.
Checks currently defined in watchmouse / CA App Synthetic Monitor:
**Note: "ops-critical-phone" is actually an email contact for "noc@" and "watchmouse@" (and watchmouse@ is further aliased to maint-announce@)**
|Enabled (Y default)|Name|Tags|URL|Type|Contact|Replacement|Retirement Status|
||API||http://en.wikipedia.org/w/api.php|HTTP|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||DNS|wiki_platform|wikipedia.org |DNS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||Dumps download||https://dumps.wikimedia.org/backup-index.html|HTTPS|ops-non-critical-mail|added to watchrat|✅ deactivated in watchmouse|
||Gerrit||https://gerrit.wikimedia.org/r/|HTTPS|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||https content - commons||https://commons.wikimedia.org/wiki/Main_Page|HTTPS|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||https services - commons|wiki_platform|https://commons.wikimedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||https services - foundationwiki|wiki_platform|https://wikimediafoundation.org/wiki/Home|HTTPS|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
||https services - loginwiki|wiki_platform|https://login.wikimedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
||https services - mediawiki|wiki_platform|https://www.mediawiki.org/wiki/MediaWiki|HTTPS|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
||https services - wikibooks|wiki_platform|https://en.wikibooks.org/wiki/Main_Page|HTTPS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||https services - wikidata|wiki_platform|https://www.wikidata.org/wiki/Wikidata:Main_Page|HTTPS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||https services - wikinews|wiki_platform|https://en.wikinews.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
||https services - wikipedia|wiki_platform|https://en.wikipedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||https services - wikiquote|wiki_platform|https://en.wikiquote.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
||https services - wikisource|wiki_platform|https://en.wikisource.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
||https services - wikiversity|wiki_platform|https://en.wikiversity.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
||https services - wikivoyage|wiki_platform|https://en.wikivoyage.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
||https services - wiktionary|wiki_platform|https://en.wiktionary.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
|N|Icinga (disabled)||http://icinga.wikimedia.org|HTTP|icinga|retire|✅ deactivated in watchmouse|
||Images & media (HTTPS)|wiki_platform|https://upload.wikimedia.org/monitoring/backend|HTTPS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||Images & media|wiki_platform|https://upload.wikimedia.org/monitoring/backend|HTTPS|ops-critical-phone|duplicate of above|✅ deactivated in watchmouse|
||IRC RecentChanges||irc.wikimedia.org:6667|CONNECT|ops-non-critical-mail|retire|✅ deactivated in watchmouse|
||Mail (SMTP)|public|mx1001.wikimedia.org|SMTP|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||Mobile site|wiki_platform|http://en.m.wikipedia.org/wiki/Main_Page|HTTP|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||Phabricator||https://phabricator.wikimedia.org/T2001|HTTPS|ops-non-critical-mail|existing check in icinga|✅ deactivated in watchmouse|
||Static assets (CSS/JS)|wiki_platform|http://meta.wikimedia.org/w/resources/assets/poweredby_mediawiki_88x31.png|HTTP|wikimedia ops|retire|✅ deactivated in watchmouse|
||Static assets (HTTPS - CSS/JS)|wiki_platform|https://meta.wikimedia.org/w/resources/assets/poweredby_mediawiki_88x31.png|HTTPS|wikimedia ops|added to watchrat|✅ deactivated in watchmouse|
||Wiki commons (s4)|wiki_platform|http://commons.wikimedia.org/wiki/Main_Page|HTTP|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
|N|Wiki commons (s4) - UNCACHED|wiki_platform|http://commons.wikimedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:de:Main Page]] (s5)|wiki_platform|http://de.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
|N|Wiki platform [[w:de:Main Page]] (s5) - UNCACHED|wiki_platform|http://de.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:dsb:Main Page]] (s3)|wiki_platform|http://dsb.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
|N|Wiki platform [[w:dsb:Main Page]] (s3) - UNCACHED|wiki_platform|http://dsb.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:en:Main Page]] (s1)|wiki_platform|http://en.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|existing checkin icinga|✅ deactivated in watchmouse|
|N|Wiki platform [[w:en:Main Page]] (s1) - UNCACHED|wiki_platform|http://en.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:en:Special:Random]]|wiki_platform|http://en.wikipedia.org/wiki/Special:Random|HTTP|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
||Wiki platform [[w:fi:Main Page]] (s2)|wiki_platform|http://fi.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
|N|Wiki platform [[w:fi:Main Page]] (s2) - UNCACHED|wiki_platform|http://fi.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:fr:Main Page]] (s6)|wiki_platform|http://fr.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
|N|Wiki platform [[w:fr:Main Page]] (s6) - UNCACHED|wiki_platform|http://fr.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:uk:Main Page]] (s7)|wiki_platform|http://uk.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|added to watchrat|✅ deactivated in watchmouse|
|N|Wiki platform [[w:uk:Main Page]] (s7) - UNCACHED|wiki_platform|http://uk.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wikimedia blog||http://blog.wikimedia.org/|HTTP|ops-non-critical-mail|existing check in icinga|✅ deactivated in watchmouse|
||wikimedia foundation mainpage|wiki_platform|http://wikimediafoundation.org/wiki/Home|HTTP|ops-critical-phone|retire (dupe of above)|✅ deactivated in watchmouse|
||donate http||http://donate.wikimedia.org/|HTTP|fundraising-critical|retire|✅ deactivated in watchmouse|
||donate https||https://donate.wikimedia.org/|HTTPS|fundraising-critical|added to watchrat (with email alert routing to fr-tech)|✅ deactivated in watchmouse|
||frdata http||http://frdata.wikimedia.org/|HTTP|fundraising-critical|retire|✅ deactivated in watchmouse|
||frdata https||https://frdata.wikimedia.org/|HTTPS|fundraising-critical|added to watchrat|✅ deactivated in watchmouse|
||payments https||https://payments.wikimedia.org/index.php/Special:SystemStatus|HTTPS|fundraising-critical|added to watchrat|✅ deactivated in watchmouse|
||payments listener||https://payments-listener.wikimedia.org/globalcollect|HTTPS|fundraising-critical|added to watchrat|✅ deactivated in watchmouse|
||banner load testing chrome8||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||banner load testing FF3.6||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||banner load testing ie6||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||banner load testing ie7||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||banner load testing ie8||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||banner load testing safari4||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||mr1-codfw OOB||mr1-codfw.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||mr1-eqiad OOB||mr1-eqiad.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||mr1-eqsin OOB||mr1-eqsin.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||mr1-esams OOB||mr1-esams.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||mr1-ulsfo OOB||mr1-ulsfo.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||Ping offload text-lb.codfw||text-lb.codfw.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||Ping offload text-lb.eqiad||text-lb.eqiad.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||secure.wikimedia.org||https://secure.wikimedia.org/wikipedia/en/wiki/Main_Page|HTTPS|ops-non-critical-mail|added to watchrat|✅ deactivated in watchmouse|
||shop.wikimedia.org|public|http://shop.wikimedia.org/|HTTP|ops-non-critical-mail|added to watchrat|✅ deactivated in watchmouse|
||en wiki login -script|||Script|ops-non-critical-mail|retire|✅ deactivated in watchmouse|
||icinga https port open||icinga.wikimedia.org:443|CONNECT|Chris Danis|retire|✅ deactivated in watchmouse|
||icinga-https||https://icinga.wikimedia.org/|HTTPS|Chris Danis|retire|✅ deactivated in watchmouse|
||wikimediafoundation.org - script|||Script|ops-non-critical-mail|retire|✅ deactivated in watchmouse|
High level todo list:
[x] set up replacement static check capability via prometheus blackbox exporter (nicknamed "watchrat")
[x] export/import relevant watchmouse checks, audit checks and move needed checks to watchrat config
[x] enable alerting
[x] disable watchmouse checks