The o11y team has discussed this internally and has decided to sunset Watchmouse as we do not seem to be obtaining great value from Watchmouse as external monitoring. Recent examples in T292603.
Historically the tool was initially used for the public status page and to expose some KPIs for external availability and an external uptime checker. Additional background: T81454, T85829, T89877, T79416
StatusPage effectively replaced Watchmouse recently (See T202061 and T285769), and the new stack is ready for production.
We also have enough redundancy with our existing Icinga and external testing infrastructure to sunset this tool without losing core functionality while reducing the scope of technology to support. In addition, we aim to improve external monitoring as part of plans and roadmap for alerting. However, we will not wait for the new solution's implementation as a dependency for moving forward with this decommission this quarter.
Checks currently defined in watchmouse / CA App Synthetic Monitor:
**Note: "ops-critical-phone" is actually an email contact for "noc@" and "watchmouse@" (and watchmouse@ is further aliased to maint-announce@)**
|Enabled (Y default)|Name|Tags|URL|Type|Contact|Replacement/Status|
||API||http://en.wikipedia.org/w/api.php|HTTP|ops-critical-phone|
||DNS|wiki_platform|wikipedia.org |DNS|ops-critical-phone|
||Dumps download||https://dumps.wikimedia.org/backup-index.html|HTTPS|ops-non-critical-mail|
||Gerrit||https://gerrit.wikimedia.org/r/|HTTPS|wikimedia ops|
||https content - commons||https://commons.wikimedia.org/wiki/Main_Page|HTTPS|wikimedia ops|
||https services - commons|wiki_platform|https://commons.wikimedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - foundationwiki|wiki_platform|https://wikimediafoundation.org/wiki/Home|HTTPS|ops-critical-phone|
||https services - loginwiki|wiki_platform|https://login.wikimedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - mediawiki|wiki_platform|https://www.mediawiki.org/wiki/MediaWiki|HTTPS|ops-critical-phone|
||https services - wikibooks|wiki_platform|https://en.wikibooks.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wikidata|wiki_platform|https://www.wikidata.org/wiki/Wikidata:Main_Page|HTTPS|ops-critical-phone|
||https services - wikinews|wiki_platform|https://en.wikinews.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wikipedia|wiki_platform|https://en.wikipedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wikiquote|wiki_platform|https://en.wikiquote.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wikisource|wiki_platform|https://en.wikisource.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wikiversity|wiki_platform|https://en.wikiversity.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wikivoyage|wiki_platform|https://en.wikivoyage.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wiktionary|wiki_platform|https://en.wiktionary.org/wiki/Main_Page|HTTPS|ops-critical-phone|
|N|Icinga (disabled)||http://icinga.wikimedia.org|HTTP|icinga|
||Images & media (HTTPS)|wiki_platform|https://upload.wikimedia.org/monitoring/backend|HTTPS|ops-critical-phone|
||Images & media|wiki_platform|https://upload.wikimedia.org/monitoring/backend|HTTPS|ops-critical-phone|
||IRC RecentChanges||irc.wikimedia.org:6667|CONNECT|ops-non-critical-mail|
||Mail (SMTP)|public|mx1001.wikimedia.org|SMTP|ops-critical-phone|
||Mobile site|wiki_platform|http://en.m.wikipedia.org/wiki/Main_Page|HTTP|wikimedia ops|
||Phabricator||https://phabricator.wikimedia.org/T2001|HTTPS|ops-non-critical-mail|
||Static assets (CSS/JS)|wiki_platform|http://meta.wikimedia.org/w/resources/assets/poweredby_mediawiki_88x31.png|HTTP|wikimedia ops|
||Static assets (HTTPS - CSS/JS)|wiki_platform|https://meta.wikimedia.org/w/resources/assets/poweredby_mediawiki_88x31.png|HTTPS|wikimedia ops|
||Wiki commons (s4)|wiki_platform|http://commons.wikimedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki commons (s4) - UNCACHED|wiki_platform|http://commons.wikimedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
||Wiki platform [[w:de:Main Page]] (s5)|wiki_platform|http://de.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki platform [[w:de:Main Page]] (s5) - UNCACHED|wiki_platform|http://de.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
||Wiki platform [[w:dsb:Main Page]] (s3)|wiki_platform|http://dsb.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki platform [[w:dsb:Main Page]] (s3) - UNCACHED|wiki_platform|http://dsb.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
||Wiki platform [[w:en:Main Page]] (s1)|wiki_platform|http://en.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki platform [[w:en:Main Page]] (s1) - UNCACHED|wiki_platform|http://en.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
||Wiki platform [[w:en:Special:Random]]|wiki_platform|http://en.wikipedia.org/wiki/Special:Random|HTTP|ops-critical-phone|
||Wiki platform [[w:fi:Main Page]] (s2)|wiki_platform|http://fi.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki platform [[w:fi:Main Page]] (s2) - UNCACHED|wiki_platform|http://fi.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
||Wiki platform [[w:fr:Main Page]] (s6)|wiki_platform|http://fr.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki platform [[w:fr:Main Page]] (s6) - UNCACHED|wiki_platform|http://fr.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
||Wiki platform [[w:uk:Main Page]] (s7)|wiki_platform|http://uk.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki platform [[w:uk:Main Page]] (s7) - UNCACHED|wiki_platform|http://uk.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
||Wikimedia blog||http://blog.wikimedia.org/|HTTP|ops-non-critical-mail|
||wikimedia foundation mainpage|wiki_platform|http://wikimediafoundation.org/wiki/Home|HTTP|ops-critical-phone|
||donate http||http://donate.wikimedia.org/|HTTP|fundraising-critical|
||donate https||https://donate.wikimedia.org/|HTTPS|fundraising-critical|
||frdata http||http://frdata.wikimedia.org/|HTTP|fundraising-critical|
||frdata https||https://frdata.wikimedia.org/|HTTPS|fundraising-critical|
||payments https||https://payments.wikimedia.org/index.php/Special:SystemStatus|HTTPS|fundraising-critical|
||payments listener||https://payments-listener.wikimedia.org/globalcollect|HTTPS|fundraising-critical|
||banner load testing chrome8||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|
||banner load testing FF3.6||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|
||banner load testing ie6||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|
||banner load testing ie7||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|
||banner load testing ie8||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|
||banner load testing safari4||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|
||mr1-codfw OOB||mr1-codfw.oob.wikimedia.org|PING|wikimedia ops|
||mr1-eqiad OOB||mr1-eqiad.oob.wikimedia.org|PING|wikimedia ops|
||mr1-eqsin OOB||mr1-eqsin.oob.wikimedia.org|PING|wikimedia ops|
||mr1-esams OOB||mr1-esams.oob.wikimedia.org|PING|wikimedia ops|
||mr1-ulsfo OOB||mr1-ulsfo.oob.wikimedia.org|PING|wikimedia ops|
||Ping offload text-lb.codfw||text-lb.codfw.wikimedia.org|PING|wikimedia ops|
||Ping offload text-lb.eqiad||text-lb.eqiad.wikimedia.org|PING|wikimedia ops|
||secure.wikimedia.org||https://secure.wikimedia.org/wikipedia/en/wiki/Main_Page|HTTPS|ops-non-critical-mail|
||shop.wikimedia.org|public|http://shop.wikimedia.org/|HTTP|ops-non-critical-mail|
||en wiki login -script|||Script|ops-non-critical-mail|
||icinga https port open||icinga.wikimedia.org:443|CONNECT|Chris Danis|
||icinga-https||https://icinga.wikimedia.org/|HTTPS|Chris Danis|
||wikimediafoundation.org - script|||Script|ops-non-critical-mail|
High level todo list:
[x] set up replacement static check capability via prometheus blackbox exporter (nicknamed "watchrat")
[] export/import relevant watchmouse checks, audit checks and move needed checks to watchrat config
[] enable alerting