The o11y team has discussed this internally and has decided to sunset Watchmouse as we do not seem to be obtaining great value from Watchmouse as external monitoring. Recent examples in T292603.
Historically the tool was initially used for the public status page and to expose some KPIs for external availability and an external uptime checker. Additional background: T81454, T85829, T89877, T79416
StatusPage effectively replaced Watchmouse recently (See T202061 and T285769), and the new stack is ready for production.
We also have enough redundancy with our existing Icinga and external testing infrastructure to sunset this tool without losing core functionality while reducing the scope of technology to support. In addition, we aim to improve external monitoring as part of plans and roadmap for alerting. However, we will not wait for the new solution's implementation as a dependency for moving forward with this decommission this quarter.
Checks currently defined in watchmouse / CA App Synthetic Monitor:
**Note: "ops-critical-phone" is actually an email contact for "noc@" and "watchmouse@" (and watchmouse@ is further aliased to maint-announce@)**
|Enabled (Y default)|Name|Tags|URL|Type|Contact|Replacement/Status|
||API||http://en.wikipedia.org/w/api.php|HTTP|ops-critical-phone|existing check in icinga|
||DNS|wiki_platform|wikipedia.org |DNS|ops-critical-phone|existing check in icinga|
||Dumps download||https://dumps.wikimedia.org/backup-index.html|HTTPS|ops-non-critical-mail|
||Gerrit||https://gerrit.wikimedia.org/r/|HTTPS|wikimedia ops|existing check in icinga|
||https content - commons||https://commons.wikimedia.org/wiki/Main_Page|HTTPS|wikimedia ops|existing check in icinga|
||https services - commons|wiki_platform|https://commons.wikimedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|existing check in icinga|
||https services - foundationwiki|wiki_platform|https://wikimediafoundation.org/wiki/Home|HTTPS|ops-critical-phone|
||https services - loginwiki|wiki_platform|https://login.wikimedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - mediawiki|wiki_platform|https://www.mediawiki.org/wiki/MediaWiki|HTTPS|ops-critical-phone|
||https services - wikibooks|wiki_platform|https://en.wikibooks.org/wiki/Main_Page|HTTPS|ops-critical-phone|existing check in icinga|
||https services - wikidata|wiki_platform|https://www.wikidata.org/wiki/Wikidata:Main_Page|HTTPS|ops-critical-phone|existing check in icinga|
||https services - wikinews|wiki_platform|https://en.wikinews.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wikipedia|wiki_platform|https://en.wikipedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|existing check in icinga|
||https services - wikiquote|wiki_platform|https://en.wikiquote.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wikisource|wiki_platform|https://en.wikisource.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wikiversity|wiki_platform|https://en.wikiversity.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wikivoyage|wiki_platform|https://en.wikivoyage.org/wiki/Main_Page|HTTPS|ops-critical-phone|
||https services - wiktionary|wiki_platform|https://en.wiktionary.org/wiki/Main_Page|HTTPS|ops-critical-phone|
|N|Icinga (disabled)||http://icinga.wikimedia.org|HTTP|icinga|retire|
||Images & media (HTTPS)|wiki_platform|https://upload.wikimedia.org/monitoring/backend|HTTPS|ops-critical-phone|existing check in icinga|
||Images & media|wiki_platform|https://upload.wikimedia.org/monitoring/backend|HTTPS|ops-critical-phone|duplicate of above|
||IRC RecentChanges||irc.wikimedia.org:6667|CONNECT|ops-non-critical-mail|
||Mail (SMTP)|public|mx1001.wikimedia.org|SMTP|ops-critical-phone|existing check in icinga|
||Mobile site|wiki_platform|http://en.m.wikipedia.org/wiki/Main_Page|HTTP|wikimedia ops|existing check in icinga|
||Phabricator||https://phabricator.wikimedia.org/T2001|HTTPS|ops-non-critical-mail|existing check in icinga|
||Static assets (CSS/JS)|wiki_platform|http://meta.wikimedia.org/w/resources/assets/poweredby_mediawiki_88x31.png|HTTP|wikimedia ops|
||Static assets (HTTPS - CSS/JS)|wiki_platform|https://meta.wikimedia.org/w/resources/assets/poweredby_mediawiki_88x31.png|HTTPS|wikimedia ops|
||Wiki commons (s4)|wiki_platform|http://commons.wikimedia.org/wiki/Main_Page|HTTP|ops-critical-phone|existing check in icinga|
|N|Wiki commons (s4) - UNCACHED|wiki_platform|http://commons.wikimedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|
||Wiki platform [[w:de:Main Page]] (s5)|wiki_platform|http://de.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki platform [[w:de:Main Page]] (s5) - UNCACHED|wiki_platform|http://de.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|
||Wiki platform [[w:dsb:Main Page]] (s3)|wiki_platform|http://dsb.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki platform [[w:dsb:Main Page]] (s3) - UNCACHED|wiki_platform|http://dsb.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|
||Wiki platform [[w:en:Main Page]] (s1)|wiki_platform|http://en.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|existing checkin icinga|
|N|Wiki platform [[w:en:Main Page]] (s1) - UNCACHED|wiki_platform|http://en.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|
||Wiki platform [[w:en:Special:Random]]|wiki_platform|http://en.wikipedia.org/wiki/Special:Random|HTTP|ops-critical-phone|
||Wiki platform [[w:fi:Main Page]] (s2)|wiki_platform|http://fi.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki platform [[w:fi:Main Page]] (s2) - UNCACHED|wiki_platform|http://fi.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|
||Wiki platform [[w:fr:Main Page]] (s6)|wiki_platform|http://fr.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki platform [[w:fr:Main Page]] (s6) - UNCACHED|wiki_platform|http://fr.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|
||Wiki platform [[w:uk:Main Page]] (s7)|wiki_platform|http://uk.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|
|N|Wiki platform [[w:uk:Main Page]] (s7) - UNCACHED|wiki_platform|http://uk.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|
||Wikimedia blog||http://blog.wikimedia.org/|HTTP|ops-non-critical-mail|existing check in icinga|
||wikimedia foundation mainpage|wiki_platform|http://wikimediafoundation.org/wiki/Home|HTTP|ops-critical-phone|
||donate http||http://donate.wikimedia.org/|HTTP|fundraising-critical|
||donate https||https://donate.wikimedia.org/|HTTPS|fundraising-critical|
||frdata http||http://frdata.wikimedia.org/|HTTP|fundraising-critical|
||frdata https||https://frdata.wikimedia.org/|HTTPS|fundraising-critical|
||payments https||https://payments.wikimedia.org/index.php/Special:SystemStatus|HTTPS|fundraising-critical|
||payments listener||https://payments-listener.wikimedia.org/globalcollect|HTTPS|fundraising-critical|
||banner load testing chrome8||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|
||banner load testing FF3.6||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|
||banner load testing ie6||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|
||banner load testing ie7||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|
||banner load testing ie8||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|
||banner load testing safari4||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|
||mr1-codfw OOB||mr1-codfw.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|
||mr1-eqiad OOB||mr1-eqiad.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|
||mr1-eqsin OOB||mr1-eqsin.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|
||mr1-esams OOB||mr1-esams.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|
||mr1-ulsfo OOB||mr1-ulsfo.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|
||Ping offload text-lb.codfw||text-lb.codfw.wikimedia.org|PING|wikimedia ops|existing check in icinga|
||Ping offload text-lb.eqiad||text-lb.eqiad.wikimedia.org|PING|wikimedia ops|existing check in icinga|
||secure.wikimedia.org||https://secure.wikimedia.org/wikipedia/en/wiki/Main_Page|HTTPS|ops-non-critical-mail|
||shop.wikimedia.org|public|http://shop.wikimedia.org/|HTTP|ops-non-critical-mail|existing check in icinga|
||en wiki login -script|||Script|ops-non-critical-mail|retire|
||icinga https port open||icinga.wikimedia.org:443|CONNECT|Chris Danis|retire|
||icinga-https||https://icinga.wikimedia.org/|HTTPS|Chris Danis|retire|
||wikimediafoundation.org - script|||Script|ops-non-critical-mail|retire|
High level todo list:
[x] set up replacement static check capability via prometheus blackbox exporter (nicknamed "watchrat")
[] export/import relevant watchmouse checks, audit checks and move needed checks to watchrat config
[] enable alerting