The o11y team has discussed this internally and has decided to sunset Watchmouse as we do not seem to be obtaining great value from Watchmouse as external monitoring. Recent examples in T292603.
Historically the tool was initially used for the public status page and to expose some KPIs for external availability and an external uptime checker. Additional background: T81454, T85829, T89877, T79416
StatusPage effectively replaced Watchmouse recently (See T202061 and T285769), and the new stack is ready for production.
We also have enough redundancy with our existing Icinga and external testing infrastructure to sunset this tool without losing core functionality while reducing the scope of technology to support. In addition, we aim to improve external monitoring as part of plans and roadmap for alerting. However, we will not wait for the new solution's implementation as a dependency for moving forward with this decommission this quarter.
Checks currently defined in watchmouse / CA App Synthetic Monitor:
**Note: "ops-critical-phone" is actually an email contact for "noc@" and "watchmouse@" (and watchmouse@ is further aliased to maint-announce@)**
|Enabled (Y default)|Name|Tags|URL|Type|Contact|Replacement|Retirement Status|
||API||http://en.wikipedia.org/w/api.php|HTTP|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||DNS|wiki_platform|wikipedia.org |DNS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||Dumps download||https://dumps.wikimedia.org/backup-index.html|HTTPS|ops-non-critical-mail|added to watchrat
||Gerrit||https://gerrit.wikimedia.org/r/|HTTPS|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||https content - commons||https://commons.wikimedia.org/wiki/Main_Page|HTTPS|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||https services - commons|wiki_platform|https://commons.wikimedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||https services - foundationwiki|wiki_platform|https://wikimediafoundation.org/wiki/Home|HTTPS|ops-critical-phone|added to watchrat|
||https services - loginwiki|wiki_platform|https://login.wikimedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|
||https services - mediawiki|wiki_platform|https://www.mediawiki.org/wiki/MediaWiki|HTTPS|ops-critical-phone|added to watchrat|
||https services - wikibooks|wiki_platform|https://en.wikibooks.org/wiki/Main_Page|HTTPS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||https services - wikidata|wiki_platform|https://www.wikidata.org/wiki/Wikidata:Main_Page|HTTPS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||https services - wikinews|wiki_platform|https://en.wikinews.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|
||https services - wikipedia|wiki_platform|https://en.wikipedia.org/wiki/Main_Page|HTTPS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||https services - wikiquote|wiki_platform|https://en.wikiquote.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|
||https services - wikisource|wiki_platform|https://en.wikisource.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|
||https services - wikiversity|wiki_platform|https://en.wikiversity.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|
||https services - wikivoyage|wiki_platform|https://en.wikivoyage.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|
||https services - wiktionary|wiki_platform|https://en.wiktionary.org/wiki/Main_Page|HTTPS|ops-critical-phone|added to watchrat|
|N|Icinga (disabled)||http://icinga.wikimedia.org|HTTP|icinga|retire|
||Images & media (HTTPS)|wiki_platform|https://upload.wikimedia.org/monitoring/backend|HTTPS|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||Images & media|wiki_platform|https://upload.wikimedia.org/monitoring/backend|HTTPS|ops-critical-phone|duplicate of above|✅ deactivated in watchmouse|
||IRC RecentChanges||irc.wikimedia.org:6667|CONNECT|ops-non-critical-mail|
||Mail (SMTP)|public|mx1001.wikimedia.org|SMTP|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
||Mobile site|wiki_platform|http://en.m.wikipedia.org/wiki/Main_Page|HTTP|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||Phabricator||https://phabricator.wikimedia.org/T2001|HTTPS|ops-non-critical-mail|existing check in icinga|✅ deactivated in watchmouse|
||Static assets (CSS/JS)|wiki_platform|http://meta.wikimedia.org/w/resources/assets/poweredby_mediawiki_88x31.png|HTTP|wikimedia ops|retire|
||Static assets (HTTPS - CSS/JS)|wiki_platform|https://meta.wikimedia.org/w/resources/assets/poweredby_mediawiki_88x31.png|HTTPS|wikimedia ops|added to watchrat|
||Wiki commons (s4)|wiki_platform|http://commons.wikimedia.org/wiki/Main_Page|HTTP|ops-critical-phone|existing check in icinga|✅ deactivated in watchmouse|
|N|Wiki commons (s4) - UNCACHED|wiki_platform|http://commons.wikimedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:de:Main Page]] (s5)|wiki_platform|http://de.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|added to watchrat|
|N|Wiki platform [[w:de:Main Page]] (s5) - UNCACHED|wiki_platform|http://de.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:dsb:Main Page]] (s3)|wiki_platform|http://dsb.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|added to watchrat|
|N|Wiki platform [[w:dsb:Main Page]] (s3) - UNCACHED|wiki_platform|http://dsb.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:en:Main Page]] (s1)|wiki_platform|http://en.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|existing checkin icinga|
|N|Wiki platform [[w:en:Main Page]] (s1) - UNCACHED|wiki_platform|http://en.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:en:Special:Random]]|wiki_platform|http://en.wikipedia.org/wiki/Special:Random|HTTP|ops-critical-phone|added to watchrat|
||Wiki platform [[w:fi:Main Page]] (s2)|wiki_platform|http://fi.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|added to watchrat|
|N|Wiki platform [[w:fi:Main Page]] (s2) - UNCACHED|wiki_platform|http://fi.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:fr:Main Page]] (s6)|wiki_platform|http://fr.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|added to watchrat|
|N|Wiki platform [[w:fr:Main Page]] (s6) - UNCACHED|wiki_platform|http://fr.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wiki platform [[w:uk:Main Page]] (s7)|wiki_platform|http://uk.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|added to watchrat|
|N|Wiki platform [[w:uk:Main Page]] (s7) - UNCACHED|wiki_platform|http://uk.wikipedia.org/wiki/Main_Page|HTTP|ops-critical-phone|retire|✅ deactivated in watchmouse|
||Wikimedia blog||http://blog.wikimedia.org/|HTTP|ops-non-critical-mail|existing check in icinga|✅ deactivated in watchmouse|
||wikimedia foundation mainpage|wiki_platform|http://wikimediafoundation.org/wiki/Home|HTTP|ops-critical-phone|retire (dupe of above)|
||donate http||http://donate.wikimedia.org/|HTTP|fundraising-critical|
||donate https||https://donate.wikimedia.org/|HTTPS|fundraising-critical|added to watchrat|
||frdata http||http://frdata.wikimedia.org/|HTTP|fundraising-critical|
||frdata https||https://frdata.wikimedia.org/|HTTPS|fundraising-critical|added to watchrat|
||payments https||https://payments.wikimedia.org/index.php/Special:SystemStatus|HTTPS|fundraising-critical|added to watchrat|
||payments listener||https://payments-listener.wikimedia.org/globalcollect|HTTPS|fundraising-critical|added to watchrat|
||banner load testing chrome8||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||banner load testing FF3.6||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||banner load testing ie6||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||banner load testing ie7||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||banner load testing ie8||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||banner load testing safari4||http://en.wikipedia.org/wiki/List_of_collective_nouns|Full-Page|(none)|retire|✅ deactivated in watchmouse|
||mr1-codfw OOB||mr1-codfw.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||mr1-eqiad OOB||mr1-eqiad.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||mr1-eqsin OOB||mr1-eqsin.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||mr1-esams OOB||mr1-esams.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||mr1-ulsfo OOB||mr1-ulsfo.oob.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||Ping offload text-lb.codfw||text-lb.codfw.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||Ping offload text-lb.eqiad||text-lb.eqiad.wikimedia.org|PING|wikimedia ops|existing check in icinga|✅ deactivated in watchmouse|
||secure.wikimedia.org||https://secure.wikimedia.org/wikipedia/en/wiki/Main_Page|HTTPS|ops-non-critical-mail|added to watchrat|
||shop.wikimedia.org|public|http://shop.wikimedia.org/|HTTP|ops-non-critical-mail||
||en wiki login -script|||Script|ops-non-critical-mail|retire|
||icinga https port open||icinga.wikimedia.org:443|CONNECT|Chris Danis|retire|
||icinga-https||https://icinga.wikimedia.org/|HTTPS|Chris Danis|retire|
||wikimediafoundation.org - script|||Script|ops-non-critical-mail|retire|
High level todo list:
[x] set up replacement static check capability via prometheus blackbox exporter (nicknamed "watchrat")
[] export/import relevant watchmouse checks, audit checks and move needed checks to watchrat config
[] enable alerting
[] disable watchmouse checks