Page MenuHomePhabricator
Create Task
Maniphest T386259

Service monitoring for FR analytics platform.
Closed, ResolvedPublic
Actions

Description

We should collect at least basic metrics on each service to prometheus and possibly alert from there. Some services might report by email, i.e. for failed job runs.

Performance metrics:

Alerting:

  • Dagster - slack notification, nagios check_procs
  • Metabase - nagios check_procs
  • Minio - nagios check_minio which scrapes exported prometheus metrics
  • Trino - API poll using nagios check_http
  • Hive Standalone Metastore

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
nsca_frack.cfg.erb create hostgroup fundraising-minio adding check-minioJgreenoperations/puppetproduction+70 -3
nsca_frack.cfg.erb break out trino hostgroup, add trino API checkJgreenoperations/puppetproduction+71 -4
icinga: frack: adjust fran* groupings and add hostDwisehauptoperations/puppetproduction+20 -7
icinga: add new frack hosts for basic monitoringDwisehauptoperations/puppetproduction+91 -0
Fix hostgroup and order for analytics checks in nsca_frack.cfg.erb.Jgreenoperations/puppetproduction+50 -50
Add fundraising-analytics hostgroup and two new checks to nsca_frack.cfg.erb.Jgreenoperations/puppetproduction+81 -11
Customize query in gerrit

Related Objects
Search...

Event Timeline

Jgreen created this task.Feb 12 2025, 7:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 12 2025, 7:50 PM
Jgreen updated the task description. (Show Details)Feb 12 2025, 7:52 PM
Jgreen updated the task description. (Show Details)Feb 13 2025, 2:01 PM
Jgreen updated the task description. (Show Details)Feb 13 2025, 2:14 PM
Jgreen updated the task description. (Show Details)Feb 13 2025, 5:08 PM
Jgreen added a parent task: T377362: EPIC: Trino/MinIO/Hive-Standalone-Metaserver/Dagster/Metabase/Superset Implementation.Feb 13 2025, 6:06 PM
Jgreen updated the task description. (Show Details)Feb 13 2025, 6:10 PM
Jgreen updated the task description. (Show Details)Feb 13 2025, 8:36 PM
Jgreen updated the task description. (Show Details)Feb 18 2025, 6:53 PM
Jgreen updated the task description. (Show Details)Feb 19 2025, 8:13 PM
Jgreen claimed this task.Feb 20 2025, 2:27 PM
Jgreen triaged this task as Medium priority.
Jgreen updated the task description. (Show Details)
Jgreen updated the task description. (Show Details)
Jgreen moved this task from Triage to In Progress on the fundraising-tech-ops board.
gerritbot added a comment.Feb 20 2025, 2:38 PM

Change #1121371 had a related patch set uploaded (by Jgreen; author: Jgreen):

[operations/puppet@production] Add fundraising-analytics hostgroup and two new checks to nsca_frack.cfg.erb.

https://gerrit.wikimedia.org/r/1121371

gerritbot added a project: Patch-For-Review.Feb 20 2025, 2:38 PM
gerritbot added a comment.Feb 20 2025, 2:42 PM

Change #1121371 merged by Filippo Giunchedi:

[operations/puppet@production] Add fundraising-analytics hostgroup and two new checks to nsca_frack.cfg.erb.

https://gerrit.wikimedia.org/r/1121371

Maintenance_bot removed a project: Patch-For-Review.Feb 20 2025, 3:30 PM
gerritbot added a comment.Feb 20 2025, 4:09 PM

Change #1121390 had a related patch set uploaded (by Jgreen; author: Jgreen):

[operations/puppet@production] Fix hostgroup and alpha order for analytics role passive checks in nsca_frack.cfg.erb

https://gerrit.wikimedia.org/r/1121390

gerritbot added a project: Patch-For-Review.Feb 20 2025, 4:09 PM
gerritbot added a comment.Feb 20 2025, 4:32 PM

Change #1121390 merged by Filippo Giunchedi:

[operations/puppet@production] Fix hostgroup and order for analytics checks in nsca_frack.cfg.erb.

https://gerrit.wikimedia.org/r/1121390

Maintenance_bot removed a project: Patch-For-Review.Feb 20 2025, 5:31 PM
Jgreen updated the task description. (Show Details)Feb 20 2025, 8:29 PM
gerritbot added a comment.May 1 2025, 9:14 PM

Change #1140541 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/puppet@production] icinga: add new frack hosts for basic monitoring

https://gerrit.wikimedia.org/r/1140541

gerritbot added a project: Patch-For-Review.May 1 2025, 9:14 PM
gerritbot added a comment.May 2 2025, 8:00 AM

Change #1140541 merged by Filippo Giunchedi:

[operations/puppet@production] icinga: add new frack hosts for basic monitoring

https://gerrit.wikimedia.org/r/1140541

Maintenance_bot removed a project: Patch-For-Review.May 2 2025, 8:31 AM
gerritbot added a comment.May 2 2025, 6:18 PM

Change #1140775 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/puppet@production] icinga: frack: adjust fran* groupings and add host

https://gerrit.wikimedia.org/r/1140775

gerritbot added a project: Patch-For-Review.May 2 2025, 6:18 PM
gerritbot added a comment.May 5 2025, 6:44 AM

Change #1140775 merged by Filippo Giunchedi:

[operations/puppet@production] icinga: frack: adjust fran* groupings and add host

https://gerrit.wikimedia.org/r/1140775

Maintenance_bot removed a project: Patch-For-Review.May 5 2025, 7:30 AM
Dwisehaupt updated the task description. (Show Details)May 8 2025, 12:13 AM
Jgreen updated the task description. (Show Details)Jun 10 2025, 7:08 PM
gerritbot added a comment.Jun 17 2025, 5:02 PM

Change #1160205 had a related patch set uploaded (by Jgreen; author: Jgreen):

[operations/puppet@production] nsca_frack.cfg.erb break out trino hostgroup, add trino API check

https://gerrit.wikimedia.org/r/1160205

gerritbot added a project: Patch-For-Review.Jun 17 2025, 5:02 PM
gerritbot added a comment.Jun 18 2025, 2:54 PM

Change #1160205 merged by Filippo Giunchedi:

[operations/puppet@production] nsca_frack.cfg.erb break out trino hostgroup, add trino API check

https://gerrit.wikimedia.org/r/1160205

Maintenance_bot removed a project: Patch-For-Review.Jun 18 2025, 3:31 PM
Jgreen updated the task description. (Show Details)Jun 18 2025, 7:32 PM
Jgreen moved this task from In Progress to Up Next on the fundraising-tech-ops board.Aug 20 2025, 7:58 PM
Jgreen moved this task from Up Next to Backlog on the fundraising-tech-ops board.
Jgreen updated the task description. (Show Details)Sep 5 2025, 4:41 PM
gerritbot added a comment.Sep 9 2025, 6:46 PM

Change #1186566 had a related patch set uploaded (by Jgreen; author: Jgreen):

[operations/puppet@production] nsca_frack.cfg.erb create hostgroup fundraising-minio adding check-minio

https://gerrit.wikimedia.org/r/1186566

gerritbot added a project: Patch-For-Review.Sep 9 2025, 6:46 PM
Jgreen updated the task description. (Show Details)Sep 9 2025, 6:53 PM
Jgreen updated the task description. (Show Details)Sep 22 2025, 1:52 PM
Jgreen closed subtask T405228: Enable prometheus-jmx-exporter for FR hive-standalone-metastore as Resolved.Sep 26 2025, 4:52 PM
Jgreen updated the task description. (Show Details)
Jgreen closed this task as Resolved.Oct 6 2025, 5:28 PM
Jgreen moved this task from Backlog to Done on the fundraising-tech-ops board.

Closing b/c I think we're close enough, although we aren't alerting for HMS outages directly we'll know quickly if there is an outage because trino will complain.

gerritbot added a comment.May 6 2026, 1:38 PM

Change #1186566 abandoned by Jgreen:

[operations/puppet@production] nsca_frack.cfg.erb create hostgroup fundraising-minio adding check-minio

Reason:

Fundraising is no longer using NSCA

https://gerrit.wikimedia.org/r/1186566

Maintenance_bot removed a project: Patch-For-Review.May 6 2026, 2:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits