Page MenuHomePhabricator
Create Task
Maniphest T301944

Web interface to navigate Prometheus alerts and their status
Closed, ResolvedPublic
Actions

Description

The alerts.w.o works fine to check firing/silenced alerts, however it'd be useful to users to be able to see all defined alerts and their status. This information is not available to alertmanager, but it is to the individual systems that send alerts to AM. In the librenms/icinga/grafana case we have the capability already, in the Prometheus case not yet.

This task tracks a solution to expose all Prometheus alerts and their status. The information is available both on the Prometheus web interface and as the ALERTS and ALERTS_FOR_STATE meta-metrics. The solution might entail either reverse-proxying the Prometheus web interface (behind SSO) and/or playing with the meta-metrics above to build suitable dashboard(s)

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

fgiunchedi created this task.Feb 17 2022, 7:53 AM
fgiunchedi updated the task description. (Show Details)
herron awarded a token.Feb 17 2022, 5:13 PM
herron added a subscriber: herron.Feb 17 2022, 5:17 PM

+1 for reverse proxying the prometheus web interface behind SSO, that seems straightforward to me and could be useful in other cases as well

gerritbot added a comment.Feb 22 2022, 9:08 PM

Change 764895 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] prometheus: sketch out proxied prometheus web with IDP

https://gerrit.wikimedia.org/r/764895

gerritbot added a project: Patch-For-Review.Feb 22 2022, 9:08 PM
gerritbot added a comment.May 3 2022, 2:57 PM

Change 788721 had a related patch set uploaded (by Herron; author: Herron):

[labs/private@master] "private" add prometheus.wm.o placeholder key

https://gerrit.wikimedia.org/r/788721

gerritbot added a comment.May 3 2022, 3:06 PM

Change 788721 merged by Herron:

[labs/private@master] "private" add prometheus.wm.o placeholder key

https://gerrit.wikimedia.org/r/788721

herron mentioned this in rLPRI0ced0b596b26: "private" add prometheus.wm.o placeholder key.May 3 2022, 3:08 PM
gerritbot added a comment.Oct 24 2022, 3:16 PM

Change 764895 merged by Herron:

[operations/puppet@production] prometheus: enable prometheus web access via proxy with IDP

https://gerrit.wikimedia.org/r/764895

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2022, 3:30 PM
gerritbot added a comment.Oct 24 2022, 5:28 PM

Change 848442 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] prometheus: update web_idp urls to prometheus-$site.wm.o

https://gerrit.wikimedia.org/r/848442

gerritbot added a project: Patch-For-Review.Oct 24 2022, 5:28 PM
gerritbot added a comment.Oct 24 2022, 5:46 PM

Change 848442 merged by Herron:

[operations/puppet@production] prometheus: update web_idp urls to prometheus-$site.wm.o

https://gerrit.wikimedia.org/r/848442

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2022, 6:30 PM
gerritbot added a comment.Oct 24 2022, 7:32 PM

Change 848480 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] prometheus: web_idp pin to prometheus(12)005

https://gerrit.wikimedia.org/r/848480

gerritbot added a project: Patch-For-Review.Oct 24 2022, 7:32 PM
gerritbot added a comment.Oct 24 2022, 7:34 PM

Change 848480 merged by Herron:

[operations/puppet@production] prometheus: web_idp pin to prometheus(12)005

https://gerrit.wikimedia.org/r/848480

gerritbot added a comment.Oct 24 2022, 7:43 PM

Change 848488 had a related patch set uploaded (by Herron; author: Herron):

[operations/dns@master] dns: add prometheus-$site.wm.o entries for prometheus web interface

https://gerrit.wikimedia.org/r/848488

gerritbot added a comment.Oct 24 2022, 7:58 PM

Change 848488 merged by Herron:

[operations/dns@master] dns: add prometheus-$site.wm.o entries for prometheus web interface

https://gerrit.wikimedia.org/r/848488

herron added a comment.EditedOct 24 2022, 8:14 PM

Prometheus web interfaces can now be accessed behind SSO with https://prometheus-site.wikimedia.org/instance/ e.g. https://prometheus-eqiad.wikimedia.org/ops/

A good next step here will be to create a landing page for the root of the site e.g. https://prometheus-eqiad.wikimedia.org/ (currently an apache default page) with high level information and links to docs, along with handling of https://prometheus.wm.o

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2022, 8:30 PM
fgiunchedi added a comment.Oct 25 2022, 7:22 AM

This is great to see -- thanks @herron for working on it!

gerritbot added a comment.Oct 31 2022, 10:47 AM

Change 851004 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] hieradata: fix Prometheus IDP entry

https://gerrit.wikimedia.org/r/851004

gerritbot added a project: Patch-For-Review.Oct 31 2022, 10:47 AM
gerritbot added a comment.Oct 31 2022, 11:04 AM

Change 851004 merged by Filippo Giunchedi:

[operations/puppet@production] hieradata: fix Prometheus IDP entry

https://gerrit.wikimedia.org/r/851004

Maintenance_bot removed a project: Patch-For-Review.Oct 31 2022, 11:30 AM
fgiunchedi added a subscriber: JMeybohm.Nov 2 2022, 11:28 AM

It was noticed today by @JMeybohm that the prometheus web interface is currently cached (and shouldn't for obvious reasons)

gerritbot added a comment.Nov 16 2022, 10:32 AM

Change 857522 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: default to valid external url

https://gerrit.wikimedia.org/r/857522

gerritbot added a project: Patch-For-Review.Nov 16 2022, 10:32 AM
gerritbot added a comment.Nov 17 2022, 7:30 PM

Change 858406 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] prometheus: disable caching of prometheus-site.wm.o

https://gerrit.wikimedia.org/r/858406

gerritbot added a comment.Nov 17 2022, 9:12 PM

Change 858406 merged by Herron:

[operations/puppet@production] prometheus: disable caching of prometheus-site.wm.o

https://gerrit.wikimedia.org/r/858406

herron added a comment.Nov 18 2022, 3:50 PM
In T301944#8362493, @fgiunchedi wrote:

It was noticed today by @JMeybohm that the prometheus web interface is currently cached (and shouldn't for obvious reasons)

With https://gerrit.wikimedia.org/r/858406 I'm now seeing e.g. x-cache: cp1089 miss, cp1079 pass on requests to the prom web interface.

Interestingly, similar cache pass rules were added at the time the prom web interface was deployed, but it just so happened to be deployed right at the same time as a refactoring of that yaml and they didn't get carried over. At any rate looking much better now!

gerritbot added a comment.Nov 21 2022, 1:19 PM

Change 857522 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: default to valid external url

https://gerrit.wikimedia.org/r/857522

gerritbot added a comment.Dec 2 2022, 7:23 PM

Change 863380 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] service::catalog: add prometheus-https

https://gerrit.wikimedia.org/r/863380

herron mentioned this in T326657: Add prometheus-https load balancer.Jan 10 2023, 5:00 PM
herron added a project: User-herron.Jan 10 2023, 6:38 PM
herron moved this task from Backlog to Working on on the User-herron board.
BCornwall changed the status of subtask T326657: Add prometheus-https load balancer from Open to In Progress.Mar 28 2023, 9:10 PM
BCornwall changed the status of subtask T326657: Add prometheus-https load balancer from In Progress to Stalled.May 1 2023, 4:31 PM
herron closed this task as Resolved.May 17 2023, 1:53 PM
herron claimed this task.
gerritbot added a comment.Jun 12 2023, 8:03 PM

Change 863380 merged by BCornwall:

[operations/puppet@production] service::catalog: add prometheus-https

https://gerrit.wikimedia.org/r/863380

Maintenance_bot removed a project: Patch-For-Review.Jun 12 2023, 8:10 PM
gerritbot added a comment.Jun 12 2023, 9:44 PM

Change 929421 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] prometheus: Disable SNI support in Envoy tlsproxy

https://gerrit.wikimedia.org/r/929421

gerritbot added a project: Patch-For-Review.Jun 12 2023, 9:44 PM
gerritbot added a comment.Jun 12 2023, 10:15 PM

Change 928942 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] Revert "service::catalog: add prometheus-https"

https://gerrit.wikimedia.org/r/928942

gerritbot added a comment.Jun 12 2023, 10:20 PM

Change 928942 merged by BCornwall:

[operations/puppet@production] Revert "service::catalog: add prometheus-https"

https://gerrit.wikimedia.org/r/928942

gerritbot added a comment.Jun 13 2023, 5:22 PM

Change 929421 merged by BCornwall:

[operations/puppet@production] prometheus: Disable SNI support in Envoy tlsproxy

https://gerrit.wikimedia.org/r/929421

gerritbot added a comment.Jun 13 2023, 5:27 PM

Change 929768 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] prometheus: Add global_cert_name to Envoy config

https://gerrit.wikimedia.org/r/929768

gerritbot added a comment.Jul 7 2023, 9:21 PM

Change 929768 abandoned by BCornwall:

[operations/puppet@production] prometheus: Disable SNI support in Envoy tlsproxy

Reason:

I5b10a4a2ad3a34b8ad2ef48052b13c93c62aedd0 supercedes this

https://gerrit.wikimedia.org/r/929768

Maintenance_bot removed a project: Patch-For-Review.Jul 7 2023, 9:30 PM
gerritbot added a comment.Jul 18 2023, 3:30 PM

Change 939326 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] service::catalog: add prometheus-https

https://gerrit.wikimedia.org/r/939326

gerritbot added a project: Patch-For-Review.Jul 18 2023, 3:30 PM
lmata moved this task from Inbox to Prioritized on the Observability-Metrics board.Jul 18 2023, 5:40 PM
BCornwall changed the status of subtask T326657: Add prometheus-https load balancer from Stalled to In Progress.Jul 19 2023, 9:34 PM
gerritbot added a comment.Jul 20 2023, 2:35 PM

Change 939326 merged by Herron:

[operations/puppet@production] service::catalog: add prometheus-https

https://gerrit.wikimedia.org/r/939326

Maintenance_bot removed a project: Patch-For-Review.Jul 20 2023, 3:10 PM
herron closed subtask T326657: Add prometheus-https load balancer as Resolved.Jul 27 2023, 3:50 PM
BCornwall reopened subtask T326657: Add prometheus-https load balancer as Open.Aug 2 2023, 10:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL