Page MenuHomePhabricator
Create Task
Maniphest T288196

Retire Prometheus 'global' instance
Closed, ResolvedPublic
Actions

Description

Since we have Thanos rule in production now (T287142) the Prometheus global instance is not really needed anymore, therefore it should be retired. Outline of steps:

  • Move global rules (mostly recording rules, possibly some alerting rules too) to Thanos rule. Either via alerts.git or Puppet
  • Audit dashboards using global instance and port them to Thanos
  • Soft-disable global by denying HTTP access
  • Remove the puppet bits related to global
  • Nuke the global LV on Prometheus hosts

Dashboards using global

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects

Event Timeline

fgiunchedi created this task.Aug 5 2021, 8:14 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 5 2021, 8:14 AM
fgiunchedi mentioned this in T287142: Deploy Thanos rule for global alerts.Aug 5 2021, 8:59 AM
• lmata edited projects, added: Observability-Metrics; removed: SRE Observability.Aug 17 2021, 3:18 PM
• lmata edited projects, added: SRE Observability (FY2021/2022-Q2); removed: Observability-Metrics.Sep 28 2021, 1:45 PM
• lmata edited projects, added: SRE Observability (FY2021/2022-Q3); removed: SRE Observability (FY2021/2022-Q2).Sep 28 2021, 3:33 PM
Krinkle added a project: Performance-Team (Radar).Sep 28 2021, 3:47 PM
Krinkle moved this task from Limbo to Watching on the Performance-Team (Radar) board.
Krinkle subscribed.
fgiunchedi moved this task from FY2021/2022-Q3 to Backlog on the SRE Observability board.Jan 17 2022, 1:54 PM
fgiunchedi edited projects, added: SRE Observability; removed: SRE Observability (FY2021/2022-Q3).
• lmata edited projects, added: Observability-Metrics; removed: SRE Observability.Jan 17 2022, 11:10 PM
fgiunchedi added a comment.Nov 15 2022, 5:11 PM

This would help reclaiming some space at least in eqiad, we have ~700G still free in the vg and getting rid of global would give us another 900G

herron subscribed.Nov 16 2022, 2:19 PM
colewhite subscribed.Nov 16 2022, 3:41 PM

Dashboards likely using the global instance:

Krinkle added a comment.Nov 16 2022, 4:03 PM

https://grafana.wikimedia.org/d/000000066/resourceloader

I'll convert this one to Thanos. From a quick clickin' around, I notice that 1) the same global/pre-aggrate metrics seem to be available in Thanos already e.g. status_cc_xc:varnish_resourceloader_resp:irate5m instead of varnish_resourceloader_resp, but also that 2) using sum(irate(varnish_resourceloader_resp[5m])) yields the same result (actually slightly more precise, which is not surprising).

Screenshot 2022-11-16 at 15.59.35.png (1,973×1,208 px, 184 KB)

Assuming that's fine, I'll convert these over and remove the source selector.

gerritbot added a comment.Nov 29 2022, 11:13 AM

Change 861825 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: move traffic rules off 'global' instance

https://gerrit.wikimedia.org/r/861825

gerritbot added a project: Patch-For-Review.Nov 29 2022, 11:13 AM
gerritbot added a comment.Nov 29 2022, 11:22 AM

Change 861826 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: deprecate traffic 'global' rules

https://gerrit.wikimedia.org/r/861826

gerritbot added a comment.Dec 1 2022, 2:32 PM

Change 861825 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: move traffic rules to 'ops' instance

https://gerrit.wikimedia.org/r/861825

Krinkle added a comment.EditedDec 5 2022, 7:01 PM
In T288196#8399876, @Krinkle wrote:

https://grafana.wikimedia.org/d/000000066/resourceloader

I'll convert this one to Thanos. From a quick clickin' around, I notice that 1) the same global/pre-aggrate metrics seem to be available in Thanos already e.g. status_cc_xc:varnish_resourceloader_resp:irate5m instead of varnish_resourceloader_resp, but also that 2) using sum(irate(varnish_resourceloader_resp[5m])) yields the same result (actually slightly more precise, which is not surprising).

Screenshot 2022-11-16 at 15.59.35.png (1,973×1,208 px, 184 KB)

Assuming that's fine, I'll convert these over and remove the source selector.

Done. I've converted all of the panels now to use Prometheus-Thanos and make the underlying raw query directly. It no longer uses the deprecated prometheus/global source.

gerritbot added a comment.Jan 11 2023, 9:52 AM

Change 878858 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] prometheus: Fix job:haproxy_frontend_http_responses_total:rate2m

https://gerrit.wikimedia.org/r/878858

gerritbot added a comment.Jan 11 2023, 10:01 AM

Change 878858 merged by Vgutierrez:

[operations/puppet@production] prometheus: Fix job:haproxy_frontend_http_responses_total:rate2m

https://gerrit.wikimedia.org/r/878858

fgiunchedi added a subscriber: Vgutierrez.Jan 11 2023, 10:03 AM

One side effect of this migration is what happens when a site is unreachable by Thanos (h/t @Vgutierrez). e.g. on https://grafana.wikimedia.org/d/000000479/cdn-frontend-traffic the other day when eqsin was unreachable then the site also disappeared from the dropdown; this is due to the fact that we're populating the dropdown based on site label from Thanos, therefore if Thanos can't reach out to eqsin then the site isn't returned.

The current behavior is counter intuitive (to me at least) as in, the site should be at least selectable and maybe no data is shown if metrics are not queryable/available.

fgiunchedi edited projects, added: User-fgiunchedi; removed: Patch-For-Review.May 19 2023, 9:20 AM
fgiunchedi added a comment.May 19 2023, 9:41 AM
In T288196#8515980, @fgiunchedi wrote:

One side effect of this migration is what happens when a site is unreachable by Thanos (h/t @Vgutierrez). e.g. on https://grafana.wikimedia.org/d/000000479/cdn-frontend-traffic the other day when eqsin was unreachable then the site also disappeared from the dropdown; this is due to the fact that we're populating the dropdown based on site label from Thanos, therefore if Thanos can't reach out to eqsin then the site isn't returned.

The current behavior is counter intuitive (to me at least) as in, the site should be at least selectable and maybe no data is shown if metrics are not queryable/available.

For site specifically what we could do is export a "fixed" metric from puppet with the list of sites, similarly to what we do for "info" metrics. This way the metric will always be available from eqiad and codfw

gerritbot added a comment.May 19 2023, 9:48 AM

Change 921248 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: remove global rules

https://gerrit.wikimedia.org/r/921248

gerritbot added a project: Patch-For-Review.May 19 2023, 9:48 AM
fgiunchedi updated the task description. (Show Details)May 19 2023, 9:54 AM
fgiunchedi updated the task description. (Show Details)May 19 2023, 10:00 AM
fgiunchedi updated the task description. (Show Details)May 19 2023, 10:13 AM
fgiunchedi updated the task description. (Show Details)May 19 2023, 12:22 PM
fgiunchedi updated the task description. (Show Details)May 19 2023, 12:37 PM
fgiunchedi updated the task description. (Show Details)May 19 2023, 12:48 PM
Krinkle removed a project: Performance-Team (Radar).May 19 2023, 12:49 PM
gerritbot added a comment.May 19 2023, 1:12 PM

Change 921347 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: soft-disable 'global' instance

https://gerrit.wikimedia.org/r/921347

gerritbot added a comment.May 19 2023, 1:12 PM

Change 921349 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: de-provision 'global' instance

https://gerrit.wikimedia.org/r/921349

gerritbot added a comment.May 19 2023, 1:12 PM

Change 921350 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: remove 'global' instance references

https://gerrit.wikimedia.org/r/921350

fgiunchedi updated the task description. (Show Details)May 19 2023, 1:44 PM
fgiunchedi moved this task from Backlog to Doing on the User-fgiunchedi board.May 22 2023, 8:17 AM
fgiunchedi updated the task description. (Show Details)May 23 2023, 9:17 AM
fgiunchedi updated the task description. (Show Details)
fgiunchedi updated the task description. (Show Details)May 23 2023, 9:26 AM
gerritbot added a comment.May 23 2023, 9:33 AM

Change 922477 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] grafana: remove varnish-aggregate-client-status-codes

https://gerrit.wikimedia.org/r/922477

fgiunchedi updated the task description. (Show Details)May 23 2023, 9:40 AM
gerritbot added a comment.May 23 2023, 12:07 PM

Change 922499 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] icinga: remove swift alerts, moved to alertmanager

https://gerrit.wikimedia.org/r/922499

gerritbot added a comment.May 23 2023, 12:12 PM

Change 922499 merged by Filippo Giunchedi:

[operations/puppet@production] icinga: remove swift alerts, moved to alertmanager

https://gerrit.wikimedia.org/r/922499

gerritbot added a comment.May 23 2023, 12:16 PM

Change 861826 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: deprecate traffic 'global' rules

https://gerrit.wikimedia.org/r/861826

gerritbot added a comment.May 23 2023, 1:31 PM

Change 921347 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: soft-disable 'global' instance

https://gerrit.wikimedia.org/r/921347

fgiunchedi updated the task description. (Show Details)May 23 2023, 1:37 PM
gerritbot added a comment.May 24 2023, 8:03 AM

Change 922477 merged by Filippo Giunchedi:

[operations/puppet@production] grafana: remove varnish-aggregate-client-status-codes

https://gerrit.wikimedia.org/r/922477

fgiunchedi updated the task description. (Show Details)May 24 2023, 8:09 AM
fgiunchedi updated the task description. (Show Details)May 24 2023, 11:58 AM
fgiunchedi updated the task description. (Show Details)May 24 2023, 12:15 PM
gerritbot added a comment.May 24 2023, 12:31 PM

Change 921248 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: remove global rules

https://gerrit.wikimedia.org/r/921248

andrea.denisse subscribed.May 24 2023, 2:10 PM
gerritbot added a comment.May 24 2023, 9:20 PM

Change 922534 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] profile: remove varnish-aggregate-client-status-codes resource

https://gerrit.wikimedia.org/r/922534

gerritbot added a comment.May 25 2023, 8:27 AM

Change 922534 merged by Filippo Giunchedi:

[operations/puppet@production] profile: ensure varnish-aggregate-client-status-codes absent

https://gerrit.wikimedia.org/r/922534

gerritbot added a comment.May 29 2023, 7:32 AM

Change 921349 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: de-provision 'global' instance

https://gerrit.wikimedia.org/r/921349

gerritbot added a comment.May 29 2023, 9:35 AM

Change 924055 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] prometheus: don't enable server when absenting

https://gerrit.wikimedia.org/r/924055

gerritbot added a comment.May 29 2023, 9:43 AM

Change 924055 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: don't enable server when absenting

https://gerrit.wikimedia.org/r/924055

fgiunchedi updated the task description. (Show Details)May 29 2023, 9:58 AM
gerritbot added a comment.May 30 2023, 8:05 AM

Change 921350 merged by Filippo Giunchedi:

[operations/puppet@production] prometheus: remove 'global' instance references

https://gerrit.wikimedia.org/r/921350

fgiunchedi changed the task status from Open to Stalled.May 30 2023, 8:11 AM
fgiunchedi updated the task description. (Show Details)

Puppet bits are gone, I'll remove the global LV in a couple of days just out of caution

• lmata added a project: SRE Observability (FY2022/2023-Q4).May 30 2023, 1:08 PM
Stashbot added a comment.Jun 1 2023, 9:18 AM

Mentioned in SAL (#wikimedia-operations) [2023-06-01T09:18:29Z] <godog> remove lv prometheus-global - T288196

fgiunchedi closed this task as Resolved.Jun 1 2023, 9:19 AM
fgiunchedi claimed this task.
fgiunchedi updated the task description. (Show Details)

The prometheus global instance is gone

• lmata awarded a token.Jun 1 2023, 9:53 PM
• lmata moved this task from Inbox to Done on the SRE Observability (FY2022/2023-Q4) board.Jul 18 2023, 5:00 PM
gerritbot added a comment.Aug 24 2023, 9:47 PM

Change 951882 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] grafana: ensure prometheus/global instances removed

https://gerrit.wikimedia.org/r/951882

gerritbot added a comment.Sep 12 2023, 9:12 PM

Change 951882 merged by Cwhite:

[operations/puppet@production] grafana: ensure prometheus/global datasources removed

https://gerrit.wikimedia.org/r/951882

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits