Page MenuHomePhabricator
Create Task
Maniphest T302995

Explore Pyrra for SLO Visualization and Management
Open, MediumPublic
Actions

Description

Today we manage SLO dashboards using an in-house jsonnet template which is rendered and deployed to grafana using grafana-grizzly. Since establishing this process, a self contained SLO management tool Pyrra (https://pyrra.dev) has seen much active development and offers several benefits including improved (dedicated) SLO visualization, search, labeling, automation of recording rules, integrated multi burn alerting, and more.

This task initially served as a placeholder to explore this (hence the patch history) and is being expanded to serve as a tracking task for Pyrra deployment and integration.

High level checklist, in rough order:

Details

SubjectRepoBranchLines +/-
services: add pyrra conftool-data and service stub entryoperations/puppetproduction+49 -4
pyrra add service dns entriesoperations/dnsmaster+6 -1
pyrra: onboard varnish-requests as pilot SLOoperations/puppetproduction+38 -1
profile::pyrra::filesystem: improve/fix lift wing pilotoperations/puppetproduction+2 -2
profile::pyrra::filesystem: add Lift Wing pilotoperations/puppetproduction+32 -0
profile::thanos: add new istio recording ruleoperations/puppetproduction+3 -0
pyrra-filesystem: enable generic recording rulesoperations/puppetproduction+1 -1
pyrra: add logstash requests slooperations/puppetproduction+36 -0
pyrra::filesystem::config: add pyrra filesystem operator config manageroperations/puppetproduction+25 -0
pyrra::filesystem: correct config permissionsoperations/puppetproduction+3 -3
pyrra: add prometheus external urloperations/puppetproduction+4 -3
thanos-rule: add pyrra filesystem operator output dir to search pathoperations/puppetproduction+2 -1
thanos::httpd: include rewrite moduleoperations/puppetproduction+1 -1
pyrra: add serveraliases and redirect to apache configoperations/puppetproduction+7 -0
pyrra: add public dns entriesoperations/dnsmaster+4 -0
pyrra: add trafficserver mappingoperations/puppetproduction+15 -0
pyrra: use load balancingoperations/puppetproduction+1 -1
titan: move pyrra off thanos roleoperations/puppetproduction+3 -2
pyrra-filesystem: add package require for file resourceoperations/puppetproduction+5 -4
pyrra-filesystem: ensure config directoryoperations/puppetproduction+3 -1
pyrra-filesystem: update prometheus folder arg nameoperations/puppetproduction+1 -1
pyrra: remove apache rewrite configoperations/puppetproduction+0 -3
pyrra: fix typo in apache configoperations/puppetproduction+1 -1
pyrra: deploy to thanos-fe hostsoperations/puppetproduction+3 -0
profile::pyrra::filesystem: add profileoperations/puppetproduction+10 -0
profile::pyrra::api: create profileoperations/puppetproduction+32 -0
pyrra: add pyrra::(api|filesystem) modulesoperations/puppetproduction+71 -0
pyrra: initial packaging for v0.6.2operations/debs/pyrramaster+227 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

herron triaged this task as Medium priority.Mar 3 2022, 6:39 PM
herron created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 3 2022, 6:39 PM
herron mentioned this in T302842: SLO dashboard refinements.Mar 3 2022, 6:49 PM
herron edited projects, added SRE Observability; removed observability.Mar 4 2022, 6:45 PM
herron moved this task from Inbox to Backlog on the SRE Observability board.
lmata edited projects, added Observability-Metrics; removed SRE Observability.Mar 25 2022, 2:05 PM
lmata moved this task from Inbox to Backlog on the Observability-Metrics board.Sep 6 2022, 7:47 PM
lmata moved this task from Backlog to Prioritized on the Observability-Metrics board.Nov 28 2022, 4:27 PM
lmata added a project: SRE Observability (FY2022/2023-Q3).
lmata moved this task from Inbox to Up next on the SRE Observability (FY2022/2023-Q3) board.Nov 30 2022, 6:40 PM
herron added a project: User-herron.Jan 13 2023, 3:11 PM
lmata edited projects, added SRE Observability (FY2022/2023-Q4); removed SRE Observability (FY2022/2023-Q3).May 2 2023, 1:41 PM
herron moved this task from Backlog to Working on on the User-herron board.May 17 2023, 1:55 PM
gerritbot added a comment.Jun 2 2023, 7:25 PM

Change 922608 had a related patch set uploaded (by Herron; author: Herron):

[operations/debs/pyrra@master] pyrra: initial packaging for v0.6.2

https://gerrit.wikimedia.org/r/922608

gerritbot added a project: Patch-For-Review.Jun 2 2023, 7:25 PM
gerritbot added a comment.Jun 6 2023, 1:33 PM

Change 922608 merged by Herron:

[operations/debs/pyrra@master] pyrra: initial packaging for v0.6.2

https://gerrit.wikimedia.org/r/922608

Maintenance_bot removed a project: Patch-For-Review.Jun 6 2023, 2:11 PM
gerritbot added a comment.Jun 13 2023, 2:28 PM

Change 929719 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra: add pyrra::(api|filesystem) modules

https://gerrit.wikimedia.org/r/929719

gerritbot added a project: Patch-For-Review.Jun 13 2023, 2:28 PM
gerritbot added a comment.Jun 13 2023, 3:12 PM

Change 929729 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] profile::pyrra::api: create profile

https://gerrit.wikimedia.org/r/929729

gerritbot added a comment.Jun 13 2023, 3:20 PM

Change 929731 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] profile::pyrra::filesystem: add profile

https://gerrit.wikimedia.org/r/929731

gerritbot added a comment.Jun 13 2023, 3:23 PM

Change 929734 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra: deploy to alert/thanos-fe hosts

https://gerrit.wikimedia.org/r/929734

gerritbot added a comment.Jun 15 2023, 1:22 PM

Change 930628 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] thanos-rule: add pyrra filesystem operator output dir to search path

https://gerrit.wikimedia.org/r/930628

fgiunchedi mentioned this in T340666: mediawiki-event-enrichment jobs alerting.Jul 7 2023, 8:14 AM
lmata added a parent task: T290924: Tooling for end-of-quarter SLO reporting.Jul 10 2023, 2:37 AM
lmata edited projects, added SRE Observability (FY2023/2024-Q1); removed SRE Observability (FY2022/2023-Q4).Jul 18 2023, 4:57 PM
lmata moved this task from Inbox to Up next on the SRE Observability (FY2023/2024-Q1) board.Jul 18 2023, 8:49 PM
gerritbot added a comment.Aug 3 2023, 5:00 PM

Change 929719 merged by Herron:

[operations/puppet@production] pyrra: add pyrra::(api|filesystem) modules

https://gerrit.wikimedia.org/r/929719

gerritbot added a comment.Aug 3 2023, 5:04 PM

Change 929729 merged by Herron:

[operations/puppet@production] profile::pyrra::api: create profile

https://gerrit.wikimedia.org/r/929729

gerritbot added a comment.Aug 3 2023, 5:06 PM

Change 929731 merged by Herron:

[operations/puppet@production] profile::pyrra::filesystem: add profile

https://gerrit.wikimedia.org/r/929731

gerritbot added a comment.Aug 3 2023, 5:20 PM

Change 929734 merged by Herron:

[operations/puppet@production] pyrra: deploy to thanos-fe hosts

https://gerrit.wikimedia.org/r/929734

gerritbot added a comment.Aug 3 2023, 5:27 PM

Change 945620 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra: fix typo in apache config

https://gerrit.wikimedia.org/r/945620

gerritbot added a comment.Aug 3 2023, 5:31 PM

Change 945620 merged by Herron:

[operations/puppet@production] pyrra: fix typo in apache config

https://gerrit.wikimedia.org/r/945620

gerritbot added a comment.Aug 3 2023, 5:38 PM

Change 945623 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra: remove apache rewrite config

https://gerrit.wikimedia.org/r/945623

gerritbot added a comment.Aug 3 2023, 5:41 PM

Change 945623 merged by Herron:

[operations/puppet@production] pyrra: remove apache rewrite config

https://gerrit.wikimedia.org/r/945623

gerritbot added a comment.Aug 3 2023, 5:49 PM

Change 945625 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra-filesystem: update prometheus folder arg name

https://gerrit.wikimedia.org/r/945625

gerritbot added a comment.Aug 3 2023, 5:50 PM

Change 945625 merged by Herron:

[operations/puppet@production] pyrra-filesystem: update prometheus folder arg name

https://gerrit.wikimedia.org/r/945625

gerritbot added a comment.Aug 3 2023, 5:59 PM

Change 945627 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra-filesystem: ensure config directory

https://gerrit.wikimedia.org/r/945627

gerritbot added a comment.Aug 3 2023, 6:04 PM

Change 945627 merged by Herron:

[operations/puppet@production] pyrra-filesystem: ensure config directory

https://gerrit.wikimedia.org/r/945627

gerritbot added a comment.Aug 3 2023, 6:13 PM

Change 945630 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra-filesystem: add package require for file resource

https://gerrit.wikimedia.org/r/945630

gerritbot added a comment.Aug 3 2023, 6:15 PM

Change 945630 merged by Herron:

[operations/puppet@production] pyrra-filesystem: add package require for file resource

https://gerrit.wikimedia.org/r/945630

gerritbot added a comment.Sep 12 2023, 12:30 PM

Change 956867 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] titan: move pyrra off thanos role

https://gerrit.wikimedia.org/r/956867

gerritbot added a comment.Sep 12 2023, 1:26 PM

Change 956867 merged by Filippo Giunchedi:

[operations/puppet@production] titan: move pyrra off thanos role

https://gerrit.wikimedia.org/r/956867

gerritbot added a comment.Sep 26 2023, 3:09 PM

Change 961128 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra: add trafficserver mapping

https://gerrit.wikimedia.org/r/961128

gerritbot added a comment.Sep 26 2023, 3:09 PM

Change 961129 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] services: add pyrra conftool-data and service stub entry

https://gerrit.wikimedia.org/r/961129

gerritbot added a comment.Sep 26 2023, 3:09 PM

Change 961130 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra: use load balancing

https://gerrit.wikimedia.org/r/961130

gerritbot added a comment.Sep 26 2023, 3:09 PM

Change 961131 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra: add serveraliases and redirect to apache config

https://gerrit.wikimedia.org/r/961131

gerritbot added a comment.Sep 26 2023, 3:10 PM

Change 961132 had a related patch set uploaded (by Herron; author: Herron):

[operations/dns@master] pyrra add service dns entries

https://gerrit.wikimedia.org/r/961132

gerritbot added a comment.Sep 26 2023, 3:10 PM

Change 961133 had a related patch set uploaded (by Herron; author: Herron):

[operations/dns@master] pyrra: add public dns entries

https://gerrit.wikimedia.org/r/961133

gerritbot added a comment.Sep 27 2023, 6:59 PM

Change 961128 merged by Herron:

[operations/puppet@production] pyrra: add trafficserver mapping

https://gerrit.wikimedia.org/r/961128

gerritbot added a comment.Sep 27 2023, 7:02 PM

Change 961133 merged by Herron:

[operations/dns@master] pyrra: add public dns entries

https://gerrit.wikimedia.org/r/961133

gerritbot added a comment.Sep 28 2023, 5:33 PM

Change 961131 merged by Herron:

[operations/puppet@production] pyrra: add serveraliases and redirect to apache config

https://gerrit.wikimedia.org/r/961131

gerritbot added a comment.Sep 28 2023, 5:39 PM

Change 961862 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] thanos::httpd: include rewrite module

https://gerrit.wikimedia.org/r/961862

gerritbot added a comment.Sep 28 2023, 5:48 PM

Change 961862 merged by Herron:

[operations/puppet@production] thanos::httpd: include rewrite module

https://gerrit.wikimedia.org/r/961862

gerritbot added a comment.Oct 3 2023, 3:33 PM

Change 930628 merged by Herron:

[operations/puppet@production] thanos-rule: add pyrra filesystem operator output dir to search path

https://gerrit.wikimedia.org/r/930628

lmata edited projects, added SRE Observability (FY2023/2024-Q2); removed SRE Observability (FY2023/2024-Q1).Oct 9 2023, 4:28 PM
lmata moved this task from Inbox to Up next on the SRE Observability (FY2023/2024-Q2) board.
Stashbot added a comment.Oct 17 2023, 10:03 PM

Mentioned in SAL (#wikimedia-operations) [2023-10-17T22:03:21Z] <herron> pyrra.wm.o upgraded to 0.7.1 T302995

gerritbot added a comment.Oct 18 2023, 5:55 PM

Change 966906 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra::filesystem::config: add pyrra filesystem operator config manager

https://gerrit.wikimedia.org/r/966906

gerritbot added a comment.Oct 18 2023, 7:28 PM

Change 966917 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra: add prometheus external url

https://gerrit.wikimedia.org/r/966917

gerritbot added a comment.Oct 18 2023, 7:33 PM

Change 966917 merged by Herron:

[operations/puppet@production] pyrra: add prometheus external url

https://gerrit.wikimedia.org/r/966917

gerritbot added a comment.Oct 18 2023, 7:36 PM

Change 966920 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra::filesystem: correct config permissions

https://gerrit.wikimedia.org/r/966920

gerritbot added a comment.Oct 18 2023, 7:42 PM

Change 966920 merged by Herron:

[operations/puppet@production] pyrra::filesystem: correct config permissions

https://gerrit.wikimedia.org/r/966920

gerritbot added a comment.Oct 18 2023, 8:19 PM

Change 966909 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra: add logstash requests slo

https://gerrit.wikimedia.org/r/966909

gerritbot added a comment.Oct 19 2023, 3:56 PM

Change 966906 merged by Herron:

[operations/puppet@production] pyrra::filesystem::config: add pyrra filesystem operator config manager

https://gerrit.wikimedia.org/r/966906

gerritbot added a comment.Oct 19 2023, 4:00 PM

Change 966909 merged by Herron:

[operations/puppet@production] pyrra: add logstash requests slo

https://gerrit.wikimedia.org/r/966909

gerritbot added a comment.Oct 19 2023, 7:01 PM

Change 967273 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra-filesystem: enable generic recording rules

https://gerrit.wikimedia.org/r/967273

gerritbot added a comment.Oct 19 2023, 7:10 PM

Change 967273 merged by Herron:

[operations/puppet@production] pyrra-filesystem: enable generic recording rules

https://gerrit.wikimedia.org/r/967273

herron renamed this task from Explore dedicated (non-grafana) SLO Visualization and Management to Explore Pyrra for SLO Visualization and Management.Oct 23 2023, 2:44 PM
herron updated the task description. (Show Details)
gerritbot added a comment.Oct 23 2023, 3:12 PM

Change 967950 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] pyrra: onboard varnish-requests as pilot SLO

https://gerrit.wikimedia.org/r/967950

lmata moved this task from Up next to In progress on the SRE Observability (FY2023/2024-Q2) board.Oct 24 2023, 2:13 PM
elukey subscribed.Nov 3 2023, 1:37 PM
herron updated the task description. (Show Details)Nov 3 2023, 1:58 PM
herron updated the task description. (Show Details)Nov 3 2023, 2:15 PM
herron updated the task description. (Show Details)Nov 13 2023, 6:11 PM
gerritbot added a comment.Nov 14 2023, 10:41 AM

Change 974148 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::thanos: add new istio recording rule

https://gerrit.wikimedia.org/r/974148

gerritbot added a comment.Nov 14 2023, 10:41 AM

Change 974149 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::pyrra::filesystem: add Lift Wing pilot

https://gerrit.wikimedia.org/r/974149

gerritbot added a comment.Nov 14 2023, 3:02 PM

Change 974148 merged by Elukey:

[operations/puppet@production] profile::thanos: add new istio recording rule

https://gerrit.wikimedia.org/r/974148

gerritbot added a comment.Nov 14 2023, 3:51 PM

Change 974149 merged by Elukey:

[operations/puppet@production] profile::pyrra::filesystem: add Lift Wing pilot

https://gerrit.wikimedia.org/r/974149

gerritbot added a comment.Nov 15 2023, 10:29 AM

Change 974496 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::pyrra::filesystem: improve/fix lift wing pilot

https://gerrit.wikimedia.org/r/974496

gerritbot added a comment.Nov 15 2023, 1:35 PM

Change 974496 merged by Elukey:

[operations/puppet@production] profile::pyrra::filesystem: improve/fix lift wing pilot

https://gerrit.wikimedia.org/r/974496

gerritbot added a comment.Dec 18 2023, 11:19 PM

Change 983950 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/dns@master] Add dyna record for community-crm

https://gerrit.wikimedia.org/r/983950

gerritbot added a comment.Dec 18 2023, 11:22 PM

Change 983951 had a related patch set uploaded (by Dwisehaupt; author: Dwisehaupt):

[operations/puppet@production] Set the cdn to pass requests for community-crm

https://gerrit.wikimedia.org/r/983951

gerritbot added a comment.Dec 19 2023, 2:38 PM

Change 967950 merged by Herron:

[operations/puppet@production] pyrra: onboard varnish-requests as pilot SLO

https://gerrit.wikimedia.org/r/967950

lmata removed a project: SRE Observability (FY2023/2024-Q2).Jan 26 2024, 1:07 AM
CodeReviewBot added a comment.Feb 22 2024, 7:34 PM

herron opened https://gitlab.wikimedia.org/repos/sre/pyrra/-/merge_requests/1

Draft: Debian/bullseye wikimedia

herron closed subtask T353691: Reload thanos-rule on new pyrra rules deployed as Resolved.Thu, Apr 4, 6:33 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL