Page MenuHomePhabricator

herron (Keith Herron)
Ops Engineer

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Tuesday

  • Clear sailing ahead.

User Details

User Since
May 30 2017, 5:25 PM (303 w, 4 d)
Availability
Available
IRC Nick
herron
LDAP User
Herron
MediaWiki User
Unknown

Recent Activity

Thu, Mar 23

herron triaged T332901: Grizzly: automatic filtering or warning about unnecessary fields as Medium priority.
Thu, Mar 23, 3:18 PM · User-herron, Observability-Metrics
herron updated the task description for T332900: Grizzly: document JSON/YAML interactions and practices for our use cases.
Thu, Mar 23, 3:00 PM · Observability-Metrics
herron updated the task description for T332892: Grizzly: upgrade to 0.2.
Thu, Mar 23, 2:59 PM · Observability-Metrics
herron created T332900: Grizzly: document JSON/YAML interactions and practices for our use cases.
Thu, Mar 23, 2:59 PM · Observability-Metrics
herron updated the task description for T332892: Grizzly: upgrade to 0.2.
Thu, Mar 23, 2:52 PM · Observability-Metrics
herron updated the task description for T332892: Grizzly: upgrade to 0.2.
Thu, Mar 23, 2:52 PM · Observability-Metrics
herron closed T332893: Grizzly: FATA[0000] recursion did not resolve in a valid Kubernetes object., a subtask of T332892: Grizzly: upgrade to 0.2, as Resolved.
Thu, Mar 23, 2:50 PM · Observability-Metrics
herron closed T332893: Grizzly: FATA[0000] recursion did not resolve in a valid Kubernetes object. as Resolved.

This deployed as a NOOP on 0.1 (production) and on 0.2 subcommands are no longer throwing recursion did not resolve errors. I think we're good here

Thu, Mar 23, 2:50 PM · Observability-Metrics
herron created T332895: Grizzly: folder attributes changing under 0.2.
Thu, Mar 23, 2:40 PM · Observability-Metrics
herron renamed T332893: Grizzly: FATA[0000] recursion did not resolve in a valid Kubernetes object. from FATA[0000] recursion did not resolve in a valid Kubernetes object. to Grizzly: FATA[0000] recursion did not resolve in a valid Kubernetes object..
Thu, Mar 23, 2:28 PM · Observability-Metrics
herron updated the task description for T332893: Grizzly: FATA[0000] recursion did not resolve in a valid Kubernetes object..
Thu, Mar 23, 2:19 PM · Observability-Metrics
herron triaged T332893: Grizzly: FATA[0000] recursion did not resolve in a valid Kubernetes object. as Medium priority.
Thu, Mar 23, 2:18 PM · Observability-Metrics
herron created T332893: Grizzly: FATA[0000] recursion did not resolve in a valid Kubernetes object..
Thu, Mar 23, 2:17 PM · Observability-Metrics
herron triaged T332892: Grizzly: upgrade to 0.2 as Medium priority.
Thu, Mar 23, 2:15 PM · Observability-Metrics

Mon, Mar 20

herron closed T332447: Grizzly: onboard home dashboard, a subtask of T331656: Grizzly: onboard "popular" dashboards as static json managed dashboards, as Resolved.
Mon, Mar 20, 8:40 PM · grizzly-sprint, Observability-Metrics
herron closed T332447: Grizzly: onboard home dashboard as Resolved.
Mon, Mar 20, 8:40 PM · grizzly-sprint

Fri, Mar 17

herron added a subtask for T331656: Grizzly: onboard "popular" dashboards as static json managed dashboards: T332447: Grizzly: onboard home dashboard.
Fri, Mar 17, 8:00 PM · grizzly-sprint, Observability-Metrics
herron added a parent task for T332447: Grizzly: onboard home dashboard: T331656: Grizzly: onboard "popular" dashboards as static json managed dashboards.
Fri, Mar 17, 8:00 PM · grizzly-sprint
herron triaged T332447: Grizzly: onboard home dashboard as Medium priority.
Fri, Mar 17, 7:59 PM · grizzly-sprint
herron added subtasks for T331656: Grizzly: onboard "popular" dashboards as static json managed dashboards: T332446: Grizzly: onboard kafka dashboard, T332445: Grizzly: onboard mail dashboard, T332444: Grizzly: onboard confd dashboard, T332443: Grizzly: onboard appservers red dashboard, T332442: Grizzly: onboard host-overview dashboard.
Fri, Mar 17, 7:58 PM · grizzly-sprint, Observability-Metrics
herron added a parent task for T332446: Grizzly: onboard kafka dashboard: T331656: Grizzly: onboard "popular" dashboards as static json managed dashboards.
Fri, Mar 17, 7:58 PM · grizzly-sprint
herron added a parent task for T332443: Grizzly: onboard appservers red dashboard: T331656: Grizzly: onboard "popular" dashboards as static json managed dashboards.
Fri, Mar 17, 7:58 PM · grizzly-sprint
herron added a parent task for T332444: Grizzly: onboard confd dashboard: T331656: Grizzly: onboard "popular" dashboards as static json managed dashboards.
Fri, Mar 17, 7:58 PM · grizzly-sprint
herron added a parent task for T332442: Grizzly: onboard host-overview dashboard: T331656: Grizzly: onboard "popular" dashboards as static json managed dashboards.
Fri, Mar 17, 7:58 PM · grizzly-sprint
herron added a parent task for T332445: Grizzly: onboard mail dashboard: T331656: Grizzly: onboard "popular" dashboards as static json managed dashboards.
Fri, Mar 17, 7:58 PM · grizzly-sprint
herron updated the task description for T331656: Grizzly: onboard "popular" dashboards as static json managed dashboards.
Fri, Mar 17, 7:57 PM · grizzly-sprint, Observability-Metrics
herron triaged T332446: Grizzly: onboard kafka dashboard as Medium priority.
Fri, Mar 17, 7:56 PM · grizzly-sprint
herron triaged T332445: Grizzly: onboard mail dashboard as Medium priority.
Fri, Mar 17, 7:56 PM · grizzly-sprint
herron triaged T332444: Grizzly: onboard confd dashboard as Medium priority.
Fri, Mar 17, 7:56 PM · grizzly-sprint
herron triaged T332443: Grizzly: onboard appservers red dashboard as Medium priority.
Fri, Mar 17, 7:56 PM · grizzly-sprint
herron triaged T332442: Grizzly: onboard host-overview dashboard as Medium priority.
Fri, Mar 17, 7:56 PM · grizzly-sprint

Thu, Mar 16

herron added a project to T331659: Grizzly: CI improvements: grizzly-sprint.
Thu, Mar 16, 7:12 PM · Patch-For-Review, grizzly-sprint, Observability-Metrics

Wed, Mar 15

herron updated the task description for T324725: Observability Bullseye upgrades.
Wed, Mar 15, 9:10 PM · SRE Observability (FY2022/2023-Q3)
herron closed T326420: Kafka-logging Bullseye Upgrades as Resolved.
Wed, Mar 15, 9:10 PM · SRE Observability (FY2022/2023-Q3), Observability-Logging, User-herron
herron closed T326420: Kafka-logging Bullseye Upgrades, a subtask of T324725: Observability Bullseye upgrades, as Resolved.
Wed, Mar 15, 9:10 PM · SRE Observability (FY2022/2023-Q3)
herron updated the task description for T326420: Kafka-logging Bullseye Upgrades.
Wed, Mar 15, 9:09 PM · SRE Observability (FY2022/2023-Q3), Observability-Logging, User-herron
herron updated the task description for T326420: Kafka-logging Bullseye Upgrades.
Wed, Mar 15, 3:08 PM · SRE Observability (FY2022/2023-Q3), Observability-Logging, User-herron
herron updated the task description for T330165: eqiad row B switches upgrade.
Wed, Mar 15, 2:25 PM · Patch-For-Review, Data Pipelines, Data-Engineering-Planning, DBA, Discovery-Search (Current work), SRE, serviceops, cloud-services-team, Machine-Learning-Team, Platform Engineering, SRE Observability, Infrastructure-Foundations, serviceops-collab, Traffic

Tue, Mar 14

herron updated the task description for T326420: Kafka-logging Bullseye Upgrades.
Tue, Mar 14, 6:52 PM · SRE Observability (FY2022/2023-Q3), Observability-Logging, User-herron
herron updated the task description for T331882: eqiad row C switches upgrade.
Tue, Mar 14, 3:29 PM · Patch-For-Review, serviceops-radar, Discovery-Search (Current work), SRE, DBA, cloud-services-team, Traffic, Infrastructure-Foundations, Machine-Learning-Team, Data-Engineering, serviceops-collab, Platform Engineering, SRE Observability

Mon, Mar 13

herron placed T331879: Investigate methods to rate-limit/discard excessive log messages closer to the producer up for grabs.
Mon, Mar 13, 2:59 PM · Wikimedia-Logstash, Observability-Logging, SRE
herron created T331879: Investigate methods to rate-limit/discard excessive log messages closer to the producer.
Mon, Mar 13, 2:59 PM · Wikimedia-Logstash, Observability-Logging, SRE

Thu, Mar 9

herron triaged T331659: Grizzly: CI improvements as Medium priority.
Thu, Mar 9, 6:39 PM · Patch-For-Review, grizzly-sprint, Observability-Metrics
herron triaged T331656: Grizzly: onboard "popular" dashboards as static json managed dashboards as Medium priority.
Thu, Mar 9, 6:26 PM · grizzly-sprint, Observability-Metrics

Mon, Mar 6

herron updated the task description for T329073: eqiad row A switches upgrade.
Mon, Mar 6, 8:01 PM · Patch-For-Review, Discovery-Search (Current work), Shared-Data-Infrastructure, Data-Engineering-Planning, DBA, SRE, Platform Engineering, Infrastructure-Foundations, Traffic, serviceops, Machine-Learning-Team, cloud-services-team, Data-Persistence, SRE Observability, serviceops-collab
herron updated the task description for T329073: eqiad row A switches upgrade.
Mon, Mar 6, 5:24 PM · Patch-For-Review, Discovery-Search (Current work), Shared-Data-Infrastructure, Data-Engineering-Planning, DBA, SRE, Platform Engineering, Infrastructure-Foundations, Traffic, serviceops, Machine-Learning-Team, cloud-services-team, Data-Persistence, SRE Observability, serviceops-collab

Fri, Mar 3

herron updated the task description for T329073: eqiad row A switches upgrade.
Fri, Mar 3, 3:10 PM · Patch-For-Review, Discovery-Search (Current work), Shared-Data-Infrastructure, Data-Engineering-Planning, DBA, SRE, Platform Engineering, Infrastructure-Foundations, Traffic, serviceops, Machine-Learning-Team, cloud-services-team, Data-Persistence, SRE Observability, serviceops-collab

Wed, Mar 1

herron added a comment to T324335: Remove logstash from the Search Elasticsearch servers.

There are a couple issues.

One is in rsyslog where 50-udp-json-logback-compat.conf loads mmjsonparse and omkafka in conflict with 30-output-kafka.conf.

Feb 16 22:08:12 relforge1003 rsyslogd[2102764]: module 'mmjsonparse' already in this config, cannot be added  [v8.2102.0 try https://www.rsyslog.com/e/2221 ]
Feb 16 22:08:12 relforge1003 rsyslogd[2102764]: module 'omkafka' already in this config, cannot be added  [v8.2102.0 try https://www.rsyslog.com/e/2221 ]
Wed, Mar 1, 2:39 PM · observability, Observability-Logging, Discovery-Search (Current work)

Feb 23 2023

herron added a comment to T324470: SRE/Oncall/Schedule: add history support.

The updated version is now managing SRE/Oncall/Schedule, will keep this open for a bit longer to track any bugs

Feb 23 2023, 7:13 PM · User-herron, SRE Observability

Feb 14 2023

herron added a comment to T328707: Update arclamp to active/active architecture.

Supporting multiple read sources would be indeed nice with the main benefit of requiring no actions on switchover. I have looked at the code and something else I didn't consider the other day is the added complexity as we would be moving from a single threaded python process to async/multiple threads. Overall I think it'd be manageable though, for example read threads per-redis all writing to a queue and the main thread in charge of reading the queue and serialize writes to files.

So I was curious how this would look like and gave it a try here (WIP but definitely reviewable, contains kinda-unrelated changes too): https://gerrit.wikimedia.org/r/q/topic:multiple-redis

The gist is that reading from pubsub.subscribe(config.get('redis_channel', 'arclamp')) is changed to read from a queue.Queue. The (bounded) queue is fed by multiple redis, each doing pubsub.subscribe. Semantics stay the same, in the sense that if the queue is empty for a certain time then the process exists. In this case however a single redis losing its connection would make the thread die, so redis reconnections are also handled as a side effect, for improved resiliency (e.g. arclamp-log.py will start up and attempt (re)connections to all redises). For ease of deployment purposes the configuration is backwards-compatible with what we have now. Let me know what you think!

Feb 14 2023, 3:45 PM · Performance-Team, Observability-Tracing

Feb 13 2023

herron added a comment to T293826: flapping icinga Letsencrypt TLS cert alerts around renewal time .
PROBLEM - mailman list info ssl expiry on lists1001 is CRITICAL: CRITICAL - Certificate lists.wikimedia.org expires in 6 day(s) (Mon 20 Feb 2023 05:31:14 AM GMT +0000). https://wikitech.wikimedia.org/wiki/Mailman/Monitoring
Feb 13 2023, 6:43 PM · Upstream, Traffic, observability, SRE

Feb 10 2023

herron closed T320749: SLO dashboards with N latency targets as Resolved.

The updated dynamic SLO dashboard template and config structure is now live. I think we're good here! If any followup is needed please reopen

Feb 10 2023, 4:35 PM · SRE Observability (FY2022/2023-Q3), User-herron, Observability-Metrics, serviceops, observability, SRE, Maps

Feb 8 2023

herron added a comment to T328917: Custom created annotations created in the GUI do not show up in the Grafana graph.

I notice that this dashboard is configured to match built-in annotations by tag. I am also able to reproduce the disappearing annotation when the tags field is left empty while adding the annotation, but it does work for me if I apply one of the tags listed in this filter (screenshot from Dashboard Settings > Annotations > Annotations & Alerts (built-in)

Feb 8 2023, 9:26 PM · Observability-Metrics
colewhite awarded T329232: kafka-logging: ensure cluster wide failure mode alerting coverage a Love token.
Feb 8 2023, 9:12 PM · SRE Observability (FY2022/2023-Q4), Observability-Alerting, Observability-Logging
herron updated the task description for T329232: kafka-logging: ensure cluster wide failure mode alerting coverage.
Feb 8 2023, 8:50 PM · SRE Observability (FY2022/2023-Q4), Observability-Alerting, Observability-Logging
herron updated the task description for T329232: kafka-logging: ensure cluster wide failure mode alerting coverage.
Feb 8 2023, 8:48 PM · SRE Observability (FY2022/2023-Q4), Observability-Alerting, Observability-Logging
herron triaged T329232: kafka-logging: ensure cluster wide failure mode alerting coverage as Medium priority.
Feb 8 2023, 8:48 PM · SRE Observability (FY2022/2023-Q4), Observability-Alerting, Observability-Logging
herron added a comment to T328784: Grafana LDAP sync fails post upgrade.

Excellent! Thanks for doing the user_auth purge!

Feb 8 2023, 2:44 PM · SRE Observability (FY2022/2023-Q3), Observability-Metrics
herron awarded T328784: Grafana LDAP sync fails post upgrade a Love token.
Feb 8 2023, 2:43 PM · SRE Observability (FY2022/2023-Q3), Observability-Metrics

Feb 7 2023

herron added a comment to T328784: Grafana LDAP sync fails post upgrade.

With all that said I think if there's consensus we can:

  • save a backup copy of grafana.db
  • delete all entries from user_auth table

This should bring back to an expected state, thoughts?

Feb 7 2023, 3:28 PM · SRE Observability (FY2022/2023-Q3), Observability-Metrics
herron updated the task description for T327925: codfw row A switches upgrade.
Feb 7 2023, 12:17 PM · Shared-Data-Infrastructure, Data-Engineering-Planning, Discovery-Search (Current work), DBA, serviceops, Traffic, Machine-Learning-Team, serviceops-collab, cloud-services-team, Platform Engineering, SRE Observability, Data-Persistence, SRE, netops, Infrastructure-Foundations

Feb 6 2023

herron added a comment to T328784: Grafana LDAP sync fails post upgrade.

Looking for nearer-term options I found that removing a user from the user_auth table will cause their user entry to no longer show 'synced via oauth' and isExternal attributes, and doing this for user 742 (from the task description) was enough to allow the sync process to complete successfully.

Feb 6 2023, 10:01 PM · SRE Observability (FY2022/2023-Q3), Observability-Metrics
herron added a comment to T328784: Grafana LDAP sync fails post upgrade.

Thanks @jbond that looks ideal, and if we can land on a working config would possibly allow us to simplify the ro/rw domain layout as well.

Feb 6 2023, 8:27 PM · SRE Observability (FY2022/2023-Q3), Observability-Metrics
herron closed T328826: "General / Phabricator" grafana board tracks decommissioned phab1001 rather than active phab1004 as Resolved.

Updated this dashboard to use phab.* in queries and instance labels in legends so this should continue working across host changes in the future (and support multiple hosts)

Feb 6 2023, 3:33 PM · observability, Phabricator

Feb 3 2023

herron added a comment to T328784: Grafana LDAP sync fails post upgrade.

Dug into this a bit, and AFAICT the "synced via oauth" and "User info cannot be updated for external Users" both relate back to isExternal:true on the user object. And isExternal:true does appear to be the case for our users populated by the grafana-ldap-users-sync script.

Feb 3 2023, 8:39 PM · SRE Observability (FY2022/2023-Q3), Observability-Metrics
herron moved T328517: Request for access for stats machines for Santhosh from Untriaged to Manager/NDA Approval/Confirmation on the SRE-Access-Requests board.
Feb 3 2023, 7:44 PM · SRE, SRE-Access-Requests
herron moved T328787: Request for SSH Access for kofori from Untriaged to Awaiting User Input on the SRE-Access-Requests board.
Feb 3 2023, 7:11 PM · SRE, SRE-Access-Requests
herron moved T328733: Requesting access to analytics-privatedata-users for Aisha Khatun from Untriaged to Awaiting User Input on the SRE-Access-Requests board.
Feb 3 2023, 4:03 PM · SRE, SRE-Access-Requests
herron updated subscribers of T328733: Requesting access to analytics-privatedata-users for Aisha Khatun.

Looping in @KFrancis for NDA confirmation as well

Feb 3 2023, 4:01 PM · SRE, SRE-Access-Requests
herron updated the task description for T320702: Jaeger secure access to OpenSearch cluster.
Feb 3 2023, 2:12 PM · SRE Observability (FY2022/2023-Q3), User-fgiunchedi, Observability-Tracing

Feb 2 2023

herron triaged T328707: Update arclamp to active/active architecture as Medium priority.
Feb 2 2023, 9:10 PM · Performance-Team, Observability-Tracing

Feb 1 2023

herron updated subscribers of T328517: Request for access for stats machines for Santhosh .

@Ottomata @odimitrijevic could you please review/approve this request for groupadd to analytics-privatedata-users?

Feb 1 2023, 4:54 PM · SRE, SRE-Access-Requests
herron updated the task description for T328517: Request for access for stats machines for Santhosh .
Feb 1 2023, 4:51 PM · SRE, SRE-Access-Requests

Jan 31 2023

herron edited projects for T328361: Route users to closest bastion host based on IP geolocation, added: Infrastructure-Foundations; removed SRE.
Jan 31 2023, 5:26 PM · Infrastructure-Foundations

Jan 30 2023

herron triaged T328145: Grant Access to 'cn=nda or cn=wmf' for ekalkst as Medium priority.
Jan 30 2023, 3:00 PM · SRE, LDAP-Access-Requests
herron moved T328145: Grant Access to 'cn=nda or cn=wmf' for ekalkst from Backlog to Awaiting User Input on the LDAP-Access-Requests board.
Jan 30 2023, 3:00 PM · SRE, LDAP-Access-Requests
herron closed T328015: Requesting access to analytics-privatedata-users for Abhas as Resolved.

Hi @Abhas, the requested access has been provisioned and will fully propagate across the fleet within 30 minutes.

Jan 30 2023, 2:57 PM · SRE, SRE-Access-Requests

Jan 26 2023

herron updated the task description for T327925: codfw row A switches upgrade.
Jan 26 2023, 6:17 PM · Shared-Data-Infrastructure, Data-Engineering-Planning, Discovery-Search (Current work), DBA, serviceops, Traffic, Machine-Learning-Team, serviceops-collab, cloud-services-team, Platform Engineering, SRE Observability, Data-Persistence, SRE, netops, Infrastructure-Foundations
herron updated the task description for T327925: codfw row A switches upgrade.
Jan 26 2023, 6:16 PM · Shared-Data-Infrastructure, Data-Engineering-Planning, Discovery-Search (Current work), DBA, serviceops, Traffic, Machine-Learning-Team, serviceops-collab, cloud-services-team, Platform Engineering, SRE Observability, Data-Persistence, SRE, netops, Infrastructure-Foundations
herron updated the task description for T327991: codfw row B switches upgrade.
Jan 26 2023, 4:52 PM · Discovery-Search (Current work), SRE, Platform Engineering, serviceops-collab, Infrastructure-Foundations, SRE Observability, Traffic, Machine-Learning-Team, cloud-services-team, Data-Persistence, DBA, serviceops, netops

Jan 24 2023

herron moved T324470: SRE/Oncall/Schedule: add history support from Backlog to Working on on the User-herron board.
Jan 24 2023, 7:04 PM · User-herron, SRE Observability
herron added a comment to T324470: SRE/Oncall/Schedule: add history support.

An updated version of the script that retains shift history is now updating https://wikitech.wikimedia.org/wiki/Sandbox-votesting

Jan 24 2023, 7:03 PM · User-herron, SRE Observability

Jan 20 2023

herron added a comment to T320702: Jaeger secure access to OpenSearch cluster.

Potential option 3: Jaeger outputs to kafka-logging as a buffer, jaeger-ingester (perhaps deployed within the logging cluster) reads from kafka-logging and persists to opensearch

Jan 20 2023, 3:54 PM · SRE Observability (FY2022/2023-Q3), User-fgiunchedi, Observability-Tracing

Jan 19 2023

herron updated the task description for T326419: Expand kafka-logging using hosts kafka-logging[12]00[45].
Jan 19 2023, 3:55 PM · Patch-For-Review, SRE Observability (FY2022/2023-Q3), User-herron, Observability-Logging

Jan 18 2023

herron awarded T313849: Q1:rack/setup/install logstash103[67] a Love token.
Jan 18 2023, 8:12 PM · SRE Observability, observability, ops-eqiad, SRE, DC-Ops

Jan 17 2023

herron added a comment to T247517: Request creation of 'sre-sandbox' VPS project.
  • did the emails informing @herron that the machine was due to be deleted go out correctly
  • where they received by @herron (spam filter etc)
  • why where they not acted upon (possibly not enough notice)
Jan 17 2023, 10:22 PM · cloud-services-team (Kanban), SRE, Cloud-VPS (Project-requests)

Jan 13 2023

herron created T326983: Expose effective prometheus blackbox exporter probe target as label.
Jan 13 2023, 7:43 PM · Observability-Metrics, User-herron
herron added a project to T302995: Explore dedicated (non-grafana) SLO Visualization and Management: User-herron.
Jan 13 2023, 3:11 PM · User-herron, SRE Observability (FY2022/2023-Q3), Observability-Metrics
herron added a project to T313230: Dispatch IRC Integration: User-herron.
Jan 13 2023, 3:09 PM · User-herron, SRE Observability (FY2022/2023-Q3), Incident Tooling
herron moved T318911: certspotter failures on alert1001 from Inbox to Backlog on the SRE Observability board.
Jan 13 2023, 3:04 PM · SRE Observability, SRE
herron moved T304481: kubernetes / calico alerts have instance with fqdn not hostname from Inbox to Backlog on the SRE Observability board.
Jan 13 2023, 3:04 PM · User-fgiunchedi, SRE Observability
herron moved T313849: Q1:rack/setup/install logstash103[67] from Inbox to Radar on the SRE Observability board.
Jan 13 2023, 3:02 PM · SRE Observability, observability, ops-eqiad, SRE, DC-Ops
herron moved T196336: Icinga passive checks go awol and downtime stops working from Inbox to Backlog on the SRE Observability board.
Jan 13 2023, 3:02 PM · SRE Observability, SRE, Icinga, observability
herron moved T316682: [cloudweb] Improve the alerts coming from the LVS setup from Inbox to Radar on the SRE Observability board.
Jan 13 2023, 2:58 PM · cloud-services-team, SRE Observability, Cloud-Services-Worktype-Maintenance, Cloud-Services-Origin-Team, User-dcaro
herron moved T324470: SRE/Oncall/Schedule: add history support from Inbox to Backlog on the SRE Observability board.
Jan 13 2023, 2:56 PM · User-herron, SRE Observability
herron moved T288623: Observability tools are easy to use, docs easy to read, help easy to find. from Inbox to Backlog on the SRE Observability board.
Jan 13 2023, 2:56 PM · SRE Observability
herron removed a project from T317316: Fine-tune dispatch SRE incident template: SRE Observability (FY2022/2023-Q3).
Jan 13 2023, 2:48 PM · User-herron, Incident Tooling
herron triaged T317316: Fine-tune dispatch SRE incident template as Medium priority.
Jan 13 2023, 2:48 PM · User-herron, Incident Tooling
herron added a comment to T317316: Fine-tune dispatch SRE incident template.

I've drafted an updated template available at the link in the description, and this is being used for creation of new incidents in the production dispatch instance.

Jan 13 2023, 2:47 PM · User-herron, Incident Tooling
herron placed T317316: Fine-tune dispatch SRE incident template up for grabs.
Jan 13 2023, 2:43 PM · User-herron, Incident Tooling