Page MenuHomePhabricator
Feed Advanced Search

Yesterday

fgiunchedi added a comment to T249597: Assess approximate total upload size for DPLA projects.

Hi @fgiunchedi. Can you please associate at least one active project with this task (via the Add Action...Change Project Tags dropdown)? This will allow others to get notified, or see this task when searching via projects. Thanks.

Tue, Apr 7, 9:43 AM
fgiunchedi created T249597: Assess approximate total upload size for DPLA projects.
Tue, Apr 7, 9:27 AM
fgiunchedi closed T248151: Big number of uploads from DPLA bot as Resolved.

I'd like to understand better how big of a dataset we're talking about for all uploads that @Dominicbm is working on, although I'll followup in another task.

Tue, Apr 7, 9:24 AM · User-fgiunchedi, Operations, SRE-swift-storage, Commons
fgiunchedi closed T240048: Make grafana-next.wm.o HTTP 302 redirect to grafana.wm.o, a subtask of T220838: Upgrade grafana to 6.4.4, as Resolved.
Tue, Apr 7, 7:53 AM · Patch-For-Review, Performance-Team (Radar), observability, Operations
fgiunchedi closed T240048: Make grafana-next.wm.o HTTP 302 redirect to grafana.wm.o as Resolved.

Tentatively resolving, grafana-next now is the standby / upgrade host

Tue, Apr 7, 7:52 AM · observability, Operations

Mon, Apr 6

fgiunchedi added a comment to T244208: Upgrade Grafana to 6.7.

Grafana 6.7.2 is running on https://grafana-next.wikimedia.org and AFAICT is running as expected. I ran into a problem (couldn't login anymore) after the database sync happened with grafana1002, although that is expected I believe. A restart of grafana-server fixed the issue temporarily.

Mon, Apr 6, 2:16 PM · Patch-For-Review, User-CDanis, Operations, observability
fgiunchedi moved T85624: Better abstractions for puppet & icinga/nagios/shinken from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:44 PM · Operations, observability
fgiunchedi moved T136311: Monitor the BMC's event log for hardware errors from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:44 PM · observability, Operations
fgiunchedi moved T222826: Leverage Grafana annotations to show events in graphs from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:41 PM · observability, Operations
fgiunchedi moved T224692: mx1001 exim queue warning from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:41 PM · Operations, observability
fgiunchedi moved T228970: Test memsniff as possible replacement of memkeys from Inbox to Radar on the observability board.
Mon, Apr 6, 12:41 PM · User-Elukey, observability, Operations, serviceops
fgiunchedi closed T174960: Varnish does not vary elasticsearch query by request body as Declined.

Tentatively resolving since things are working as intended

Mon, Apr 6, 12:39 PM · observability, Operations, Traffic, Wikimedia-Logstash
fgiunchedi moved T223483: Logstash stops processing messages if a single output becomes blocked from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:37 PM · observability, Operations, Wikimedia-Logstash
fgiunchedi moved T231274: Have a link to the alert in the icinga alert email from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:37 PM · Icinga, observability, Operations
fgiunchedi moved T225125: Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:36 PM · Patch-For-Review, Discovery-Search (Current work), observability, Elasticsearch, Operations, Wikimedia-Logstash
fgiunchedi moved T227080: Deprecate all non-Kafka logstash inputs from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:36 PM · Patch-For-Review, observability, Wikimedia-Logstash, User-fgiunchedi, Operations
fgiunchedi moved T205856: Retire udp2log: onboard its producers and consumers to the logging pipeline from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:36 PM · Analytics, observability, Wikimedia-Logstash, Operations
fgiunchedi moved T231079: check_prometheus_metric: show errors in icinga, make "warning" optional from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:36 PM · observability
fgiunchedi moved T233089: Export zuul metrics to Prometheus from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:35 PM · Patch-For-Review, Continuous-Integration-Infrastructure, Release-Engineering-Team (CI & Testing services), Release-Engineering-Team-TODO, observability, Operations
fgiunchedi moved T233474: Ensure graphs used by Performance account for Varnish-to-ATS migration from Inbox to Radar on the observability board.
Mon, Apr 6, 12:35 PM · Traffic, Performance-Team, Operations, observability
fgiunchedi moved T178690: Better organization for SRE grafana dashboards from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:35 PM · User-CDanis, Patch-For-Review, User-fgiunchedi, observability, Operations
fgiunchedi moved T243065: Provision plaintext syslog collectors in esams/ulsfo/eqsin from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:34 PM · netops, observability, Operations
fgiunchedi moved T244624: Beta puppet patch "prometheus: make ferm DNS record type configurable" from Inbox to Radar on the observability board.
Mon, Apr 6, 12:34 PM · Beta-Cluster-Infrastructure, Operations, observability
fgiunchedi moved T222795: Re-evaluate service-runner's (ab)use of statsd timing metric for nodejs GC stats from Inbox to Radar on the observability board.
Mon, Apr 6, 12:34 PM · observability, serviceops-radar, Patch-For-Review, Services (later), service-runner, Operations
fgiunchedi moved T247962: Migrate role::prometheus to Buster from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:33 PM · observability
fgiunchedi moved T247963: Migrate role::graphite::production to Buster from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:33 PM · observability
fgiunchedi moved T247966: Migrate role::alerting_host to Buster from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:33 PM · observability
fgiunchedi moved T247967: Migrate role::netmon to Buster from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:33 PM · netops, observability, Operations
fgiunchedi moved T247968: Migrate logging::webrequest::ops to Buster from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:33 PM · observability
fgiunchedi moved T233956: Deploy Thanos (long-term storage) stateless components: sidecar and query from Inbox to Backlog on the observability board.
Mon, Apr 6, 12:33 PM · Patch-For-Review, User-fgiunchedi, Goal, observability
fgiunchedi added a comment to T240048: Make grafana-next.wm.o HTTP 302 redirect to grafana.wm.o.

Different approach/idea: point grafana-next to standby grafana: https://gerrit.wikimedia.org/r/c/operations/puppet/+/586344

Mon, Apr 6, 12:31 PM · observability, Operations
fgiunchedi removed a project from T243129: deployment-cache-upload05: Several millions of logstash error entries: observability.

Untagging observability as this is a service / beta issue (also seems fixed?)

Mon, Apr 6, 12:30 PM · Operations, Beta-Cluster-Infrastructure
fgiunchedi moved T247820: Decide on `service-runner` aggregated prometheus metrics and use of `service` label from Inbox to In progress on the observability board.
Mon, Apr 6, 12:27 PM · Performance-Team (Radar), observability, Operations
fgiunchedi updated the task description for T233956: Deploy Thanos (long-term storage) stateless components: sidecar and query.
Mon, Apr 6, 7:45 AM · Patch-For-Review, User-fgiunchedi, Goal, observability

Fri, Apr 3

fgiunchedi updated subscribers of T249311: Deploy Prometheus Push Gateway.
Fri, Apr 3, 8:21 AM · observability
fgiunchedi created T249311: Deploy Prometheus Push Gateway.
Fri, Apr 3, 8:20 AM · observability

Thu, Apr 2

fgiunchedi moved T189333: Changing Kibana filters is ridiculously slow from Backlog to Radar on the User-fgiunchedi board.
Thu, Apr 2, 10:48 AM · Developer Productivity, User-fgiunchedi, observability, Traffic, Operations, User-Addshore, Wikimedia-Logstash
fgiunchedi moved T233956: Deploy Thanos (long-term storage) stateless components: sidecar and query from Backlog to Up next on the User-fgiunchedi board.
Thu, Apr 2, 10:48 AM · Patch-For-Review, User-fgiunchedi, Goal, observability
fgiunchedi closed T236075: Evaluate, suggest and choose an alert escalation solution as Resolved.

VO is the selected tool, resolving

Thu, Apr 2, 10:28 AM · User-fgiunchedi, observability
fgiunchedi closed T246997: smartd not starting properly on gen9 + buster as Resolved.

smartd.service will be masked on Buster + hpsa, resolving

Thu, Apr 2, 10:27 AM · User-fgiunchedi, Operations
fgiunchedi added a project to T86969: Send scap log directly to logstash via syslog input: User-fgiunchedi.
Thu, Apr 2, 9:25 AM · User-fgiunchedi, Patch-For-Review, Scap
fgiunchedi added a comment to T247820: Decide on `service-runner` aggregated prometheus metrics and use of `service` label.

e.g. golang services), and we seem to be fine without it?

If they were using statsd, they'd wouldn't be fine without it, right? If we weren't using k8s, they wouldn't be fine without it?
service-runner is a generic nodejs library that can and is run without prometheus and without k8s.

Thu, Apr 2, 9:23 AM · Performance-Team (Radar), observability, Operations
fgiunchedi added a comment to T246793: Scap no longer sets 'level' in Logstash messages.

this looks like it's likely the culprit: rMSCA3ca8820b7a68: Support for logging json to syslog

@fgiunchedi do you think ^ is a likely cause of this regression?

Thu, Apr 2, 9:10 AM · Developer Productivity, observability, Regression, Release-Engineering-Team-TODO, Scap
fgiunchedi updated subscribers of T249174: ms-be1023 crashed / Smart Storage Battery failure.

Thanks @Volans for taking a look! Indeed seems like the battery failed, maybe we can try a reseat first @Cmjohnson @Jclark-ctr next time you get a chance? The host *seems* otherwise fine to me in terms of normal operation.

Thu, Apr 2, 9:05 AM · Operations, ops-eqiad, SRE-swift-storage
fgiunchedi renamed T249174: ms-be1023 crashed / Smart Storage Battery failure from ms-be1023 crashed to ms-be1023 crashed / Smart Storage Battery failure.
Thu, Apr 2, 9:04 AM · Operations, ops-eqiad, SRE-swift-storage

Tue, Mar 31

fgiunchedi added a comment to T247820: Decide on `service-runner` aggregated prometheus metrics and use of `service` label.

Another example:

chart: eventgate
app: eventgate-wikimedia # (or maybe just eventgate?)
service: eventgate-analytics
chart: eventgate
app: eventgate-wikimedia # (or maybe just eventgate?)
service: eventgate-main
# etc.

Should the application add service among its labels for Prometheus metrics?

I'd argue that it is necessary for the work that Cole is doing in service-runner for the app to know the service name. Filippo has made a point that it might not be necessary for prometheus metrics, as the service label can be added by the prometheus server side configs. However, the metrics abstraction makes the service-runner metrics API look the same for both statsd and prometheus, and statsd needs the service name in its metric names (since it doesn't have labels). We could manually avoid setting it as a prometheus label, but that seems a bit awkward and would add some special case code needed to be added to service-runner to do this.
service-runner should be WMF infrastructure agnostic, so I think adding the service name label here makes sense. We can and should make k8s exporter and prometheus server added labels override any app provided ones.

Tue, Mar 31, 1:28 PM · Performance-Team (Radar), observability, Operations

Mon, Mar 30

fgiunchedi triaged T239090: Restbase logging indexing conflict on 'res' and 'body' logging fields as High priority.

@fgiunchedi can you update the priority for this task?

Mon, Mar 30, 2:44 PM · Core Platform Team Workboards (Clinic Duty Team), User-fgiunchedi, Wikimedia-Logstash, RESTBase
fgiunchedi moved T234900: Setup bacula backup monitoring from Inbox to Radar on the observability board.
Mon, Mar 30, 2:33 PM · Patch-For-Review, Availability, observability, Goal, Operations
fgiunchedi closed T239837: prometheus hosts try to start rsync and fails on every puppet run as Invalid.

Has been fixed since

Mon, Mar 30, 2:30 PM · observability
fgiunchedi closed T240379: Explicitly state the timezone in grafana as Invalid.

Not a bug, timezone is displayed

Mon, Mar 30, 2:26 PM · Upstream, observability
fgiunchedi closed T152445: Move prometheus entry point off port 80 as Invalid.

We're moving Prometheus on its own dedicated hosts everywhere, I see no reason not to leave the current entry point as is now (also we moved to apache in the meantime)

Mon, Mar 30, 2:19 PM · observability, Prometheus-metrics-monitoring, Operations
fgiunchedi closed T247538: Icinga latency is skyrocketing and commands ignored as Resolved.

With https://gerrit.wikimedia.org/r/580985 merged I'm resolving this task since check latency is doing better now, and we're alerting on excessive latency.

Mon, Mar 30, 2:13 PM · User-fgiunchedi, fundraising-tech-ops, observability, Operations
fgiunchedi added a comment to T248858: Create prometheus metrics for Maps OSM data disk usage.

We have https://github.com/wrouesnel/postgres_exporter deployed on the maps hosts, I believe some/all of the metrics you are looking for are available in grafana/prometheus. You can get a preview of those from the host itself if you wish: curl -s localhost:9187/metrics or use Grafana's "explore" function (while logged in), the metrics will start with pg_, hope that helps!

Mon, Mar 30, 2:00 PM · Product-Infrastructure-Team-Backlog (Kanban), Maps (Maps-data)
fgiunchedi moved T248151: Big number of uploads from DPLA bot from Backlog to Radar on the User-fgiunchedi board.
Mon, Mar 30, 9:55 AM · User-fgiunchedi, Operations, SRE-swift-storage, Commons
fgiunchedi moved T217142: [WIP] [Proposal] Use the Kafka-Logstash logging infrastructure to log client-side errors from Up next to Doing on the User-fgiunchedi board.
Mon, Mar 30, 9:54 AM · observability, User-fgiunchedi, Better Use Of Data, MW-1.34-notes (1.34.0-wmf.15; 2019-07-23), Patch-For-Review, User-herron, Product-Infrastructure-Team-Backlog, Wikimedia-Logstash

Thu, Mar 26

fgiunchedi lowered the priority of T247538: Icinga latency is skyrocketing and commands ignored from High to Medium.

Lowering priority as things I believe are better now, pending https://gerrit.wikimedia.org/r/c/operations/puppet/+/580985 as the last attempt at lowering check latency further.

Thu, Mar 26, 1:58 PM · User-fgiunchedi, fundraising-tech-ops, observability, Operations
fgiunchedi added a comment to T248174: Request increased quota for monitoring Cloud VPS project.

Thanks @Andrew ! Appreciate it

Thu, Mar 26, 9:00 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)

Tue, Mar 24

fgiunchedi updated subscribers of T247820: Decide on `service-runner` aggregated prometheus metrics and use of `service` label.

Myself, @akosiaris @colewhite and @Ottomata met today to bikesh^W understand better what service means and other related labels.

Tue, Mar 24, 4:26 PM · Performance-Team (Radar), observability, Operations

Mon, Mar 23

fgiunchedi updated the task description for T248174: Request increased quota for monitoring Cloud VPS project.
Mon, Mar 23, 10:09 AM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
fgiunchedi added a project to T248151: Big number of uploads from DPLA bot: User-fgiunchedi.
Mon, Mar 23, 9:46 AM · User-fgiunchedi, Operations, SRE-swift-storage, Commons

Fri, Mar 20

fgiunchedi updated subscribers of T248151: Big number of uploads from DPLA bot.

Summary of the IRC chat: the current batch of uploads is about halfway finished and will likely be done by early next week, although no byte size estimates are available. Bots don't seem to have upload rate limits enforced (thanks @Reedy) which I filed as T248177. The one file per page approach is fine as is, depending on the source we do get cases like that.

Fri, Mar 20, 3:00 PM · User-fgiunchedi, Operations, SRE-swift-storage, Commons
fgiunchedi created T248177: Enforce upload rate limits for bots on commons.
Fri, Mar 20, 2:35 PM · Wikimedia-Site-requests, Patch-For-Review, Operations, Commons
fgiunchedi created T248174: Request increased quota for monitoring Cloud VPS project.
Fri, Mar 20, 1:39 PM · cloud-services-team (Kanban), Cloud-VPS (Quota-requests)
fgiunchedi moved T246030: Enable client side error logging in prod for small wiki from Backlog to Radar on the User-fgiunchedi board.
Fri, Mar 20, 1:32 PM · MW-1.35-notes (1.35.0-wmf.23; 2020-03-10), Product-Infrastructure-Team-Backlog (Kanban), Patch-For-Review, Analytics-Kanban, Performance-Team (Radar), Desktop Improvements, observability, Wikimedia-Logstash, User-fgiunchedi, Better Use Of Data, Epic, Analytics
fgiunchedi moved T246997: smartd not starting properly on gen9 + buster from Backlog to Doing on the User-fgiunchedi board.
Fri, Mar 20, 1:31 PM · User-fgiunchedi, Operations
fgiunchedi moved T247538: Icinga latency is skyrocketing and commands ignored from Backlog to Doing on the User-fgiunchedi board.
Fri, Mar 20, 1:31 PM · User-fgiunchedi, fundraising-tech-ops, observability, Operations
fgiunchedi added a project to T246997: smartd not starting properly on gen9 + buster: User-fgiunchedi.
Fri, Mar 20, 1:31 PM · User-fgiunchedi, Operations
fgiunchedi added a comment to T248151: Big number of uploads from DPLA bot.

Hi, this is me! 😳If it's easier, I can get on Telegram or IRC to chat with you about my project. Obviously, I've been going at a high rate, but I don't really want to break Wikimedia!

Fri, Mar 20, 1:28 PM · User-fgiunchedi, Operations, SRE-swift-storage, Commons
fgiunchedi added a project to T247538: Icinga latency is skyrocketing and commands ignored: User-fgiunchedi.
Fri, Mar 20, 1:25 PM · User-fgiunchedi, fundraising-tech-ops, observability, Operations
fgiunchedi renamed T244208: Upgrade Grafana to 6.7 from Upgrade Grafana to 6.6 to Upgrade Grafana to 6.7.
Fri, Mar 20, 1:21 PM · Patch-For-Review, User-CDanis, Operations, observability
fgiunchedi added a comment to T248131: Prometheus jobs reduced availability alerts for Icinga exporter.

The flapping seems to have started with the latest version of the exporter AFAICS (around March 18 end of UTC day) maybe that's a lead too? Also it happens in codfw exclusively, I'm assuming due to the periodic icinga restarts/reloads we do there

Fri, Mar 20, 9:22 AM · observability
fgiunchedi created T248151: Big number of uploads from DPLA bot.
Fri, Mar 20, 9:09 AM · User-fgiunchedi, Operations, SRE-swift-storage, Commons

Thu, Mar 19

fgiunchedi placed T111540: Clean up labs graphite datapoints up for grabs.
Thu, Mar 19, 3:47 PM · cloud-services-team (Kanban), Operations, Cloud-VPS, Shinken, Graphite
fgiunchedi placed T86546: graphite-web logs are not rotated up for grabs.
Thu, Mar 19, 3:46 PM · audits-data-retention, Operations, Graphite
fgiunchedi placed T88997: Improve graphite failover up for grabs.
Thu, Mar 19, 3:46 PM · Performance-Team (Radar), Operations, Graphite
fgiunchedi placed T95429: rsync errors slowing down object-replicator up for grabs.
Thu, Mar 19, 3:45 PM · Operations, SRE-swift-storage
fgiunchedi closed T99233: limit the impact of many new metrics being pushed to graphite as Declined.

Not relevant anymore as we're dialing down our graphite usage across the board

Thu, Mar 19, 3:45 PM · Operations, Graphite
fgiunchedi closed T99234: improve graphite operational documentation as Resolved.

Docs have been expanded and available at https://wikitech.wikimedia.org/wiki/Graphite

Thu, Mar 19, 3:44 PM · Operations, Graphite
fgiunchedi closed T101141: UDP rcvbuferrors and inerrors on graphite hosts, a subtask of T105218: check_graphite - "UNKNOWN: More than half of the datapoints are undefined ", as Resolved.
Thu, Mar 19, 3:43 PM · Patch-For-Review, Operations, HHVM, Graphite, observability
fgiunchedi closed T101141: UDP rcvbuferrors and inerrors on graphite hosts as Resolved.

Resolving since we have significantly lessened the load of udp traffic

Thu, Mar 19, 3:43 PM · observability, MW-1.27-release-notes, MW-1.27-release (WMF-deploy-2016-04-12_(1.27.0-wmf.21)), Operations, Graphite
fgiunchedi placed T102575: document graphite failover/backfill procedures up for grabs.
Thu, Mar 19, 3:42 PM · Documentation, Operations, Graphite
fgiunchedi closed T119718: Make it easier to ban misbehaving dashboards from graphite as Declined.

Declining as we haven't been experiencing this problem anymore (less dashboards on graphite)

Thu, Mar 19, 3:42 PM · Graphite, Operations
fgiunchedi placed T119719: Enforce a minimum refresh period for grafana dashboards hitting graphite up for grabs.
Thu, Mar 19, 3:41 PM · Operations, Graphite
fgiunchedi added a comment to T244208: Upgrade Grafana to 6.7.

See also T119719 when Grafana 6.7 is released

Thu, Mar 19, 3:39 PM · Patch-For-Review, User-CDanis, Operations, observability
fgiunchedi added a comment to T119719: Enforce a minimum refresh period for grafana dashboards hitting graphite.

GH issue is resolved, and the feature will be available in Grafana 6.7: https://github.com/grafana/grafana/blob/master/CHANGELOG.md#670-beta1-2020-03-12

Thu, Mar 19, 3:38 PM · Operations, Graphite
fgiunchedi placed T130709: authoritative copy of 'root' files for upload.wikimedia.org is only in swift up for grabs.
Thu, Mar 19, 3:36 PM · SRE-swift-storage, Operations
fgiunchedi placed T138821: extend existing graphite whisper files retention to five years up for grabs.
Thu, Mar 19, 3:35 PM · Graphite, Operations
fgiunchedi placed T169316: Thumbor should alert/page when thumbs aren't rendered up for grabs.
Thu, Mar 19, 3:34 PM · Performance-Team (Radar), Icinga, observability, Thumbor
fgiunchedi placed T159830: Sanity check global-multiwrite logs for ConfirmEdit usage up for grabs.
Thu, Mar 19, 3:33 PM · Operations, SRE-swift-storage
fgiunchedi created T248093: Renew certs for mcrouter on all application servers..
Thu, Mar 19, 2:09 PM · Operations, serviceops
fgiunchedi removed a project from T247968: Migrate logging::webrequest::ops to Buster: Epic.
Thu, Mar 19, 10:15 AM · observability
fgiunchedi removed a project from T247967: Migrate role::netmon to Buster: Epic.
Thu, Mar 19, 10:15 AM · netops, observability, Operations
fgiunchedi removed a project from T247966: Migrate role::alerting_host to Buster: Epic.
Thu, Mar 19, 10:15 AM · observability
fgiunchedi removed a project from T247963: Migrate role::graphite::production to Buster: Epic.
Thu, Mar 19, 10:15 AM · observability
fgiunchedi added a comment to T247820: Decide on `service-runner` aggregated prometheus metrics and use of `service` label.

Good idea forking the original task. Thanks for that!

Thu, Mar 19, 10:13 AM · Performance-Team (Radar), observability, Operations

Wed, Mar 18

fgiunchedi added a comment to T247538: Icinga latency is skyrocketing and commands ignored.

Top 50 checks as of today, with a little longer time horizon than the previous audit

Wed, Mar 18, 3:30 PM · User-fgiunchedi, fundraising-tech-ops, observability, Operations
fgiunchedi created T247968: Migrate logging::webrequest::ops to Buster.
Wed, Mar 18, 12:41 PM · observability
fgiunchedi created T247967: Migrate role::netmon to Buster.
Wed, Mar 18, 12:40 PM · netops, observability, Operations
fgiunchedi edited projects for T247963: Migrate role::graphite::production to Buster, added: observability; removed Operations.
Wed, Mar 18, 12:38 PM · observability
fgiunchedi edited projects for T247966: Migrate role::alerting_host to Buster, added: observability; removed Operations.
Wed, Mar 18, 12:38 PM · observability