Page MenuHomePhabricator
Feed Advanced Search

Today

fgiunchedi updated the task description for T266019: Several days of metrics not uploaded to Thanos object storage from Prometheus on PoPs.
Thu, Oct 22, 3:56 PM · User-fgiunchedi, observability
fgiunchedi added a comment to T246004: Spike: Can/should Swift be used as Flink checkpoint backend?.

Thank you all for swift (pun intended) action!

Thu, Oct 22, 1:29 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service, Epic
fgiunchedi reopened T221904: swift backend decomms / rebalances are noisy as "Open".

Unfortunately reopening, we've been seeing failures (e.g. systemd, ssh) during latest codfw rebalances

Thu, Oct 22, 12:14 PM · observability, SRE-swift-storage, Operations
fgiunchedi assigned T266214: Degraded RAID on ms-be2017 to Papaul.

@Papaul this disk is busted, it is a 4TB drive, please replace with a spare, the led should be blinking. Thank you!

Thu, Oct 22, 8:37 AM · SRE-swift-storage, Operations, ops-codfw

Yesterday

fgiunchedi updated the task description for T261281: Improve performance of Thanos (+ Prometheus).
Wed, Oct 21, 12:47 PM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi moved T265649: PuppetDB grafana graphs not matching logs from Doing to Radar on the User-fgiunchedi board.
Wed, Oct 21, 12:43 PM · User-fgiunchedi, User-jbond, Operations, observability, Puppet
fgiunchedi committed rLPRI61531417ca29: Add wqds swift account (authored by fgiunchedi).
Add wqds swift account
Wed, Oct 21, 8:30 AM

Tue, Oct 20

fgiunchedi updated the task description for T266019: Several days of metrics not uploaded to Thanos object storage from Prometheus on PoPs.
Tue, Oct 20, 2:49 PM · User-fgiunchedi, observability
fgiunchedi closed T234698: ms-be1020 - firmware upgrade: (was: host went down) as Declined.

Not really, host is going to be decom'd soon.

Tue, Oct 20, 2:00 PM · ops-eqiad, SRE-swift-storage, Operations
fgiunchedi moved T266016: Refresh and expand Swift hardware capacity from Backlog to Doing on the User-fgiunchedi board.
Tue, Oct 20, 1:55 PM · User-fgiunchedi, SRE-swift-storage
fgiunchedi moved T266017: Implement alerting roadmap phase 2 from Backlog to Up next on the User-fgiunchedi board.
Tue, Oct 20, 1:55 PM · User-fgiunchedi, observability
fgiunchedi moved T266019: Several days of metrics not uploaded to Thanos object storage from Prometheus on PoPs from Backlog to Up next on the User-fgiunchedi board.
Tue, Oct 20, 1:55 PM · User-fgiunchedi, observability
fgiunchedi closed T163673: Some swift disks wrongly mounted on 5 ms-be hosts as Resolved.

Yes let's resolve, thanks @Marostegui

Tue, Oct 20, 1:54 PM · Patch-Needs-Improvement, User-fgiunchedi, Operations
fgiunchedi closed T163673: Some swift disks wrongly mounted on 5 ms-be hosts, a subtask of T160640: Rack and Setup ms-be1028-ms-1039, as Resolved.
Tue, Oct 20, 1:54 PM · Patch-For-Review, User-fgiunchedi, Operations
fgiunchedi created T266019: Several days of metrics not uploaded to Thanos object storage from Prometheus on PoPs.
Tue, Oct 20, 1:53 PM · User-fgiunchedi, observability
fgiunchedi closed T265632: Alert on Thanos sidecar not uploading blocks as Resolved.

This is complete!

Tue, Oct 20, 1:50 PM · observability
fgiunchedi added a comment to T246004: Spike: Can/should Swift be used as Flink checkpoint backend?.

I don't know! @fgiunchedi how does one access the cluster? @elukey can check the network VLAN ACLs and update accordingly.

Tue, Oct 20, 1:47 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service, Epic
fgiunchedi created T266017: Implement alerting roadmap phase 2.
Tue, Oct 20, 1:37 PM · User-fgiunchedi, observability
fgiunchedi added a parent task for T261633: Put ms-be2057 (Dell R740xd2) in service: T266016: Refresh and expand Swift hardware capacity.
Tue, Oct 20, 1:34 PM · Patch-For-Review, User-fgiunchedi, Operations, SRE-swift-storage
fgiunchedi added a parent task for T265093: (Need By: TBD) rack/setup/install ms-be106[0-3]: T266016: Refresh and expand Swift hardware capacity.
Tue, Oct 20, 1:34 PM · Operations, ops-eqiad, DC-Ops
fgiunchedi added a parent task for T265419: (Need By: TBD) rack/setup/install ms-be20[58-61]: T266016: Refresh and expand Swift hardware capacity.
Tue, Oct 20, 1:34 PM · Operations, ops-codfw, DC-Ops
fgiunchedi added subtasks for T266016: Refresh and expand Swift hardware capacity: Unknown Object (Task), Unknown Object (Task), Unknown Object (Task), T265419: (Need By: TBD) rack/setup/install ms-be20[58-61], T265093: (Need By: TBD) rack/setup/install ms-be106[0-3], T261633: Put ms-be2057 (Dell R740xd2) in service.
Tue, Oct 20, 1:34 PM · User-fgiunchedi, SRE-swift-storage
fgiunchedi added a project to T266016: Refresh and expand Swift hardware capacity: User-fgiunchedi.
Tue, Oct 20, 1:33 PM · User-fgiunchedi, SRE-swift-storage
fgiunchedi created T266016: Refresh and expand Swift hardware capacity.
Tue, Oct 20, 1:33 PM · User-fgiunchedi, SRE-swift-storage
fgiunchedi moved T249311: Deploy Prometheus Push Gateway from Backlog to Doing on the User-fgiunchedi board.
Tue, Oct 20, 10:43 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi added a project to T249311: Deploy Prometheus Push Gateway: User-fgiunchedi.
Tue, Oct 20, 10:43 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi moved T265712: Sync users and permissions from LDAP to Grafana from Up next to Doing on the User-fgiunchedi board.
Tue, Oct 20, 10:26 AM · Patch-For-Review, User-fgiunchedi, observability, Operations
fgiunchedi moved T199406: rsyslog's in:imtcp thread stuck on recvfrom loop from down/rebooted hosts from Doing to Up next on the User-fgiunchedi board.
Tue, Oct 20, 10:26 AM · User-fgiunchedi, Operations
fgiunchedi moved T259780: rsyslog occasional segfault on centrallog hosts from Doing to Up next on the User-fgiunchedi board.
Tue, Oct 20, 10:26 AM · User-fgiunchedi, observability, Operations
fgiunchedi moved T265649: PuppetDB grafana graphs not matching logs from Backlog to Doing on the User-fgiunchedi board.
Tue, Oct 20, 10:25 AM · User-fgiunchedi, User-jbond, Operations, observability, Puppet
fgiunchedi updated the task description for T259780: rsyslog occasional segfault on centrallog hosts.
Tue, Oct 20, 10:24 AM · User-fgiunchedi, observability, Operations
fgiunchedi added a comment to T265649: PuppetDB grafana graphs not matching logs.

Looking at the panel's query, AFAICT that's the rate of replace facts operations:

Tue, Oct 20, 9:37 AM · User-fgiunchedi, User-jbond, Operations, observability, Puppet

Mon, Oct 19

fgiunchedi added a project to T265649: PuppetDB grafana graphs not matching logs: User-fgiunchedi.
Mon, Oct 19, 3:21 PM · User-fgiunchedi, User-jbond, Operations, observability, Puppet
fgiunchedi added a project to T97861: Provide centralized logging (logstash) for Toolforge: observability.

cc observability for visibility

Mon, Oct 19, 8:43 AM · observability, cloud-services-team (Kanban), Toolforge
fgiunchedi added a comment to T143162: Reduce task notification noise/frequency of changes to associated open patchsets.

Yeah adding Patch-For-Review only sounds good to me, less visually noisy for sure (albeit still spammy via email I believe?).

Mon, Oct 19, 8:24 AM · Developer Productivity, Release-Engineering-Team (Development services), Release-Engineering-Team-TODO, Diffusion

Fri, Oct 16

fgiunchedi moved T265712: Sync users and permissions from LDAP to Grafana from Backlog to Up next on the User-fgiunchedi board.
Fri, Oct 16, 4:06 PM · Patch-For-Review, User-fgiunchedi, observability, Operations
fgiunchedi created T265712: Sync users and permissions from LDAP to Grafana.
Fri, Oct 16, 10:15 AM · Patch-For-Review, User-fgiunchedi, observability, Operations
fgiunchedi moved T78135: Provide a pxe-bootable rescue image from Backlog to Radar on the User-fgiunchedi board.
Fri, Oct 16, 10:02 AM · User-fgiunchedi, Operations
fgiunchedi added a comment to T199406: rsyslog's in:imtcp thread stuck on recvfrom loop from down/rebooted hosts.

Note that with the latest rsyslog version (cfr T259780: rsyslog occasional segfault on centrallog hosts) this bug might be fixed too, and thus we can remove the bandaid

Fri, Oct 16, 10:00 AM · User-fgiunchedi, Operations
fgiunchedi moved T261281: Improve performance of Thanos (+ Prometheus) from Up next to Doing on the User-fgiunchedi board.
Fri, Oct 16, 10:00 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi added a comment to T259780: rsyslog occasional segfault on centrallog hosts.
Fri, Oct 16, 9:58 AM · User-fgiunchedi, observability, Operations
fgiunchedi updated subscribers of T86969: Send scap log directly to logstash via syslog input.

@LarsWirzenius re: the next scap release to get this change included, is there a timeline ? thanks!

Fri, Oct 16, 9:35 AM · User-fgiunchedi, Patch-For-Review, Scap
fgiunchedi added a comment to T265590: ulog: filter out diffscan from ulog.

If the figure of 7.5M logs/day just from ulog dropping expected internal scans is accurate then it does make a difference (about +10% overall log daily volume)

Fri, Oct 16, 9:32 AM · observability, Security, Operations, netops, User-jbond
fgiunchedi added a comment to T265647: codfw: relocate thanos-fe2003 to create space for new ms-be servers .

Proposal sounds good to me @Papaul. Host can be powered off at any time, and must be depooled first

Fri, Oct 16, 8:25 AM · Operations, ops-codfw

Thu, Oct 15

fgiunchedi created T265632: Alert on Thanos sidecar not uploading blocks.
Thu, Oct 15, 4:05 PM · observability
fgiunchedi closed T265208: Degraded RAID on ms-be2036 as Resolved.

Disk is rebuilding! Thank you @Papaul

Thu, Oct 15, 3:01 PM · Operations, ops-codfw
fgiunchedi added a project to T265590: ulog: filter out diffscan from ulog: observability.
Thu, Oct 15, 12:19 PM · observability, Security, Operations, netops, User-jbond
fgiunchedi added a comment to T225140: Icinga alerts that should open tasks instead of alerting.

Thank you for the feedback and use cases, we have this feature on the roadmap (namely to connect alerts to phabricator) and will get prioritized. One of the benefits of alertmanager is that it is making it possible to add this feature easily (e.g. https://github.com/knyar/phalerts)

Thu, Oct 15, 8:11 AM · observability

Wed, Oct 14

fgiunchedi committed rODTHce0fc5ee910a: debian: ship thanos-receive (authored by fgiunchedi).
debian: ship thanos-receive
Wed, Oct 14, 5:51 PM
fgiunchedi committed rODTH13c56b11fdda: Ship thanos receive, rule, query-frontend (authored by fgiunchedi).
Ship thanos receive, rule, query-frontend
Wed, Oct 14, 5:51 PM
fgiunchedi committed rODTHbfefda8c4871: debian: fix github links in service units (authored by fgiunchedi).
debian: fix github links in service units
Wed, Oct 14, 5:51 PM
fgiunchedi committed rODTH201f9e000199: New upstream release (authored by fgiunchedi).
New upstream release
Wed, Oct 14, 5:51 PM
fgiunchedi committed rODTH230750c25ff2: Merge tag 'upstream/0.16.0_rc0' into debian/buster-wikimedia (authored by fgiunchedi).
Merge tag 'upstream/0.16.0_rc0' into debian/buster-wikimedia
Wed, Oct 14, 5:51 PM
fgiunchedi closed T225613: Swift / puppet interaction can fill up root filesystem as Resolved.

This is fixed, permissions will be set depending on whether the directory is a mountpoint or not. Unrelated but distinct issue is the interaction between drive-audit and systemd in T265450: flip/flop mounting filesystems between systemd and swift-drive-audit

Wed, Oct 14, 1:43 PM · User-fgiunchedi, SRE-swift-storage
fgiunchedi added a comment to T261281: Improve performance of Thanos (+ Prometheus).

The last two patches are to enable compactions back in Prometheus and leave the rest unchanged. I asked upstream about what's the recommended solution in our case and just reenabling compactions should work as expected.

Wed, Oct 14, 12:45 PM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi created T265450: flip/flop mounting filesystems between systemd and swift-drive-audit.
Wed, Oct 14, 8:01 AM · SRE-swift-storage
fgiunchedi added a comment to T265208: Degraded RAID on ms-be2036.

Thanks @Papaul! It looks like the sdd disk is in trouble, do you have a spare or can order one? Thank you!

Wed, Oct 14, 7:48 AM · Operations, ops-codfw

Tue, Oct 13

fgiunchedi added a comment to T246004: Spike: Can/should Swift be used as Flink checkpoint backend?.

Ok, thank you for the information. It doesn't seem we have an isolated test environment anyways so even though I'm reluctant we'll have to test on the production swift cluster. A middle ground I suppose would be to create the accounts on the thanos swift cluster first, which is functionally the same as production but not in the hot-path for serving content. Let me know how you'd like to proceed!

Tue, Oct 13, 2:24 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service, Epic
fgiunchedi reopened T209709: Feature: enable prometheus-nginx-exporter for nginx metrics as "Open".

I'm reopening the task because we indeed have many nginx instances deployed which should have metrics (however basic, from the non-commercial nginx).

Tue, Oct 13, 2:20 PM · observability
fgiunchedi added a comment to T265208: Degraded RAID on ms-be2036.

Plus these usb recurring messages in dmesg

Tue, Oct 13, 1:40 PM · Operations, ops-codfw
fgiunchedi added a comment to T265208: Degraded RAID on ms-be2036.

I spotted a problem with sdd in dmesg too, perhaps that disk isn't healthy

Tue, Oct 13, 12:27 PM · Operations, ops-codfw
fgiunchedi added a comment to T253655: Document and/or improve navigation of the various HTTP frontend Grafana dashboards.

I wanted to mention/add that as part of T263536: Aggregated metrics for ats-tls <-> clients ttfb percentiles there's also https://grafana.wikimedia.org/d/8T2XA-5Gz/frontend-ats-tls-ttfb-latency which should have actionable insights

Tue, Oct 13, 10:41 AM · Sustainability (Incident Followup), Performance-Team (Radar), Traffic, observability, Operations
fgiunchedi added a comment to T263536: Aggregated metrics for ats-tls <-> clients ttfb percentiles.

With 50 percentile added I'm considering this closed!

As a demo/playground I've started https://grafana.wikimedia.org/d/fD5qC_KGz/filippo-ats-latency for anyone interested

Tue, Oct 13, 10:40 AM · Patch-For-Review, User-fgiunchedi, Operations, observability, Traffic
fgiunchedi created T265338: DNS race in WMCS VPS when deleting and re-provisioning an instance with the same name.
Tue, Oct 13, 9:46 AM · cloud-services-team (Kanban)

Mon, Oct 12

fgiunchedi added a comment to T265208: Degraded RAID on ms-be2036.

Thanks! Indeed that's what happened, I've unmounted the filesystems and delete the files

Mon, Oct 12, 3:13 PM · Operations, ops-codfw
fgiunchedi assigned T265208: Degraded RAID on ms-be2036 to Papaul.

@Papaul I've updated the hw raid firmware to 6.88 and rebooted to apply the upgrade. On reboot the message below was still there, what do you think ? Feel free to upgrade the ilo firmware too, the host can be powered down (from Linux) at any time. Thank you!

Mon, Oct 12, 8:20 AM · Operations, ops-codfw
fgiunchedi added a comment to T265208: Degraded RAID on ms-be2036.

The host rebooted into Linux OK, however there were error messages at boot. Looks like related to both ilo and the hw raid firmware.

Mon, Oct 12, 8:08 AM · Operations, ops-codfw

Fri, Oct 9

fgiunchedi updated the task description for T265117: Consider swift ring management automation.
Fri, Oct 9, 8:09 AM · SRE-swift-storage
fgiunchedi created T265117: Consider swift ring management automation.
Fri, Oct 9, 8:06 AM · SRE-swift-storage
fgiunchedi created T265114: Deploy swift-health-exporter.
Fri, Oct 9, 7:32 AM · SRE-swift-storage

Thu, Oct 8

fgiunchedi removed a project from T265015: TypeError: $.widget is not a function -- User script error: Wikimedia-Logstash.

Removing wikimedia-logstash since it doesn't seem to be a logstash-specific issue, please add back if that's not the case!

Thu, Oct 8, 12:41 PM · Wikidata
fgiunchedi removed a project from T265035: Gadget Error: "Uncaught TypeError: $(...).css(...).draggable is not a function": Wikimedia-Logstash.

Removing wikimedia-logstash since it doesn't seem to be a logstash-specific issue, please add back if that's not the case!

Thu, Oct 8, 12:40 PM · Wikidata
fgiunchedi removed a project from T265037: User Script error: "ReferenceError: wikibase is not defined": Wikimedia-Logstash.

Removing wikimedia-logstash since it doesn't seem to be a logstash-specific issue, please add back if that's not the case!

Thu, Oct 8, 12:40 PM · Wikidata
fgiunchedi removed a project from T265022: Client (Gadget?) Error: "NS_ERROR_FILE_CORRUPTED: ": Wikimedia-Logstash.

Removing wikimedia-logstash since it doesn't seem to be a logstash-specific issue, please add back if that's not the case!

Thu, Oct 8, 12:39 PM · Wikidata
fgiunchedi moved T199406: rsyslog's in:imtcp thread stuck on recvfrom loop from down/rebooted hosts from Backlog to Doing on the User-fgiunchedi board.
Thu, Oct 8, 10:34 AM · User-fgiunchedi, Operations
fgiunchedi moved T259780: rsyslog occasional segfault on centrallog hosts from Backlog to Doing on the User-fgiunchedi board.
Thu, Oct 8, 10:12 AM · User-fgiunchedi, observability, Operations
fgiunchedi created T264998: Some object-replicator log lines not making it to centrallog.
Thu, Oct 8, 9:48 AM · User-fgiunchedi, Operations, SRE-swift-storage
fgiunchedi added a project to T259780: rsyslog occasional segfault on centrallog hosts: User-fgiunchedi.
Thu, Oct 8, 8:33 AM · User-fgiunchedi, observability, Operations

Wed, Oct 7

fgiunchedi closed T245603: Move termbox to the logging pipeline, a subtask of T227080: Deprecate all non-Kafka logstash inputs, as Resolved.
Wed, Oct 7, 1:59 PM · Patch-For-Review, observability, Wikimedia-Logstash, User-fgiunchedi, Operations
fgiunchedi closed T245603: Move termbox to the logging pipeline as Resolved.

Can confirm that termbox is no longer using gelf, thank you @akosiaris

Wed, Oct 7, 1:58 PM · Wikidata-Termbox, observability, Wikimedia-Logstash
fgiunchedi closed T226703: rsyslog output modules (fwd, kafka) failures should not affect each other as Resolved.

This is complete now, we have local queues for all rsyslog outputs

Wed, Oct 7, 10:12 AM · Patch-For-Review, User-fgiunchedi, observability
fgiunchedi closed T264588: Report swift-object server per-method latencies, a subtask of T261633: Put ms-be2057 (Dell R740xd2) in service, as Resolved.
Wed, Oct 7, 8:40 AM · Patch-For-Review, User-fgiunchedi, Operations, SRE-swift-storage
fgiunchedi closed T264588: Report swift-object server per-method latencies as Resolved.

This is complete!

Wed, Oct 7, 8:40 AM · User-fgiunchedi, Operations, SRE-swift-storage

Tue, Oct 6

fgiunchedi closed T263536: Aggregated metrics for ats-tls <-> clients ttfb percentiles as Resolved.
Tue, Oct 6, 4:13 PM · Patch-For-Review, User-fgiunchedi, Operations, observability, Traffic
fgiunchedi added a comment to T263536: Aggregated metrics for ats-tls <-> clients ttfb percentiles.

With 50 percentile added I'm considering this closed!

Tue, Oct 6, 4:13 PM · Patch-For-Review, User-fgiunchedi, Operations, observability, Traffic
fgiunchedi added a comment to T246004: Spike: Can/should Swift be used as Flink checkpoint backend?.

@fgiunchedi Currently, Flink pipeline resides on the Analytics Hadoop cluster. As for the question whether Flink creates it's containers - I think not, it did complain when there was no container, so I assume it expects one.

Tue, Oct 6, 2:28 PM · Discovery-Search (Current work), Wikidata, Wikidata-Query-Service, Epic
MSantos awarded T245604: Move wikifeeds to the logging pipeline a Party Time token.
Tue, Oct 6, 1:47 PM · Wikifeeds, observability, Wikimedia-Logstash, Operations
fgiunchedi edited P12929 (An Untitled Masterwork).
Tue, Oct 6, 10:14 AM
fgiunchedi created P12929 (An Untitled Masterwork).
Tue, Oct 6, 10:13 AM
fgiunchedi added a comment to T227080: Deprecate all non-Kafka logstash inputs.

As far as gelf is concerned we're left with:

Tue, Oct 6, 7:52 AM · Patch-For-Review, observability, Wikimedia-Logstash, User-fgiunchedi, Operations
fgiunchedi closed T245604: Move wikifeeds to the logging pipeline, a subtask of T227080: Deprecate all non-Kafka logstash inputs, as Resolved.
Tue, Oct 6, 7:40 AM · Patch-For-Review, observability, Wikimedia-Logstash, User-fgiunchedi, Operations
fgiunchedi closed T245604: Move wikifeeds to the logging pipeline as Resolved.

@MSantos I went ahead and deployed wikifeed chart 0.0.19, resolving

Tue, Oct 6, 7:40 AM · Wikifeeds, observability, Wikimedia-Logstash, Operations

Mon, Oct 5

fgiunchedi closed T219919: Move citoid logging to new logging pipeline as Resolved.

Citoid is using only k8s-native logging now, resolving.

Mon, Oct 5, 2:13 PM · observability, Citoid, Platform Team Legacy (Watching / External), Services (watching), service-runner, Wikimedia-Logstash, Operations
fgiunchedi closed T219919: Move citoid logging to new logging pipeline, a subtask of T211125: Move service-runner to new logging infrastructure, as Resolved.
Mon, Oct 5, 2:13 PM · observability, Platform Team Legacy (Watching / External), Patch-For-Review, service-runner, Wikimedia-Logstash, Operations
fgiunchedi moved T264588: Report swift-object server per-method latencies from Backlog to Doing on the User-fgiunchedi board.
Mon, Oct 5, 1:19 PM · User-fgiunchedi, Operations, SRE-swift-storage
fgiunchedi closed T200984: Stop introducing new code expanded from erb templates as Invalid.

This seems somewhat but not exactly a duplicate of T254480. The other ticket would be resolved once we have removed all cases of it from the repo.

Then this ticket would be still here to prevent it from happening again. So does it mean this is about making jenkins-bot vote -1 if it detects such an attempt?

Mon, Oct 5, 10:50 AM · Operations, Puppet
fgiunchedi added a comment to T229584: Run swift-object-expirer as part of the swift cluster.

Merged the patch above, apologies for the late action on this.

Mon, Oct 5, 10:04 AM · Analytics-Radar, Recommendation-API, Operations, Discovery
fgiunchedi updated the task description for T264291: Swift users and their usage.
Mon, Oct 5, 9:47 AM · SRE-swift-storage
fgiunchedi created T264588: Report swift-object server per-method latencies.
Mon, Oct 5, 9:29 AM · User-fgiunchedi, Operations, SRE-swift-storage
fgiunchedi closed T258948: Deploy Alertmanager for alerting infrastructure phase 1 as Resolved.

This is complete!

Mon, Oct 5, 8:41 AM · Patch-For-Review, User-fgiunchedi, observability