Page MenuHomePhabricator

colewhite (cwhite)
User

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Wednesday

  • Clear sailing ahead.

User Details

User Since
Aug 21 2018, 6:05 PM (223 w, 5 d)
Availability
Available
LDAP User
Cwhite
MediaWiki User
Unknown

Recent Activity

Fri, Dec 2

colewhite updated the task description for T321410: Upgrade logstash to bullseye.
Fri, Dec 2, 12:07 AM · Patch-For-Review, Observability-Logging

Wed, Nov 30

colewhite added a comment to T323357: Spam graphite metrics from MediaWiki.objectcache.

No new uuid entries have appeared in the namespace for a week. Should we consider this task resolved?

Wed, Nov 30, 12:03 AM · SecTeam-Processed, Platform Team Workboards (MW Expedition), Parsoid, Security-Team, Security, Observability-Metrics, MediaWiki-General

Tue, Nov 22

colewhite added a comment to T320468: Logging spam from revscoring deploys.

@colewhite after a lot of research I think that this will go away with Istio 1.15.3, and we'll upgrade to it probably/hopefully during the next quarter (as part of the k8s 1.23 upgrade). Would it be ok to wait until that time?

Tue, Nov 22, 5:16 PM · Machine-Learning-Team, SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging

Mon, Nov 21

colewhite added a comment to T323357: Spam graphite metrics from MediaWiki.objectcache.

https://gerrit.wikimedia.org/r/c/mediawiki/core/+/859045 has greatly reduced the number of metric namespaces generated. Thank you!

Mon, Nov 21, 5:40 PM · SecTeam-Processed, Platform Team Workboards (MW Expedition), Parsoid, Security-Team, Security, Observability-Metrics, MediaWiki-General

Fri, Nov 18

colewhite added a comment to T323357: Spam graphite metrics from MediaWiki.objectcache.

Deploy from around the same time: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/855029

Fri, Nov 18, 11:06 PM · SecTeam-Processed, Platform Team Workboards (MW Expedition), Parsoid, Security-Team, Security, Observability-Metrics, MediaWiki-General
colewhite triaged T323357: Spam graphite metrics from MediaWiki.objectcache as High priority.
Fri, Nov 18, 10:35 PM · SecTeam-Processed, Platform Team Workboards (MW Expedition), Parsoid, Security-Team, Security, Observability-Metrics, MediaWiki-General
colewhite set Security to security-bug on T323357: Spam graphite metrics from MediaWiki.objectcache.

I was able to locate the source for the [01]_[a-z0-9]{8}_[a-z0-9]{4}_11ed_[a-z0-9]{4}_[a-z0-9]{12} metrics coming from SimpleParsoidOutputStash:49. These metrics can be generated by requesting data from DiscussionTools extension api endpoint discussiontoolspageinfo.

Fri, Nov 18, 10:22 PM · SecTeam-Processed, Platform Team Workboards (MW Expedition), Parsoid, Security-Team, Security, Observability-Metrics, MediaWiki-General

Wed, Nov 16

colewhite added a comment to T288196: Retire Prometheus 'global' instance.

Dashboards likely using the global instance:

Wed, Nov 16, 3:42 PM · Patch-For-Review, Observability-Metrics, Performance-Team (Radar)

Thu, Nov 10

colewhite edited Description on Phatality.
Thu, Nov 10, 10:53 PM
colewhite edited Description on Phatality.
Thu, Nov 10, 10:53 PM

Nov 4 2022

colewhite updated the task description for T321410: Upgrade logstash to bullseye.
Nov 4 2022, 10:45 PM · Patch-For-Review, Observability-Logging
colewhite updated the task description for T322448: Volumes stuck in "Reserved" state.
Nov 4 2022, 10:10 PM · Cloud-VPS
colewhite attached a referenced file: F35707238: Screenshot from 2022-11-04 21-05-12.png.
Nov 4 2022, 10:09 PM · Cloud-VPS
colewhite created T322448: Volumes stuck in "Reserved" state.
Nov 4 2022, 10:09 PM · Cloud-VPS
colewhite changed the status of T321410: Upgrade logstash to bullseye from Open to In Progress.
Nov 4 2022, 8:01 PM · Patch-For-Review, Observability-Logging
colewhite edited projects for T226970: Use white version of Wikimedia logo for grafana, added: Observability-Metrics; removed SRE Observability.
Nov 4 2022, 3:23 PM · Observability-Metrics, SRE, UI-Standardization
lmata awarded T304440: Test and upgrade OpenSearch to 2.2.0 a Like token.
Nov 4 2022, 11:26 AM · SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging

Nov 3 2022

colewhite closed T304440: Test and upgrade OpenSearch to 2.2.0 as Resolved.

Phatality is not available on eqiad pending a deploy, but the upgrade is complete.

Nov 3 2022, 10:42 PM · SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging
colewhite updated the task description for T304440: Test and upgrade OpenSearch to 2.2.0.
Nov 3 2022, 10:41 PM · SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging

Oct 26 2022

colewhite updated subscribers of T304440: Test and upgrade OpenSearch to 2.2.0.

After the switchover to the 2.2.0 instance in codfw, the phatality button has disappeared from the dashboards.

There was a patch merged to update the plugin to use the latest version: https://gerrit.wikimedia.org/r/c/releng/phatality/+/822664. Is there a chance something needs to be configured in the opensearch instance instead?

Oct 26 2022, 2:25 PM · SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging

Oct 25 2022

colewhite updated the task description for T304440: Test and upgrade OpenSearch to 2.2.0.
Oct 25 2022, 7:40 PM · SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging

Oct 21 2022

colewhite claimed T321410: Upgrade logstash to bullseye.
Oct 21 2022, 7:42 PM · Patch-For-Review, Observability-Logging
colewhite moved T304440: Test and upgrade OpenSearch to 2.2.0 from Inbox to In progress on the SRE Observability (FY2022/2023-Q2) board.
Oct 21 2022, 7:37 PM · SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging
colewhite added a project to T304440: Test and upgrade OpenSearch to 2.2.0: SRE Observability (FY2022/2023-Q2).
Oct 21 2022, 7:37 PM · SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging
colewhite moved T321335: Put logstash203[67] in service from Inbox to Backlog on the Observability-Logging board.
Oct 21 2022, 7:36 PM · SRE Observability (FY2022/2023-Q2), Observability-Logging
colewhite moved T321410: Upgrade logstash to bullseye from Inbox to Backlog on the Observability-Logging board.
Oct 21 2022, 7:36 PM · Patch-For-Review, Observability-Logging
colewhite created T321410: Upgrade logstash to bullseye.
Oct 21 2022, 7:35 PM · Patch-For-Review, Observability-Logging

Oct 20 2022

hashar awarded T304440: Test and upgrade OpenSearch to 2.2.0 a Barnstar token.
Oct 20 2022, 8:48 PM · SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging
colewhite updated the task description for T304440: Test and upgrade OpenSearch to 2.2.0.
Oct 20 2022, 7:16 PM · SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging
colewhite updated the task description for T304440: Test and upgrade OpenSearch to 2.2.0.
Oct 20 2022, 7:16 PM · SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging
colewhite triaged T321335: Put logstash203[67] in service as Medium priority.
Oct 20 2022, 5:48 PM · SRE Observability (FY2022/2023-Q2), Observability-Logging
colewhite created T321335: Put logstash203[67] in service.
Oct 20 2022, 5:48 PM · SRE Observability (FY2022/2023-Q2), Observability-Logging

Oct 19 2022

colewhite moved T320468: Logging spam from revscoring deploys from In progress to Done on the SRE Observability (FY2022/2023-Q2) board.
Oct 19 2022, 2:36 PM · Machine-Learning-Team, SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging
colewhite moved T320636: smart-data-dump fails occasionally due to facter timeouts from In progress to Done on the SRE Observability (FY2022/2023-Q2) board.
Oct 19 2022, 2:36 PM · SRE Observability (FY2022/2023-Q2), Observability-Alerting

Oct 18 2022

colewhite edited Description on Phatality.
Oct 18 2022, 11:33 PM

Oct 17 2022

colewhite moved T320468: Logging spam from revscoring deploys from Inbox to Radar on the Observability-Logging board.
Oct 17 2022, 9:57 PM · Machine-Learning-Team, SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging

Oct 13 2022

colewhite moved T320468: Logging spam from revscoring deploys from Inbox to In progress on the SRE Observability (FY2022/2023-Q2) board.
Oct 13 2022, 9:16 PM · Machine-Learning-Team, SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging
colewhite added a project to T320468: Logging spam from revscoring deploys: SRE Observability (FY2022/2023-Q2).
Oct 13 2022, 9:16 PM · Machine-Learning-Team, SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging
colewhite moved T320636: smart-data-dump fails occasionally due to facter timeouts from Inbox to In progress on the SRE Observability (FY2022/2023-Q2) board.
Oct 13 2022, 9:15 PM · SRE Observability (FY2022/2023-Q2), Observability-Alerting
colewhite added a project to T320636: smart-data-dump fails occasionally due to facter timeouts: SRE Observability (FY2022/2023-Q2).
Oct 13 2022, 9:15 PM · SRE Observability (FY2022/2023-Q2), Observability-Alerting
colewhite claimed T320636: smart-data-dump fails occasionally due to facter timeouts.

Rolled back changes to use facter back to calling raid.rb. Updated raid.rb to look at the new metric.

Oct 13 2022, 9:14 PM · SRE Observability (FY2022/2023-Q2), Observability-Alerting
colewhite triaged T320636: smart-data-dump fails occasionally due to facter timeouts as Medium priority.
Oct 13 2022, 9:14 PM · SRE Observability (FY2022/2023-Q2), Observability-Alerting

Oct 12 2022

colewhite closed T313099: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy, a subtask of T295939: Logstash throttler does not apply to k8s logs, as Resolved.
Oct 12 2022, 10:14 PM · Observability-Logging
colewhite closed T313099: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy as Resolved.

Clean up as the indexes age is needed to gracefully handle what has been ingested already. With the sampling in place I think its safe to say this is resolved for now.

Oct 12 2022, 10:14 PM · SRE Observability (FY2022/2023-Q1), Patch-For-Review, serviceops, Observability-Logging
colewhite added a comment to T320468: Logging spam from revscoring deploys.

Grafana link. Drop rule added to the logstash pipeline pending upstream mitigation.

Oct 12 2022, 7:39 PM · Machine-Learning-Team, SRE Observability (FY2022/2023-Q2), Patch-For-Review, Observability-Logging
colewhite added a project to T320620: Port openapi/swagger checks/alerts to Prometheus: Observability-Alerting.
Oct 12 2022, 2:18 PM · Observability-Alerting, observability, serviceops
colewhite created T320636: smart-data-dump fails occasionally due to facter timeouts.
Oct 12 2022, 1:38 PM · SRE Observability (FY2022/2023-Q2), Observability-Alerting

Oct 7 2022

colewhite claimed T300130: Move Kafka logging to the new intermediate PKI.
Oct 7 2022, 11:38 AM · SRE Observability (FY2022/2023-Q2)

Sep 21 2022

colewhite moved T300937: Evaluate storing logs from applications in yarn with the typical logging infrastructure from Inbox to Radar on the Observability-Logging board.
Sep 21 2022, 12:47 PM · Observability-Logging, Analytics-Clusters, Wikimedia-Logstash
colewhite moved T248884: Documentation of client side error logging capabilities on mediawiki from Inbox to Radar on the Observability-Logging board.
Sep 21 2022, 10:59 AM · Instrument-ClientError, Observability-Logging, observability, Analytics-Radar, Documentation, Performance-Team (Radar), Wikimedia-Logstash, Better Use Of Data
colewhite moved T261225: Set strict CSP rule on Kibana logstash.wikimedia.org from Inbox to Backlog on the Observability-Logging board.
Sep 21 2022, 10:58 AM · Observability-Logging, observability, Security, Wikimedia-Logstash
colewhite moved T292682: Develop tooling for quickly parsing 5xx and sampled-1000 logs from Inbox to Blocked on the Observability-Logging board.
Sep 21 2022, 10:56 AM · Observability-Logging, SRE
colewhite moved T301110: Ingest webrequest sampled 1000 into logstash from Inbox to Blocked on the Observability-Logging board.
Sep 21 2022, 10:54 AM · SRE, Observability-Logging
colewhite moved T315500: Eliminate field collisions between syslog_cee and ECS-formatted logs from Inbox to Prioritized on the Observability-Logging board.
Sep 21 2022, 10:54 AM · Observability-Logging
colewhite moved T316992: ClientError fieldname 'tags' is conflicting from Inbox to Radar on the Observability-Logging board.
Sep 21 2022, 10:53 AM · Instrument-ClientError, Observability-Logging
colewhite claimed T313099: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy.
Sep 21 2022, 10:53 AM · SRE Observability (FY2022/2023-Q1), Patch-For-Review, serviceops, Observability-Logging
colewhite moved T313099: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy from Inbox to Prioritized on the Observability-Logging board.
Sep 21 2022, 10:53 AM · SRE Observability (FY2022/2023-Q1), Patch-For-Review, serviceops, Observability-Logging
colewhite closed T222826: Leverage Grafana annotations to show events in graphs as Resolved.

MVP achieved. Further iterations and features should come in separately.

Sep 21 2022, 10:50 AM · Observability-Logging, Patch-For-Review, SRE
colewhite added a project to T223934: Add annotations from ops vendor maintenance calendar to Grafana: Observability-Metrics.

Tagging observability-metrics because while logging could handle it, but it may not be the most efficient way to get this information in. We have SimpleJSON and JSON API datasource plugins that might be a better fit.

Sep 21 2022, 10:48 AM · Observability-Metrics, SRE
colewhite removed a subtask for T222826: Leverage Grafana annotations to show events in graphs: T223934: Add annotations from ops vendor maintenance calendar to Grafana.
Sep 21 2022, 10:42 AM · Observability-Logging, Patch-For-Review, SRE
colewhite removed a parent task for T223934: Add annotations from ops vendor maintenance calendar to Grafana: T222826: Leverage Grafana annotations to show events in graphs.
Sep 21 2022, 10:42 AM · Observability-Metrics, SRE
colewhite placed T305090: Add logstash diagnostics up for grabs.
Sep 21 2022, 10:38 AM · Observability-Logging
colewhite lowered the priority of T305090: Add logstash diagnostics from Medium to Low.

Can provide deep information about how the pipeline is working
Which input did <event> come from
Which filters did <event> hit?
...

Logstash does not currently support it without a lot of manual configuration. Possibly explore adding a feature to add the plugin id to an array for all filters that evaluated successfully (possibly @metadata).

Sep 21 2022, 10:38 AM · Observability-Logging
colewhite moved T314098: Update Phatality to reference ECS fields from Inbox to Prioritized on the Observability-Logging board.
Sep 21 2022, 8:21 AM · Patch-For-Review, Observability-Logging, Phatality

Sep 15 2022

colewhite closed T251293: Facter is slow on a few hosts as Resolved.

Rolled back the changes and only one host experienced a regression. Created T317924 to handle that host.

Sep 15 2022, 9:00 PM · Patch-For-Review, Infrastructure-Foundations, Puppet, SRE
colewhite added a subtask for T251293: Facter is slow on a few hosts: T317924: raid_mgmt_tools cannot detect raid on clouddb1021.
Sep 15 2022, 8:59 PM · Patch-For-Review, Infrastructure-Foundations, Puppet, SRE
colewhite added a parent task for T317924: raid_mgmt_tools cannot detect raid on clouddb1021: T251293: Facter is slow on a few hosts.
Sep 15 2022, 8:59 PM · SRE, Infrastructure-Foundations
colewhite created T317924: raid_mgmt_tools cannot detect raid on clouddb1021.
Sep 15 2022, 8:58 PM · SRE, Infrastructure-Foundations
colewhite added a project to T271138: Some Observability clusters do not support IPv6.: Wikimedia-Incident.
Sep 15 2022, 8:29 PM · Observability-Metrics, Wikimedia-Incident, IPv6, User-crusnov
colewhite updated the task description for T271138: Some Observability clusters do not support IPv6..
Sep 15 2022, 3:40 PM · Observability-Metrics, Wikimedia-Incident, IPv6, User-crusnov
colewhite closed T271138: Some Observability clusters do not support IPv6. as Resolved.

All indicated hosts have ipv6 records now.

Sep 15 2022, 3:40 PM · Observability-Metrics, Wikimedia-Incident, IPv6, User-crusnov
colewhite closed T271138: Some Observability clusters do not support IPv6., a subtask of T253173: Some clusters do not have DNS for IPv6 addresses (TRACKING TASK), as Resolved.
Sep 15 2022, 3:40 PM · Infrastructure-Foundations, IPv6, User-jbond, netbox
colewhite awarded T317887: Upgrade to Grafana 9 a Love token.
Sep 15 2022, 3:23 PM · Observability-Metrics

Sep 14 2022

colewhite added a comment to T251293: Facter is slow on a few hosts.

raid_mgmt_tools does not detect raid on clouddb1021

cwhite@clouddb1021:~$ sudo /usr/bin/ruby /var/lib/puppet/lib/facter/raid.rb | jq .
{
  "raid": [
    "megaraid"
  ]
}
cwhite@clouddb1021:~$ sudo /usr/bin/facter --puppet --json -l error raid | jq .
{
  "raid": [
    "megaraid"
  ]
}
cwhite@clouddb1021:~$ sudo /usr/bin/facter --puppet --json -l error raid_mgmt_tools | jq .
{
  "raid_mgmt_tools": []
}
Sep 14 2022, 9:34 PM · Patch-For-Review, Infrastructure-Foundations, Puppet, SRE
colewhite added a member for Wikimedia-Logstash: colewhite.
Sep 14 2022, 6:58 PM
colewhite added a member for Phatality: colewhite.
Sep 14 2022, 6:57 PM
colewhite added a comment to T271138: Some Observability clusters do not support IPv6..

mwlog* hosts added to netbox and sre.dns.netbox cookbook has been run.

Sep 14 2022, 4:10 PM · Observability-Metrics, Wikimedia-Incident, IPv6, User-crusnov
colewhite updated the task description for T271138: Some Observability clusters do not support IPv6..
Sep 14 2022, 4:10 PM · Observability-Metrics, Wikimedia-Incident, IPv6, User-crusnov
colewhite updated the task description for T271138: Some Observability clusters do not support IPv6..
Sep 14 2022, 4:03 PM · Observability-Metrics, Wikimedia-Incident, IPv6, User-crusnov
colewhite updated the task description for T271138: Some Observability clusters do not support IPv6..
Sep 14 2022, 4:02 PM · Observability-Metrics, Wikimedia-Incident, IPv6, User-crusnov
colewhite added a comment to T271138: Some Observability clusters do not support IPv6..

logstash-* hosts have been added to netbox and the sre.dns.netbox cookbook has been run.

Sep 14 2022, 3:59 PM · Observability-Metrics, Wikimedia-Incident, IPv6, User-crusnov

Sep 13 2022

colewhite committed rOSECefc9c0ef85ef: bugfix: pin markupsafe to compatible version 2.0.1 (authored by colewhite).
bugfix: pin markupsafe to compatible version 2.0.1
Sep 13 2022, 6:18 PM
colewhite added a project to T313230: Dispatch IRC Integration: Incident Tooling.
Sep 13 2022, 4:28 PM · SRE Observability (FY2022/2023-Q2), Incident Tooling
colewhite added a project to T313228: Deploy Dispatch for SRE incident workflow automation: Incident Tooling.
Sep 13 2022, 4:28 PM · SRE Observability (FY2022/2023-Q2), Incident Tooling
colewhite added a member for Observability-Metrics: colewhite.
Sep 13 2022, 4:27 PM
colewhite added a member for Observability-Logging: colewhite.
Sep 13 2022, 4:26 PM

Sep 12 2022

colewhite added a comment to T300130: Move Kafka logging to the new intermediate PKI.

@colewhite does it sound good?

Sep 12 2022, 9:36 PM · SRE Observability (FY2022/2023-Q2)
colewhite awarded T316996: Degraded RAID on logstash2027 a Love token.
Sep 12 2022, 5:09 PM · Observability-Logging, SRE, ops-codfw
colewhite added a comment to T251293: Facter is slow on a few hosts.

[Removed]

Sep 12 2022, 3:25 PM · Patch-For-Review, Infrastructure-Foundations, Puppet, SRE

Sep 9 2022

colewhite added a comment to T313099: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy.

I don't think I have much to add, side from the fact, that I wouldn't find it improbable that people are cooperating on debugging some issue sharing links to specific log entries in logstash (it has happened to me a lot in the past). At just 31 days, some of these debugging sessions might end up broken to the surprise of the debuggers. I don't have a counter proposal though as far as the number of days go.

Sep 9 2022, 4:11 PM · SRE Observability (FY2022/2023-Q1), Patch-For-Review, serviceops, Observability-Logging
colewhite removed a subtask for T277816: Improve Logstash's throttling capabilities: T313099: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy.
Sep 9 2022, 3:10 PM · SRE Observability (FY2022/2023-Q2), Observability-Logging, observability, Wikimedia-Logstash
colewhite added a subtask for T295939: Logstash throttler does not apply to k8s logs: T313099: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy.
Sep 9 2022, 3:10 PM · Observability-Logging
colewhite edited parent tasks for T313099: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy, added: T295939: Logstash throttler does not apply to k8s logs; removed: T277816: Improve Logstash's throttling capabilities.
Sep 9 2022, 3:10 PM · SRE Observability (FY2022/2023-Q1), Patch-For-Review, serviceops, Observability-Logging
colewhite added a subtask for T277816: Improve Logstash's throttling capabilities: T313099: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy.
Sep 9 2022, 3:09 PM · SRE Observability (FY2022/2023-Q2), Observability-Logging, observability, Wikimedia-Logstash
colewhite added a parent task for T313099: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy: T277816: Improve Logstash's throttling capabilities.
Sep 9 2022, 3:09 PM · SRE Observability (FY2022/2023-Q1), Patch-For-Review, serviceops, Observability-Logging
colewhite added a comment to T313099: Increase of ~50 million access logs per day from mobileapps-production-tls-proxy.

I 've been meaning to ask regarding this, can we sample really heavily those logs? We want to use them just for debugging mobileapps incidents, we definitely not need the entirety of them. If we sampled, say 1:100, it would still work pretty nicely probably.

Sep 9 2022, 3:05 PM · SRE Observability (FY2022/2023-Q1), Patch-For-Review, serviceops, Observability-Logging

Sep 8 2022

colewhite added a comment to T300130: Move Kafka logging to the new intermediate PKI.

apifeatureusage now using the new pki truststore and appears to be working.

Sep 8 2022, 3:00 PM · SRE Observability (FY2022/2023-Q2)

Sep 7 2022

colewhite moved T300130: Move Kafka logging to the new intermediate PKI from Inbox to Prioritized on the Observability-Logging board.
Sep 7 2022, 7:44 PM · SRE Observability (FY2022/2023-Q2)
colewhite added a comment to T300130: Move Kafka logging to the new intermediate PKI.
  • rsyslog: /etc/ssl/certs/wmf-ca-certificates.crt
    • logstash collectors: /etc/ssl/localcerts/wmf-java-cacerts
    • kafkatee on centrallog: /etc/ssl/certs/wmf-ca-certificates.crt
    • !!! apifeatureusage collectors: /etc/logstash/kafka_logging-eqiad.truststore.jks
Sep 7 2022, 7:29 PM · SRE Observability (FY2022/2023-Q2)

Sep 6 2022

colewhite edited projects for T300130: Move Kafka logging to the new intermediate PKI, added: Observability-Logging, SRE Observability (FY2022/2023-Q1); removed observability.

Followed up offline. @elukey and I are scheduling a time to complete this.

Sep 6 2022, 4:42 PM · SRE Observability (FY2022/2023-Q2)