Page MenuHomePhabricator
Feed Advanced Search

Wed, Sep 15

herron awarded T279601: decommission icinga1001.wikimedia.org a Party Time token.
Wed, Sep 15, 8:02 PM · SRE, DC-Ops, ops-eqiad, SRE Observability (FY2021/2022-Q1), decommission-hardware

Tue, Sep 14

herron added a comment to T288620: Document path forward for how to Retire all non-Kafka Logstash inputs.

After exploring the NPM approach a bit on https://gerrit.wikimedia.org/r/c/operations/puppet/+/720110/ it's clear that we would be better off to look for an alternate tool written in another language with less convoluted dependencies, and which is easier to audit and maintain in the long term.

Tue, Sep 14, 2:52 PM · Patch-For-Review, Goal, SRE Observability (FY2021/2022-Q1)
herron closed T289036: Use Grizzly for Varnish SLO Grafana dashboard as Resolved.

the cluster dropdown should only list cache_text and cache_upload, while it currently includes clusters such as appserver and bastion which obviously don't make much sense for a Varnish SLO dashboard. Other than that I think we look good

Tue, Sep 14, 1:52 PM · SRE Observability (FY2021/2022-Q1), SRE, Traffic

Fri, Sep 10

herron added a comment to T288620: Document path forward for how to Retire all non-Kafka Logstash inputs.

So far so good testing logagent. Confirmed that it can indeed ingest/parse GELF logs from our elasticsearch-gelf config and output them json formatted to stdout. By wrapping this config in a systemd unit we should be able to pick up these logs with rsyslog and send them onward to kafka logging.

Fri, Sep 10, 2:20 PM · Patch-For-Review, Goal, SRE Observability (FY2021/2022-Q1)

Thu, Sep 9

herron added a comment to T288620: Document path forward for how to Retire all non-Kafka Logstash inputs.

I'm tempted to try shimming these using an rsyslog listener that emulates gelf and routes these logs to the kafka logging pipeline until the longer-term/upgraded elastic config is in place.

Thu, Sep 9, 2:14 PM · Patch-For-Review, Goal, SRE Observability (FY2021/2022-Q1)

Wed, Sep 8

herron added a comment to T288620: Document path forward for how to Retire all non-Kafka Logstash inputs.

According to the logstash input type distributions graph we're down to elastisearch via gelf for non-kafka inputs.

Wed, Sep 8, 4:56 PM · Patch-For-Review, Goal, SRE Observability (FY2021/2022-Q1)
herron added a comment to T286911: Upgrade MXes to Bullseye.

Seeing errors like this in the paniclog unfortunately

Wed, Sep 8, 3:29 PM · SRE, Patch-For-Review, Infrastructure-Foundations, Mail

Wed, Sep 1

herron awarded T284215: Splunk oncall / victorops mobile app logout tracking a Party Time token.
Wed, Sep 1, 8:26 PM · SRE Observability (FY2021/2022-Q1)

Tue, Aug 31

herron updated the task description for T289624: Q1: (Need By: TBD) rack/setup/install centrallog2002.codfw.wmnet.
Tue, Aug 31, 4:20 PM · SRE, observability, SRE Observability (FY2021/2022-Q1), ops-codfw, DC-Ops
herron added a comment to T289036: Use Grizzly for Varnish SLO Grafana dashboard.

Thanks @ema! This is helpful feedback

Tue, Aug 31, 2:56 PM · SRE Observability (FY2021/2022-Q1), SRE, Traffic

Mon, Aug 30

herron added a comment to T290031: New VictorOps user request for Arnoldokoth .

Hi @Arnoldokoth welcome! I've just created an account for you, and you should see a VictorOps invite in your email shortly.

Mon, Aug 30, 8:25 PM · SRE Observability (FY2021/2022-Q1)
herron added a comment to T225125: Migrate Elasticsearch from deprecated Gelf logstash input to rsyslog Kafka logging pipeline.

With the elastic SSPL changes that happened this year (T272111 T272238 etc.) is ES7 still a part of the roadmap for moving these logs away from gelf?

Mon, Aug 30, 6:42 PM · SRE Observability, observability, Discovery-Search, Elasticsearch, SRE, Wikimedia-Logstash
herron created T290012: Add service SLO URL to template.
Mon, Aug 30, 3:32 PM · SRE Observability (FY2021/2022-Q1)
herron created T290009: Add Budget Burndown Panels to SLO Dashboard Template.
Mon, Aug 30, 3:07 PM · Patch-For-Review, SRE Observability (FY2021/2022-Q1)

Thu, Aug 26

herron awarded T283507: decommission logstash102[012] a Party Time token.
Thu, Aug 26, 2:51 PM · SRE, DC-Ops, ops-eqiad, observability, decommission-hardware

Tue, Aug 24

herron added a comment to T288989: beta logstash servers run out of disk space.

As a short term stopgap I've cleaned daemon.log manually on deployment-logstash0[456] (same thing done on all)

Tue, Aug 24, 7:50 PM · Release-Engineering-Team (Radar), SRE Observability (FY2021/2022-Q2), Beta-Cluster-Infrastructure
herron created T289615: Migrate existing SLO related metrics to recording rules.
Tue, Aug 24, 7:19 PM · Patch-For-Review, SRE Observability (FY2021/2022-Q1)

Mon, Aug 23

herron added a comment to T252773: Move kafkamon hosts to Debian Buster.

I opted to remove role::kafka::monitoring in favor of role::kafka::monitoring_buster so the config wouldn't be disrupted when retiring the old hosts. Will upload a patch to update the cumin alias.

Mon, Aug 23, 6:24 PM · SRE Observability (FY2021/2022-Q1), Analytics-Clusters, Analytics-Radar, SRE

Aug 19 2021

herron closed T252773: Move kafkamon hosts to Debian Buster as Resolved.

Old hosts have been retired and the duplicate role cleaned up, resolving!

Aug 19 2021, 6:03 PM · SRE Observability (FY2021/2022-Q1), Analytics-Clusters, Analytics-Radar, SRE
herron closed T252773: Move kafkamon hosts to Debian Buster, a subtask of T234629: Move the Analytics infrastructure to Debian Buster, as Resolved.
Aug 19 2021, 6:03 PM · Analytics-Clusters, Analytics-Kanban
herron updated the task description for T252773: Move kafkamon hosts to Debian Buster.
Aug 19 2021, 6:01 PM · SRE Observability (FY2021/2022-Q1), Analytics-Clusters, Analytics-Radar, SRE

Aug 18 2021

herron added a comment to T171482: Programmatic generation of grafana dashboards.

Sounds good, yes grizzly deploys the jsonnet/grafonnet approach outlined in the task description and good progress has been made putting that in place.

Aug 18 2021, 5:02 PM · Observability-Metrics, SRE Observability (FY2021/2022-Q1), User-fgiunchedi, SRE
herron moved T274374: Extend Retention of Alerts (Icinga) in Logstash from In progress to Done on the SRE Observability (FY2021/2022-Q1) board.
Aug 18 2021, 2:40 PM · SRE Observability (FY2021/2022-Q1)
herron closed T274374: Extend Retention of Alerts (Icinga) in Logstash as Resolved.
Aug 18 2021, 2:26 PM · SRE Observability (FY2021/2022-Q1)
herron closed T274374: Extend Retention of Alerts (Icinga) in Logstash, a subtask of T274372: Improve Automation for Alert Reviews, as Resolved.
Aug 18 2021, 2:26 PM · Observability-Alerting

Aug 16 2021

herron closed T284233: kafka-logging hosts running out of space on /srv as Resolved.

Disk util on kafka-logging hosts has been stable for 70+ days now, resolving

Aug 16 2021, 3:52 PM · SRE Observability (FY2021/2022-Q1)
herron moved T288028: Remove the "Long running screen/tmux" Icinga check from Inbox to In progress on the SRE Observability (FY2021/2022-Q1) board.
Aug 16 2021, 3:49 PM · Observability-Alerting, Patch-For-Review, SRE
herron moved T288620: Document path forward for how to Retire all non-Kafka Logstash inputs from Inbox to In progress on the SRE Observability (FY2021/2022-Q1) board.
Aug 16 2021, 3:49 PM · Patch-For-Review, Goal, SRE Observability (FY2021/2022-Q1)
herron closed T251644: Icinga refresh hardware selection (2020) as Resolved.
Aug 16 2021, 3:49 PM · SRE Observability (FY2021/2022-Q1), SRE
herron closed T288122: New VictorOps user request as Resolved.

Hi @MatthewVernon, I see your VO account is now active and you are present in the SRE Batphone rotation as well.

Aug 16 2021, 3:48 PM · SRE Observability (FY2021/2022-Q1)

Aug 13 2021

herron added a comment to T288825: Rebalance kafka partitions in main-{eqiad,codfw} clusters.

Nice! Regarding upstream improvements, on a related note there will hopefully in the future be better control over partition movement within Kafka itself with https://cwiki.apache.org/confluence/display/KAFKA/KIP-435%3A+Internal+Partition+Reassignment+Batching and similar work (although afaict is currently stalled). But splitting them out manually seems fine for now.

Aug 13 2021, 6:21 PM · Services (watching), User-herron, SRE

Aug 11 2021

herron closed T287938: Expand logstash SSD tier in codfw/eqiad as Resolved.

Resolving as new hosts and extended SSD retention are in place now. Let's reopen if any issues arise.

Aug 11 2021, 1:53 PM · Patch-For-Review, SRE Observability (FY2021/2022-Q1)
herron moved T287938: Expand logstash SSD tier in codfw/eqiad from Up next to In progress on the SRE Observability (FY2021/2022-Q1) board.
Aug 11 2021, 1:51 PM · Patch-For-Review, SRE Observability (FY2021/2022-Q1)

Aug 5 2021

herron added a comment to T287938: Expand logstash SSD tier in codfw/eqiad.

New hosts are live in both sites, and shards are relocating onto the new hosts. Next step will be to increase retention on the SSD tier, I think we can safely double it.

Aug 5 2021, 7:22 PM · Patch-For-Review, SRE Observability (FY2021/2022-Q1)
herron added a watcher for SRE Observability (FY2021/2022-Q3): herron.
Aug 5 2021, 4:22 PM
herron added a watcher for SRE Observability (FY2021/2022-Q2): herron.
Aug 5 2021, 4:22 PM
herron added a watcher for SRE Observability (FY2021/2022-Q1): herron.
Aug 5 2021, 4:22 PM
herron added a comment to T288258: ~35k /var/log/logstash/logstash_jvm_gc* files on some logstash hosts.

Removing the pid from the filename would help keep these under control, and we could increase the filecount to keep more history on disk if needed. Are there any dependencies on the filename format?

Aug 5 2021, 4:20 PM · SRE Observability (FY2021/2022-Q1)
herron triaged T288258: ~35k /var/log/logstash/logstash_jvm_gc* files on some logstash hosts as Medium priority.
Aug 5 2021, 4:15 PM · SRE Observability (FY2021/2022-Q1)

Aug 4 2021

herron added a comment to T288028: Remove the "Long running screen/tmux" Icinga check.

+1 to removing the check. We also have since enabled shell TMOUT which helps clean up cases where shells are left idle. Currently that's a 5 day timeout.

Aug 4 2021, 7:24 PM · Observability-Alerting, Patch-For-Review, SRE
herron added a comment to T288122: New VictorOps user request.

Welcome @MatthewVernon! I've created an account for you in VO with SRE team membership, and you should be receiving an invite via email.

Aug 4 2021, 7:02 PM · SRE Observability (FY2021/2022-Q1)
herron added a comment to T225005: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345].

Plan looks good to me!

Aug 4 2021, 5:06 PM · Analytics-Radar, Patch-For-Review, Services (watching), Platform Team Legacy (Watching / External), User-herron, SRE

Aug 3 2021

herron updated the task description for T287938: Expand logstash SSD tier in codfw/eqiad.
Aug 3 2021, 2:37 PM · Patch-For-Review, SRE Observability (FY2021/2022-Q1)
herron added a comment to T287938: Expand logstash SSD tier in codfw/eqiad.

Along with deploying these we should extend retention on the SSD tier

Aug 3 2021, 2:34 PM · Patch-For-Review, SRE Observability (FY2021/2022-Q1)
herron renamed T287938: Expand logstash SSD tier in codfw/eqiad from Put logstash hardware in service to Expand logstash SSD tier in codfw/eqiad.
Aug 3 2021, 2:32 PM · Patch-For-Review, SRE Observability (FY2021/2022-Q1)
herron closed T234854: Upgrade ELK Stack to version 7 as Resolved.

alerting and several other cleanup patches merged

Aug 3 2021, 2:26 PM · SRE Observability (FY2021/2022-Q1), observability, Patch-For-Review, SRE, Wikimedia-Logstash
herron closed T234854: Upgrade ELK Stack to version 7, a subtask of T272655: Phatality doesn't work with Kibana 7, as Resolved.
Aug 3 2021, 2:25 PM · SRE Observability, observability, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Wikimedia-Logstash, Phatality

Aug 2 2021

herron closed T287793: mx1001 alerting for 2043 mails in exim queue as Resolved.

This alert has cleared and the queue is now ~50% below the icinga threshold.

Aug 2 2021, 4:11 PM · Infrastructure-Foundations, SRE, Mail

Jul 27 2021

herron closed T281266: Decommission old ELK5 Logstash cluster as Resolved.

All elk5 hardware has been decommed at this point.

Jul 27 2021, 5:39 PM · SRE Observability (FY2021/2022-Q1), Patch-For-Review, SRE
herron added a parent task for T287496: decommission servers logstash202[012].codfw.wmnet: T281266: Decommission old ELK5 Logstash cluster.
Jul 27 2021, 5:30 PM · decommission-hardware
herron added a subtask for T281266: Decommission old ELK5 Logstash cluster: T287496: decommission servers logstash202[012].codfw.wmnet.
Jul 27 2021, 5:30 PM · SRE Observability (FY2021/2022-Q1), Patch-For-Review, SRE
herron assigned T287496: decommission servers logstash202[012].codfw.wmnet to Papaul.
Jul 27 2021, 5:29 PM · decommission-hardware
herron updated the task description for T287496: decommission servers logstash202[012].codfw.wmnet.
Jul 27 2021, 5:29 PM · decommission-hardware
herron triaged T287496: decommission servers logstash202[012].codfw.wmnet as Medium priority.
Jul 27 2021, 4:31 PM · decommission-hardware

Jul 22 2021

herron updated the task description for T286065: Switch buffer re-partition - Eqiad Row C.
Jul 22 2021, 3:10 PM · Patch-For-Review, DBA, Analytics, Infrastructure-Foundations, SRE, netops
herron updated the task description for T286065: Switch buffer re-partition - Eqiad Row C.
Jul 22 2021, 2:49 PM · Patch-For-Review, DBA, Analytics, Infrastructure-Foundations, SRE, netops

Jul 19 2021

herron added a comment to T253810: Alert on ECC warnings in SEL.

I've PoC this with check_ipmi_sensor which supports checking SEL
...
The downside of this approach is potentially old SEL entries that we'll have to clear as they are surfaced on first deployment. Going forward, the SEL will need clearing for such errors to let the icinga alert actually clear. Since if we deploy this we'll be routinely clear the SEL on errors, I think it is important to log its entries elsewhere too and for that we can deploy freeipmi-ipmiseld which polls SEL and logs to syslog.

Jul 19 2021, 4:56 PM · User-MoritzMuehlenhoff, Wikimedia-Incident, observability, SRE
herron added a comment to T225005: Replace and expand kafka main hosts (kafka[12]00[123]) with kafka-main[12]00[12345].

sure, sounds good to me!

Jul 19 2021, 2:53 PM · Analytics-Radar, Patch-For-Review, Services (watching), Platform Team Legacy (Watching / External), User-herron, SRE
herron added a comment to T286911: Upgrade MXes to Bullseye.

+1 for option 2, I think that will be a more straightforward approach overall.

Jul 19 2021, 2:26 PM · SRE, Patch-For-Review, Infrastructure-Foundations, Mail

Jul 1 2021

herron triaged T285949: Redirect https://lists.wikimedia.org/pipermail/foobar/ to https://lists.wikimedia.org/hyperkitty/list/foobar@lists.wikimedia.org/ as Medium priority.
Jul 1 2021, 5:34 PM · SRE, Wikimedia-Mailing-lists
herron triaged T285569: Automated uploads of minimal & comprehensible timeseries metrics for statuspage display as Medium priority.
Jul 1 2021, 5:29 PM · User-jbond, SRE-OnFire, Patch-For-Review, observability, SRE
herron triaged T285769: Ensure SRE team has a good understanding of how & when to declare an outage on the status page; & it is easy to do so as Medium priority.
Jul 1 2021, 5:29 PM · SRE Observability (FY2021/2022-Q1), SRE-OnFire, SRE
herron triaged T285931: Grant Access to mediawiki gerrit group for divec as Medium priority.
Jul 1 2021, 5:27 PM · MediaWiki-Gerrit-Group-Requests, SRE
herron triaged T285936: Please add btullis@wikimedia.org to the analytics-alerts mailing list as Medium priority.
Jul 1 2021, 5:27 PM · SRE
herron added a comment to T285936: Please add btullis@wikimedia.org to the analytics-alerts mailing list.

Hi @BTullis, sure, I've just added you to analytics-alerts and you should be receiving these emails now.

Jul 1 2021, 5:26 PM · SRE
herron triaged T285927: Add the possibility to deploy calico on kubernetes master nodes as Medium priority.
Jul 1 2021, 5:18 PM · Patch-For-Review, Kubernetes, Machine-Learning-Team, SRE, serviceops
herron triaged T285835: Thanos bucket operations sporadic errors as High priority.
Jul 1 2021, 5:17 PM · Patch-For-Review, SRE Observability (FY2021/2022-Q1), User-fgiunchedi, SRE
herron triaged T285534: mtail testing infrastructure prints python deprecation warnings as Medium priority.
Jul 1 2021, 5:16 PM · good first task, SRE, observability
herron triaged T285533: mtail testing infrastructure does not report Runtime errors as Medium priority.
Jul 1 2021, 5:15 PM · observability, SRE
herron triaged T256641: Delay spinner showing for graphs for 1s as Medium priority.
Jul 1 2021, 5:13 PM · Patch-For-Review, serviceops, SRE, Graphoid
herron closed T285580: Grant Access to ldap/wmf for fgoodwin as Resolved.

Hi @FGoodwin, your ldap account has been added to group wmf. I'll transition this to resolved now, but please don't hesitate to reopen if any followup is needed. Thanks!

Jul 1 2021, 2:57 PM · SRE, LDAP-Access-Requests
herron triaged T285899: Root access to AQS cluster as Medium priority.
Jul 1 2021, 2:42 PM · SRE, Platform Engineering, SRE-Access-Requests
herron moved T285899: Root access to AQS cluster from Untriaged to SRE Meeting Review on the SRE-Access-Requests board.

Looks reasonable to me, and thanks much for writing the patch!

Jul 1 2021, 2:41 PM · SRE, Platform Engineering, SRE-Access-Requests
herron closed T285877: New production ssh key for sbassett as Resolved.

Key updated, but gerrit unable to update task due to policy. Resolving!

Jul 1 2021, 2:12 PM · SecTeam-Processed, SRE-Access-Requests, SRE, Security

Jun 30 2021

herron added a comment to T285877: New production ssh key for sbassett.

Verified face to face via a google meet session

Jun 30 2021, 7:38 PM · SecTeam-Processed, SRE-Access-Requests, SRE, Security
herron closed T285326: Grant Access to ldap/wmf for TChin as Resolved.

Hi @tchin, your ldap account is now a member of the wmf group. I'll transition to resolved now but please don't hesitate to reopen if any follow-up is needed. Thanks!

Jun 30 2021, 6:38 PM · SRE, LDAP-Access-Requests
herron added a comment to T285754: Requesting access to analytics cluster for Ben Tullis.

@herron, so we should do step 1 and then help Ben do step 2?

Jun 30 2021, 4:01 PM · LDAP-Access-Requests, SRE, SRE-Access-Requests
herron added a comment to T285754: Requesting access to analytics cluster for Ben Tullis.

I also have an item on my checklist to say that I should be in the cn=ops LDAP group.

There are instructions on how I can add myself to that group, but only once I have sudo access.

Can anyone confirm this requirement? If so, can it be done on this ticket, or should I raise a new one?

Jun 30 2021, 3:02 PM · LDAP-Access-Requests, SRE, SRE-Access-Requests

Jun 29 2021

herron added a comment to T285754: Requesting access to analytics cluster for Ben Tullis.

Shell account has been created, and ldap account has been added to group wmf

Jun 29 2021, 6:53 PM · LDAP-Access-Requests, SRE, SRE-Access-Requests
herron updated the task description for T285754: Requesting access to analytics cluster for Ben Tullis.
Jun 29 2021, 6:19 PM · LDAP-Access-Requests, SRE, SRE-Access-Requests
herron moved T285754: Requesting access to analytics cluster for Ben Tullis from Untriaged to Manager/NDA Approval/Confirmation on the SRE-Access-Requests board.

Sure I'll go ahead and prep a patch. I may have missed it, but what realname should be used for btullis?

Jun 29 2021, 6:10 PM · LDAP-Access-Requests, SRE, SRE-Access-Requests
herron added a comment to T285754: Requesting access to analytics cluster for Ben Tullis.

@razzi will take care of this, and I will follow up with SRE on enabling root access after the initial access is granted.

Jun 29 2021, 5:22 PM · LDAP-Access-Requests, SRE, SRE-Access-Requests
herron updated the task description for T285754: Requesting access to analytics cluster for Ben Tullis.
Jun 29 2021, 5:15 PM · LDAP-Access-Requests, SRE, SRE-Access-Requests

Jun 28 2021

herron removed a project from T277629: Create new group for root access to snapshot*, dumpsdata* and labstore1006,7 with holger in it: SRE-Access-Requests.

Hey @ArielGlenn, Since this has been idling in the access request queue for some time I'm going to untag SRE-Access-Requests for the time being. If any follow up is needed please do re-tag. Thanks!

Jun 28 2021, 7:36 PM · SRE, Dumps-Generation
herron moved T285436: Access request to superset for user natalia-rodriguez from Awaiting User Input to Manager Approval Pending on the LDAP-Access-Requests board.
Jun 28 2021, 7:29 PM · SRE, LDAP-Access-Requests
herron moved T285580: Grant Access to ldap/wmf for fgoodwin from Awaiting User Input to Manager Approval Pending on the LDAP-Access-Requests board.
Jun 28 2021, 7:29 PM · SRE, LDAP-Access-Requests
herron assigned T285326: Grant Access to ldap/wmf for TChin to tchin.

Hi @tchin could you please coordinate obtaining a comment of approval on this task from your manager?

Jun 28 2021, 7:29 PM · SRE, LDAP-Access-Requests
herron updated the task description for T285580: Grant Access to ldap/wmf for fgoodwin.
Jun 28 2021, 7:26 PM · SRE, LDAP-Access-Requests
herron reassigned T285580: Grant Access to ldap/wmf for fgoodwin from MNadrofsky to FGoodwin.

Hi @FGoodwin could you please coordinate obtaining a comment of approval on this task from your manager?

Jun 28 2021, 7:26 PM · SRE, LDAP-Access-Requests
herron moved T285436: Access request to superset for user natalia-rodriguez from Backlog to Awaiting User Input on the LDAP-Access-Requests board.
Jun 28 2021, 6:55 PM · SRE, LDAP-Access-Requests
herron assigned T285436: Access request to superset for user natalia-rodriguez to NRodriguez.

Hi @NRodriguez there are a couple steps to check off in order to move forward on this request. When you have a moment could you please...

Jun 28 2021, 2:44 PM · SRE, LDAP-Access-Requests
herron updated the task description for T285436: Access request to superset for user natalia-rodriguez.
Jun 28 2021, 2:30 PM · SRE, LDAP-Access-Requests
herron updated the task description for T285436: Access request to superset for user natalia-rodriguez.
Jun 28 2021, 2:30 PM · SRE, LDAP-Access-Requests

Jun 24 2021

herron closed T279342: Migrate colocated kafka-logging brokers to dedicated kafka-logging hosts as Resolved.
Jun 24 2021, 6:52 PM · Patch-For-Review, observability
herron updated the task description for T279342: Migrate colocated kafka-logging brokers to dedicated kafka-logging hosts.
Jun 24 2021, 6:51 PM · Patch-For-Review, observability

Jun 14 2021

herron closed Restricted Task, a subtask of T234854: Upgrade ELK Stack to version 7, as Resolved.
Jun 14 2021, 5:02 PM · SRE Observability (FY2021/2022-Q1), observability, Patch-For-Review, SRE, Wikimedia-Logstash
herron closed T234854: Upgrade ELK Stack to version 7 as Resolved.
Jun 14 2021, 3:58 PM · SRE Observability (FY2021/2022-Q1), observability, Patch-For-Review, SRE, Wikimedia-Logstash
herron updated the task description for T234854: Upgrade ELK Stack to version 7.
Jun 14 2021, 3:58 PM · SRE Observability (FY2021/2022-Q1), observability, Patch-For-Review, SRE, Wikimedia-Logstash
herron closed T234854: Upgrade ELK Stack to version 7, a subtask of T272655: Phatality doesn't work with Kibana 7, as Resolved.
Jun 14 2021, 3:57 PM · SRE Observability, observability, Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), Wikimedia-Logstash, Phatality
herron closed T247014: ELK7 shards failed errors when loading saved objects, e.g. "field expansion matches too many fields, limit: 1024, got: 1726" as Resolved.
Jun 14 2021, 3:56 PM · observability, Patch-For-Review, SRE, Wikimedia-Logstash