Page MenuHomePhabricator
Create Task
Maniphest T390215

Logstash is overwhelmed
Open, MediumPublic
Actions

Assigned To
colewhite
Authored By
colewhite
Mar 27 2025, 4:35 PM
Tags
Referenced Files
F62305955: analyzed_data_2024.12.27-2025.06.12.png
Jun 12 2025, 8:05 PM
F59947935: Screenshot From 2025-05-13 21-53-17.png
May 13 2025, 9:57 PM
F59947936: Screenshot From 2025-05-13 21-52-41.png
May 13 2025, 9:57 PM
F59947937: Screenshot From 2025-05-13 21-51-25.png
May 13 2025, 9:57 PM
F58931465: Screenshot from 2025-03-27 16-33-07.png
Mar 27 2025, 4:35 PM
Subscribers
Aklapper
andrea.denisse
brennen
colewhite
Daimona
elukey
fgiunchedi
View All 10 Subscribers

Description

logstash-k8s indexes have nearly quadrupled in size since 60d ago leading to curator forcemerge failures and running out of space on the ssd hosts.

Screenshot from 2025-03-27 16-33-07.png (370×1 px, 67 KB)

X-Axis is day, Y-Axis is in GB

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
logstash: remove logstash-ml indices after 70doperations/puppetproduction+36 -0
logstash: move logstash-ml logs to hdd-class nodes after 7doperations/puppetproduction+2 -2
logstash: drop netdev logspamoperations/puppetproduction+5 -0
logstash: reduce logstash-ml index utilization on ssd-class nodesoperations/puppetproduction+47 -0
logstash: drop 99% eventgate-analytics-external logsoperations/puppetproduction+8 -0
logstash: drop logspamoperations/puppetproduction+5 -0
logstash: reroute mobileapps webrequestsoperations/puppetproduction+61 -0
logstash: reduce replicas for high-volume logstash-ml logsoperations/puppetproduction+36 -2
logstash: drop mobileapps detail fieldoperations/puppetproduction+8 -7
logstash: relocate ecs-default indexes to hdd nodes after 8 weeksoperations/puppetproduction+50 -0
logstash: Reroute apache.access logs to webrequest partitionoperations/puppetproduction+78 -1
logstash: drop logs from DumpIndex.phpoperations/puppetproduction+3 -6
logstash: mw-script remove heading and outgoing_link fieldoperations/puppetproduction+1 -1
logstash: mw-script remove text fieldoperations/puppetproduction+1 -1
logstash: remove mw-script large fieldsoperations/puppetproduction+7 -0
logstash: drop request.query fieldoperations/puppetproduction+9 -2
logstash: create partition for ml logsoperations/puppetproduction+9 -0
logstash: expand conditionaloperations/puppetproduction+1 -1
logstash: also remove outRequest fieldoperations/puppetproduction+2 -2
logstash: drop out_request fieldoperations/puppetproduction+8 -0
logstash: move err field mitigation to effective placeoperations/puppetproduction+55 -125
logstash: temporarily disable curator forcemergeoperations/puppetproduction+20 -17
api-gateway: Lower ratelimit log leveloperations/deployment-chartsmaster+3 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
colewhite triaged this task as High priority.Mar 27 2025, 4:37 PM

First suspect is kubernetes.container_name:"production-ratelimit" generating debug logs at around 1500/sec.

colewhite updated the task description. (Show Details)Mar 27 2025, 4:43 PM
colewhite updated the task description. (Show Details)
gerritbot added a comment.Mar 27 2025, 4:44 PM

Change #1131777 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] api-gateway: Lower ratelimit log level

https://gerrit.wikimedia.org/r/1131777

gerritbot added a project: Patch-For-Review.Mar 27 2025, 4:44 PM
gerritbot added a comment.Mar 27 2025, 4:49 PM

Change #1131777 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: Lower ratelimit log level

https://gerrit.wikimedia.org/r/1131777

hnowlan mentioned this in T388804: Update api-gateway ratelimit version.Mar 27 2025, 4:53 PM
colewhite changed the task status from In Progress to Open.Mar 27 2025, 5:10 PM
colewhite lowered the priority of this task from High to Medium.

Log volume is down quite a bit since raising the production-ratelimit log level filter. Will check back in on this in a few days to re-evaluate in case there are other applications doing the same thing.

Maintenance_bot removed a project: Patch-For-Review.Mar 27 2025, 5:31 PM
colewhite added a subscriber: elukey.EditedMar 28 2025, 7:05 PM

Checking back today, istio-ingressgateway is logging about 1000/logs/sec these days.

istio-ingressgateway was last discussed in July 2023, during its early stages. I think its time for another look as istio-ingressgateway is solidly half of all logs we get via kubernetes.

@elukey would it be reasonable to either reduce the number of istio-ingressgateway logs stored either via sampling or can we implement reduced retention to less than 90d? Either way is reasonably easy to accomplish.

Spoke too soon about istio - they're not in the k8s indexes!

The biggest contributor to the logstash-k8s indexes now is revertrisk-language-agnostic-predictor-default-00025 at more than half of all logs from k8s at ~378/logs/sec (~32 million per day).

We'll check back in next week to see if the index size is overall back to within reasonable.

gerritbot added a comment.Apr 10 2025, 2:31 PM

Change #1135740 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: move err field mitigation to effective place

https://gerrit.wikimedia.org/r/1135740

gerritbot added a project: Patch-For-Review.Apr 10 2025, 2:31 PM
gerritbot added a comment.Apr 10 2025, 2:56 PM

Change #1135748 had a related patch set uploaded (by Cwhite; author: Filippo Giunchedi):

[operations/puppet@production] logstash: temporarily disable curator forcemerge

https://gerrit.wikimedia.org/r/1135748

gerritbot added a comment.Apr 10 2025, 2:58 PM

Change #1135748 merged by Cwhite:

[operations/puppet@production] logstash: temporarily disable curator forcemerge

https://gerrit.wikimedia.org/r/1135748

gerritbot added a comment.Apr 10 2025, 3:17 PM

Change #1135740 merged by Cwhite:

[operations/puppet@production] logstash: move err field mitigation to effective place

https://gerrit.wikimedia.org/r/1135740

Maintenance_bot removed a project: Patch-For-Review.Apr 10 2025, 3:31 PM
andrea.denisse subscribed.Apr 10 2025, 3:37 PM
gerritbot added a comment.Apr 11 2025, 12:38 PM

Change #1135948 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] grafana: set max_source_resolution=auto for thanos ds

https://gerrit.wikimedia.org/r/1135948

gerritbot added a project: Patch-For-Review.Apr 11 2025, 12:38 PM
fgiunchedi subscribed.Apr 15 2025, 1:01 PM
In T390215#10730310, @gerritbot wrote:

Change #1135740 merged by Cwhite:

[operations/puppet@production] logstash: move err field mitigation to effective place

https://gerrit.wikimedia.org/r/1135740

FTR turns out this wasn't the cause, in fact disabling forcemerge had little impact one way or another AFAICT. I've followed up on T391661 re: enabling it again

fgiunchedi added a comment.Apr 16 2025, 10:19 AM

We're at it again today, best correlation (i.e. not necessarily causation!) is indexing time vs time spent querying vs kafka lag, e.g. here for today's events https://w.wiki/DpuD

My current theory is that heavy queries over large indices all with one shard cause opensearch to slow down significantly, enough to cause indexing to slow down and thus kafka backpressure.

What might help to further research this theory is turning on shard slow logs: https://docs.opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/#shard-slow-logs

Ideally also search request slow logs, though that's a opensearch 2.12 feature: https://docs.opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/#search-request-slow-logs

kostajh subscribed.Apr 16 2025, 7:08 PM
gerritbot added a comment.Apr 16 2025, 7:18 PM

Change #1137057 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop out_request field

https://gerrit.wikimedia.org/r/1137057

gerritbot added a comment.Apr 16 2025, 7:23 PM

Change #1137057 merged by Cwhite:

[operations/puppet@production] logstash: drop out_request field

https://gerrit.wikimedia.org/r/1137057

matmarex subscribed.EditedApr 16 2025, 7:39 PM

Noting for future reference that Logstash has not received almost any events from MediaWiki in the past two hours or so (since 17:05 UTC), and this is apparently the reason.

matmarex mentioned this in T392142: Office Wiki credentials inexplicably stop working.Apr 16 2025, 7:42 PM
gerritbot added a comment.Apr 16 2025, 7:59 PM

Change #1137064 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: also remove outRequest field

https://gerrit.wikimedia.org/r/1137064

gerritbot added a comment.Apr 16 2025, 8:02 PM

Change #1137064 merged by Cwhite:

[operations/puppet@production] logstash: also remove outRequest field

https://gerrit.wikimedia.org/r/1137064

Daimona subscribed.Apr 16 2025, 8:08 PM
gerritbot added a comment.Apr 16 2025, 8:09 PM

Change #1137065 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: expand conditional

https://gerrit.wikimedia.org/r/1137065

gerritbot added a comment.Apr 16 2025, 8:13 PM

Change #1137065 merged by Cwhite:

[operations/puppet@production] logstash: expand conditional

https://gerrit.wikimedia.org/r/1137065

colewhite added a comment.Apr 16 2025, 9:55 PM

Update: we saw immediate ingest improvement after removing the out_request and outRequest fields which were generated by mobileapps. We should watch for more inexplicable dips in throughput in the coming days if this was an incomplete mitigation.

We uncovered these fields as problematic by chance. These fields happened to show up in an analysis of the DLQ as they exceeded the 2048 index key limit.

To detection, more graphs should be provisioned to watch OpenSearch for spikes in time spent parsing and writing to index.

Adoption of ECS and improvements around schema validation will prevent issues like this from occurring in the future.

matmarex unsubscribed.Apr 16 2025, 10:01 PM
fgiunchedi changed the status of subtask T391661: Weekly indices are force-merged by curator every day for a week from Open to Stalled.Apr 23 2025, 9:16 AM
fgiunchedi changed the status of subtask T391687: Consider sharding big logging indices from Open to Stalled.May 6 2025, 12:42 PM
fgiunchedi added a comment.May 6 2025, 12:54 PM
In T390215#10746778, @fgiunchedi wrote:

We're at it again today, best correlation (i.e. not necessarily causation!) is indexing time vs time spent querying vs kafka lag, e.g. here for today's events https://w.wiki/DpuD

My current theory is that heavy queries over large indices all with one shard cause opensearch to slow down significantly, enough to cause indexing to slow down and thus kafka backpressure.

What might help to further research this theory is turning on shard slow logs: https://docs.opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/#shard-slow-logs

Ideally also search request slow logs, though that's a opensearch 2.12 feature: https://docs.opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/#search-request-slow-logs

FWIW I have held off on those for now, easy enough to turn slow logs whenever we need it though

colewhite added a comment.May 13 2025, 9:57 PM

Checking back on this, it seems the situation hasn't improved much.

Screenshot From 2025-05-13 21-51-25.png (363×1 px, 79 KB)

I looked around at the logs and found that the ml hosts are more than half the logs in the k8s indexes. I think this may be a good place to partition.

Screenshot From 2025-05-13 21-52-41.png (482×1 px, 35 KB)

Screenshot From 2025-05-13 21-53-17.png (482×1 px, 35 KB)

gerritbot added a comment.May 13 2025, 10:05 PM

Change #1145339 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: create partition for ml logs

https://gerrit.wikimedia.org/r/1145339

colewhite edited projects, added SRE Observability (FY2024/2025-Q4); removed SRE Observability (FY2024/2025-Q3).May 13 2025, 10:49 PM
gerritbot added a comment.May 15 2025, 4:16 PM

Change #1145339 merged by Cwhite:

[operations/puppet@production] logstash: create partition for ml logs

https://gerrit.wikimedia.org/r/1145339

gerritbot added a comment.May 29 2025, 3:09 PM

Change #1152079 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop request.query field

https://gerrit.wikimedia.org/r/1152079

gerritbot added a comment.May 29 2025, 3:15 PM

Change #1152079 merged by Cwhite:

[operations/puppet@production] logstash: drop request.query field

https://gerrit.wikimedia.org/r/1152079

gerritbot added a comment.May 29 2025, 3:48 PM

Change #1152088 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove mw-script large fields

https://gerrit.wikimedia.org/r/1152088

gerritbot added a comment.May 29 2025, 3:53 PM

Change #1152088 merged by Cwhite:

[operations/puppet@production] logstash: remove mw-script large fields

https://gerrit.wikimedia.org/r/1152088

brennen merged a task: T395579: https://logstash.wikimedia.org/app/dashboards#/view/mediawiki-errors not showing new data.May 29 2025, 3:55 PM
brennen added subscribers: dancy, brennen.

Having merged T395579: https://logstash.wikimedia.org/app/dashboards#/view/mediawiki-errors not showing new data in here, I note a) maybe that was premature, and b) that one's UBN from RelEng perspective, since it's a blocker for deployments.

gerritbot added a comment.May 29 2025, 4:07 PM

Change #1152093 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: mw-script remove text field

https://gerrit.wikimedia.org/r/1152093

gerritbot added a comment.May 29 2025, 4:12 PM

Change #1152093 merged by Cwhite:

[operations/puppet@production] logstash: mw-script remove text field

https://gerrit.wikimedia.org/r/1152093

gerritbot added a comment.May 29 2025, 4:18 PM

Change #1152095 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove heading field

https://gerrit.wikimedia.org/r/1152095

gerritbot added a comment.May 29 2025, 4:26 PM

Change #1152095 merged by Cwhite:

[operations/puppet@production] logstash: mw-script remove heading and outgoing_link field

https://gerrit.wikimedia.org/r/1152095

Jdforrester-WMF subscribed.May 29 2025, 5:01 PM
gerritbot added a comment.May 29 2025, 5:25 PM

Change #1152108 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop logs from DumpIndex.php

https://gerrit.wikimedia.org/r/1152108

jnuche subscribed.May 30 2025, 7:38 AM
gerritbot added a comment.Jun 3 2025, 4:11 PM

Change #1153279 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: Reroute apache.access logs to webrequest partition

https://gerrit.wikimedia.org/r/1153279

dancy added a comment.Jun 9 2025, 3:39 PM

@colewhite Can you give us a summary of the state of this problem?

colewhite added a comment.Jun 12 2025, 6:30 PM

This task's scope expanded a bit since its initial inception. Some applications were logging large JSON objects that would slow down parsing within OpenSearch generating backpressure on Logstash and reducing ingest capacity to well below the rate of enqueue. This has largely been mitigated and a permanent fix is in the works.

Current state is we're still ingesting a volume of logs that keeps us closer to overrunning available disk than is comfortable.

dancy added a comment.Jun 12 2025, 6:42 PM

Thank you!

colewhite added a comment.Jun 12 2025, 8:05 PM

analyzed_data_2024.12.27-2025.06.12.png (400×1 px, 105 KB)

gerritbot added a comment.Jun 16 2025, 7:34 PM

Change #1153279 merged by Cwhite:

[operations/puppet@production] logstash: Reroute apache.access logs to webrequest partition

https://gerrit.wikimedia.org/r/1153279

gerritbot added a comment.Jun 17 2025, 6:24 PM

Change #1160230 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: relocate ecs-default indexes to hdd nodes after 8 weeks

https://gerrit.wikimedia.org/r/1160230

herron changed the status of subtask T391714: Review logging cluster merge pressure from Open to Stalled.Jun 18 2025, 2:44 PM
gerritbot added a comment.Jun 18 2025, 4:09 PM

Change #1160875 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop mobileapps detail field

https://gerrit.wikimedia.org/r/1160875

gerritbot added a comment.Jun 18 2025, 4:13 PM

Change #1160875 merged by Cwhite:

[operations/puppet@production] logstash: drop mobileapps detail field

https://gerrit.wikimedia.org/r/1160875

colewhite added a comment.EditedJun 18 2025, 10:11 PM

7 weeks of ecs-default indexes have been moved to the HDD nodes and replication reduced on one week of late May indexes to mitigate imminent high watermark problems.

colewhite mentioned this in T397366: Degraded RAID on logstash2035.Jun 23 2025, 8:54 PM
gerritbot added a comment.Jun 23 2025, 10:27 PM

Change #1163053 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: reduce replicas for high-volume logstash-ml logs

https://gerrit.wikimedia.org/r/1163053

gerritbot added a comment.Jun 23 2025, 10:54 PM

Change #1163053 merged by Cwhite:

[operations/puppet@production] logstash: reduce replicas for high-volume logstash-ml logs

https://gerrit.wikimedia.org/r/1163053

colewhite edited projects, added SRE Observability (FY2025/2026-Q1); removed SRE Observability (FY2024/2025-Q4).Jun 30 2025, 10:49 PM
gerritbot added a comment.Jul 1 2025, 1:57 PM

Change #1165532 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: reroute mobileapps webrequests

https://gerrit.wikimedia.org/r/1165532

gerritbot added a comment.Jul 1 2025, 2:03 PM

Change #1165532 merged by Cwhite:

[operations/puppet@production] logstash: reroute mobileapps webrequests

https://gerrit.wikimedia.org/r/1165532

colewhite moved this task from Inbox to In progress on the SRE Observability (FY2025/2026-Q1) board.Jul 2 2025, 2:11 PM
gerritbot added a comment.Aug 4 2025, 8:33 PM

Change #1175597 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop logspam

https://gerrit.wikimedia.org/r/1175597

gerritbot added a comment.Aug 4 2025, 8:36 PM

Change #1175597 merged by Cwhite:

[operations/puppet@production] logstash: drop logspam

https://gerrit.wikimedia.org/r/1175597

gerritbot added a comment.Sep 18 2025, 5:10 PM

Change #1189534 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop 99% eventgate-analytics-external logs

https://gerrit.wikimedia.org/r/1189534

gerritbot added a comment.Sep 18 2025, 5:16 PM

Change #1189534 merged by Cwhite:

[operations/puppet@production] logstash: drop 99% eventgate-analytics-external logs

https://gerrit.wikimedia.org/r/1189534

gerritbot added a comment.Nov 4 2025, 11:06 PM

Change #1201827 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: reduce logstash-ml index utilization on ssd-class nodes

https://gerrit.wikimedia.org/r/1201827

gerritbot added a comment.Nov 4 2025, 11:08 PM

Change #1201827 merged by Cwhite:

[operations/puppet@production] logstash: reduce logstash-ml index utilization on ssd-class nodes

https://gerrit.wikimedia.org/r/1201827

gerritbot added a comment.Dec 1 2025, 11:09 PM

Change #1213589 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop netdev logspam

https://gerrit.wikimedia.org/r/1213589

gerritbot added a comment.Dec 1 2025, 11:11 PM

Change #1213589 merged by Cwhite:

[operations/puppet@production] logstash: drop netdev logspam

https://gerrit.wikimedia.org/r/1213589

colewhite added a subtask: T412143: ~5k/logs/sec from netdev.Dec 9 2025, 5:36 PM
gerritbot added a comment.Dec 11 2025, 2:54 PM

Change #1217539 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: move logstash-ml logs to hdd-class nodes after 7d

https://gerrit.wikimedia.org/r/1217539

gerritbot added a comment.Dec 11 2025, 2:59 PM

Change #1217539 merged by Cwhite:

[operations/puppet@production] logstash: move logstash-ml logs to hdd-class nodes after 7d

https://gerrit.wikimedia.org/r/1217539

dancy unsubscribed.Dec 15 2025, 4:43 PM
gerritbot added a comment.Tue, Jan 20, 9:53 PM

Change #1229198 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove logstash-ml indices after 70d

https://gerrit.wikimedia.org/r/1229198

gerritbot added a comment.Wed, Jan 21, 11:08 PM

Change #1229198 merged by Cwhite:

[operations/puppet@production] logstash: remove logstash-ml indices after 70d

https://gerrit.wikimedia.org/r/1229198

colewhite added a subtask: T415289: logging-hd nodes evict-rejoin troubles.Thu, Jan 22, 4:59 PM
Ladsgroup mentioned this in T412637: Remove support for deprecated revisions without rvslots.Fri, Jan 30, 12:42 PM
Ladsgroup mentioned this in T416384: Reduce logstash logs from machine learning infra.Tue, Feb 3, 6:29 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits