Change #1135748 had a related patch set uploaded (by Cwhite; author: Filippo Giunchedi):

[operations/puppet@production] logstash: temporarily disable curator forcemerge

https://gerrit.wikimedia.org/r/1135748

Change #1135748 merged by Cwhite:

[operations/puppet@production] logstash: temporarily disable curator forcemerge

https://gerrit.wikimedia.org/r/1135748

Change #1135740 merged by Cwhite:

[operations/puppet@production] logstash: move err field mitigation to effective place

https://gerrit.wikimedia.org/r/1135740

Change #1135948 had a related patch set uploaded (by Filippo Giunchedi; author: Filippo Giunchedi):

[operations/puppet@production] grafana: set max_source_resolution=auto for thanos ds

https://gerrit.wikimedia.org/r/1135948

In T390215#10730310, @gerritbot wrote:

Change #1135740 merged by Cwhite:

[operations/puppet@production] logstash: move err field mitigation to effective place

https://gerrit.wikimedia.org/r/1135740

FTR turns out this wasn't the cause, in fact disabling forcemerge had little impact one way or another AFAICT. I've followed up on T391661 re: enabling it again

We're at it again today, best correlation (i.e. not necessarily causation!) is indexing time vs time spent querying vs kafka lag, e.g. here for today's events https://w.wiki/DpuD

My current theory is that heavy queries over large indices all with one shard cause opensearch to slow down significantly, enough to cause indexing to slow down and thus kafka backpressure.

What might help to further research this theory is turning on shard slow logs: https://docs.opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/#shard-slow-logs

Ideally also search request slow logs, though that's a opensearch 2.12 feature: https://docs.opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/#search-request-slow-logs

Change #1137057 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop out_request field

https://gerrit.wikimedia.org/r/1137057

Change #1137057 merged by Cwhite:

[operations/puppet@production] logstash: drop out_request field

https://gerrit.wikimedia.org/r/1137057

Noting for future reference that Logstash has not received almost any events from MediaWiki in the past two hours or so (since 17:05 UTC), and this is apparently the reason.

Change #1137064 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: also remove outRequest field

https://gerrit.wikimedia.org/r/1137064

Change #1137064 merged by Cwhite:

[operations/puppet@production] logstash: also remove outRequest field

https://gerrit.wikimedia.org/r/1137064

Change #1137065 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: expand conditional

https://gerrit.wikimedia.org/r/1137065

Change #1137065 merged by Cwhite:

[operations/puppet@production] logstash: expand conditional

https://gerrit.wikimedia.org/r/1137065

Update: we saw immediate ingest improvement after removing the out_request and outRequest fields which were generated by mobileapps. We should watch for more inexplicable dips in throughput in the coming days if this was an incomplete mitigation.

We uncovered these fields as problematic by chance. These fields happened to show up in an analysis of the DLQ as they exceeded the 2048 index key limit.

To detection, more graphs should be provisioned to watch OpenSearch for spikes in time spent parsing and writing to index.

Adoption of ECS and improvements around schema validation will prevent issues like this from occurring in the future.

In T390215#10746778, @fgiunchedi wrote:

We're at it again today, best correlation (i.e. not necessarily causation!) is indexing time vs time spent querying vs kafka lag, e.g. here for today's events https://w.wiki/DpuD

My current theory is that heavy queries over large indices all with one shard cause opensearch to slow down significantly, enough to cause indexing to slow down and thus kafka backpressure.

What might help to further research this theory is turning on shard slow logs: https://docs.opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/#shard-slow-logs

Ideally also search request slow logs, though that's a opensearch 2.12 feature: https://docs.opensearch.org/docs/latest/install-and-configure/configuring-opensearch/logs/#search-request-slow-logs

FWIW I have held off on those for now, easy enough to turn slow logs whenever we need it though

Checking back on this, it seems the situation hasn't improved much.

I looked around at the logs and found that the ml hosts are more than half the logs in the k8s indexes. I think this may be a good place to partition.

Change #1145339 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: create partition for ml logs

https://gerrit.wikimedia.org/r/1145339

Change #1145339 merged by Cwhite:

[operations/puppet@production] logstash: create partition for ml logs

https://gerrit.wikimedia.org/r/1145339

Change #1152079 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop request.query field

https://gerrit.wikimedia.org/r/1152079

Change #1152079 merged by Cwhite:

[operations/puppet@production] logstash: drop request.query field

https://gerrit.wikimedia.org/r/1152079

Change #1152088 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove mw-script large fields

https://gerrit.wikimedia.org/r/1152088

Change #1152088 merged by Cwhite:

[operations/puppet@production] logstash: remove mw-script large fields

https://gerrit.wikimedia.org/r/1152088

Change #1152093 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: mw-script remove text field

https://gerrit.wikimedia.org/r/1152093

Change #1152093 merged by Cwhite:

[operations/puppet@production] logstash: mw-script remove text field

https://gerrit.wikimedia.org/r/1152093

Change #1152095 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove heading field

https://gerrit.wikimedia.org/r/1152095

Change #1152095 merged by Cwhite:

[operations/puppet@production] logstash: mw-script remove heading and outgoing_link field

https://gerrit.wikimedia.org/r/1152095

Change #1152108 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop logs from DumpIndex.php

https://gerrit.wikimedia.org/r/1152108

Change #1153279 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: Reroute apache.access logs to webrequest partition

https://gerrit.wikimedia.org/r/1153279

@colewhite Can you give us a summary of the state of this problem?

This task's scope expanded a bit since its initial inception. Some applications were logging large JSON objects that would slow down parsing within OpenSearch generating backpressure on Logstash and reducing ingest capacity to well below the rate of enqueue. This has largely been mitigated and a permanent fix is in the works.

Current state is we're still ingesting a volume of logs that keeps us closer to overrunning available disk than is comfortable.

Thank you!

Change #1153279 merged by Cwhite:

[operations/puppet@production] logstash: Reroute apache.access logs to webrequest partition

https://gerrit.wikimedia.org/r/1153279

Change #1160230 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: relocate ecs-default indexes to hdd nodes after 8 weeks

https://gerrit.wikimedia.org/r/1160230

Change #1160875 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop mobileapps detail field

https://gerrit.wikimedia.org/r/1160875

Change #1160875 merged by Cwhite:

[operations/puppet@production] logstash: drop mobileapps detail field

https://gerrit.wikimedia.org/r/1160875

7 weeks of ecs-default indexes have been moved to the HDD nodes and replication reduced on one week of late May indexes to mitigate imminent high watermark problems.

Change #1163053 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: reduce replicas for high-volume logstash-ml logs

https://gerrit.wikimedia.org/r/1163053

Change #1163053 merged by Cwhite:

[operations/puppet@production] logstash: reduce replicas for high-volume logstash-ml logs

https://gerrit.wikimedia.org/r/1163053

Change #1165532 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: reroute mobileapps webrequests

https://gerrit.wikimedia.org/r/1165532

Change #1165532 merged by Cwhite:

[operations/puppet@production] logstash: reroute mobileapps webrequests

https://gerrit.wikimedia.org/r/1165532

Change #1175597 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop logspam

https://gerrit.wikimedia.org/r/1175597

Change #1175597 merged by Cwhite:

[operations/puppet@production] logstash: drop logspam

https://gerrit.wikimedia.org/r/1175597

Change #1189534 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop 99% eventgate-analytics-external logs

https://gerrit.wikimedia.org/r/1189534

Change #1189534 merged by Cwhite:

[operations/puppet@production] logstash: drop 99% eventgate-analytics-external logs

https://gerrit.wikimedia.org/r/1189534

Change #1201827 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: reduce logstash-ml index utilization on ssd-class nodes

https://gerrit.wikimedia.org/r/1201827

Change #1201827 merged by Cwhite:

[operations/puppet@production] logstash: reduce logstash-ml index utilization on ssd-class nodes

https://gerrit.wikimedia.org/r/1201827

Change #1213589 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: drop netdev logspam

https://gerrit.wikimedia.org/r/1213589

Change #1213589 merged by Cwhite:

[operations/puppet@production] logstash: drop netdev logspam

https://gerrit.wikimedia.org/r/1213589

Change #1217539 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: move logstash-ml logs to hdd-class nodes after 7d

https://gerrit.wikimedia.org/r/1217539

Change #1217539 merged by Cwhite:

[operations/puppet@production] logstash: move logstash-ml logs to hdd-class nodes after 7d

https://gerrit.wikimedia.org/r/1217539

Change #1229198 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: remove logstash-ml indices after 70d

https://gerrit.wikimedia.org/r/1229198

Change #1229198 merged by Cwhite:

[operations/puppet@production] logstash: remove logstash-ml indices after 70d

https://gerrit.wikimedia.org/r/1229198

The istio-system namespace is logging ~980 events/sec. Many are just istio-ingressgateway for authority:page-analytics.discovery.wmnet (~833 events/sec).

In T390215#11784911, @colewhite wrote:

The istio-system namespace is logging ~980 events/sec. Many are just istio-ingressgateway for authority:page-analytics.discovery.wmnet (~833 events/sec).

@colewhite afaics those are valid HTTP access logs, the downstream client seems to be the rest gateway for WikiFeeds: traffic

Change #1289423 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: restore sampling of webrequest logs

https://gerrit.wikimedia.org/r/1289423

Change #1289423 merged by Cwhite:

[operations/puppet@production] logstash: add sampling to page-analytics.discovery.wmnet istio logs

https://gerrit.wikimedia.org/r/1289423

@colewhite I was thinking that we could also lower down the retention of the istio access logs to say one or two weeks, if it is 30 days or more now. We really don't care about history at the moment, a couple of weeks or even one should be ok (2 better of course). What do you think?

In T390215#11955723, @elukey wrote:

@colewhite I was thinking that we could also lower down the retention of the istio access logs to say one or two weeks, if it is 30 days or more now. We really don't care about history at the moment, a couple of weeks or even one should be ok (2 better of course). What do you think?

I think it's a good idea. They share indexes with the overall webrequest partition so we'll have to configure a new index pattern.

Change #1298904 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: add drop for istio ratelimit-media.svc

https://gerrit.wikimedia.org/r/1298904

Change #1298904 merged by Cwhite:

[operations/puppet@production] logstash: add drop for istio ratelimit-media.svc

https://gerrit.wikimedia.org/r/1298904

Change #1298907 had a related patch set uploaded (by Cwhite; author: Cwhite):

[operations/puppet@production] logstash: move page-analytics and ratelimit webrequest log filters

https://gerrit.wikimedia.org/r/1298907

Change #1298907 merged by Cwhite:

[operations/puppet@production] logstash: move page-analytics and ratelimit webrequest log filters

https://gerrit.wikimedia.org/r/1298907