Also if you check UA, most logs are simply from "MediaWiki/1.46.0-wmf.13" or "ChangePropagation/WMF". Can we sample these?

I can see that most logs have "kubernetes.pod_name" values like:
"controller-..."
"webhook-..."
"kserve-controller-manager-..."
"istio-ingressgateway-..."
These are system logs generated by kserve, which seem to be the majority.
@DPogorzelski-WMF Is there a way to reduce the system logs from kserve?
@elukey sorry to ping you, but maybe you have some insights here

I think that the best course of action is to split logs by namespace:

• istio-system - https://logstash.wikimedia.org/goto/2b96a9c732952692351003f0b5229bd7 - is likely something related to the Istio ingress, and the volume is not that massive (compared to the total).
• knative-serving - https://logstash.wikimedia.org/goto/07c24f8578daed61f3008dbb51ad86dd - those are related to the knative control plane, I believe we can attempt to tune them down via configuration option (if any).
• kserve - https://logstash.wikimedia.org/goto/182fff55e1474600719ddc8d63d42647 ` those are related to the kserve control plan, same as above.

I think we should focus to find config options for knative and kserve at the moment, it should reduce the firehose by a lot. Most of those logs are something that we don't need, they are emitted by those pods by default IIRC.

Change #1247995 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: reduce logs emitted by knative components on ml-serve

https://gerrit.wikimedia.org/r/1247995

Tested staging and the knative traffic volume dropped:

Change #1247995 merged by Elukey:

[operations/deployment-charts@master] admin_ng: reduce logs emitted by knative components on ml-serve

https://gerrit.wikimedia.org/r/1247995

Knative should be good now:

For the kserve controller we sadly cannot do much from what I can see, since we'd need https://github.com/kserve/kserve/commit/5e7207ca0870e37e72e119845f7f8933ad57ca6a that is available from 0.12 onward (and we have 0.11.2). We could backport it to our image, so after that we'll have all the zap-logging options to use.

@DPogorzelski-WMF @klausman we already have kserve 0.13 in production-images, so in theory we could simply upgrade the control plane + helm chart to include the above commit and reduce the spam to logstash considerably. Lemme know how you want to proceed :)

Change #1250573 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::logstash: drop kserve-controller's logs

https://gerrit.wikimedia.org/r/1250573