We can surely come up with a new config to drop buckets that are not needed, but there is a caveat, namely that the annotation to use to reduce/customize the buckets (needs to be applied to all pods running istio) is available from 1.19 onward, and we are running 1.15.x.

- name: sidecar.istio.io/statsHistogramBuckets
  featureStatus: Alpha
  description: Specifies the custom histogram buckets with a prefix matcher to separate the Istio mesh metrics from the Envoy stats, e.g. {"istio":[1,5,10,50,100,500,1000,5000,10000],"envoy":[1,5,10,25,50,100,250,500,1000,2500,5000,10000]}. Default buckets are [0.5,1,5,10,25,50,100,250,500,1000,2500,5000,10000,30000,60000,300000,600000,1800000,3600000].

We are going to upgrade soon-ish when all the k8s clusters will be upgrade as well, but it may take some months. In the meantime, we could simply drop the unnecessary buckets via labeldrop, but we should verify it the procedure is sound.

The only concern I have with dropping metrics based on a given label is that the _sum and _count values will no longer reflect the actual buckets stored in the TSDB.
It might be pointless for us, but it's just something to keep in mind

In T392886#10780265, @tappof wrote:

The only concern I have with dropping metrics based on a given label is that the _sum and _count values will no longer reflect the actual buckets stored in the TSDB.
It might be pointless for us, but it's just something to keep in mind

@tappof definitely, we should try to figure out if we care about it or not. In my head _sum and _count would still make sense (namely lead to consistent results) even if we dropped some labels, but I am not sure if something could become inconsistent. Do you have anything on top of your head that could affect us?

I found https://istio.io/latest/docs/ops/common-problems/upgrade-issues/#use-a-proxy-annotation-to-customize-the-histogram-bucket-sizes that could be useful, namely applying an envoy filter in our Istio version. It is known that metrics and envoy filters sometimes don't play well (see https://github.com/istio/istio/issues/39772 for example) but for our use case it should be fine. I'd be inclined to test it on ml-staging to see how it goes.

Change #1143584 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: allow Istio gateways to customize histogram buckets

https://gerrit.wikimedia.org/r/1143584

Change #1144612 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/docker-images/production-images@master] istio: introduce legacy images to backport features

https://gerrit.wikimedia.org/r/1144612

Change #1144612 abandoned by Elukey:

[operations/docker-images/production-images@master] istio: introduce legacy images to backport features

Reason:

Doesn't work sigh, more detailed explanation in the task.

https://gerrit.wikimedia.org/r/1144612

Change #1143584 abandoned by Elukey:

[operations/deployment-charts@master] admin_ng: allow Istio gateways to customize histogram buckets

https://gerrit.wikimedia.org/r/1143584