Reporting a conversation on IRC:

08:23  <claime> elukey: can I have your take on this ? https://phabricator.wikimedia.org/T338357
08:27  <elukey> claime: o/ I think that we'd need more details about it, like how much time is being waited (is it 
                unreasonable or not?) and if there are specific jobs that have wait times etc..
08:28  <elukey> a while ago we spread more the load on kafka brokers and increased partitions for some big topics
08:28  <elukey> but the improvement in perfs was a side effect, it wasn't part of why we did the rebalance
08:29  <elukey> we do have some brokers doing less than others, see
08:29  <elukey> https://grafana.wikimedia.org/d/000000027/kafka?orgId=1&viewPanel=48
08:30  <elukey> so we could think about a rebalance but I'd prefer more details before scheduling the work, so far 
                it seems that we are trying something with Kafka without many proofs about it being the culprit
08:30  <elukey> I'll write something in the task

s/kafka/Eventgate/

Eventgate might be the issue here, given its history of flaky performance and random erroring out. Maybe it's kafka indeed, but more investigation is needed.

There look to be some pretty big p99 latency spikes on the eventgate side of things, at least in the last 3 hours, sometimes bumping up to 2.5 seconds.

These correlate mostly with kafka RTT spikes to kafka-main1002, as well as with spikes in produce requests from the 'restbase' envoy cluster, as well as spikes in produce requests for the change-prop.transcludes.resource-change stream.

This (eqiad) topic has 3 partitions:

13:00:38 [@kafka-main1002:/home/otto] $ kafka topics --describe --topic eqiad.change-prop.transcludes.resource-change
kafka-topics --zookeeper conf1007.eqiad.wmnet,conf1008.eqiad.wmnet,conf1009.eqiad.wmnet/kafka/main-eqiad --describe --topic eqiad.change-prop.transcludes.resource-change
Topic:eqiad.change-prop.transcludes.resource-change	PartitionCount:3	ReplicationFactor:3	Configs:
	Topic: eqiad.change-prop.transcludes.resource-change	Partition: 0	Leader: 1003	Replicas: 1003,1004,1002	Isr: 1002,1003,1004
	Topic: eqiad.change-prop.transcludes.resource-change	Partition: 1	Leader: 1001	Replicas: 1001,1005,1003	Isr: 1001,1003,1005
	Topic: eqiad.change-prop.transcludes.resource-change	Partition: 2	Leader: 1002	Replicas: 1002,1003,1004	Isr: 1002,1003,1004

Only one of which (Partition 2) has kafka-main1002 as the leader broker.

There are also corresponding spikes in resource-purge messages too, (but I don' see these reflected in the eventgate-main dash?)

resource-purge has 5 partitions, balanced across all 5 brokers. change-prop.transcludes.resource-change has 3 partitions, only using 3 of the brokers.

Its possible that one of or both of this change-prop.transcludes.resource-change and/or eventgate-main should be scaled out. I'm not sure that load on this partition is this problem, but it could be?

I would consider:

• bumping both eqiad.change-prop.transcludes.resource-change and codfw.change-prop.transcludes.resource-change topics to 5 partitions in both Kafka main clusters, and rebalancing the topic.
• increasing # of eventgate-main replicas.

Change 928566 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/deployment-charts@master] eventgate-main: Increase number of replicas

https://gerrit.wikimedia.org/r/928566

Change 928566 merged by jenkins-bot:

[operations/deployment-charts@master] eventgate-main: Increase number of replicas

https://gerrit.wikimedia.org/r/928566

kafka topics --topic eqiad.change-prop.transcludes.resource-change --alter --partitions 5
kafka topics --topic codfw.change-prop.transcludes.resource-change --alter --partitions 5

Given the size of the topic I think it is good to expand the partitions to 5, the commands above should do it. Shall I proceed?

Mentioned in SAL (#wikimedia-operations) [2023-06-09T09:57:08Z] <elukey> increase {eqiad,codfw}.change-prop.transcludes.resource-change topic partitions (3->5) on kafka main clusters - T338357

I rechecked the eqiad topic metrics after a day and it seems not using 5 partitions, but 3. I think that the ChangeProp's producer client is not able to use the two extra ones, so effectively the load is not spread in 5 partitions yet :(

Nope I was wrong, the 5 partitions are used, but the partition leaders are still 1001->1003:

Topic:eqiad.change-prop.transcludes.resource-change     PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 0    Leader: 1003    Replicas: 1003,1004,1002        Isr: 1002,1003,1004
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 1    Leader: 1001    Replicas: 1001,1005,1003        Isr: 1001,1003,1005
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 2    Leader: 1002    Replicas: 1002,1003,1004        Isr: 1002,1003,1004
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 3    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 4    Leader: 1002    Replicas: 1002,1003,1004        Isr: 1002,1003,1004

By the way, the capacity increase (from 5 to 8 replicas) didn't change the picture in any noticeable way.

HTTPS p99 by source (via envoy)

the partition leaders are still 1001->1003

Rats.
https://wikitech.wikimedia.org/wiki/Kafka/Administration#Rebalance_topic_partitions_to_new_brokers

expand the partitions to 5, the commands above should do it. Shall I proceed?

@elukey just verifying that you did this for both kafka main-eqiad and main-codfw?

In T338357#8922718, @Ottomata wrote:

the partition leaders are still 1001->1003

Rats.
https://wikitech.wikimedia.org/wiki/Kafka/Administration#Rebalance_topic_partitions_to_new_brokers

Yeah I think we'd need to manually re-assign.. There are ISRs on 1004 and 1005, but they are not really used as partition leaders.

expand the partitions to 5, the commands above should do it. Shall I proceed?

@elukey just verifying that you did this for both kafka main-eqiad and main-codfw?

Yep correct!

I think that past me wrote https://wikitech.wikimedia.org/wiki/Kafka/Administration#Rebalance_topic_partitions, so we could probably run something like (1004 and 1005 should become partition leaders for the partition 3 and 4):

{
  "partitions": [
    {"topic": "eqiad.change-prop.transcludes.resource-change", "partition": 0, "replicas": [1003,1004,1002]},
    {"topic": "eqiad.change-prop.transcludes.resource-change", "partition": 1, "replicas": [1005,1001,1003]},
    {"topic": "eqiad.change-prop.transcludes.resource-change", "partition": 2, "replicas": [1002,1003,1004]},
    {"topic": "eqiad.change-prop.transcludes.resource-change", "partition": 3, "replicas": [1001,1002,1003]},
    {"topic": "eqiad.change-prop.transcludes.resource-change", "partition": 4, "replicas": [1004,1003,1002]}
  ],
  "version":1
}

The current status is:

Topic:eqiad.change-prop.transcludes.resource-change     PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 0    Leader: 1003    Replicas: 1003,1004,1002        Isr: 1002,1003,1004
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 1    Leader: 1001    Replicas: 1001,1005,1003        Isr: 1001,1003,1005
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 2    Leader: 1002    Replicas: 1002,1003,1004        Isr: 1002,1003,1004
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 3    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 4    Leader: 1002    Replicas: 1002,1003,1004        Isr: 1002,1003,1004

Then we repeat the same for all 4 topics in eqiad/codfw. Thoughts?

Might be easiest to keep partitions 0-2 on their current leaders, and change the default leadership for the new partitions 4 and 5? So maybe:

{
  "partitions": [
    {"topic": "eqiad.change-prop.transcludes.resource-change", "partition": 0, "replicas": [1003,1004,1002]},
    {"topic": "eqiad.change-prop.transcludes.resource-change", "partition": 1, "replicas": [1001,1005,1003]},
    {"topic": "eqiad.change-prop.transcludes.resource-change", "partition": 2, "replicas": [1002,1003,1004]},
    {"topic": "eqiad.change-prop.transcludes.resource-change", "partition": 3, "replicas": [1005,1001,1003]},
    {"topic": "eqiad.change-prop.transcludes.resource-change", "partition": 4, "replicas": [1004,1003,1002]}
  ],
  "version":1
}

I just swapped your assignment of partition 1 with partition 3, to keep partition 1 the same.

That way kafka doesn't have to move the older stuff?

But either way, this topic isn't that big, and shouldn't take that long to move things, if your assignment is more balanced for some reason.

Then we repeat the same for all 4 topics in eqiad/codfw. Thoughts?

+1 go for it

Mentioned in SAL (#wikimedia-operations) [2023-06-13T15:00:29Z] <elukey> run kafka re-assign partitions for eqiad.change-prop.transcludes.resource-change on kafka-main1001 - T338357

Ran a preferred replica election and:

Topic:eqiad.change-prop.transcludes.resource-change     PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 0    Leader: 1003    Replicas: 1003,1004,1002        Isr: 1002,1003,1004
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 1    Leader: 1001    Replicas: 1001,1005,1003        Isr: 1001,1003,1005
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 2    Leader: 1002    Replicas: 1002,1003,1004        Isr: 1002,1003,1004
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 3    Leader: 1005    Replicas: 1005,1001,1003        Isr: 1001,1003,1005
        Topic: eqiad.change-prop.transcludes.resource-change    Partition: 4    Leader: 1004    Replicas: 1004,1003,1002        Isr: 1002,1003,1004

Next steps:

• Check if anything changes in the latency graphs, and in case update the other related topics (codfw/eqiad).

for the last hour (15:00 UTC) it is now at 5.9% from 30%: https://performance.wikimedia.org/arclamp/svgs/hourly/2023-06-13_15.excimer-wall.RunSingleJob.reversed.svgz?x=555.0&y=1301 which is good but still high tbh

The following topic may need more partitions:

Topic:eqiad.mediawiki.job.RecordLintJob PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: eqiad.mediawiki.job.RecordLintJob        Partition: 0    Leader: 1005    Replicas: 1005,1001,1002        Isr: 1001,1002,1005

From this link (isolate the topic for a better view) it seems that the volume of messages is high enough to warrant 3/5 partitions.

I am going to execute:

kafka topics --topic eqiad.mediawiki.job.RecordLintJob --alter --partitions 5
kafka topics --topic codfw.mediawiki.job.RecordLintJob --alter --partitions 5

On both clusters.

Same for:

Topic:eqiad.mediawiki.job.refreshLinks  PartitionCount:1        ReplicationFactor:3     Configs:
        Topic: eqiad.mediawiki.job.refreshLinks Partition: 0    Leader: 1004    Replicas: 1004,1003,1001        Isr: 1001,1003,1004

I'd say yes to both. I guess we'll need to restart changeprop-jobqueue to make it effective though?

New layout:

Topic:eqiad.mediawiki.job.RecordLintJob PartitionCount:5        ReplicationFactor:3     Configs:
        Topic: eqiad.mediawiki.job.RecordLintJob        Partition: 0    Leader: 1005    Replicas: 1005,1001,1002        Isr: 1001,1002,1005
        Topic: eqiad.mediawiki.job.RecordLintJob        Partition: 1    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1001,1002,1003
        Topic: eqiad.mediawiki.job.RecordLintJob        Partition: 2    Leader: 1002    Replicas: 1002,1003,1004        Isr: 1002,1003,1004
        Topic: eqiad.mediawiki.job.RecordLintJob        Partition: 3    Leader: 1003    Replicas: 1003,1004,1001        Isr: 1003,1004,1001
        Topic: eqiad.mediawiki.job.RecordLintJob        Partition: 4    Leader: 1004    Replicas: 1004,1001,1002        Isr: 1004,1001,1002

Mentioned in SAL (#wikimedia-operations) [2023-07-05T13:55:50Z] <elukey> expand kafka topic partitions from 1 to 5 for {codfw,eqiad}.mediawiki.job.RecordLintJob and {eqiad,codfw}.mediawiki.job.refreshLinks on kafka-main eqiad/codfw - T338357

In T338357#8990672, @Joe wrote:

I'd say yes to both. I guess we'll need to restart changeprop-jobqueue to make it effective though?

In theory no, from this graph it seems that jobs are nicely produced in multiple partitions :)

In T338357#8990722, @elukey wrote:

In T338357#8990672, @Joe wrote:

I'd say yes to both. I guess we'll need to restart changeprop-jobqueue to make it effective though?

In theory no, from this graph it seems that jobs are nicely produced in multiple partitions :)

To keep archives happy - I was wrong, we needed to roll restart changeprop-jobqueue (the above link was for changeprop, wrong one sorry).

Some thoughts from IRC:

15:26  <elukey> there are two things that we may try to set in node-rdkafka:
15:26  <elukey> https://github.com/Blizzard/node-rdkafka/blob/master/config.d.ts#L683
15:26  <elukey> https://github.com/Blizzard/node-rdkafka/blob/master/config.d.ts#L739
15:27  <elukey> linger.ms states how much time a producer (in this case, changeprop) waits to collect messages before sending
15:27  <elukey> and batch.size states how many msgs to collect before sending to the broker
15:27  <elukey> IIUC the first limit that is hit wins
15:28  <elukey> we don't set anything afaics, so we have the defaults:
15:28  <elukey> linger.ms: 5
15:28  <elukey> batch.size: 1000000
15:28  <elukey> I don't see the latter being hit in our case, the former is maybe a little tight?

Maybe we could try to experiment with linger.ms? Something like 10/15 to start with, and see if anything changes in our metrics.

This seems orthogonal to the problem stated in the description, but I am wondering if Changeprop causes pressure on the Kafka brokers that in turn are slower for EventGate.

Opened T341140 to see if we can upgrade the node-rdkafka package version as well (we run a version from 3 years ago).

Change 935772 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] changeprop: increase the linger.ms value

https://gerrit.wikimedia.org/r/935772

Change 935772 merged by Elukey:

[operations/deployment-charts@master] changeprop: increase the linger.ms value

https://gerrit.wikimedia.org/r/935772

Mentioned in SAL (#wikimedia-operations) [2023-07-06T15:54:24Z] <elukey> changeprop's kafka linger.ms set to 20s - T338357 (was 5ms, now changeprop waits a bit more to batch messages to send to kafka in one go)

The changeprop linger.ms increase doesn't seem to have changed much, but I'd like to see how it goes when I'll apply it to the jobqueue's changeprop as well.

Anyway, I think that we should start using a metric to concentrate on, and so far the EventGate's P99 seemed to be the most promising.

If we concentrate on EventGate, this needs to be remembered:

# Producer type specific overrides.
# If you need to configure some producer specific settings,
# e.g. different batch settings, you can provide them here.
hasty:
  conf:
    # HastyProducer doesn't block HTTP clients, so we can
    # afford to wait for a largish batch size.
    queue.buffering.max.ms: 1000
    # A custom kafka config. Will call producer.setPollInterval
    # with this value if set.
    producer.poll.interval.ms: 100
guaranteed:
  conf:
    # GuaranteedProducer does block HTTP clients, so we attempt to send
    # the produce request as soon as possible, rather than waiting
    # for larger batches.
    queue.buffering.max.ms: 0
    # Custom kafka config, will call producer.setPollInterval
    # with this value if set.
    producer.poll.interval.ms: 10

Most of our producers are guaranteed afaics from the metrics, and queue.buffering.max.ms is an alias for linger.ms (see https://github.com/Blizzard/node-rdkafka/blob/master/config.d.ts). This means that EventGate's guaranteed sends messages as soon as it has any in the queue, potentially hitting the kafka brokers with short bursts of requests that may slow them down (raising up the latency etc..).

What we should do in my opinion is to find a metric that matches with the P99 increase, either on the broker side or on the EventGate's side, to figure out what part of the chain slows down (otherwise we may end up rolling out a ton of features without any benefit).

To be clear - I didn't say that it is EventGate's fault, just that we should try to work on it since it shows a clear high latency at P99. We had another task like this one, that was solved by T288825, that points me in the direction of the Kafka brokers' threads being saturated by bursts of traffic.

Change 936304 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::kafka: update prometheus config

https://gerrit.wikimedia.org/r/936304

Some useful pointers:

• https://github.com/confluentinc/librdkafka/issues/1836
• https://www.confluent.io/blog/5-common-pitfalls-when-using-apache-kafka/#key-broker-metrics

The former states how setting queue.buffering.max.ms: 0 may lead to some extra load on the broker side, and the latter is about metrics that we should care.

I added request/response queue size, network processor avg idle percent, request handler idle percent (still not working, see https://gerrit.wikimedia.org/r/936304), Produce/FetchFollower/FetchConsumer time.

I need to review graphs and metrics, but overall there are some considerations:

• The request queue size (max 500 by default IIUC) sometimes shows values up to 140:

https://grafana-rw.wikimedia.org/d/000000027/kafka?orgId=1&forceLogin&viewPanel=69

Other clusters don't have such a queue, see for example Jumbo (that gets way more produce requests / second in theory).

• We have the default network threads set for kafka, IIUC either 3 or 5 for each broker, but they don't seem to be busy all the time.

https://grafana-rw.wikimedia.org/d/000000027/kafka?orgId=1&forceLogin&var-datasource=thanos&var-kafka_cluster=main-eqiad&var-cluster=kafka_main&var-kafka_broker=All&var-disk_device=All&viewPanel=74

• Produce request time (total) seems to be very high, but I am not 100% sure why (in other clusters is way lower, maybe it is a PEBCAK from me since it is Friday).

https://grafana-rw.wikimedia.org/d/000000027/kafka?orgId=1&forceLogin&var-datasource=thanos&var-kafka_cluster=main-eqiad&var-cluster=kafka_main&var-kafka_broker=All&var-disk_device=All&viewPanel=71

@Ottomata I am wondering if EventGate's queue.buffering.max.ms: 0 may be too tight, not enough to saturate the request queue but enough to cause some delays every now and then. I understand its impact for Mediawiki's clients, but would some value like 5 considerable to see how Kafka behaves? I mean if the queues start to get down after it.

. I understand its impact for MediaWiki's clients, but would some value like 5 considerable to see how Kafka behaves? I mean if the queues start to get down after it.

Yeah, for eventgate-main? I'd so go for it. It actually shouldn't matter to much for EventBus emitted events, as those are sent in a deferred HTTP request anyway, after the MediaWiki HTTP response has been closed.

Change 936515 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] services: allow kafka batches in EventGate's main producer

https://gerrit.wikimedia.org/r/936515

Change 936304 merged by Elukey:

[operations/puppet@production] profile::kafka: update prometheus config

https://gerrit.wikimedia.org/r/936304

Really nice graph:

https://grafana-rw.wikimedia.org/d/000000027/kafka?forceLogin&orgId=1&from=now-3h&to=now&var-datasource=thanos&var-kafka_cluster=main-eqiad&var-cluster=kafka_main&var-kafka_broker=All&var-disk_device=All&viewPanel=75

It should represent how idle kafka threads are when processing requests, and it is clearly visible that kafka-main100[1-3] are busy and kafka-main100[45] are way less. If we check the distribution of the partitions there is a correlation, and this also explains why in the past we improved latencies moving partitions around.

Change 936515 merged by Elukey:

[operations/deployment-charts@master] services: allow kafka batches in EventGate's main producer

https://gerrit.wikimedia.org/r/936515

I have deployed the change to eventgate-main, so far no visible differences on kafka broker metrics but let's see if anything gets better with the envoy latency.

Given the clear imbalance in business for brokers, I'd suggest to have another round of T288825 to move partitions around

The eventgate p99 envoy latencies looks better in my opinion right after the batching change, I see way less high spikes than before (SAL 2023-07-10 13:27 - https://sal.toolforge.org/log/DXP8P4kBxE1_1c7sgV8G)

Kafka max latencies are also showing a big change :)

@Ladsgroup do you see any difference in the jobrunner's metrics by any chance?

Change 937041 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] services: increase kafka batch wait time for eventgate-main

https://gerrit.wikimedia.org/r/937041

Can I just say that this is pretty awesome? Especially the max latencies for kafka are pretty telling. Keep up the good work on this one!

Change 937041 merged by Elukey:

[operations/deployment-charts@master] services: increase kafka batch wait time for eventgate-main

https://gerrit.wikimedia.org/r/937041

In T338357#9003992, @elukey wrote:

The eventgate p99 envoy latencies looks better in my opinion right after the batching change, I see way less high spikes than before (SAL 2023-07-10 13:27 - https://sal.toolforge.org/log/DXP8P4kBxE1_1c7sgV8G)

Kafka max latencies are also showing a big change :)

@Ladsgroup do you see any difference in the jobrunner's metrics by any chance?

The latest hour I could find says 6.8% is being spent on pushing jobs: https://performance.wikimedia.org/arclamp/svgs/hourly/2023-07-11_08.excimer-wall.RunSingleJob.reversed.svgz?x=555.0&y=1301
Let's wait for a bit until new flamegraphs trickle in.

Change 937432 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] eventgate: set a more performant default for queue.buffering.max.ms

https://gerrit.wikimedia.org/r/937432

In T338357#9004520, @Ladsgroup wrote:

In T338357#9003992, @elukey wrote:

The eventgate p99 envoy latencies looks better in my opinion right after the batching change, I see way less high spikes than before (SAL 2023-07-10 13:27 - https://sal.toolforge.org/log/DXP8P4kBxE1_1c7sgV8G)

Kafka max latencies are also showing a big change :)

@Ladsgroup do you see any difference in the jobrunner's metrics by any chance?

The latest hour I could find says 6.8% is being spent on pushing jobs: https://performance.wikimedia.org/arclamp/svgs/hourly/2023-07-11_08.excimer-wall.RunSingleJob.reversed.svgz?x=555.0&y=1301
Let's wait for a bit until new flamegraphs trickle in.

Ok I have finally found where to check for the percentage in the flame graph, I'll stop asking the same question again :)

I see from https://performance.wikimedia.org/arclamp/svgs/hourly/2023-07-11_14.excimer-wall.RunSingleJob.reversed.svgz?x=475.4&y=933 that we are still at 7%, but we cannot do much more on the EventGate side.

Next step is T341558 :)

Another really nice graph:

The above is the latency observed by kafka producers towards kafka main-eqiad.

The change in latency starts around the 11th, exactly when we completed the eventagate's deployment (increased batch):
https://sal.toolforge.org/log/giozRIkBGiVuUzOd7jQG

After the first round of rebalances in kafka main-eqiad I didn't see any noticeable change in the average idle time for the kafka request threads, so something else is causing more load to 100[1-3].

See this graph for more visual infos.

The load on brokers is way nicer now, the issue was that we didn't have enough threads to handle requests from producers. The latency is now around one second (on avg), still not perfect but way better than before.

I checked some flame graphs and we are still at 6/7% when pushing events to the job queues, at this point not sure how much performance we can get in tuning kafka configs.

I have some hopes from T341140, namely upgrading the old changeprop's kafka client, but it may also not lead to any improvements.

@Ladsgroup do we have a target percentage in the flame graphs that we want to reach? What is something that you'd consider acceptable? To organize next steps :)

I'm sorry for the delay. I will write a script to download flamegraphs in the past two years to get a sense of what has been the usual value.

Hope this helps

It shows clear improvement but has not gone back to the previous state

Honestly I find it weird that the baseline of time share in jobs is 5% for just queuing further jobs, what are queuing? airplanes?

In T338357#9051781, @Ladsgroup wrote:

Honestly I find it weird that the baseline of time share in jobs is 5% for just queuing further jobs, what are queuing? airplanes?

Not saying it's because of kafka, I think maybe some parameters for some jobs are massive. I'll need to dig

Change 937432 merged by Elukey:

[operations/deployment-charts@master] eventgate: set a more performant default for queue.buffering.max.ms

https://gerrit.wikimedia.org/r/937432

@Ladsgroup should we close this ticket or do you think that more work is needed?

In T338357#9252759, @elukey wrote:

@Ladsgroup should we close this ticket or do you think that more work is needed?

good question, it's much better but not back to the April values (T338357#9051034). At least at the time of the comment: T338357#9051034. I can run it again to see if it has improved since then. I think some investigation is needed why it's slower than April (or any time before that) but I wonder if we can improve something in mediawiki side to make queuing jobs faster. It can be a dedicated ticket maybe?