Linking some really good thoughts from Javier from the parent task: T421216#11792886
...and responding here.

I was unsure if increasing partitions is what will solve our html flink pipeline woes, and advised waiting to try it. But, Javier makes some really good points about how even if increasing partitions will not necessarily solve our problems, increasing to more than one partition is probably just a good idea in general.

Okay! I'm convinced!

Ideally, we should replicate the configuration to Main too.

Since we intend one day too add more partitions to kafka main topics too, we should check with other consumers of mediawiki.page_change.v1 topics too. Specifically Search Update Pipeline (Search Platform Team), WDQS (Wikidata platform team), and other change-prop based usages, which I think are mostly just LiftWing (ML platform team).

I don't expect any of these to have issues with more partitions, but we should verify with them just in case.

@dcausse @gmodena @gkyziridis please provide your respective team input.

In T422511#11800369, @Ahoelzl wrote:

@dcausse @gmodena @gkyziridis please provide your respective team input.

+1 from me. Do you have a timeline in mind?

We'll need to coordinate a deployment of rdf-streaming-updater that enables partition discovery (and possibly parallelism tweaks - checking).

Flink 1.20 has partition discovery enabled by default (and we don't override that behavior). Parallelism, TM replicas and TM slots settings for the production deployment seem well aligned with the partition increase.

We might need to update staging since it reads from the main kafka topics with parallelism=1, but no big deal. I'll file a task on Wikidata Platform.

Do you have a timeline in mind?

It's very quick, so as soon as people agree with doing it, we can do it.

In T422511#11800369, @Ahoelzl wrote:

@dcausse @gmodena @gkyziridis please provide your respective team input.

Unless flink is doing something special during dynamic partition discovery it might use its default offset reset strategy? If yes and they use latest offsets they might lose events the time the app is restarted or partitions discovered:

• IIRC wdqs uses latest and may lose events for at most 5min (default partition discovery interval)
• search update pipeline is latest as well and might lose 10s of events (kafka-source-config.partition.discovery.interval.ms explicit set to 10000)

I have a question regarding canary events, if the canary events only flow to a single partition this might create an imbalance on flink applications using it, esp. some partitions will rely on source idleness and some will rely on the canary to have the watermark move when the topic goes idle, this might not be ideal I think because the way to wake up/ignore an idle page-change topic might vary depending on the partition.

if the canary events only flow to a single partition this might create an imbalance on flink applications

Uh, wow no I have never considered that we might need canaries on all topic partitions. It is not impossible to do if this is a real requirement.

But, perhaps it would be better if applications didn't rely on canaries for watermark advancement? They are more intended for a pipeline health check than a pipeline 'liveness probe' :)

In T422511#11805626, @Ottomata wrote:

But, perhaps it would be better if applications didn't rely on canaries for watermark advancement? They are more intended for a pipeline health check than a pipeline 'liveness probe' :)

Flink apps generally set source idleness and don't expect canary events but unfortunately they can't avoid these events to still wake-up idle sources (unless there are ways to apply a filter very early at the source level?).
I actually don't know if this is a problem to have partitions properly idle and a single one being woken up at regular intervals, it might just be OK but sounded a bit odd to have only one partition receiving this traffic.

Based on a meeting today (notes here) it sounded like adding more partitions may actually hinder our current efforts in T421216: HTML Enrichment - Tuning & Backfilling configuration. @JMonton-WMF should we hold on actually doing this?

Ok! we can wait until everything works. I still believe 3 partitions for a topic like this is better than 1, from the Kafka perspective, but if we are not gonna get any benefits now, we can wait and asses this again.

Unless flink is doing something special during dynamic partition discovery it might use its default offset reset strategy? If yes and they use latest offsets they might lose events the time the app is restarted or partitions discovered:

• IIRC wdqs uses latest and may lose events for at most 5min (default partition discovery interval)

Urgh. You are right.

FLIP-288 suggests that later discovered partitions will reset from earliest. But this has been implemented in kafka connector >= 4.0.0 (we are still on 3.x.x).

In T422511#11807568, @gmodena wrote:

Unless flink is doing something special during dynamic partition discovery it might use its default offset reset strategy? If yes and they use latest offsets they might lose events the time the app is restarted or partitions discovered:

• IIRC wdqs uses latest and may lose events for at most 5min (default partition discovery interval)

Urgh. You are right.

FLIP-288 suggests that later discovered partitions will reset from earliest. But this has been implemented in kafka connector >= 4.0.0 (we are still on 3.x.x).

Oh nice, thanks for the link! yes defaulting to the initial reset strategy was kind of counter intuitive, glad to see they fixed this.
Something I just remember and perhaps to keep in mind when we do the partitioning change, the wdqs updater might flag new partition messages as late events if the discovery happens late, so possibly reducing the discovery interval to something like 30s might be interesting for wdqs to avoid a flood of late events being reconciled.

Through the task: https://phabricator.wikimedia.org/T429127 we stablished a process to decide partitions based on size. This topic is 30GB and the applications we were building are already working with 1 partition. I think we can close this task.