I wanted to understand how multi-DC ness relates to all the pieces here. Just writing down what I found:

• Kafka jumbo-eqiad is only in eqiad
• LiftWing is only in eqiad (right?)
• Cassandra is multi DC
• change-prop is multi DC

Because MediaWiki its active/passive, mediawiki.page_change.v1 will (mostly) only be produced to one of its DC prefixed topics at a time. E.g. if eqiad is active, only the eqiad.mediawiki.page_change.v1 topic will have events produced to it.

change-prop is deployed multi DC in wikikube k8s in eqiad and codfw. In each datacenter, it only consumes from its local DC prefixed topics. E.g. in eqiad change-prop only consumes from eqiad.mediawiki.page_change.v1. It then calls out to LiftWing, which I believe only exists in eqiad(?) Example rule config here: https://gitlab.wikimedia.org/-/snippets/105#L892

LiftWing is only in eqiad, so it will only write to Cassandra in eqiad. (I don't know much about Cassandra Multi-DCness).

So, if we do Option B. or Option C. (allow change-prop to consume from Kafka jumbo), when codfw is the active datacenter, the data flow will go like this:

MediaWiki in codfw 
-> Kafka main-codfw topic codfw.mediawiki.page_change.v1
-> mediawiki_page_content_change_enrich Flink Job in codfw produce
-> |cross DC| Kafka jumbo-eqiad `codfw.mediawiki.page_content_change.v1` topic
-> |cross DC| codfw change-prop consumes `codfw.mediawiki.page_content_change.v1` from Kafka jumbo-eqiad
-> |cross DC| call LiftWing in eqiad
-> LiftWing write to Cassandra in eqiad

That is 3 cross DC hops for this pipeline.

(There may also be a cross DC write when producing weighted tags events from LiftWing too, but I think there isn't. cc @dcausse)

If we do Option A. Produce mediawiki.page_content_change.v1 to Kafka main, this looks like:

Change-Prop and liftwing do:

MediaWiki in codfw 
-> Kafka main-codfw topic codfw.mediawiki.page_change.v1
-> mediawiki_page_content_change_enrich Flink Job in codfw produce
-> Kafka main-codfw `codfw.mediawiki.page_content_change.v1` topic
-> codfw change-prop consumes `codfw.mediawiki.page_content_change.v1` from Kafka main-codfw
-> |cross DC| call LiftWing in eqiad
-> LiftWing write to Cassandra in eqiad

Only one crossing of the DC boundary (because LiftWing is only in eqiad).

LiftWing is only in eqiad (right?)

LiftWing is in both eqiad and codfw

Option C. New change-prop rule consuming from Kafka jumbo

I was looking into this. Would it just be adding the kafka-jumbo brokers (kafka-jumbo101x.eqiad.wmnet) into the broker_list in changeprop's values-eqiad.yaml, without requiring a new rule in the helm chart? @elukey, since we worked on change-prop's chart before, wdyt?

Option A would require some talk with SRE but given the size of the topic and the current /srv usage in main-eqiad / codfw I don't see any big opposition in having the stream hosted there (especially if we advertise that ML will not need to query the mediawiki API as direct consequence for the use case). It would probably be the most clean and reliable option in my opinion.

Option C should work, in theory both Changeprops (eqiad and codfw) will listen/pull from eqiad/codfw prefixed topics without duplicating work, the only downside would be the cross-dc request from say Changeprop codfw to kafka Jumbo (as Andrew anticipated). After checking the helmfile settings I fear that the chart assumes only one kafka cluster to use, so either main or jumbo, but we'd need to dig a little bit more to confirm or deny this assumption :)

Option D is also fine in my opinion, we'll pay an extra call to the mediawiki API but this stream is low volume and we don't really need any strict latency deadlines/perfomance to handle it.

Option A would require some talk with SRE but given the size of the topic and the current /srv usage in main-eqiad / codfw I don't see any big opposition in having the stream hosted there (especially if we advertise that ML will not need to query the mediawiki API as direct consequence for the use case). It would probably be the most clean and reliable option in my opinion.

I agree Option A would be the preferred option. To move forward, what's the process for getting approval from SRE ServiceOps? Who should we talk to? cc @klausman

@Ottomata, what work is required to produce mediawiki.page_content_change.v1 to Kafka main? I'm expecting just some helmfile changes, for example, in the mw-page-content-change-enrich/values-codfw.yaml, and not requiring changes in mediawiki event enrichment code, right?

what work is required to produce mediawiki.page_content_change.v1 to Kafka main? I'm expecting just some helmfile changes, for example, in the mw-page-content-change-enrich/values-codfw.yaml, and not requiring changes in mediawiki event enrichment code, right?

Yeah, it should just be a helmfile change.

Hi @Joe! The Machine Learning and Growth teams are collaborating on a GrowthExperiments newcomer task for revising tone (associated hypotheses are WE1.1.2 & WE1.1.17).

@Ottomata mentioned that they considered producing mediawiki.page_content_change.v1 to Kafka main but deferred it since use cases were mostly data-lake oriented. We now have a real-time use case and want to revisit that.

Producing mediawiki.page_content_change.v1 to Kafka main would enable real-time updates for derived data products like the tone-suggestion workflow. We prefer this approach since it avoids relying on the MediaWiki API. While the original concern was large event bodies, current topic size and /srv usage suggest it should be fine as @elukey noted in the comment. It's likely the cleanest and most reliable option.

I'd appreciate hearing your thoughts on this. I'm reaching out to you since I found your discussion with the Data Engineering team from two years ago in T330507#8735183.

@Ottomata, what work is required to produce mediawiki.page_content_change.v1 to Kafka main? I'm expecting just some helmfile changes, for example, in the mw-page-content-change-enrich/values-codfw.yaml, and not requiring changes in mediawiki event enrichment code, right?

As the PyFlink application already lives in the main K8s cluster, and already reads from the main Kafka cluster, I believe that changing the sink to write into the main Kafka cluster should be enough. It means it will stop writing in Jumbo, start writing in a new topic in Main, and Mirror Maker will copy all new messages into the Jumbo cluster, in the same topic as the PyFlink application is writing now. Offsets should be fine as the application and consumer are the same.

I don't think we can have both versions of PyFlink working at the same time, Mirror Maker would copy messages already processed, making everything duplicated.

Changing the sink in the application shouldn't affect other applications reading the topics in Jumbo either.

If we get approval to move it to Kafka main, I can create the MR with the changes.

@achou, do you have a timeline for this initiative? So we can prioritize it.

If pushing to kafka-main you might need to increase broker's message.max.bytes see T344688.

Would we also need to explicitly create the topics in main? Is auto topic creation enabled there?

In T409469#11374597, @tchin wrote:

Would we also need to explicitly create the topics in main? Is auto topic creation enabled there?

indeed, and even if auto topic creation is enabled we probably want to set the number of partitions to 5 like other "big" topics?

That sounds good. Then, we could consider increasing the partitions in Jumbo too, codfw.mediawiki.page_content_change.v1 and eqiad.mediawiki.page_content_change.v1 both have 3 partitions right now. I'm not sure if any consumer In Jumbo relies on the message ordering that could be affected by the change in partitions, I'm guessing not.

Thank you for the discussion everyone! Reading through, I would suggest proceeding with Option D for the time being. This approach not only unblocks the work without requiring any significant changes to Kafka, but also allows us to observe the workflow in practice and better understand its requirements.
That said, we can later define a set of performance expectations (eg for latency), which will then help us to assess whether any of the other options would provide sufficient benefit to justify any additional efforts. Thoughts?

Hi, thanks all for the input. :) Due to our tight timeline, ML team has decided to move forward with Option D for now.

That said, we can later define a set of performance expectations (eg for latency), which will then help us to assess whether any of the other options would provide sufficient benefit to justify any additional efforts. Thoughts?

I agree! We should follow up on this and revisit the topic in the future. ML team would really like to see this work happen, as we will have other similar use cases that could benefit from mediawiki.page_content_change.v1 in Kafka main.

I'm very happy you're going with the option @jijiki recommended, which sounds like both the path of least resistance and the best option.

Having said that, let me state this for posterity: I don't think that, as it stands, moving such a big topic to kavka-main is an option. If we want to have such fat topics (on top of all the other very busy topics on -main) in that cluster, we need an hardware expansion. I don't think it's an option also because it seems like there's a lot of duplication with the topic already existing on kafka-main.

So if we ever want to move forward with moving that topic, we must reason about why we'd need both page_content_change and page_change as topics, and consolidate the needs in a single flow.