Is that equivalent in k8s (having a CPU request and limit of 1 core per Job- and TaskManager pod)?

yes I believe so.

I believe we will have 2 TaskManager pods, so 3 cores (1 per pod).

Do we need a full core? Probably not?

https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#resource

Hi, I have a few questions, if the plan is we move this event flux to kafka-main:

• why is kafka-main a better fit than kafka-jumbo?
• what is the expected rate of production of messages to the topics on kafka-main?
• how many topics?
• Currently we allow a maximum size for messages of 1 MB. Would that be enough for this new set of topics?

I tend to be protective of kafka-main because it's now *absolutely vital* to the correct functioning of all of our websites.

Separately, I assume you have specific long-term plans besides just deploying this service; can you point me to your design documents so that I can maybe answer some of the above questions myself?

why is kafka-main a better fit than kafka-jumbo?

Multi DC. This service is sort of like change prop, except that instead of just hitting an external API, it submits new events itself.
We may be able to do multi DC with kafka stretch, but we haven't really evaluated this fully yet, and the current Kafka multi DC single compute pattern we currently use works well with this kind of enrichment job.

what is the expected rate of production of messages to the topics on kafka-main?

Same as mediawiki.page_change, so ~30 messages per second. Message size will be larger though. In the Kafka stretch evaluation ticket, we did some napkin estimation of the average byte size of this stream, and came up with about 0.5MB / second.

For 2021, the average revision size was 20623 bytes. (The average is quite inflated by big outliers). Let's just round this up and say 30000 bytes. In 2021 there were 533764132 revisions, so 533764132*30000/365/24/60/60 == 507767 bytes / second

how many topics?

For now 4: the two DC prefixed ones: eqiad.mediawki.page_content_change and codfw.mediawki.page_content_change, as well as 2 DC prefixed 'error' topics (name TBD). In the normal case, the error stream will be very low volume. It should only contain events when for whatever reason the MW API fails after retries, or because of some other bug in our pipeline.

However, we may later want to experiment with per-wiki topics (to reduce the amount of data consumers need to consume if they only want changes from small wikis) and also with compacted topics for maintaining and bootstrapping current state into downstream systems (WDQS, Search, etc.). In either case, those topics would be subsets of the two we will create now. It may also be possible to use Kafka jumbo for those use cases. Anyway, we aren't requesting those now.

Currently we allow a maximum size for messages of 1 MB. Would that be enough for this new set of topics?

I believe the max is 4MB. This should be enough for the majority of use cases. For the edge cases where content is huge, we don't (currently) plan to add the content in the event, but instead indicate that the consumer must get the content themselves.

I tend to be protective of kafka-main because it's now *absolutely vital* to the correct functioning of all of our websites.

Ya understood. FWIW, the output mediawiki.page_content_change stream will be used by the search index and WDQS update pipeline, so it is also related to the functioning of websites.

We would be fine any Kafka multi DC kafka cluster. Kafka main is just the only multi DC one we have.

can you point me to your design documents

• T291120: MediaWiki Event Carried State Transfer - Problem Statement
• MediaWiki Event Carried State Transfer: Comprehensiveness - Decision Record
• Event Platform Value Stream - mediawiki.org
• Event Driven Use cases
• Informal pyflink + event enrichment design doc
• Email thread snippet that answers some of the 'what and why Flink' questions

Also, clearly we will not be ready to deploy this by end of March now. We are investigating some python related memory issues in T332166.

Sorry, I'm getting confused; to my understanding, WDQS/search will use mediawiki.page_change which AIUI are generated from mediawiki, not mediawiki.page_content_change.

I would prefer avoiding adding new topics on kafka unless they have a current consumer. Is there a current consumer of this new enriched stream, or any client that plans to use it?

In T330507#8732991, @Joe wrote:

Sorry, I'm getting confused; to my understanding, WDQS/search will use mediawiki.page_change which AIUI are generated from mediawiki, not mediawiki.page_content_change.

this is correct, search & wdqs require a very specific rendering of the page that only MW can produce and thus we will be using the mediawiki.page_change stream + a call to the MW api. If someday the rendering function is externalized out of mediawiki it might perhaps make sense to use the enriched stream mediawiki.page_content_change.

Oh, I misunderstood, I thought that WDQS updater was using the Wikidata content?

In T330507#8734369, @Ottomata wrote:

Oh, I misunderstood, I thought that WDQS updater was using the Wikidata content?

It is, but in its RDF form not JSON (fetched&transformed via Special:EntityData).

@Joe we discussed the use of page_content_change in Kafka main today, and decided that we don't need it for now after all. We may one day, but it looks like all the immediate use cases of it are more data lakey (incremental dumps, mediawiki history, etc.), so Kafka jumbo will be fine.

We'd still like to deploy to wikikube, as DSE is not multi DC and also would prefer not to run long lived apps as this time. This means that while the app would consume page_change from local DC Kafka main, it would always produce page_content_change to Kafka jumbo-eqiad. This means that when MW is pooled in codfw, most page_content_change events would be produced cross DC to Kafka jumbo. FWIW, this data would be produced cross DC links if we were using Kafka main too (via MirrorMaker). However, in this case, the Kafka producer in the app would have to deal with any cross DC latencies, so we'll have to be careful with producer buffering.

(Given that webrequest Kafka producers produce cross DC, I'm hoping we'll be okay :) )

In T330507#8735183, @Ottomata wrote:

@Joe we discussed the use of page_content_change in Kafka main today, and decided that we don't need it for now after all. We may one day, but it looks like all the immediate use cases of it are more data lakey (incremental dumps, mediawiki history, etc.), so Kafka jumbo will be fine.

We'd still like to deploy to wikikube, as DSE is multi DC and also would prefer not to run long lived apps as this time. This means that while the app would consume page_change from local DC Kafka main, it would always produce page_content_change to Kafka jumbo-eqiad. This means that when MW is pooled in codfw, most page_content_change events would be produced cross DC to Kafka jumbo. FWIW, this data would be produced cross DC links if we were using Kafka main too (via MirrorMaker). However, in this case, the Kafka producer in the app would have to deal with any cross DC latencies, so we'll have to be careful with producer buffering.

(Given that webrequest Kafka producers produce cross DC, I'm hoping we'll be okay :) )

Yes I wouldn't be worried about the cross-dc latency or traffic volume for this. Basically it would work as follows:

• mediawiki produces events (page_change) to the kafka-main in the mediawiki master DC
• Flink in that same DC will pick up the events, enrich them, and generate another event to kafka-jumbo in eqiad

Is that correct?

if that's the case, I think we're all set, minus some decisions regarding how to run the flink operator AIUI - but I'll leave @JMeybohm and @akosiaris weigh in on that.

Is that correct?

Correct!

In T330507#8737310, @Joe wrote:

if that's the case, I think we're all set, minus some decisions regarding how to run the flink operator AIUI - but I'll leave @JMeybohm and @akosiaris weigh in on that.

Can we have a separate service request'ish task for deploying please? I feel like it's a dedicated thing which uses its own config/deployment steps, resources and decisions to be made (like T331283: [Event Platform] [NEEDS GROOMING] Store Flink HA metadata in Zookeeper).

Done: T333464: New Service Request: flink-kubernetes-operator

Caveat: on DSE we run the application with the same memory settings, and noticed a slightly higher memory footprint (in the order of 20%). Might be due to different Java versions (java8 on YARN, java11 on k8s). We have tasks for load testing and tuning planned for upcoming sprints.

In the past weeks we gained more experience running this job, on YARN and k8s, with traffic from all wikis. We also improved instrumentation.
Based on what we measured so far, we expect a memory footprint increase in the Python portion of the application. Java memory allocation is in line with what we reported in OP.

We see the Python process grow in the 500MB to 1500MB range, depending on tunables and traffic. We are still experimenting with said tunables to find a compromise between memory use and pipeline stability.
On YARN we are able to get the pipeline to run fine within the 3072MB budget. On k8s (DSE) we encounter occasional container OOMKills during traffic spikes, which we are currently investigating.

I'll give an update on this task as our perf tuning progresses.

Change 895241 merged by jenkins-bot:

[operations/deployment-charts@master] New wikikube service: mediawiki-page-content-change-enrichment - staging

https://gerrit.wikimedia.org/r/895241

Change 920382 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] Create mediawiki-page-content-change-enrichment namespaces in staging

https://gerrit.wikimedia.org/r/920382

Change 920382 merged by jenkins-bot:

[operations/deployment-charts@master] Create mediawiki-page-content-change-enrichment namespaces in staging

https://gerrit.wikimedia.org/r/920382

Change 920778 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] staging - Move flink-operator values to release specific values files

https://gerrit.wikimedia.org/r/920778

Change 920778 merged by jenkins-bot:

[operations/deployment-charts@master] staging - Move flink-operator values to release specific values files

https://gerrit.wikimedia.org/r/920778

@JMeybohm, I just tried to apply ^ in staging-codfw and got:

Error: UPGRADE FAILED: release namespace-certificates failed, and has been rolled back due to atomic being set: failed to create resource: admission webhook "webhook.cert-manager.io" denied the request: spec.commonName: Too long: must have at most 64 bytes

I'm guessing this is because this commonName of "mediawiki-page-content-change-enrichment.k8s-staging.discovery.wmnet" is too long?

+ # Source: raw/templates/resources.yaml
+ apiVersion: cert-manager.io/v1
+ kind: Certificate
+ metadata:
+   labels:
+     app: raw
+     chart: raw-0.3.0
+     heritage: Helm
+     release: namespace-certificates
+   name: mediawiki-page-content-change-enrichment
+   namespace: istio-system
+ spec:
+   commonName: mediawiki-page-content-change-enrichment.k8s-staging.discovery.wmnet
+   dnsNames:
+   - mediawiki-page-content-change-enrichment.k8s-staging.discovery.wmnet
+   issuerRef:
+     group: cfssl-issuer.wikimedia.org
+     kind: ClusterIssuer
+     name: discovery
+   revisionHistoryLimit: 2
+   secretName: mediawiki-page-content-change-enrichment-tls-certificate

Should we use a shorter namespace name?

Change 920782 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] mediawiki-page-content-change-enrichment - Revert deploy in staging

https://gerrit.wikimedia.org/r/920782

Change 920782 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki-page-content-change-enrichment - Revert deploy in staging

https://gerrit.wikimedia.org/r/920782

Maybe we can abbreviate as 'mw-page-content-change-enrich'?

cc @gmodena for naming of swift and zookeper paths in T336656 and T331283.

Change 921056 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] page_content_chnage - Shorten name of service and namespace in wikikube

https://gerrit.wikimedia.org/r/921056

Change 921056 merged by jenkins-bot:

[operations/deployment-charts@master] page_content_chnage - Shorten name of service and namespace in wikikube

https://gerrit.wikimedia.org/r/921056

Change 921058 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] kubernetes::deployment_server::services - define mw-page-content-change-enrich

https://gerrit.wikimedia.org/r/921058

Change 921058 merged by Ottomata:

[operations/puppet@production] kubernetes::deployment_server::services - define mw-page-content-change-enrich

https://gerrit.wikimedia.org/r/921058

Change 921060 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] page_content_change - fix kafka SSL setting in wikikube staging

https://gerrit.wikimedia.org/r/921060

Change 921060 merged by Ottomata:

[operations/deployment-charts@master] page_content_change - fix kafka SSL setting in wikikube staging

https://gerrit.wikimedia.org/r/921060

Change 921061 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] page_content_change - move common kafka SSL settings into values-main.yaml

https://gerrit.wikimedia.org/r/921061

Change 921061 merged by Ottomata:

[operations/deployment-charts@master] page_content_change - move common kafka SSL settings into values-main.yaml

https://gerrit.wikimedia.org/r/921061

DEPLOYED IN wikikube staging yeehaw!

@Ottomata should we adopt this naming conventions also for DSE?

I think it will be easier to not change in in DSE for now, since our intention is to (soon!) delete the deployment there.

Oh but let's set job_name to match the same in wikikube as we decided in https://phabricator.wikimedia.org/T336656#8864759

Change 922138 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] Undeploy flink-operator and uncreate service namespace in staging-codfw

https://gerrit.wikimedia.org/r/922138

otto opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/56

Upgrade to eventutilities 0.8.0

otto merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/56

Upgrade to eventutilities 0.8.0

Change 922593 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] mw-page-content-change-enrich - bump image to v1.17.0

https://gerrit.wikimedia.org/r/922593

Change 922593 merged by jenkins-bot:

[operations/deployment-charts@master] mw-page-content-change-enrich - bump image to v1.17.0

https://gerrit.wikimedia.org/r/922593

Change 922839 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] mw-page-content-change-enrich - deploy in eqiad and codfw

https://gerrit.wikimedia.org/r/922839

FYI: I just added swift access key to wikikube main mw-page-content-change-enrich private repo values.

Change 922138 abandoned by Ottomata:

[operations/deployment-charts@master] Undeploy flink-operator and uncreate service namespace in staging-codfw

Reason:

comment from https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/922874/ "operator (not the apps) should be/stay deployed in staging-codfw as well"

https://gerrit.wikimedia.org/r/922138

Change 923693 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] mw-page-content-change-enrich - set proper value of error sink stream name

https://gerrit.wikimedia.org/r/923693

Change 923693 merged by Ottomata:

[operations/deployment-charts@master] mw-page-content-change-enrich - set proper value of error sink stream name

https://gerrit.wikimedia.org/r/923693

Change 922839 merged by jenkins-bot:

[operations/deployment-charts@master] mw-page-content-change-enrich - deploy in eqiad and codfw

https://gerrit.wikimedia.org/r/922839

Mentioned in SAL (#wikimedia-operations) [2023-05-31T13:39:21Z] <ottomata> destroy mw-page-content-change-enrich deployment in dse-k8s-eqiad in order to deploy in wikikube - T330507

We are live in wikikube eqiad and codfw!