Page MenuHomePhabricator

mediawiki_event_enrichment - update default params and tests to use mediawiki/page_change 1.3.0 (latest) schema
Closed, ResolvedPublic

Description

In T403664, we upgraded the mediawiki/page_change schema to include user_central_id.

In https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1196179, the Flink mediawiki_event_enrichment configs were bumped to use this schema version. This effectively applied the change in production.

However, the test fixtures and default parameters (ex 1., ex 2. -- this even uses the wrong stream name ;) ) in mediawiki_event_enrichment still reference old schema versions. We should update these too so dev and CI use the same versions as prod.

Done is

  • mediawiki_event_enrichment jobs that use mediawiki/page_change schemas use 1.3.0 (or later) for default stream descriptor params.
  • test fixtures are updated to use events with version mediawiki/page_change/1.3.0 (or later) with user_central_id in them.

Event Timeline

JMonton-WMF updated Other Assignee, added: JMonton-WMF.
JMonton-WMF updated Other Assignee, removed: JMonton-WMF.

I've created a MR with the changes.

As it's my first time changing this repository, I have a concern about changing the job name as I don't know what can be affected.
I see there is a topic in Kafka with that job name, to keep errors, and none with the new one, but as Kafka has the "auto create topics" enabled, I'm guessing a new topic will be automatically created. If that's the case, I can remove the old topic after deploying.

I added a test to the new field user_central_id, but I'm not sure if that's expected to be tested, as it doesn't seem to be part of any enrichment.

I have a concern about changing the job name as I don't know what can be affected.

Oo good catch. You will be changing the defaults, and it looks like in production things like the stream_manager.error_sink.stream are explicitly set in helmfile values (see also this).

So, the good news is that the local default dev error stream name doesn't really matter. I'll leave a comment on the MR about what we should do for default (dev) values.

Change #1202068 had a related patch set uploaded (by JavierMonton; author: JavierMonton):

[operations/deployment-charts@master] mediawiki-event-enrichment: Deploy new version 1.43.0

https://gerrit.wikimedia.org/r/1202068

Change #1202068 merged by jenkins-bot:

[operations/deployment-charts@master] mediawiki-event-enrichment: Deploy new version 1.43.0

https://gerrit.wikimedia.org/r/1202068