The ML team is willing to deprecate the mediawiki.revision-score stream in favor of a stream per model (T317768). CirrusSearch data pipelines should be using mediawiki.revision_score_drafttopic instead of mediawiki.revision-score one (see T328576).
AC:
| Project | Branch | Lines +/- | Subject | |
|---|---|---|---|---|
| operations/deployment-charts | master | +1 -1 | services: change lift wing's kafka topic in changeprop's config |
| Reference | Source Branch | Dest Branch | Author | Title | |
|---|---|---|---|---|---|
| repos/data-engineering/airflow-dags!381 | work/ebernhardson/drafttopic-t333468 | main | ebernhardson | Use the drafttopic specific stream |
@Gehel not super urgent, we have the deadline of August/September to deprecate ORES, any time before that would be good. If it happens this quarter it would be awesome for us :)
ebernhardson opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/381
Use the drafttopic specific stream
@Ottomata Hi! The Search team is ready to use the new draft topic stream, but it is based on rc1 page_change and I am wondering if we should wait for a more stable state (to avoid kafka client offset changes and holes in the data in Hive etc..). Is there a target date to move page change out of RC status?
I want to do it imminently, but I keep hoping for a decision soon on MW user types. I think this is not going to happen, so I'm considering anticipating in in the schema...need to talk to other event platform folks about this.
@elukey we are ready to go. T336817: Release mediawiki.page_change.v1 stream
Okay if we just do this and you can change your configs to use mediawiki.page_change.v1 once we're done?
Change 920696 had a related patch set uploaded (by Elukey; author: Elukey):
[operations/deployment-charts@master] services: change lift wing's kafka topic in changeprop's config
Change 920696 merged by Elukey:
[operations/deployment-charts@master] services: change lift wing's kafka topic in changeprop's config
Commented in https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/381 the the draft topic stream is now using the new page_change source, we should be ready to proceed. Thanks!
ebernhardson merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/381
Use the drafttopic specific stream
The drafttopic stream has had it's inputs changed. Note that articletopic is still being read from the event.mediawiki_revision_score table. This is intended to be replaced with a different model but it hasn't changed over yet. The articletopic work is tracked in T328276.