Page MenuHomePhabricator
Create Task
Maniphest T375821

Migrate streaming updater event schema to the standard schema repository
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

The schema that SUP uses to communicate between the producer and consumer is still marked in development and referenced in the schema config as development/cirrussearch/update_pipeline/update. Is it stable enough now that we can move it into the schema repository and make it work like the other events? This would, for example, have ensured the new private variants of the streams get auto-created by canary events instead of doing it manually.

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Allow reading from "legacy" update streamsrepos/search-platform/cirrus-streaming-updater!170dcausseT375821-allow-reading-legacy-update-streamsmain
Produce errors to the fetch_error.v1 streamrepos/search-platform/cirrus-streaming-updater!169dcausseT375821-produce-to-fetch-error-v1main
Add --page-weighted-tags-change-legacy-streamrepos/search-platform/cirrus-streaming-updater!168dcausseT375821-add-page-weighted-tags-change-legacy-streammain
cirrussearch: promote SUP schemas to stablerepos/data-engineering/schemas-event-primary!9dcaussecirrussearch-promote-sup-schemas-to-stablemaster
Customize query in GitLab

Related Objects
Search...

Event Timeline

EBernhardson created this task.Sep 26 2024, 9:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 26 2024, 9:21 PM
dr0ptp4kt triaged this task as Medium priority.Sep 30 2024, 3:08 PM
dr0ptp4kt moved this task from needs triage to Current work on the Discovery-Search board.
dr0ptp4kt edited projects, added Discovery-Search (Current work); removed Discovery-Search.
Gehel set the point value for this task to 3.Oct 14 2024, 3:48 PM
Gehel moved this task from Incoming to Ready for Dev -- SWE on the Discovery-Search (Current work) board.
pfischer subscribed.Oct 14 2024, 4:02 PM

So two things have to happen:

  1. Rename the schema:
    • copy from development to mediawiki to make sure old events can still be validated (development copy can only be removed after retention period of kafka is over)
    • adapt schema_title in ext-EventStreamConfig.php accordingly
  2. Rename stream: This may require two phases
    1. Phase A:
      • Add the new topics (without rc0 suffix) to the stream config cirrussearch.update_pipeline.update.rc0 in ext-EventStreamConfig.php
      • Configure SUP producer to write to the new topic
    2. Phase B: (once no more messages are written to old topic and kafka retention period is over)
      • Strip rc0 suffix from stream name and remove topics with rc0 suffix in ext-EventStreamConfig.php
      • Configure SUP producer and consumer to use stream without rc0 suffix
dcausse claimed this task.Jan 17 2025, 4:24 PM
dcausse moved this task from Ready for Dev -- SWE to In Progress on the Discovery-Search (Current work) board.
dcausse added a comment.Jan 28 2025, 2:05 PM

I'm not sure we can safely change the schema_title of an existing stream so we might have to create separate streams and adapt the pipeline to consume from multiple update streams.

CodeReviewBot added a project: Patch-For-Review.Jan 28 2025, 4:03 PM

dcausse opened https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/merge_requests/9

cirrussearch: promote SUP schemas to stable

CodeReviewBot added a comment.Jan 28 2025, 8:30 PM

dcausse merged https://gitlab.wikimedia.org/repos/data-engineering/schemas-event-primary/-/merge_requests/9

cirrussearch: promote SUP schemas to stable

gerritbot added a comment.Jan 29 2025, 9:15 AM

Change #1114955 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/mediawiki-config@master] cirrus: add v1 stream for the search update pipeline

https://gerrit.wikimedia.org/r/1114955

gerritbot added a comment.Jan 29 2025, 9:15 AM

Change #1114956 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/mediawiki-config@master] cirrus: drop rc0 streams

https://gerrit.wikimedia.org/r/1114956

dcausse mentioned this in T385005: Build all search platform plugins for opensearch 1.3.20.Jan 29 2025, 9:44 AM
CodeReviewBot added a comment.Jan 30 2025, 11:18 AM

dcausse opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/168

Add --page-weighted-tags-change-legacy-stream

CodeReviewBot added a comment.Jan 30 2025, 2:40 PM

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/168

Add --page-weighted-tags-change-legacy-stream

CodeReviewBot added a comment.Jan 30 2025, 5:28 PM

dcausse opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/169

Produce errors to the fetch_error.v1 stream

CodeReviewBot added a comment.Jan 31 2025, 4:17 PM

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/169

Produce errors to the fetch_error.v1 stream

CodeReviewBot added a comment.Jan 31 2025, 4:42 PM

dcausse opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/170

Allow reading from "legacy" update streams

Gehel edited projects, added Discovery-Search (2025.02.10 - 2025.02.28); removed Discovery-Search (Current work).Feb 11 2025, 2:19 PM
Gehel moved this task from Incoming to In Progress on the Discovery-Search (2025.02.10 - 2025.02.28) board.
Gehel edited projects, added Discovery-Search (2025.03.01 - 2025.03.21); removed Discovery-Search (2025.02.10 - 2025.02.28).Feb 28 2025, 1:37 PM
Gehel moved this task from Incoming to In Progress on the Discovery-Search (2025.03.01 - 2025.03.21) board.
gerritbot added a comment.Mar 4 2025, 8:12 AM

Change #1114955 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: add v1 stream for the search update pipeline

https://gerrit.wikimedia.org/r/1114955

Stashbot added a comment.Mar 4 2025, 8:13 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-04T08:13:29Z] <dcausse@deploy2002> Started scap sync-world: Backport for [[gerrit:1114955|cirrus: add v1 stream for the search update pipeline (T375821)]]

Stashbot added a comment.Mar 4 2025, 8:29 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-04T08:29:16Z] <dcausse@deploy2002> dcausse: Backport for [[gerrit:1114955|cirrus: add v1 stream for the search update pipeline (T375821)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Mar 4 2025, 8:54 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-04T08:54:47Z] <dcausse@deploy2002> Finished scap sync-world: Backport for [[gerrit:1114955|cirrus: add v1 stream for the search update pipeline (T375821)]] (duration: 41m 17s)

Stashbot added a comment.Mar 4 2025, 8:55 AM

Mentioned in SAL (#wikimedia-operations) [2025-03-04T08:55:34Z] <dcausse> restarting eventgate-main to pickup to new streams (T375821)

dcausse added a subtask: T387863: Repartition [eqiad|codfw].cirrussearch.update_pipeline.update.v1 topics in kafka-main@[eqiad|codfw].Mar 4 2025, 2:19 PM
dcausse added a subtask: T387865: Repartition [eqiad|codfw].mediawiki.cirrussearch.page_weighted_tags_change.v1 topics in kafka-main@[eqiad|codfw].Mar 4 2025, 2:30 PM
CodeReviewBot added a comment.Mar 4 2025, 3:30 PM

dcausse merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/170

Allow reading from "legacy" update streams

gerritbot added a comment.Mar 4 2025, 5:16 PM

Change #1124483 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] cirrus-streaming-updater: add explicit consumer/producer streams

https://gerrit.wikimedia.org/r/1124483

gerritbot added a comment.Mar 4 2025, 5:16 PM

Change #1124484 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] cirrus-streaming-updater: consume from new v1 & legacy rc0 streams

https://gerrit.wikimedia.org/r/1124484

gerritbot added a comment.Mar 4 2025, 5:16 PM

Change #1124485 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] cirrus-streaming-updater: produce to v1 update streams

https://gerrit.wikimedia.org/r/1124485

gerritbot added a comment.Mar 4 2025, 5:16 PM

Change #1124486 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] cirrus-streaming-updater: stop consuming from legacy streams

https://gerrit.wikimedia.org/r/1124486

gerritbot added a comment.Mar 4 2025, 5:45 PM

Change #1124483 merged by jenkins-bot:

[operations/deployment-charts@master] cirrus-streaming-updater: add explicit consumer/producer streams

https://gerrit.wikimedia.org/r/1124483

gerritbot added a comment.Mar 5 2025, 10:15 AM

Change #1124742 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Produce weighted_tags to mediawiki.cirrussearch.page_weighted_tags_change.v1

https://gerrit.wikimedia.org/r/1124742

dcausse moved this task from In Progress to Blocked / Waiting on the Discovery-Search (2025.03.01 - 2025.03.21) board.Mar 5 2025, 10:26 AM

Moving to blocked waiting for the two subtasks to be addressed

dcausse mentioned this in T271776: Allow limiting lexeme searches by language.Mar 6 2025, 3:17 PM
brouberol closed subtask T387863: Repartition [eqiad|codfw].cirrussearch.update_pipeline.update.v1 topics in kafka-main@[eqiad|codfw] as Resolved.Mar 14 2025, 9:28 AM
brouberol closed subtask T387865: Repartition [eqiad|codfw].mediawiki.cirrussearch.page_weighted_tags_change.v1 topics in kafka-main@[eqiad|codfw] as Resolved.Mar 18 2025, 10:35 AM
dcausse moved this task from Blocked / Waiting to In Progress on the Discovery-Search (2025.03.01 - 2025.03.21) board.Mar 18 2025, 1:28 PM
gerritbot added a comment.Mar 20 2025, 2:10 PM

Change #1124484 merged by jenkins-bot:

[operations/deployment-charts@master] cirrus-streaming-updater: consume from new v1 & legacy rc0 streams

https://gerrit.wikimedia.org/r/1124484

gerritbot added a comment.Mar 20 2025, 4:44 PM

Change #1129889 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/alerts@master] cirrus: update alerts based on rc0 topics

https://gerrit.wikimedia.org/r/1129889

gerritbot added a comment.Mar 24 2025, 8:34 AM

Change #1124485 merged by jenkins-bot:

[operations/deployment-charts@master] cirrus-streaming-updater: produce to v1 update streams

https://gerrit.wikimedia.org/r/1124485

gerritbot added a comment.Mar 24 2025, 9:21 AM

Change #1130529 had a related patch set uploaded (by DCausse; author: DCausse):

[machinelearning/liftwing/inference-services@main] search weighted_tags: allow producing to the "v1" stream

https://gerrit.wikimedia.org/r/1130529

gerritbot added a comment.Mar 24 2025, 9:21 AM

Change #1130530 had a related patch set uploaded (by DCausse; author: DCausse):

[machinelearning/liftwing/inference-services@main] search weighted_tags: drop BC for rc0 weighted_tag stream

https://gerrit.wikimedia.org/r/1130530

gerritbot added a comment.Mar 24 2025, 9:41 AM

Change #1129889 merged by jenkins-bot:

[operations/alerts@master] cirrus: update alerts based on rc0 topics

https://gerrit.wikimedia.org/r/1129889

gerritbot added a comment.Mar 24 2025, 1:56 PM

Change #1130529 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] search weighted_tags: allow producing to the "v1" stream

https://gerrit.wikimedia.org/r/1130529

kevinbazira mentioned this in rMLISde93bf8305cb: search weighted_tags: allow producing to the "v1" stream.Mar 24 2025, 1:56 PM
dcausse moved this task from In Progress to Needs Review on the Discovery-Search (2025.03.01 - 2025.03.21) board.Mar 24 2025, 3:10 PM
gerritbot added a comment.Mar 24 2025, 4:10 PM

Change #1124742 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Produce weighted_tags to mediawiki.cirrussearch.page_weighted_tags_change.v1

https://gerrit.wikimedia.org/r/1124742

Gehel edited projects, added Discovery-Search (2025.03.22 - 2025.04.11); removed Discovery-Search (2025.03.01 - 2025.03.21).Mar 24 2025, 4:49 PM
Gehel moved this task from Incoming to Needs Review on the Discovery-Search (2025.03.22 - 2025.04.11) board.
ReleaseTaggerBot added a project: MW-1.44-notes (1.44.0-wmf.22; 2025-03-25).Mar 24 2025, 5:00 PM
dcausse moved this task from Needs Review to To be Deployed on the Discovery-Search (2025.03.22 - 2025.04.11) board.Mar 28 2025, 8:38 AM

All rc0 streams are now empty, we should be able to stop consuming from these legacy streams.

gerritbot added a comment.Mar 31 2025, 12:45 PM

Change #1124486 merged by jenkins-bot:

[operations/deployment-charts@master] cirrus-streaming-updater: stop consuming from legacy streams

https://gerrit.wikimedia.org/r/1124486

gerritbot added a comment.Mar 31 2025, 4:01 PM

Change #1130530 merged by jenkins-bot:

[machinelearning/liftwing/inference-services@main] search weighted_tags: drop BC for rc0 weighted_tag stream

https://gerrit.wikimedia.org/r/1130530

kevinbazira mentioned this in rMLISadcad459c7b1: search weighted_tags: drop BC for rc0 weighted_tag stream.Mar 31 2025, 4:02 PM
dcausse closed this task as Resolved.Mar 31 2025, 4:17 PM
dcausse moved this task from To be Deployed to Done on the Discovery-Search (2025.03.22 - 2025.04.11) board.
gerritbot added a comment.Apr 1 2025, 4:55 AM

Change #1132804 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: update article-country image

https://gerrit.wikimedia.org/r/1132804

gerritbot added a comment.Apr 1 2025, 7:22 AM

Change #1132804 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: update article-country image

https://gerrit.wikimedia.org/r/1132804

dcausse mentioned this in T390853: Consider using upgradeMode=savepoint for the cirrus-streaming-updater.Apr 2 2025, 12:57 PM
Gehel moved this task from Done to Reported on the Discovery-Search (2025.03.22 - 2025.04.11) board.Apr 4 2025, 9:41 AM
gerritbot added a comment.Aug 20 2025, 8:47 AM

Change #1180506 had a related patch set uploaded (by DCausse; author: DCausse):

[operations/deployment-charts@master] ml-services: stop using weighted_tags.rc0 stream

https://gerrit.wikimedia.org/r/1180506

gerritbot added a comment.Aug 21 2025, 6:59 AM

Change #1180506 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: stop using weighted_tags.rc0 stream

https://gerrit.wikimedia.org/r/1180506

gerritbot added a comment.Aug 22 2025, 1:51 PM

Change #1114956 merged by jenkins-bot:

[operations/mediawiki-config@master] cirrus: drop rc0 streams

https://gerrit.wikimedia.org/r/1114956

Maintenance_bot removed a project: Patch-For-Review.Aug 22 2025, 2:32 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits