The SUP producer is populating the cirrussearch.update_pipeline.update.rc0 with events coming from private wikis. We read this stream to index cloudelastic but since we do not allow index autocreation no data is leaked but we should probably better isolate this data to limit the risk.
AC:
Change #1073565 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):
[operations/mediawiki-config@master] Add a private variant of the cirrus update stream
Change #1073566 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):
[operations/deployment-charts@master] [WIP] cirrus: Read from public and private streams
ebernhardson opened https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/163
producer: Split private events into their own stream
ebernhardson merged https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests/163
producer: Split private events into their own stream
Change #1073566 merged by jenkins-bot:
[operations/deployment-charts@master] cirrus: Read from public and private streams
Change #1075055 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson):
[operations/deployment-charts@master] Revert^2 "cirrus: Read from public and private streams"
Patch was merged, but in testing against staging found it is not yet ready for deploy. We need to ship the mediawiki-config patch first. I won't be available during the backport window today, and I'm out of office for the next two days, will try to ship this on Thursday.
Change #1073565 merged by jenkins-bot:
[operations/mediawiki-config@master] Add a private variant of the cirrus update stream
Mentioned in SAL (#wikimedia-operations) [2024-09-24T07:03:20Z] <dcausse@deploy1003> Started scap sync-world: Backport for [[gerrit:1073565|Add a private variant of the cirrus update stream (T374335)]]
Mentioned in SAL (#wikimedia-operations) [2024-09-24T07:07:07Z] <dcausse@deploy1003> dcausse, ebernhardson: Backport for [[gerrit:1073565|Add a private variant of the cirrus update stream (T374335)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
Mentioned in SAL (#wikimedia-operations) [2024-09-24T07:27:31Z] <dcausse@deploy1003> Finished scap sync-world: Backport for [[gerrit:1073565|Add a private variant of the cirrus update stream (T374335)]] (duration: 24m 11s)
Change #1075055 merged by jenkins-bot:
[operations/deployment-charts@master] Revert^2 "cirrus: Read from public and private streams"
This is unfortunately still stalled, for some reason the new private streams aren't being mirrored between datacenters (but the existing ones are). That means if i deploy to prod eqiad will stop getting updates for private wikis (as they are produced in codfw). Still looking into how to get that going.
Not clear what made mirrormaker start working. Over the weekend the codfw-main -> eqiad-main replication started working for the codfw.* topic, but eqiad-main -> eqiad-jumbo and eqiad-main -> codfw-main hadn't picked the eqiad.* topic yet. Produced another message to the eqiad.* topic using kafkacat this morning and the mirroring looks to have picked up as expected now.
Went ahead and deployed the updated chart+image to eqiad and codfw. Made a test edit in officewiki, verified it showed up in the codfw private stream and that the update landed in both eqiad and codfw clusters.