Change 963080 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] mw-page-content-change-enrich - set replicas: 2 for eqiad and codfw

https://gerrit.wikimedia.org/r/963080

^ oops, wrong task.

A quick comment to mention that consumers of fully idle streams (streams without canary events) are currently prone to https://issues.apache.org/jira/browse/KAFKA-4682. I believe that enabling canary events would help to workaround this issue.
My understanding is that KAFKA-4682 might cause loss of events if the consumer is configured to reset to LATEST or re-process many events if configured to EARLIEST.

Change 968344 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] Enable canary events for all MediaWiki event streams

https://gerrit.wikimedia.org/r/968344

We'd like to do this soon. I'm aiming for November 6th. Let's see if this date will work for some folks we know might be affected, and then we can send an announcement out.

cc @Xqt for pywikibot.
cc @REsquito-WMF @prabhat for Enterprise
cc @dcausse for search
cc @xcollazo for Data Products
cc @Krinkle in case you know of any reason not to do this for mediawiki.recentchange stream.

cc @xcollazo for Data Products

Sounds good to me.

Draft announcement: https://docs.google.com/document/d/1Lwr5IYps2eio49-PJIkMtKbHRfu62vwrt2wRh_STkTA/edit

@Ottomata

WE have created ticket https://phabricator.wikimedia.org/T349697 and will be grooming, unfortently with so such notice i'm not sure we can deliver by the 6th of November. Hopefully by the 14th

Please reach out to me, @prabhat and @HShaikh to discuss timelines if needed

Okay, we can delay timelines as needed. We've delayed since 2020, so why not another week or two. (This is why I reached out before sending announcement :) )

Thanks @Ottomata :)

We'll update here once we have this Live and Ready

In T266798#9277763, @Ottomata wrote:

cc @dcausse for search

All good, all search jobs should be using a filter on meta.domain

In T266798#9277763, @Ottomata wrote:

cc @Krinkle in case you know of any reason not to do this for mediawiki.recentchange stream.

Do any of the topics that power EventStreams today have canary events enabled?

Would they be filtered out by the EventStreams proxy or will they be exposed to outside clients over HTTP-SSE?

Is there a Wikimedia EventStreams docs and demos today, and a majority of third-party EventSource consumers, that likely knows to handle these correctly?

From a quick look at https://wikitech.wikimedia.org/wiki/EventStreams, I could not find mention of "canary" on that page in terms of docs. Using the demo at https://codepen.io/ottomata/pen/LYpPpxj?editors=1010, I could not find a topic listed there that matches one that (in the LHS context of Andrew's patch) is listed as having canary events enabled today. Grepping the https://github.com/ChlodAlejandro/wikimedia-streams repo, I could not find mention of "canary" either (if handled correctly, might be worth a test case at least).

xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-dump/-/merge_requests/19

Modify visibility SQL to account for canary events.

 Do any of the topics that power EventStreams today have canary events enabled?

Yes, any ones that have been created since we started emitting canary events by default as part of T251609}.

Would they be filtered out by the EventStreams proxy or will they be exposed to outside clients over HTTP-SSE?

The are exposed client side as events. We could consider adding code to filter them out, but it might be nice for clients to be able to use these events in the same way we do: differentiate between a buggy client and an empty stream.

Is there a Wikimedia EventStreams docs and demos today, and a majority of third-party EventSource consumers, that likely knows to handle these correctly?

No, good point. We should probably add these, I suppose to https://wikitech.wikimedia.org/wiki/EventStreams.

We were planning to handle this 'API' change via an announcement, we should def update docs first. Will do.

https://github.com/ChlodAlejandro/wikimedia-streams repo

COOL! Don't think I was aware of this. Added to https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams/Powered_By

cc @Chlod for JavaScript wikimedia-streams client.

xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-dump/-/merge_requests/19

Modify visibility SQL to account for canary events.

xcollazo opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/531

Bump mediawiki dumps artifact to pickup canary fix.

cc @Chlod for JavaScript wikimedia-streams client.

Thanks for the heads up! I've started a PR for this at https://github.com/ChlodAlejandro/wikimedia-streams/pull/87, but I'll have to finish this up maybe tomorrow since my connection right now is spotty. Having the behavior/use for canary events documented on Wikitech would also be a great help for anyone looking for the info in the JSDoc.

xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/531

Bump mediawiki dumps artifact to pickup canary fix.

Canary events Docs updates for EventStreams:
https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams#Canary_Events

@REsquito-WMF @Chlod, we'd like to make an announcement, and to do that we should have a deadline.

What would be a good deadline for you all? Would December 11th 2023 be enough time?

@Ottomata That should work for us,

we have https://phabricator.wikimedia.org/T349697 as part of our current sprint.

you can keep an yes on it ... but should be closed over the next 2 weeks

@Ottomata Also good here. Canary event filtering was released as part of v2.1.0 in wikimedia-streams, and around a month of time post-announcement should be enough for (maintained) dependent projects to update.

Announcement sent out today. We will enable canary events on Monday December 11th.

Announcement:
https://wikimedia.slack.com/archives/C01R06P8D1B/p1699383613240779?thread_ts=1698177111.829059&cid=C01R06P8D1B

Think this might be worth a User-notice, given that it affects volunteer-run tools as well (can't definitively call this as I don't have analytics access). Tool maintainers need to update their package versions to properly filter out the canary events (to v2.1.0 for npm@wikimedia-streams and to whichever version closes T350756 for Pywikibot, when it gets closed). Volunteers like me also don't have access to the Slack announcement, so that probably won't be enough to get the word out to volunteer developers.

Possible wording for this would be: "Heartbeat events will be sent to Wikimedia event streams from December 11. Streams users are advised to filter out these events. Updates to Pywikibot or wikimedia-streams will discard these events by default." Not sure if that gets the 'you need to update soon' vibe that well, though.

If the Pywikibot patch gets merged soon, then this could maybe go into the next edition, in the "Future changes" section.

@Chlod thanks. TIL about User-notice.

Updated description for User-notice.

Volunteers like me also don't have access to the Slack announcement, so that probably won't be enough to get the word out to volunteer developers.

Indeed! I did send an email annoucement to

• analytics-announce@lists.wikimedia.org
• tech-all@wikimedia.org
• data-engineering@wikimedia.org
• data-platform-engineering@wikimedia.org

In hindsight I should sent to wikitech-l. Should I do that, or allow User-notice to handle that?

Oh, looks like I haven't subscribed to any of those lists. 😅 Re: wikitech-l, Tech News is sent there every week, but it wouldn't hurt to also have a dedicated announcement for it so it's easier to find.

K, sent to wikitech-l.

Re: User-notice - Thank you for the draft-wording, that is greatly appreciated! I suggest changing the Pywikibot link from github to mediawiki-wiki, mostly so that we could link to a translatable page.
I initially didn't notice that it's still waiting for a patch-merge, so I drafted the entry. Please either ping me when it's ready (before mid-Friday) or add it directly (before mid-Friday), whichever week that happens.

The raw wikitext for all of that is:

• [[File:Octicons-tools.svg|12px|link=|alt=|<translate>Advanced item</translate>]] <translate>Canary (also known as heartbeat) events will be produced into [<tvar name="1">https://stream.wikimedia.org/?doc#/streams</tvar> Wikimedia event streams] from December 11. Streams users are advised to filter out these events, by discarding all events where <tvar name="code-equals-string"><bdi lang="zxx" dir="ltr"><code><nowiki>meta.domain == "canary"</nowiki></code></bdi></tvar>. Updates to [[<tvar name="3">mw:Special:MyLanguage/Manual:Pywikibot</tvar>|Pywikibot]] or [<tvar name="4">https://github.com/ChlodAlejandro/wikimedia-streams</tvar> wikimedia-streams] will discard these events by default.</translate> [https://phabricator.wikimedia.org/T266798]

Sidenote: Tech News is sent to wikitech-ambassadors@lists.wikimedia.org each week, but not to wikitech-l@lists.wikimedia.org.
Sidenote2: The list analytics-announce@lists.wikimedia.org is good for anyone to join, but the other mailing lists Ottomata mentioned are internal team/group addresses, hence they wouldn't seem familiar!

Pywikibot link from github to mediawiki-wiki

+1, thank you.

I initially didn't notice that it's still waiting for a patch-merge, so I drafted the entry. Please either ping me when it's ready (before mid-Friday) or add it directly (before mid-Friday), whichever week that happens.

What do you mean by 'it's still waiting for patch-merge'?

In T266798#9320909, @Ottomata wrote:

What do you mean by 'it's still waiting for patch-merge'?

I was referring to this comment (possibly I misunderstood!):

In T266798#9317254, @Chlod wrote:

If the Pywikibot patch gets merged soon, then this could maybe go into the next edition, in the "Future changes" section.

In other words: If it's ready to go into this upcoming edition (which will be sent on Monday), then it needs to be added to the draft-page within the next 24-hours (or just revert my removal!). Or if it belongs in the November 20th edition instead, then we'd add it next week.

I think it would be good to include it in the earliest edition we can, to give users a chance to modify their code. A future version of pywikibot may automatically do this for users of that codebase.

@Ottomata We have deployed our changes to prod.

Is there any place or anyhow we can test this before going live?

Thanks

Is there any place or anyhow we can test this before going live?

@REsquito-WMF hm, not really? But, feel free to inject artificial canary events (meta.domain: "canary") into your pipeline and verify that they are discarded.

I plan to deploy T351247: [Event Platform] change propagation should discard canary events on Monday Dec 11, and then if all goes well, will enable these canary events on Monday or Tuesday.

FYI, Pywikibot version 8.6.0 includes the change to discard canary events by default.

Change 968344 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable canary events for all MediaWiki event streams

https://gerrit.wikimedia.org/r/968344

Change 982122 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/mediawiki-config@master] Revert accidental portals submodule change

https://gerrit.wikimedia.org/r/982122

Change 982122 merged by jenkins-bot:

[operations/mediawiki-config@master] Revert accidental portals submodule change

https://gerrit.wikimedia.org/r/982122

Mentioned in SAL (#wikimedia-operations) [2023-12-11T16:10:34Z] <ottomata> enabling canary events for all mediawiki state change event streams - T266798

Mentioned in SAL (#wikimedia-operations) [2023-12-11T16:19:06Z] <otto@deploy2002> Synchronized wmf-config/ext-EventStreamConfig.php: Config: [[gerrit:968344|Enable canary events for all MediaWiki event streams (T266798)]] (duration: 08m 25s)

This has been done!