To me, this approach makes a lot of sense. Anyone that needs something like mediawiki.revision-score-combined can just join all the relevant mediawiki.revision-score-<model-name> streams. One use case like that came to mind, WME, so I'm cc-ing @RBrounley_WMF.

The only possible problem would be if we need that -combined stream to happen as quickly as possible in some cases. If that's important enough, we can always generate it in step 1 of your proposal above (with ChangeProp (or Flink or Benthos or similar)). So I think this is future-proof and makes a lot of sense. +1.

+1 for me as well - I see only benefits in this move except for the changes of already existing consumers.

Tagging @Tgr as someone who might know more about Growth's consumption of the articletopic scores as described on Wikitech, which is a major Product consumer of this stream.

My general input: separate streams makes much more sense to me (it's much more transparent / scalable even if potentially more complicated to query). Thanks for doing this work!

Maybe out of scope, but I'm also aware of the ores_model and ores_classification tables on MariaDB (documentation). I assume this won't change because it seems to be about ORES scores show up in RecentChanges as opposed to this streaming pipeline, but curious if those are expected to be continued to be supported or deprecated? It's very nice having public history of model scores but it's also a second set of data that needs maintenance and frankly in the past I have found them very opaque as far as what is contained within them and how they are structured.

In T317768#8238625, @Isaac wrote:

Maybe out of scope, but I'm also aware of the ores_model and ores_classification tables on MariaDB (documentation). I assume this won't change because it seems to be about ORES scores show up in RecentChanges as opposed to this streaming pipeline, but curious if those are expected to be continued to be supported or deprecated? It's very nice having public history of model scores but it's also a second set of data that needs maintenance and frankly in the past I have found them very opaque as far as what is contained within them and how they are structured.

Hi Isaac! Thanks for your inputs :) The ORES extension use case is being discussed in T312518, in theory we could move the extension to the new Lift Wing endpoint but we are wondering who is using the feature and if it is worth to support it (or maybe change its format etc..). A lot of traffic going through ORES is duplicate between the revision-score stream and the ORES extension, reducing the overlapping would be great but it may take time. If you want to follow up in T312518 please do, the more feedback the better!

In T317768#8238625, @Isaac wrote:

Tagging @Tgr as someone who might know more about Growth's consumption of the articletopic scores as described on Wikitech, which is a major Product consumer of this stream.

Happy to provide details on that if needed, but AIUI this wouldn't change anything for users of the score information, it would just make the process of getting the information to where it needs to be more modular. In our case the place where it needs to be is the ElasticSearch index, so Search team members (who are already subscribed to this task) might be able to provide feedback on whether/how this change would affect them. We wouldn't be affected and I don't know much about the current pipeline, so I don't have an opinion either way (other than the issue below).

Maybe out of scope, but I'm also aware of the ores_model and ores_classification tables on MariaDB (documentation).

Good point. These are needed for SQL joins for change lists (recent changes, user contributions etc). Granted we probably shouldn't use SQL joins to construct those lists (see e.g. T307328: Scalability issues of recentchanges table), but that's a whole other conversation and a big effort to fix. So for now keeping those DB tables functional is important.

IIRC currently they are created on demand, by MediaWiki querying the ORES API and storing results. That workflow is reliant on the ORES API being cached (as otherwise it would be very slow). Maybe it would make sense to flip it and push the information to MediaWiki as edits happen, instead of having it pull them.

A lot of traffic going through ORES is duplicate between the revision-score stream and the ORES extension, reducing the overlapping would be great but it may take time. If you want to follow up in T312518 please do, the more feedback the better!

Makes sense. And added my comment - thanks @elukey !

This convo (also discussed in T301878: Send score to eventgate when requested) is very relevant to T307959: [Event Platform] Design and Implement realtime enrichment pipeline for MW page change with content, in that we'd like to standardize a a process (or at least stream/topic layout) for consuming streams, enriching them with extra data (like ORES scores), and producing new streams. One particularly relevant bit will be the stream/topic layout as suggested by @JAllemandou here. In that case, we are talking about page change streams with wiki content in them, which will make the events quite large. In that case, will likely not want to have all wiki projects in the same topics, as it would mean that someone wanting to consume changes for e.g. wiktionary will also consume changes for wikidata. For non-content streams, this isn't a big deal, as the events are small enough, and filtering out unwanted events is fine. Anyway, more convos to have.

@Ottomata noted, will review the previous conversations! And if you need me in future ones please let me know :)

One thing that I didn't get - do you think that this task's proposal can be implemented, or is it clashing with the aforementioned work so better to have more discussions? I am asking since I am trying to come up with a plan to deprecate ORES in favor of Lift Wing, and the revision-score use case is the biggest one to manage. Lemme know!

do you think that this task's proposal can be implemented

Ya absolutely! It can be done now with the current revision-score data model, or, if you are okay with waiting until we figure out page change schema, as discussed in https://phabricator.wikimedia.org/T301878#8008932, we can work together to make a nice unified data model for changes to pages, with a standardized way to add enriched info, like scores.

But also, I don't want to block, so proceed with the current data model too!

In T317768#8250242, @Ottomata wrote:

do you think that this task's proposal can be implemented

Ya absolutely! It can be done now with the current revision-score data model, or, if you are okay with waiting until we figure out page change schema, as discussed in https://phabricator.wikimedia.org/T301878#8008932, we can work together to make a nice unified data model for changes to pages, with a standardized way to add enriched info, like scores.

But also, I don't want to block, so proceed with the current data model too!

I am absolutely fine to wait for the new page change schema, but I'd ask more info about the timeline of its rollout. Our goal is to deprecate ORES asap during the next months, so we'd prefer to avoid waiting too much (OOW hardware, old models, git-lfs, etc..). Lemme know :)

We put it in our current sprint to get a WIP 'test topic' version of it deployed, at least in beta, but maybe testwiki too. I've been traveling a lot recently and not had time to work on it. But I hope to really focus on it in the next month or two...so perhaps that is the timeline?

I'd hope to feel good about you basing a WIP revision score model on it by Oct 17.

In T317768#8281631, @Ottomata wrote:

We put it in our current sprint to get a WIP 'test topic' version of it deployed, at least in beta, but maybe testwiki too. I've been traveling a lot recently and not had time to work on it. But I hope to really focus on it in the next month or two...so perhaps that is the timeline?

I'd hope to feel good about you basing a WIP revision score model on it by Oct 17.

Perfect! The ideal sunset time for ORES is during the next 6 months, so I think we can definitely work together on the new stream. Thanks!

FYI, we have deployed a rc0.mediawiki.page_change stream to group0 wikis! Example event here. It has the development/mediawiki/page/change schema. We put this in development/ as we wanted to indicate that it is still a WIP and subject to change.

Anyway, to extend this schema scores, check out the development/mediawiki/page/change/current.yaml file. It $refs the /development/fragment/mediawiki/state/change/page/1.0.0 schema, and then adds any extra information it needs, like the content_slots field.

You could do the same for new /development/mediawiki/page_score_change/current.yaml schema (name to be bikeshed :) ) like:

allOf:
  - $ref: /fragment/common/2.0.0#
  - $ref: /development/fragment/mediawiki/state/change/page/1.0.0
properties:
  revision:
    type: object
    properties:
      # your scores model here, whatever that might be
      scores:
        type: object
        properties:
          model_name: 
            type: string
          score:
            type: number
          prediction:
            type: string

If you really wanted to keep these based on revision entities instead of page entities, you can $ref the revision entity model, and/or any of the other mediawiki entity models we made. You could also put your scores model into its own reusable fragment schema, similar to how this content model has its own fragment schema.

@Ottomata recognizing that this might be long past the time when you'd want this feedback but a question about an additional field:

Similar to is_redirect, we often use whether an article is a disambiguation / list page as a determination for how to handle with ML models -- e.g., it's not intended behavior to run many models like add-a-link or the topic model on disambiguation / list pages. While I don't think list article is easy to determine without making a call to Wikidata (I assume that's out of the question), disambiguation pages are tracked by Mediawiki -- e.g., https://en.wikipedia.org/w/api.php?action=query&titles=Albert&prop=pageprops&format=json&ppprop=disambiguation.

What would be the process to consider whether this could be included as part of the page info in the event?

@Isaac, for sake of continuity, let's have this discussion over on T308017: Design Schema for page state and page state with content (enriched) streams.

Hi all, we have a use case here T326179. These models are already hosted on LiftWing. The suggested end-point could be mediawiki-revision-score-revert-risk-la

What would be the procedure/timeline to have that stream up? Currently we are building an app (T318634) that would benefit from that stream, and also the Android team (ping: @JTannerWMF , @Dbrant) might be interested on using that stream.

Hi @diego! I opened T327302 to investigate a way to provide streams like the ones that you are asking, I'll report back asap if the idea works. If not we'll need to adopt a new stream processor and it will likely take more time (we can sync about this during the ML/Research meeting this week).

Change 884155 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/mediawiki-config@master] wmf-config: add new revision-score streams for EventGate main

https://gerrit.wikimedia.org/r/884155

Hi folks! Getting back to this task so everybody interested can chime in. We have been testing ChangeProp as "stream processor" for this job and it seems working really well, so I filed https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/884155 to create new revision-score-schema-based streams as described in the task's proposal.

The above patch will need to be deployed and then after that EventGate will be able to accept new events for the streams. The idea is still to eventually deprecate mediawiki.revision-score with ORES, so if you are a consumer of the stream please come forward so we can figure out how to proceed.

@dcausse tagging you since IIRC the Search team uses the revision-score stream for https://wikitech.wikimedia.org/wiki/Search/articletopic, lemme know if the above proposal makes sense :)

@elukey thanks for the ping! If the plan is to still keep the mediawiki.revision-score for a while then the migration should be relatively easy for us (we do currently read corresponding hive tables not the kafka topic). In the future we might read the kafka topics directly but nothing is written yet.

@dcausse could you loop me in when you start on migration for this? I've been working with @KStoller-WMF with the Growth team (as the main consumer of the articletopic data) and we'd like to migrate from the old ORES model to a new one (that should also have scores available once changeprop is running). That overhead should be quite minimal - the schema is all the same -- and hopefully it should reduce some technical debt as that model natively supports all Wikipedia language editions and not just English. Relevant task: T315994

In T317768#8565463, @Isaac wrote:

@dcausse could you loop me in when you start on migration for this? I've been working with @KStoller-WMF with the Growth team (as the main consumer of the articletopic data) and we'd like to migrate from the old ORES model to a new one (that should also have scores available once changeprop is running). That overhead should be quite minimal - the schema is all the same -- and hopefully it should reduce some technical debt as that model natively supports all Wikipedia language editions and not just English. Relevant task: T315994

Sure! we are very much interested in this model to pay back some tech-dept too, filed T328276 to track this work and also attempt to clarify what has to be done.

Change 884155 merged by jenkins-bot:

[operations/mediawiki-config@master] wmf-config: add new revision-score streams for EventGate main

https://gerrit.wikimedia.org/r/884155

Mentioned in SAL (#wikimedia-operations) [2023-02-01T14:40:04Z] <awight@deploy1002> Started scap: Backport for [[gerrit:884155|wmf-config: add new revision-score streams for EventGate main (T317768)]]

Mentioned in SAL (#wikimedia-operations) [2023-02-01T14:41:56Z] <awight@deploy1002> elukey and awight: Backport for [[gerrit:884155|wmf-config: add new revision-score streams for EventGate main (T317768)]] synced to the testservers: mwdebug2001.codfw.wmnet, mwdebug1002.eqiad.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet

Mentioned in SAL (#wikimedia-operations) [2023-02-01T14:48:29Z] <awight@deploy1002> Finished scap: Backport for [[gerrit:884155|wmf-config: add new revision-score streams for EventGate main (T317768)]] (duration: 08m 25s)