Page MenuHomePhabricator

Add all models to fakewiki
Open, LowestPublic

Description

https://ores.wikimedia.org/v3/scores/fakewiki is very helpful for local development because you can give it any arbitrary revision number and it will give you a random score. This way I don't have to train any models on my local, when all I need are scores (doesn't matter if they are accurate). However not all models are enabled on fakewiki, so I'm unable to work on features that involve the missing models.

In particular, fakewiki is missing wp10, reverted, draftquality, and articlequality: https://ores.wikimedia.org/v3/scores/fakewiki/?models=damaging|goodfaith|reverted|articlequality|wp10|draftquality&revids=123&format=json

Could we add these, since I assume ORES isn't really doing any processing anyway for fakewiki?

Event Timeline

Restricted Application added a project: Scoring-platform-team. · View Herald TranscriptJul 23 2019, 7:36 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Well need to come up with some clever way to produce deterministic output.

Right how the editquality models use the revision Id to come up with a "prediction". We probably want to follow that pattern somehow. For reverted this is straightforward.

For wp10, we could take the last two digits, divide by 100, and just stick that into some distribution across the classes. I'm thinking a beta distribution would work well.

For draftquality, I'm not sure I can think of a clever way to do this. Maybe we can use the revision ID to be a random seed and work from there. That ought to work.

Now for draft topic, we want 3-5 target classes to have high probability and everything else to have low probability. I like the random seed idea. I wonder how predictable that can be.

Halfak triaged this task as Lowest priority.Wed, Sep 11, 9:15 PM