Page MenuHomePhabricator

Enable wp10 and draftquality models for testwiki
Closed, ResolvedPublic

Description

Background

My team is working on enabling ORES filters on Special:NewPagesFeed, so that users can filters articles and drafts by ORES predicted issue/class (see T196178 and subtasks).

After enabling the ores role in MediaWiki Vagrant, I attempted to run mwscript extensions/ORES/maintenance/PopulateDatabase.php. This script works fine if you use damaging or goodfaith as the models, but if you enable wp10 and draftquality you get an exception, and the error message from the URL is {"error": {"code": "not found", "message": "Models ('wp10', 'draftquality') not available for testwiki"}}

Request

I'd like to be able to run PopulateDatabase.php and store accurate data for wp10 and draftquality in my MW-Vagrant database. (As an aside, if you want to test this in Vagrant you'll need to adjust your config per the patch in T198989)

Event Timeline

kostajh created this task.Jul 6 2018, 8:39 PM
Restricted Application added a project: Scoring-platform-team. · View Herald TranscriptJul 6 2018, 8:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
SBisson added a subscriber: SBisson.Jul 9 2018, 7:56 PM

Note, we're working to enable the ORES extension on testwiki.wikipedia.org (T199913).

if @Halfak signs this off, I make it happen. Basically the question boils down to how we produce random homogeneous scores for a multi-class classifier.

+1 for enabling. I made a change in T199239: Implement named aliases in ScorerModels that allows us to safely (and efficiently) configure models to be used across contexts.

@Ladsgroup, make sure that you use the same model key (e.g. enwiki_draftquality and enwiki_wp10) in the configuration for the scoring context of testwiki. Otherwise, everything should work fine :)

Relatedly, we can do this for the damaging/goodfaith models as well. The result would be that testwiki edits are actually scored by the enwiki models rather than using a deterministic scoring strategy. I'm not sure if that would be useful or if it would disrupt any ongoing work.

I'm not sure about this approach, It's good for production but the task description asks about having wp10 testwiki model in vagrant which we don't have access to enwiki models there and we can only go with deterministic approaches. I like it too because of better decoupling (if I want to test wp10 model in my localhost, I should not need to pull in editquality repo)

Not sure this is the right task to bring it up but we would really like to have real scoring for 'wp10' and 'draftquality' on test.wikipedia.org. We plan on doing user testing of the PageTRiage/ORES integration on that site.

Using the models trained on enwiki production sounds reasonable since article content would likely be imported from there.

Yes, thanks @SBisson. It is definitely important for us to be able to test that the right scores are being populated to the New Pages Feed as part of the work in T193782. The reason this is particularly relevant for this work is because we want to have actual English Wikipedia users testing this feature in test wiki, and it will make a big difference if the scores they see are realistic.

Ladsgroup claimed this task.Aug 1 2018, 6:00 PM

Just putting it in my plate, hopefully will get to it rather quick.

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptAug 1 2018, 6:00 PM

Change 450392 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/services/ores/deploy@master] Add wp10 and draftquality for testwiki

https://gerrit.wikimedia.org/r/450392

Change 450392 merged by Halfak:
[mediawiki/services/ores/deploy@master] Add wp10 and draftquality for testwiki

https://gerrit.wikimedia.org/r/450392

Change 451222 had a related patch set uploaded (by Catrope; owner: Catrope):
[operations/mediawiki-config@master] Enable wp10 and draftquality ORES models on testwiki

https://gerrit.wikimedia.org/r/451222

Change 450392 merged by Halfak:
[mediawiki/services/ores/deploy@master] Add wp10 and draftquality for testwiki
https://gerrit.wikimedia.org/r/450392

@Ladsgroup Is this scheduled to be deployed during tomorrow's ORES window at 20:00 UTC? If so, we could deploy the above config patch in the same window, or in the following SWAT window at 23:00 UTC.

Halfak added a subscriber: awight.

@awight is working on come updates to some of our models. I believe that is the only blocker to the next ORES deployment. I've just created T201518: ORES deployment (Early August) to track the changes that will go out with this deployment.

Change 451222 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable wp10 and draftquality ORES models on testwiki

https://gerrit.wikimedia.org/r/451222

Mentioned in SAL (#wikimedia-operations) [2018-08-13T23:13:01Z] <catrope@deploy1001> Synchronized wmf-config/InitialiseSettings.php: Enable wp10 and draftquality ORES models on testwiki (T198997) (duration: 00m 51s)

Catrope closed this task as Resolved.Aug 13 2018, 11:19 PM