Page MenuHomePhabricator
Create Task
Maniphest T199357

New Pages Feed: score draftquality on most recent revision
Closed, ResolvedPublic
Actions

Description

In T195796, @Ladsgroup helped us understand how the draftquality model is being scored on pages. The current business rule is that draftquality is only scored on the first revision of a page:

In T195796#4266861, @Ladsgroup wrote:
In T195796#4266332, @kaldari wrote:

@Ladsgroup: What do you think about storing the draftquality score for the most recent revision rather than the first revision? This would help on a number of fronts.

I thought about it and it's not hard to treat it like wp10 model but we end up with lots of useless scores in the database for articles that have been created ages ago but got recently edited (like article of "Barack Obama") which doesn't make sense. I can't find any not-expensive way to prevent that now but ideas are more than welcome. FWIW, Some analysis showed the scores are not very different between the first and the latest version.

For the New Pages Feed improvements work, we've decided that we prefer for the model to be scored on the latest revision of the page. That's because when pages are new, they can be improved quickly, either by the original author or a community collaborator. We want those latest changes to be reflected in model scores so that reviewers continue to prioritize review on drafts that need attention soonest.

Therefore, the user story for this task is:

  • As a reviewer, I need the New Pages Feed to reflect the latest draftquality score for each page in the feed.

That said, we don't want to be scoring and storing scores for many irrelevant pages. The engineer who takes this task will first have to figure out an approach that will work. @Catrope recommended a hybrid approach in which we use the ORES API to score new revisions of pages that are in the PageTriage queue.

Details

SubjectRepoBranchLines +/-
Score 'draftquality' and 'wp10' for new revs of pages in queuemediawiki/extensions/PageTriagewmf/1.32.0-wmf.16+31 -5
Score 'draftquality' and 'wp10' for new revs of pages in queuemediawiki/extensions/PageTriagemaster+31 -5
Introducing ORESCheckModels hookmediawiki/extensions/ORESmaster+59 -4
CleanupParent for draftquality model when PageTriage is usedoperations/mediawiki-configmaster+4 -4
Score latest revision on 'draftquality'mediawiki/extensions/PageTriagemaster+64 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

MMiller_WMF created this task.Jul 11 2018, 6:48 PM
MMiller_WMF moved this task from Incoming to Ready for Development on the Growth-Team (Sprint 0 (Growth Team)) board.
MMiller_WMF added a subscriber: SBisson.

@SBisson -- FYI that this task is now created and in the "To Do" column of the sprint board.

Catrope added a comment.Jul 11 2018, 9:16 PM

@SBisson is https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ORES/+/444252 related to this task?

• TBolliger unsubscribed.Jul 11 2018, 9:34 PM
SBisson added a comment.Jul 11 2018, 11:17 PM
In T199357#4417481, @Catrope wrote:

@SBisson is https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ORES/+/444252 related to this task?

No, this patch is to ease filtering in the PageTriage extension.

SBisson added a comment.Jul 12 2018, 1:23 PM

For this task we talked about having the PageTriage extension submit new FetchScoreJobs to score and save draftquality when an article it is monitoring gets a new revision.

This solution would retain the scores for all revisions, not just first and last like we hoped. Configuring cleanParent: true would clean the first revision as well. I don't see a good way to identifiy the first revision of a page other than storing it in pagetriage_page.

• Niharika unsubscribed.Jul 12 2018, 4:43 PM
MaxSem unsubscribed.Jul 12 2018, 6:30 PM
SBisson claimed this task.Jul 18 2018, 6:40 PM
SBisson moved this task from Ready for Development to In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.
gerritbot subscribed.Jul 19 2018, 7:31 PM

Change 446892 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/PageTriage@master] Score latest revision on 'draftquality'

https://gerrit.wikimedia.org/r/446892

gerritbot added a project: Patch-For-Review.Jul 19 2018, 7:31 PM
MusikAnimal unsubscribed.Jul 19 2018, 7:33 PM
SBisson moved this task from In Progress to Code Review on the Growth-Team (Sprint 0 (Growth Team)) board.Jul 19 2018, 7:51 PM
gerritbot added a comment.Jul 30 2018, 7:21 PM

Change 449274 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/ORES@master] [POC] Introducing a hook to decide what to score

https://gerrit.wikimedia.org/r/449274

gerritbot added a comment.Jul 30 2018, 7:21 PM

Change 449275 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[mediawiki/extensions/PageTriage@master] [POC] Let ORES score 'draftquality' for ne revs of pages in queue

https://gerrit.wikimedia.org/r/449275

gerritbot added a comment.Jul 30 2018, 8:42 PM

Change 446892 abandoned by Sbisson:
Score latest revision on 'draftquality'

https://gerrit.wikimedia.org/r/446892

gerritbot added a comment.Jul 31 2018, 10:40 AM

Change 449437 had a related patch set uploaded (by Sbisson; owner: Sbisson):
[operations/mediawiki-config@master] CleanupParent for draftquality model when PageTriage is used

https://gerrit.wikimedia.org/r/449437

SBisson mentioned this in T200578: AfC - Review (AFCH) added categories change ORES scoring for the same draft.Jul 31 2018, 2:20 PM
gerritbot added a comment.Aug 1 2018, 11:50 AM

Change 449437 merged by jenkins-bot:
[operations/mediawiki-config@master] CleanupParent for draftquality model when PageTriage is used

https://gerrit.wikimedia.org/r/449437

Stashbot subscribed.Aug 1 2018, 11:54 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-01T11:54:17Z] <zfilipin@deploy1001> Synchronized wmf-config/: SWAT: [[gerrit:449437|CleanupParent for draftquality model when PageTriage is used (T199357)]] (duration: 00m 56s)

SBisson mentioned this in T199158: New Pages Feed: list ORES categories in the feed (3.4).Aug 1 2018, 3:53 PM
gerritbot added a comment.Aug 4 2018, 10:29 AM

Change 449274 merged by jenkins-bot:
[mediawiki/extensions/ORES@master] Introducing ORESCheckModels hook

https://gerrit.wikimedia.org/r/449274

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)).Aug 4 2018, 11:00 AM
Catrope moved this task from Code Review to QA on the Growth-Team (Sprint 0 (Growth Team)) board.Aug 7 2018, 7:06 PM
gerritbot added a comment.Aug 7 2018, 8:44 PM

Change 449275 merged by jenkins-bot:
[mediawiki/extensions/PageTriage@master] Score 'draftquality' and 'wp10' for new revs of pages in queue

https://gerrit.wikimedia.org/r/449275

ReleaseTaggerBot edited projects, added MW-1.32-notes (WMF-deploy-2018-08-14 (1.32.0-wmf.17)); removed MW-1.32-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)).Aug 7 2018, 9:00 PM
Etonkovidova subscribed.Aug 10 2018, 7:35 PM

@MMiller_WMF I verified that only the latest revision scores for Predicted issues (draftquality) and on Predicted class (wp10) will be stored.
Two minor notices:

(1) damaging and goodfaith results are stored for all revisions. Since undo revision creates a new entry in the revision table and triggers new ORES re-evaluation, there probably no point of keeping ORES scores for revisions.

(2) 'Undo' revision action will not return to the previous revision scores - the 'undo' action creates a new revision which will be re-evaluated again. The same simple mechanism is in place - any new revision in the revision table will have new scores in ores table. I did not notice significant discrepancies in scores for undid revision though, but something to keep in mind if we would have complaints about differently scored revisions that were un-done.

Etonkovidova added a comment.Aug 10 2018, 9:30 PM

--dry-run for extensions/ORES/maintenance/BackfillPageTriageQueue.php looks fine.

MMiller_WMF moved this task from QA to Needs PM Review on the Growth-Team (Sprint 0 (Growth Team)) board.Aug 13 2018, 4:19 PM
Jdforrester-WMF edited projects, added MW-1.32-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)); removed MW-1.32-notes (WMF-deploy-2018-08-14 (1.32.0-wmf.17)).Aug 14 2018, 8:54 PM
gerritbot added a comment.Aug 14 2018, 10:33 PM

Change 452850 had a related patch set uploaded (by Catrope; owner: Sbisson):
[mediawiki/extensions/PageTriage@wmf/1.32.0-wmf.16] Score 'draftquality' and 'wp10' for new revs of pages in queue

https://gerrit.wikimedia.org/r/452850

gerritbot added a comment.Aug 14 2018, 11:33 PM

Change 452850 merged by jenkins-bot:
[mediawiki/extensions/PageTriage@wmf/1.32.0-wmf.16] Score 'draftquality' and 'wp10' for new revs of pages in queue

https://gerrit.wikimedia.org/r/452850

ReleaseTaggerBot edited projects, added MW-1.32-notes (WMF-deploy-2018-08-07 (1.32.0-wmf.16)); removed MW-1.32-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)).Aug 15 2018, 12:00 AM
Stashbot added a comment.Aug 15 2018, 12:12 AM

Mentioned in SAL (#wikimedia-operations) [2018-08-15T00:12:14Z] <catrope@deploy1001> Synchronized php-1.32.0-wmf.16/extensions/PageTriage/: SWAT: PageTriage fixes (T199357, T201812, T201560, T201373, T201253) (duration: 00m 51s)

Stashbot mentioned this in T201812: Running Echo as dependency of PageTriage fails unit tests.Aug 15 2018, 12:12 AM
Stashbot mentioned this in T201560: NewPagesFeed in AFC mode shows each page 4 times when all four "predicted issues" checkboxes are checked.
Stashbot mentioned this in T201373: New Pages Feed: "created date" value changes depending on sort.
Stashbot mentioned this in T201253: Special:NewPagesFeed should ignore ptrp_reviewed status for Drafts.
MMiller_WMF moved this task from Needs PM Review to Epics In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.Aug 16 2018, 5:14 PM

I just checked this out by repeatedly changing page content and watching scores change in Test Wiki. I believe that it is scoring the latest revision, so that's good! This ticket is done.

Catrope closed this task as Resolved.Aug 16 2018, 9:45 PM
DannyS712 moved this task from Backlog to PageCuration & NewPagesFeed on the English-Wikipedia-New-Pages-Patrol board.Dec 2 2019, 9:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL