Page MenuHomePhabricator

Re-label huwiki damaging and badfaith edits
Closed, ResolvedPublic

Description

It seems to be a common theme these days. We should probably have a good process for this. But for now, let's just give it a try.

Let's create a campaign for labeling the target edits of huwiki again.

It would be great if we could have a way to make sure that the second label comes from someone other than the first labeler. But that might need to be future work.

Event Timeline

Halfak created this task.May 20 2019, 9:22 AM
Restricted Application added a project: artificial-intelligence. · View Herald TranscriptMay 20 2019, 9:22 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Tgr added a comment.May 20 2019, 11:41 AM

As long as the size of the edit set is significantly smaller than the original one (hundreds), we can just ask the people who have reviewed more than, say, 100 edits to not participate this time, to make this training set mostly independent.

Harej triaged this task as Normal priority.Jun 4 2019, 9:14 PM
Harej moved this task from Untriaged to New development on the Scoring-platform-team board.
Tgr added a comment.Jun 29 2019, 11:46 AM

@Halfak can I help move this along? If you can specify the conditions and the expected format, I can produce a dataset.

Halfak added a comment.Jul 1 2019, 6:49 PM

I created a new campaign with 500 items marked damaging or badfaith. See https://labels.wmflabs.org/ui/huwiki/

I named the campaign "Edit quality re-label (500)". I can update that with a translated string if you can give me the text.

Tgr added a comment.EditedJul 2 2019, 6:28 PM

Thanks @Halfak! "Edit quality re-label" is "Szerkesztési minőség újraosztályozása".

Since we want to avoid the same people dominating the labeling, we depend on T223899: Information about finished campaigns should be accessible in Wikilabels or some other way of getting a list of top labelers.

Halfak added a comment.Jul 2 2019, 6:43 PM

OK the name is updated. We have a change that should allow the finished campaigns to be visible. I'll work on a deployment.

Halfak added a comment.Jul 2 2019, 7:33 PM

New deployment is out. You can see stats on the old campaign here: https://labels.wmflabs.org/stats/huwiki/33

Tgr added a comment.Jul 5 2019, 12:25 PM

@Halfak seems like something is wrong with wikilabels, after creating a new workset the request for workset contents (https://labels.wmflabs.org/campaigns/huwiki/<user>/<workset>/?tasks=&campaign=) returns tasks: [] so wikilabels shows a 0/0 progress bar and throws Could not select task. Index 0 out of bounds. at TaskList.selectByIndex on the JS console and Nem sikerült betölteni a(z) „damaging_and_goodfaith” űrlapot: $2 gets disoplayed to the user. Several other users have reproduced this (but not everyone who tried). Should I make a separate bug report?

Tgr added a comment.Jul 11 2019, 12:39 PM

@Halfak labeling is now mostly done (474 out of 500), all the remaining items need privileged access, and none of the admins are able to use the labeling tool due to the aforementioned bug (which seems to only affect a subset of the users).

OK that's great. We can work with this.

Halfak closed this task as Resolved.Jul 16 2019, 7:37 PM

Are there any new information/update?

Halfak added a comment.Aug 5 2019, 2:03 PM

We should have the updated model deployed early this week. Sorry for the delay! We had a couple minor hiccups last week with this deployment because there's a few other things going out with it.