Page MenuHomePhabricator

Investigate issues with vandalism detection on Water (Q283)
Closed, ResolvedPublic

Description

This card is done when a report is post to a wiki about the changes.

Event Timeline

Halfak raised the priority of this task from to Needs Triage.
Halfak updated the task description. (Show Details)
Halfak moved this task to Backlog on the Machine-Learning-Team (Active Tasks) board.
Halfak added a subscriber: Halfak.
Halfak set Security to None.

From #wikimedia-ai:

[13:18:32] <halfak> Oh! I should check the model against water.
[13:18:34] <halfak> One sec.
[13:19:12] <halfak> Wikidata search is awful!
[13:20:22] <halfak> Amir1, looks like we're scoring edits to water with a bit less extreme scores.
[13:20:55] <Amir1> :))))
[13:21:22] <halfak> We're still scoring highly, but not at the 99-100% level.
[13:22:10] <halfak> Last 5 edits: 0.79, 0.94, 0.86, 0.81, 0.92
[13:22:25] <Amir1> that's better
[13:22:32] <Amir1> but we still need to work on them
[13:22:52] <halfak> Compared to 0.98, 1.00, 1.00, 0.98, 0.99

AUC = 0.8467 for Wikidata with user.age -- Still need to test against Water.

Just got a new dataset for training against from @Ladsgroup.

https://tools.wmflabs.org/dexbot/damaging_73k.tsv is list of 73K edits randomly sampled, balanced and ready to be sampled and fed to the training system. (the second row is damaging or not)

How was that detected? Does "is damaging" means "was reverted" here?

Just got a new dataset for training against from @Ladsgroup.

https://tools.wmflabs.org/dexbot/damaging_73k.tsv is list of 73K edits randomly sampled, balanced and ready to be sampled and fed to the training system. (the second row is damaging or not)