Notice that the "reverted" model was trained to detect reverts, not to detect vandalism (in other words, not all reverted edits are vandalism). It is likely that many edits which simply add a link to an article were in fact reverted, and the model learned this behavior. The "damaging" and "goodfaith" models, once trained for hewiki should provide better predictions for the "vandalism" user case.

BTW: you might be interested in this table of hewiki edits, still not reverted, with high "reverted" scores: https://he.wikipedia.org/w/index.php?oldid=17753480#Scores

It is likely that many edits which simply add a link to an article were in fact reverted, and the model learned this behavior.

I guess the model learned it due to bad weighting of the samples / non representative training set. Anyway, even if the training set is representative, this is a bug and the model doesn't work well.

BTW- the still not reverted with high "reverted" score include all edits of removing interwiki due to wikidata e.g https://he.wikipedia.org/w/index.php?diff=13744418

bad weighting of the samples / non representative training set

This model is based on a random sample of 20k edits from hewiki over the last year. So I don't think that non-representativeness is the problem.

Generally, I don't think the presence of false positives suggests that "the model doesn't work well". We will always have false positives -- it's a matter of how many. The AUC metric that we use to evaluate models is a robust measurement of false-negatives/false-positives across various probabilities. The [reverted model we have deployed for hewiki](http://ores.wmflabs.org/scores/hewiki/) gets an AUC of 0.874 which is pretty good considering that the state of the art in the academic literature is around 0.84 for realtime features.

Regardless, this is still a problem and we're looking into it. It could be that the user.is_anon feature is just too highly weighted in the model. I think that this predictor is problematic for a lot of reasons, but the sad truth is that it is an *effective* discriminator. We're working on different feature extraction strategies that will allow us to discriminate between problematic anon edits better.

LinearSVC assumes linear weighting of the features, so user.is_anon may give too high weight. I guess other models that don't assume linearity will not fail here (either RBF kernel, or decision tree/RF model)

Actually, the "linear" in Linear SVC only refers to the kernel that is used -- not the weighting.

See this discussion of the differences between linear and non-linear SVM kernels: http://stats.stackexchange.com/questions/73032/linear-kernel-and-non-linear-kernel-for-support-vector-machine

I should note that we can still run tests for this. I've been working on our revert labeling strategy. See https://github.com/wiki-ai/editquality/pull/1 Once I have a new set of reverts labeled and features extracted, I'll make sure to run tests with RBFSVC and RF models to check your hypothesis.

After announcing revscoring system is enabled for hewiki on Village pump here: https://he.wikipedia.org/w/index.php?title=%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%9E%D7%96%D7%A0%D7%95%D7%9F&oldid=17782721#.D7.9B.D7.9C.D7.99_.D7.97.D7.93.D7.A9_.D7.9C.D7.96.D7.99.D7.94.D7.95.D7.99_.D7.94.D7.A9.D7.97.D7.AA.D7.95.D7.AA

Most of the users who tried it (with frontend being mw-gadget-ScoredRevisions) claimed revscoring isn't reliable or had different issues after giving it few days of trial. The main issues were:

1. It seems that revscoring is too sensitive to anonymous and standard edit summaries of mobile version ("Fixed typo", "added content")
2. (maybe the same issue) @IKhitron claims 19/20 of high scored edits that he checked were good edits
3. (Frontend) Almost all anonymous edits are marked as red which is too noisy, and it isn't possible to remove the marking from already patrolled edits. ( @He7d3r - see https://github.com/he7d3r/mw-gadget-ScoredRevisions/pull/3 )

Indeed, @eranroz, from these 20 at least a dosen had such a sumaries.

Sorry it's been so long since I picked this up. It looks like the "Linear SVC" models struggle to learn nuance when it comes to anon edits. We're having more success with GradientBoosting strategies. I've trained a new model for hewiki that uses a this gradient boosting algorithm and it seems to score the edits in question better.

$ revscoring score submodules/editquality/models/hewiki.reverted.gradient_boosting.model --host https://he.wikipedia.org 17763753
2016-03-30 11:43:26,009 INFO:revscoring.extractors.api.extractor -- Batch requesting 1 revision from the API
2016-03-30 11:43:26,027 INFO:requests.packages.urllib3.connectionpool -- Starting new HTTPS connection (1): he.wikipedia.org
2016-03-30 11:43:26,633 INFO:revscoring.extractors.api.extractor -- Batch requesting 1 revision.parent from the API
2016-03-30 11:43:26,789 INFO:revscoring.extractors.api.extractor -- Batch requesting 0 revision.user.info from the API
17763753	{"probability": {"false": 0.520149441941693, "true": 0.47985055805830706}, "prediction": false}

I'd like to deploy this soon, but since it uses a different modeling strategy, the ranges of relevant probabilies changes too. We'll need to deploy a change to ScoredRevisions to change the thresholds for hewiki to correspond to the GradientBoosting model. There should be an announcement about that soon.

Forgot to note that this was deployed a long time ago.

I'm going to resolve this for now. Please re-open if it is still an issue.