Looks like the issue is here:
https://github.com/wiki-ai/editquality/blob/master/editquality/utilities/fetch_labels.py#L100
Here's the label where we discovered the issue:
{
"campaign_id": 4,
"data": {
"rev_id": 637600969
},
"id": 194738,
"labels": [
{
"data": {
"damaging": true,
"goodfaith": null,
"unsure": false
},
"timestamp": 1432144745.16389,
"user_id": 42074979
}
]
},| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Natalia | T171491 Unlabeled goodfaith observations are assumed "false" -- should be "true" | |||
| Resolved | Natalia | T171497 Review training set to check strange examples of labels |
For English wikipedia, the change to assuming goodfaith edit when the actual label is null worked well. The AUC score grew from 0.909 to 0.928:
| model | mean(scores) | std(scores) | params |
| :--------------------------- | ---------------: | --------------: | :------------------------------------------------------------------------------- |
| GradientBoostingClassifier | 0.928 | 0.006 | learning_rate=0.01, n_estimators=700, max_depth=5, max_features="log2" |
| GradientBoostingClassifier | 0.927 | 0.008 | learning_rate=0.01, n_estimators=500, max_depth=5, max_features="log2" |
| GradientBoostingClassifier | 0.926 | 0.008 | learning_rate=0.01, n_estimators=500, max_depth=7, max_features="log2" |
| GradientBoostingClassifier | 0.926 | 0.008 | learning_rate=0.01, n_estimators=700, max_depth=7, max_features="log2" |
| GradientBoostingClassifier | 0.925 | 0.005 | learning_rate=0.01, n_estimators=700, max_depth=3, max_features="log2" |
| RandomForestClassifier | 0.925 | 0.006 | min_samples_leaf=5, max_features="log2", n_estimators=320, criterion="entropy" |
| GradientBoostingClassifier | 0.925 | 0.009 | learning_rate=0.01, n_estimators=300, max_depth=7, max_features="log2" |
| GradientBoostingClassifier | 0.925 | 0.007 | learning_rate=0.1, n_estimators=100, max_depth=3, max_features="log2" |
| GradientBoostingClassifier | 0.924 | 0.007 | learning_rate=0.01, n_estimators=300, max_depth=5, max_features="log2" |
| GradientBoostingClassifier | 0.924 | 0.004 | learning_rate=0.5, n_estimators=300, max_depth=1, max_features="log2" |
For Russian wiki, the AUC haven't really changed:
| model | mean(scores) | std(scores) | params |
| :--------------------------- | ---------------: | --------------: | :-------------------------------------------------------------------------------- |
| GradientBoostingClassifier | 0.935 | 0.007 | max_depth=7, n_estimators=700, learning_rate=0.01, max_features="log2" |
| RandomForestClassifier | 0.935 | 0.005 | criterion="entropy", n_estimators=640, min_samples_leaf=5, max_features="log2" |
| GradientBoostingClassifier | 0.935 | 0.006 | max_depth=7, n_estimators=500, learning_rate=0.01, max_features="log2" |
| RandomForestClassifier | 0.935 | 0.005 | criterion="entropy", n_estimators=640, min_samples_leaf=7, max_features="log2" |
| RandomForestClassifier | 0.934 | 0.005 | criterion="entropy", n_estimators=640, min_samples_leaf=13, max_features="log2" |
| RandomForestClassifier | 0.934 | 0.005 | criterion="entropy", n_estimators=320, min_samples_leaf=5, max_features="log2" |
| RandomForestClassifier | 0.934 | 0.005 | criterion="entropy", n_estimators=320, min_samples_leaf=13, max_features="log2" |
| RandomForestClassifier | 0.934 | 0.005 | criterion="entropy", n_estimators=320, min_samples_leaf=7, max_features="log2" |
| RandomForestClassifier | 0.934 | 0.005 | criterion="entropy", n_estimators=640, min_samples_leaf=3, max_features="log2" |
| GradientBoostingClassifier | 0.934 | 0.006 | max_depth=7, n_estimators=300, learning_rate=0.01, max_features="log2" |
https://github.com/wiki-ai/editquality/pull/87 is merged. Now all that is left is retraining the models!