Page MenuHomePhabricator
Create Task
Maniphest T156503

Build damaging/goodfaith models for Romanian Wikipedia
Closed, ResolvedPublic
Actions

Related Objects
Search...

Event Timeline

Halfak created this task.Jan 27 2017, 5:33 PM
Halfak added a subtask: T156357: Deploy edit quality campaign for Romanian Wikipedia.
Halfak added a parent task: T156518: Deploy ORES Review Tool on Romanian Wikipedia.Jan 27 2017, 7:31 PM
Halfak mentioned this in T156357: Deploy edit quality campaign for Romanian Wikipedia.Jan 27 2017, 7:33 PM
Halfak added a subtask: T156517: Complete Romanian Wikipedia edit quality campaign.Feb 2 2017, 3:26 PM
Halfak removed a subtask: T156357: Deploy edit quality campaign for Romanian Wikipedia.
Halfak triaged this task as Low priority.Feb 2 2017, 3:31 PM
Halfak moved this task from Unsorted to New development on the Machine-Learning-Team board.
Halfak added a parent task: T130213: [Epic] Edit quality models (damaging/goodfaith).Apr 14 2017, 4:44 PM
Halfak edited projects, added Machine-Learning-Team (Active Tasks); removed Machine-Learning-Team.Jun 24 2017, 6:17 PM
Restricted Application added a project: artificial-intelligence. · View Herald TranscriptJun 24 2017, 6:17 PM
Halfak assigned this task to Sumit.Jun 26 2017, 4:23 PM
Sumit added a comment.Jun 27 2017, 8:48 AM

Damaging:

make models/rowiki.damaging.gradient_boosting.model       
cat datasets/rowiki.labeled_revisions.w_cache.20k_2016.json | \
        revscoring cv_train \
                revscoring.scorer_models.GradientBoosting \
                editquality.feature_lists.rowiki.damaging \
                damaging \
                --version=0.3.0 \
                -p 'max_depth=5' \
                -p 'learning_rate=0.01' \
                -p 'max_features="log2"' \
                -p 'n_estimators=700' \
                -s 'table' -s 'accuracy' -s 'precision' -s 'recall' -s 'pr' -s 'roc' -s 'recall_at_fpr(max_fpr=0.10)' -s 'filter_rate_at_recall(min_recall=0.9)' -s 'filt
er_rate_at_recall(min_recall=0.75)' -s 'recall_at_precision(min_precision=0.995)' -s 'recall_at_precision(min_precision=0.99)' -s 'recall_at_precision(min_precision=0.98
)' -s 'recall_at_precision(min_precision=0.90)' -s 'recall_at_precision(min_precision=0.75)' -s 'recall_at_precision(min_precision=0.60)' -s 'recall_at_precision(min_pre
cision=0.45)' -s 'recall_at_precision(min_precision=0.15)' \
                --balance-sample-weight \
                --center --scale > models/rowiki.damaging.gradient_boosting.model
2017-06-27 08:00:43,699 INFO:revscoring.utilities.cv_train -- Cross-validating model statistics for 10 folds...
2017-06-27 08:00:44,352 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 1...
2017-06-27 08:03:13,756 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 2...
2017-06-27 08:05:56,730 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 3...
2017-06-27 08:08:40,903 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 4...
2017-06-27 08:11:27,209 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 5...
2017-06-27 08:14:17,733 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 6...
2017-06-27 08:17:35,238 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 7...
2017-06-27 08:20:17,584 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 8...
2017-06-27 08:23:33,992 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 9...
2017-06-27 08:26:46,826 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 10...
2017-06-27 08:29:07,741 INFO:revscoring.utilities.cv_train -- Training model on all data...
ScikitLearnClassifier
 - type: GradientBoosting
 - params: max_leaf_nodes=null, learning_rate=0.01, min_samples_split=2, verbose=0, center=true, warm_start=false, n_estimators=700, presort="auto", balanced_sample_weig
ht=true, loss="deviance", min_samples_leaf=1, balanced_sample=false, init=null, random_state=null, subsample=1.0, max_features="log2", scale=true, min_weight_fraction_le
af=0.0, max_depth=5
 - version: 0.3.0
 - trained: 2017-06-27T08:29:32.376824

Table:
                 ~False    ~True
        -----  --------  -------
        False     17212     1678
        True        109      828

Accuracy: 0.91
Precision:
        -----  -----
        False  0.994
        True   0.33
        -----  -----

Recall:
        -----  -----
        False  0.911
        True   0.881
        -----  -----

PR-AUC:
        -----  -----
        False  0.994
        True   0.54
        -----  -----

ROC-AUC:
        -----  -----
        False  0.959
        True   0.963
        -----  -----

Recall @ 0.1 false-positive rate:
        label      threshold    recall    fpr
        -------  -----------  --------  -----
        False          0.548     0.904  0.094
        True           0.444     0.916  0.098

Filter rate @ 0.9 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.569          0.138     0.9
        True           0.455          0.866     0.906

Filter rate @ 0.75 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.915          0.284     0.75
        True           0.777          0.915     0.752

Recall @ 0.995 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.564     0.901        0.995
        True           0.962     0.038        1

Recall @ 0.99 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.331     0.934         0.99
        True           0.962     0.038         1

Recall @ 0.98 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.128     0.968         0.98
        True           0.962     0.038         1

Recall @ 0.9 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.959     0.051        0.981

Recall @ 0.75 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.945     0.164        0.787

Recall @ 0.6 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.923     0.351        0.608

Recall @ 0.45 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.827     0.706        0.454

Recall @ 0.15 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.105     0.981        0.189
Sumit added a comment.Jun 27 2017, 1:42 PM

Goodfaith:

make models/rowiki.goodfaith.gradient_boosting.model                                     [97/1922]
cat datasets/rowiki.labeled_revisions.w_cache.20k_2016.json | \
        revscoring cv_train \
                revscoring.scorer_models.GradientBoosting \
                editquality.feature_lists.rowiki.goodfaith \
                goodfaith \
                --version=0.3.0 \
                -p 'max_depth=3' \
                -p 'learning_rate=0.1' \
                -p 'max_features="log2"' \
                -p 'n_estimators=300' \
                -s 'table' -s 'accuracy' -s 'precision' -s 'recall' -s 'pr' -s 'roc' -s 'recall_at_fpr(max_fpr=0.10)' -s 'filter_rate_at_recall(min_recall=0.9)' -s 'filt
er_rate_at_recall(min_recall=0.75)' -s 'recall_at_precision(min_precision=0.995)' -s 'recall_at_precision(min_precision=0.99)' -s 'recall_at_precision(min_precision=0.98
)' -s 'recall_at_precision(min_precision=0.90)' -s 'recall_at_precision(min_precision=0.75)' -s 'recall_at_precision(min_precision=0.60)' -s 'recall_at_precision(min_pre
cision=0.45)' -s 'recall_at_precision(min_precision=0.15)' \
                --balance-sample-weight \
                --center --scale > models/rowiki.goodfaith.gradient_boosting.model
2017-06-27 13:11:03,053 INFO:revscoring.utilities.cv_train -- Cross-validating model statistics for 10 folds...
2017-06-27 13:11:03,907 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 1...
2017-06-27 13:13:54,482 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 2...
2017-06-27 13:17:12,485 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 3...
2017-06-27 13:19:46,401 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 4...
2017-06-27 13:22:17,370 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 5...
2017-06-27 13:25:08,119 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 6...
2017-06-27 13:27:31,615 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 7...
2017-06-27 13:29:51,620 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 8...
2017-06-27 13:32:09,126 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 9...
2017-06-27 13:34:20,776 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 10...
2017-06-27 13:36:25,349 INFO:revscoring.utilities.cv_train -- Training model on all data...
ScikitLearnClassifier
 - type: GradientBoosting
 - params: max_features="log2", min_samples_leaf=1, min_weight_fraction_leaf=0.0, warm_start=false, balanced_sample=false, balanced_sample_weight=true, center=true, loss
="deviance", min_samples_split=2, max_leaf_nodes=null, verbose=0, max_depth=3, random_state=null, n_estimators=300, learning_rate=0.1, scale=true, subsample=1.0, init=nu
ll, presort="auto"
 - version: 0.3.0
 - trained: 2017-06-27T13:36:32.777290

Table:
                 ~False    ~True
        -----  --------  -------
        False       489       81
        True       1393    17864

Accuracy: 0.926
Precision:
        -----  -----
        False  0.26
        True   0.995
        -----  -----

Recall:
        -----  -----
        False  0.856
        True   0.928
        -----  -----

PR-AUC:
        -----  -----
        False  0.469
        True   0.994
        -----  -----

ROC-AUC:
        -----  -----
        False  0.964
        True   0.962
        -----  -----

Recall @ 0.1 false-positive rate:
        label      threshold    recall    fpr
        -------  -----------  --------  -----
        False          0.278     0.93   0.097
        True           0.634     0.914  0.093

Filter rate @ 0.9 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.37           0.89      0.907
        True           0.738          0.124     0.9

Filter rate @ 0.75 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.76           0.933     0.754
        True           0.972          0.271     0.75

Recall @ 0.995 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.987     0.04         1
        True           0.414     0.936        0.995

Recall @ 0.99 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.987     0.04         1
        True           0.174     0.964        0.991

Recall @ 0.98 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.987     0.04          1
        True           0.059     0.991         0.98

Recall @ 0.9 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.986     0.052        0.99
        True           0.013     1            0.972

Recall @ 0.75 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.977     0.107        0.808
        True           0.013     1            0.972

Recall @ 0.6 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.951     0.296        0.616
        True           0.013     1            0.972

Recall @ 0.45 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.914      0.49        0.46
        True           0.013      1           0.972

Recall @ 0.15 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.098     0.968        0.161
        True           0.013     1            0.972
Sumit added a comment.Jun 27 2017, 2:54 PM

need to retrain the models after the regex update, PR soon.

Sumit added a comment.Jun 27 2017, 6:39 PM

https://github.com/wiki-ai/editquality/pull/78

Sumit moved this task from Parked to Review on the Machine-Learning-Team (Active Tasks) board.Jun 27 2017, 7:01 PM
Halfak added a parent task: T166045: Scoring platform team FY18 Q1.Jun 28 2017, 5:03 PM
Sumit moved this task from Review to Completed on the Machine-Learning-Team (Active Tasks) board.Jul 7 2017, 7:27 PM
Halfak added a parent task: T170485: ORES deployment - Mid July, 2017.Jul 12 2017, 9:30 PM
Halfak closed this task as Resolved.Jul 12 2017, 10:49 PM
Halfak closed subtask T156517: Complete Romanian Wikipedia edit quality campaign as Resolved.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL