Page MenuHomePhabricator

Build damaging/goodfaith models for Romanian Wikipedia
Closed, ResolvedPublic

Event Timeline

Halfak triaged this task as Low priority.Feb 2 2017, 3:31 PM
Halfak moved this task from Untriaged to New development on the Scoring-platform-team board.
Restricted Application added a project: artificial-intelligence. · View Herald TranscriptJun 24 2017, 6:17 PM
Halfak assigned this task to Sumit.Jun 26 2017, 4:23 PM
Sumit added a comment.Jun 27 2017, 8:48 AM

Damaging:

make models/rowiki.damaging.gradient_boosting.model       
cat datasets/rowiki.labeled_revisions.w_cache.20k_2016.json | \
        revscoring cv_train \
                revscoring.scorer_models.GradientBoosting \
                editquality.feature_lists.rowiki.damaging \
                damaging \
                --version=0.3.0 \
                -p 'max_depth=5' \
                -p 'learning_rate=0.01' \
                -p 'max_features="log2"' \
                -p 'n_estimators=700' \
                -s 'table' -s 'accuracy' -s 'precision' -s 'recall' -s 'pr' -s 'roc' -s 'recall_at_fpr(max_fpr=0.10)' -s 'filter_rate_at_recall(min_recall=0.9)' -s 'filt
er_rate_at_recall(min_recall=0.75)' -s 'recall_at_precision(min_precision=0.995)' -s 'recall_at_precision(min_precision=0.99)' -s 'recall_at_precision(min_precision=0.98
)' -s 'recall_at_precision(min_precision=0.90)' -s 'recall_at_precision(min_precision=0.75)' -s 'recall_at_precision(min_precision=0.60)' -s 'recall_at_precision(min_pre
cision=0.45)' -s 'recall_at_precision(min_precision=0.15)' \
                --balance-sample-weight \
                --center --scale > models/rowiki.damaging.gradient_boosting.model
2017-06-27 08:00:43,699 INFO:revscoring.utilities.cv_train -- Cross-validating model statistics for 10 folds...
2017-06-27 08:00:44,352 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 1...
2017-06-27 08:03:13,756 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 2...
2017-06-27 08:05:56,730 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 3...
2017-06-27 08:08:40,903 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 4...
2017-06-27 08:11:27,209 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 5...
2017-06-27 08:14:17,733 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 6...
2017-06-27 08:17:35,238 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 7...
2017-06-27 08:20:17,584 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 8...
2017-06-27 08:23:33,992 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 9...
2017-06-27 08:26:46,826 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 10...
2017-06-27 08:29:07,741 INFO:revscoring.utilities.cv_train -- Training model on all data...
ScikitLearnClassifier
 - type: GradientBoosting
 - params: max_leaf_nodes=null, learning_rate=0.01, min_samples_split=2, verbose=0, center=true, warm_start=false, n_estimators=700, presort="auto", balanced_sample_weig
ht=true, loss="deviance", min_samples_leaf=1, balanced_sample=false, init=null, random_state=null, subsample=1.0, max_features="log2", scale=true, min_weight_fraction_le
af=0.0, max_depth=5
 - version: 0.3.0
 - trained: 2017-06-27T08:29:32.376824

Table:
                 ~False    ~True
        -----  --------  -------
        False     17212     1678
        True        109      828

Accuracy: 0.91
Precision:
        -----  -----
        False  0.994
        True   0.33
        -----  -----

Recall:
        -----  -----
        False  0.911
        True   0.881
        -----  -----

PR-AUC:
        -----  -----
        False  0.994
        True   0.54
        -----  -----

ROC-AUC:
        -----  -----
        False  0.959
        True   0.963
        -----  -----

Recall @ 0.1 false-positive rate:
        label      threshold    recall    fpr
        -------  -----------  --------  -----
        False          0.548     0.904  0.094
        True           0.444     0.916  0.098

Filter rate @ 0.9 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.569          0.138     0.9
        True           0.455          0.866     0.906

Filter rate @ 0.75 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.915          0.284     0.75
        True           0.777          0.915     0.752

Recall @ 0.995 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.564     0.901        0.995
        True           0.962     0.038        1

Recall @ 0.99 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.331     0.934         0.99
        True           0.962     0.038         1

Recall @ 0.98 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.128     0.968         0.98
        True           0.962     0.038         1

Recall @ 0.9 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.959     0.051        0.981

Recall @ 0.75 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.945     0.164        0.787

Recall @ 0.6 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.923     0.351        0.608

Recall @ 0.45 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.827     0.706        0.454

Recall @ 0.15 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.038     1            0.954
        True           0.105     0.981        0.189
Sumit added a comment.Jun 27 2017, 1:42 PM

Goodfaith:

make models/rowiki.goodfaith.gradient_boosting.model                                     [97/1922]
cat datasets/rowiki.labeled_revisions.w_cache.20k_2016.json | \
        revscoring cv_train \
                revscoring.scorer_models.GradientBoosting \
                editquality.feature_lists.rowiki.goodfaith \
                goodfaith \
                --version=0.3.0 \
                -p 'max_depth=3' \
                -p 'learning_rate=0.1' \
                -p 'max_features="log2"' \
                -p 'n_estimators=300' \
                -s 'table' -s 'accuracy' -s 'precision' -s 'recall' -s 'pr' -s 'roc' -s 'recall_at_fpr(max_fpr=0.10)' -s 'filter_rate_at_recall(min_recall=0.9)' -s 'filt
er_rate_at_recall(min_recall=0.75)' -s 'recall_at_precision(min_precision=0.995)' -s 'recall_at_precision(min_precision=0.99)' -s 'recall_at_precision(min_precision=0.98
)' -s 'recall_at_precision(min_precision=0.90)' -s 'recall_at_precision(min_precision=0.75)' -s 'recall_at_precision(min_precision=0.60)' -s 'recall_at_precision(min_pre
cision=0.45)' -s 'recall_at_precision(min_precision=0.15)' \
                --balance-sample-weight \
                --center --scale > models/rowiki.goodfaith.gradient_boosting.model
2017-06-27 13:11:03,053 INFO:revscoring.utilities.cv_train -- Cross-validating model statistics for 10 folds...
2017-06-27 13:11:03,907 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 1...
2017-06-27 13:13:54,482 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 2...
2017-06-27 13:17:12,485 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 3...
2017-06-27 13:19:46,401 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 4...
2017-06-27 13:22:17,370 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 5...
2017-06-27 13:25:08,119 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 6...
2017-06-27 13:27:31,615 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 7...
2017-06-27 13:29:51,620 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 8...
2017-06-27 13:32:09,126 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 9...
2017-06-27 13:34:20,776 INFO:revscoring.scorer_models.sklearn_classifier -- Performing cross-validation 10...
2017-06-27 13:36:25,349 INFO:revscoring.utilities.cv_train -- Training model on all data...
ScikitLearnClassifier
 - type: GradientBoosting
 - params: max_features="log2", min_samples_leaf=1, min_weight_fraction_leaf=0.0, warm_start=false, balanced_sample=false, balanced_sample_weight=true, center=true, loss
="deviance", min_samples_split=2, max_leaf_nodes=null, verbose=0, max_depth=3, random_state=null, n_estimators=300, learning_rate=0.1, scale=true, subsample=1.0, init=nu
ll, presort="auto"
 - version: 0.3.0
 - trained: 2017-06-27T13:36:32.777290

Table:
                 ~False    ~True
        -----  --------  -------
        False       489       81
        True       1393    17864

Accuracy: 0.926
Precision:
        -----  -----
        False  0.26
        True   0.995
        -----  -----

Recall:
        -----  -----
        False  0.856
        True   0.928
        -----  -----

PR-AUC:
        -----  -----
        False  0.469
        True   0.994
        -----  -----

ROC-AUC:
        -----  -----
        False  0.964
        True   0.962
        -----  -----

Recall @ 0.1 false-positive rate:
        label      threshold    recall    fpr
        -------  -----------  --------  -----
        False          0.278     0.93   0.097
        True           0.634     0.914  0.093

Filter rate @ 0.9 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.37           0.89      0.907
        True           0.738          0.124     0.9

Filter rate @ 0.75 recall:
        label      threshold    filter_rate    recall
        -------  -----------  -------------  --------
        False          0.76           0.933     0.754
        True           0.972          0.271     0.75

Recall @ 0.995 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.987     0.04         1
        True           0.414     0.936        0.995

Recall @ 0.99 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.987     0.04         1
        True           0.174     0.964        0.991

Recall @ 0.98 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.987     0.04          1
        True           0.059     0.991         0.98

Recall @ 0.9 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.986     0.052        0.99
        True           0.013     1            0.972

Recall @ 0.75 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.977     0.107        0.808
        True           0.013     1            0.972

Recall @ 0.6 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.951     0.296        0.616
        True           0.013     1            0.972

Recall @ 0.45 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.914      0.49        0.46
        True           0.013      1           0.972

Recall @ 0.15 precision:
        label      threshold    recall    precision
        -------  -----------  --------  -----------
        False          0.098     0.968        0.161
        True           0.013     1            0.972
Sumit added a comment.Jun 27 2017, 2:54 PM

need to retrain the models after the regex update, PR soon.