Test draftquality sentiment feature on Editquality
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Sumit
	Jul 10 2017, 5:38 PM

Project Tags

Referenced Files

	F8975004: Enwiki_1000_damaging_notdamaging_edits_sentiment
	Aug 7 2017, 3:45 AM

Subscribers

Aklapper

Sumit

Description

To generate a tuning report of editquality on enwiki using the sentiment( https://github.com/wiki-ai/draftquality/pull/9 ) feature and test its viability.

Related Objects

Mentioned In: Blog Post: Status update (October 6, 2017)
T171386: Investigate small loss in fitness with the new data in fawiki

Event Timeline

Sumit created this task.Jul 10 2017, 5:38 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 10 2017, 5:38 PM

Sumit added a project: draftquality-modeling.Jul 13 2017, 3:38 PM

Restricted Application added a project: artificial-intelligence. · View Herald TranscriptJul 13 2017, 3:38 PM

Enwiki damaging gives slight rise in accuracy:

Model tuning report

Revscoring version: 1.3.17
Features: editquality.feature_lists.enwiki.damaging
Date: 2017-07-20T02:42:12.884370
Observations: 19460
Labels: [false, true]
Scoring: roc_auc
Folds: 5

Top scoring configurations

model	mean(scores)	std(scores)	params
:---------------------------	---------------:	--------------:	:-----------------------------------------------------------------------
GradientBoostingClassifier	0.922	0.007	n_estimators=700, max_depth=5, learning_rate=0.01, max_features="log2"
GradientBoostingClassifier	0.921	0.007	n_estimators=500, max_depth=5, learning_rate=0.01, max_features="log2"
GradientBoostingClassifier	0.921	0.009	n_estimators=700, max_depth=7, learning_rate=0.01, max_features="log2"
GradientBoostingClassifier	0.92	0.008	n_estimators=500, max_depth=7, learning_rate=0.01, max_features="log2"
GradientBoostingClassifier	0.919	0.005	n_estimators=100, max_depth=3, learning_rate=0.1, max_features="log2"
GradientBoostingClassifier	0.919	0.009	n_estimators=300, max_depth=3, learning_rate=0.1, max_features="log2"
GradientBoostingClassifier	0.919	0.006	n_estimators=700, max_depth=3, learning_rate=0.01, max_features="log2"
GradientBoostingClassifier	0.918	0.008	n_estimators=300, max_depth=7, learning_rate=0.01, max_features="log2"
GradientBoostingClassifier	0.918	0.007	n_estimators=700, max_depth=1, learning_rate=0.1, max_features="log2"
GradientBoostingClassifier	0.918	0.006	n_estimators=300, max_depth=5, learning_rate=0.01, max_features="log2"

Models

GradientBoostingClassifier

mean(scores)	std(scores)	params
---------------:	--------------:	:-----------------------------------------------------------------------
0.922	0.007	n_estimators=700, max_depth=5, learning_rate=0.01, max_features="log2"
0.921	0.007	n_estimators=500, max_depth=5, learning_rate=0.01, max_features="log2"
0.921	0.009	n_estimators=700, max_depth=7, learning_rate=0.01, max_features="log2"
0.92	0.008	n_estimators=500, max_depth=7, learning_rate=0.01, max_features="log2"
0.919	0.005	n_estimators=100, max_depth=3, learning_rate=0.1, max_features="log2"
0.919	0.009	n_estimators=300, max_depth=3, learning_rate=0.1, max_features="log2"
0.919	0.006	n_estimators=700, max_depth=3, learning_rate=0.01, max_features="log2"
0.918	0.008	n_estimators=300, max_depth=7, learning_rate=0.01, max_features="log2"
0.918	0.007	n_estimators=700, max_depth=1, learning_rate=0.1, max_features="log2"
0.918	0.006	n_estimators=300, max_depth=5, learning_rate=0.01, max_features="log2"
0.918	0.008	n_estimators=100, max_depth=5, learning_rate=0.1, max_features="log2"
0.917	0.007	n_estimators=500, max_depth=1, learning_rate=0.1, max_features="log2"
0.917	0.006	n_estimators=300, max_depth=1, learning_rate=0.5, max_features="log2"
0.917	0.012	n_estimators=500, max_depth=3, learning_rate=0.1, max_features="log2"
0.916	0.006	n_estimators=500, max_depth=3, learning_rate=0.01, max_features="log2"
0.916	0.009	n_estimators=500, max_depth=1, learning_rate=0.5, max_features="log2"
0.915	0.006	n_estimators=300, max_depth=1, learning_rate=0.1, max_features="log2"
0.914	0.008	n_estimators=100, max_depth=1, learning_rate=0.5, max_features="log2"
0.914	0.01	n_estimators=700, max_depth=3, learning_rate=0.1, max_features="log2"
0.914	0.006	n_estimators=100, max_depth=7, learning_rate=0.01, max_features="log2"
0.914	0.009	n_estimators=700, max_depth=1, learning_rate=0.5, max_features="log2"
0.913	0.006	n_estimators=300, max_depth=3, learning_rate=0.01, max_features="log2"
0.912	0.012	n_estimators=300, max_depth=5, learning_rate=0.1, max_features="log2"
0.912	0.006	n_estimators=100, max_depth=5, learning_rate=0.01, max_features="log2"
0.912	0.008	n_estimators=100, max_depth=7, learning_rate=0.1, max_features="log2"
0.909	0.012	n_estimators=700, max_depth=5, learning_rate=0.1, max_features="log2"
0.908	0.011	n_estimators=500, max_depth=5, learning_rate=0.1, max_features="log2"
0.907	0.013	n_estimators=300, max_depth=7, learning_rate=0.1, max_features="log2"
0.907	0.011	n_estimators=700, max_depth=7, learning_rate=0.1, max_features="log2"
0.905	0.011	n_estimators=500, max_depth=7, learning_rate=0.1, max_features="log2"
0.903	0.008	n_estimators=100, max_depth=3, learning_rate=0.01, max_features="log2"
0.903	0.007	n_estimators=100, max_depth=1, learning_rate=0.1, max_features="log2"
0.902	0.007	n_estimators=700, max_depth=1, learning_rate=0.01, max_features="log2"
0.897	0.021	n_estimators=100, max_depth=1, learning_rate=1, max_features="log2"
0.895	0.007	n_estimators=500, max_depth=1, learning_rate=0.01, max_features="log2"
0.892	0.013	n_estimators=100, max_depth=3, learning_rate=0.5, max_features="log2"
0.892	0.013	n_estimators=300, max_depth=3, learning_rate=0.5, max_features="log2"
0.892	0.016	n_estimators=500, max_depth=1, learning_rate=1, max_features="log2"
0.886	0.011	n_estimators=300, max_depth=1, learning_rate=0.01, max_features="log2"
0.885	0.03	n_estimators=300, max_depth=1, learning_rate=1, max_features="log2"
0.883	0.017	n_estimators=700, max_depth=5, learning_rate=0.5, max_features="log2"
0.882	0.016	n_estimators=500, max_depth=3, learning_rate=0.5, max_features="log2"
0.881	0.019	n_estimators=700, max_depth=1, learning_rate=1, max_features="log2"
0.876	0.026	n_estimators=700, max_depth=3, learning_rate=0.5, max_features="log2"
0.875	0.018	n_estimators=500, max_depth=5, learning_rate=0.5, max_features="log2"
0.871	0.014	n_estimators=100, max_depth=1, learning_rate=0.01, max_features="log2"
0.867	0.021	n_estimators=500, max_depth=7, learning_rate=0.5, max_features="log2"
0.862	0.021	n_estimators=100, max_depth=5, learning_rate=0.5, max_features="log2"
0.856	0.046	n_estimators=300, max_depth=5, learning_rate=0.5, max_features="log2"
0.847	0.06	n_estimators=100, max_depth=7, learning_rate=0.5, max_features="log2"
0.837	0.05	n_estimators=100, max_depth=3, learning_rate=1, max_features="log2"
0.836	0.092	n_estimators=300, max_depth=7, learning_rate=0.5, max_features="log2"
0.835	0.072	n_estimators=700, max_depth=7, learning_rate=0.5, max_features="log2"
0.806	0.016	n_estimators=100, max_depth=5, learning_rate=1, max_features="log2"
0.779	0.055	n_estimators=300, max_depth=3, learning_rate=1, max_features="log2"
0.776	0.06	n_estimators=500, max_depth=3, learning_rate=1, max_features="log2"
0.713	0.066	n_estimators=100, max_depth=7, learning_rate=1, max_features="log2"
0.711	0.101	n_estimators=300, max_depth=5, learning_rate=1, max_features="log2"
0.696	0.154	n_estimators=700, max_depth=3, learning_rate=1, max_features="log2"
0.682	0.073	n_estimators=300, max_depth=7, learning_rate=1, max_features="log2"
0.645	0.118	n_estimators=700, max_depth=5, learning_rate=1, max_features="log2"
0.58	0.152	n_estimators=500, max_depth=5, learning_rate=1, max_features="log2"
0.559	0.182	n_estimators=700, max_depth=7, learning_rate=1, max_features="log2"
0.467	0.123	n_estimators=500, max_depth=7, learning_rate=1, max_features="log2"

GaussianNB

mean(scores)	std(scores)	params
---------------:	--------------:	:---------
0.777	0.03

LogisticRegression

mean(scores)	std(scores)	params
---------------:	--------------:	:--------------------
0.903	0.004	penalty="l1", C=10
0.903	0.005	penalty="l1", C=1
0.896	0.008	penalty="l1", C=0.1
0.846	0.007	penalty="l2", C=1
0.845	0.011	penalty="l2", C=0.1
0.842	0.01	penalty="l2", C=10

BernoulliNB

mean(scores)	std(scores)	params
---------------:	--------------:	:---------
0.828	0.012

RandomForestClassifier

mean(scores)	std(scores)	params
---------------:	--------------:	:--------------------------------------------------------------------------------
0.917	0.006	criterion="entropy", n_estimators=640, min_samples_leaf=13, max_features="log2"
0.917	0.005	criterion="entropy", n_estimators=160, min_samples_leaf=7, max_features="log2"
0.916	0.006	criterion="entropy", n_estimators=640, min_samples_leaf=7, max_features="log2"
0.916	0.005	criterion="entropy", n_estimators=320, min_samples_leaf=13, max_features="log2"
0.916	0.006	criterion="entropy", n_estimators=640, min_samples_leaf=5, max_features="log2"
0.915	0.007	criterion="entropy", n_estimators=320, min_samples_leaf=5, max_features="log2"
0.915	0.007	criterion="entropy", n_estimators=160, min_samples_leaf=13, max_features="log2"
0.915	0.006	criterion="entropy", n_estimators=320, min_samples_leaf=7, max_features="log2"
0.914	0.007	criterion="entropy", n_estimators=640, min_samples_leaf=3, max_features="log2"
0.914	0.006	criterion="entropy", n_estimators=320, min_samples_leaf=3, max_features="log2"
0.914	0.006	criterion="entropy", n_estimators=80, min_samples_leaf=13, max_features="log2"
0.913	0.008	criterion="entropy", n_estimators=160, min_samples_leaf=5, max_features="log2"
0.912	0.006	criterion="gini", n_estimators=160, min_samples_leaf=13, max_features="log2"
0.912	0.006	criterion="gini", n_estimators=640, min_samples_leaf=13, max_features="log2"
0.912	0.004	criterion="entropy", n_estimators=640, min_samples_leaf=1, max_features="log2"
0.912	0.006	criterion="gini", n_estimators=640, min_samples_leaf=7, max_features="log2"
0.912	0.006	criterion="gini", n_estimators=320, min_samples_leaf=13, max_features="log2"
0.911	0.006	criterion="entropy", n_estimators=80, min_samples_leaf=7, max_features="log2"
0.911	0.006	criterion="gini", n_estimators=640, min_samples_leaf=3, max_features="log2"
0.911	0.006	criterion="gini", n_estimators=640, min_samples_leaf=5, max_features="log2"
0.911	0.005	criterion="gini", n_estimators=320, min_samples_leaf=7, max_features="log2"
0.911	0.004	criterion="gini", n_estimators=320, min_samples_leaf=5, max_features="log2"
0.91	0.008	criterion="entropy", n_estimators=160, min_samples_leaf=3, max_features="log2"
0.91	0.007	criterion="entropy", n_estimators=40, min_samples_leaf=13, max_features="log2"
0.91	0.006	criterion="gini", n_estimators=160, min_samples_leaf=7, max_features="log2"
0.909	0.004	criterion="entropy", n_estimators=80, min_samples_leaf=5, max_features="log2"
0.908	0.005	criterion="gini", n_estimators=320, min_samples_leaf=3, max_features="log2"
0.908	0.007	criterion="gini", n_estimators=40, min_samples_leaf=13, max_features="log2"
0.908	0.004	criterion="entropy", n_estimators=320, min_samples_leaf=1, max_features="log2"
0.908	0.006	criterion="gini", n_estimators=80, min_samples_leaf=13, max_features="log2"
0.908	0.007	criterion="gini", n_estimators=160, min_samples_leaf=5, max_features="log2"
0.907	0.005	criterion="gini", n_estimators=80, min_samples_leaf=5, max_features="log2"
0.907	0.008	criterion="gini", n_estimators=80, min_samples_leaf=7, max_features="log2"
0.907	0.006	criterion="entropy", n_estimators=40, min_samples_leaf=7, max_features="log2"
0.906	0.006	criterion="entropy", n_estimators=40, min_samples_leaf=5, max_features="log2"
0.905	0.006	criterion="gini", n_estimators=160, min_samples_leaf=3, max_features="log2"
0.904	0.008	criterion="gini", n_estimators=640, min_samples_leaf=1, max_features="log2"
0.904	0.005	criterion="gini", n_estimators=320, min_samples_leaf=1, max_features="log2"
0.904	0.006	criterion="entropy", n_estimators=160, min_samples_leaf=1, max_features="log2"
0.903	0.008	criterion="gini", n_estimators=40, min_samples_leaf=7, max_features="log2"
0.903	0.005	criterion="gini", n_estimators=40, min_samples_leaf=5, max_features="log2"
0.902	0.008	criterion="entropy", n_estimators=80, min_samples_leaf=3, max_features="log2"
0.901	0.003	criterion="entropy", n_estimators=40, min_samples_leaf=3, max_features="log2"
0.901	0.009	criterion="entropy", n_estimators=20, min_samples_leaf=13, max_features="log2"
0.901	0.004	criterion="gini", n_estimators=160, min_samples_leaf=1, max_features="log2"
0.9	0.006	criterion="gini", n_estimators=80, min_samples_leaf=3, max_features="log2"
0.899	0.006	criterion="gini", n_estimators=40, min_samples_leaf=3, max_features="log2"
0.899	0.009	criterion="gini", n_estimators=20, min_samples_leaf=7, max_features="log2"
0.897	0.009	criterion="entropy", n_estimators=20, min_samples_leaf=7, max_features="log2"
0.897	0.007	criterion="entropy", n_estimators=80, min_samples_leaf=1, max_features="log2"
0.896	0.009	criterion="gini", n_estimators=20, min_samples_leaf=13, max_features="log2"
0.895	0.008	criterion="entropy", n_estimators=20, min_samples_leaf=5, max_features="log2"
0.894	0.005	criterion="gini", n_estimators=20, min_samples_leaf=5, max_features="log2"
0.89	0.009	criterion="entropy", n_estimators=10, min_samples_leaf=13, max_features="log2"
0.89	0.01	criterion="gini", n_estimators=10, min_samples_leaf=13, max_features="log2"
0.888	0.005	criterion="gini", n_estimators=80, min_samples_leaf=1, max_features="log2"
0.883	0.009	criterion="entropy", n_estimators=20, min_samples_leaf=3, max_features="log2"
0.883	0.019	criterion="gini", n_estimators=10, min_samples_leaf=7, max_features="log2"
0.883	0.006	criterion="gini", n_estimators=20, min_samples_leaf=3, max_features="log2"
0.88	0.005	criterion="entropy", n_estimators=10, min_samples_leaf=7, max_features="log2"
0.879	0.002	criterion="gini", n_estimators=40, min_samples_leaf=1, max_features="log2"
0.876	0.012	criterion="entropy", n_estimators=10, min_samples_leaf=5, max_features="log2"
0.874	0.015	criterion="gini", n_estimators=10, min_samples_leaf=5, max_features="log2"
0.873	0.015	criterion="entropy", n_estimators=40, min_samples_leaf=1, max_features="log2"
0.859	0.013	criterion="entropy", n_estimators=10, min_samples_leaf=3, max_features="log2"
0.852	0.021	criterion="entropy", n_estimators=20, min_samples_leaf=1, max_features="log2"
0.846	0.004	criterion="gini", n_estimators=10, min_samples_leaf=3, max_features="log2"
0.844	0.011	criterion="gini", n_estimators=20, min_samples_leaf=1, max_features="log2"
0.806	0.009	criterion="entropy", n_estimators=10, min_samples_leaf=1, max_features="log2"
0.802	0.012	criterion="gini", n_estimators=10, min_samples_leaf=1, max_features="log2"

Goodfaith model shows a slight fall in accuracy:

Model tuning report

Revscoring version: 1.3.17
Features: editquality.feature_lists.enwiki.goodfaith
Date: 2017-07-20T15:34:07.565220
Observations: 19460
Labels: [false, true]
Scoring: roc_auc
Folds: 5

Top scoring configurations

model	mean(scores)	std(scores)	params
:---------------------------	---------------:	--------------:	:-----------------------------------------------------------------------
GradientBoostingClassifier	0.907	0.012	max_features="log2", max_depth=5, learning_rate=0.01, n_estimators=700
GradientBoostingClassifier	0.906	0.014	max_features="log2", max_depth=7, learning_rate=0.01, n_estimators=500
GradientBoostingClassifier	0.905	0.013	max_features="log2", max_depth=5, learning_rate=0.01, n_estimators=500
GradientBoostingClassifier	0.905	0.011	max_features="log2", max_depth=3, learning_rate=0.01, n_estimators=700
GradientBoostingClassifier	0.904	0.015	max_features="log2", max_depth=7, learning_rate=0.01, n_estimators=700
GradientBoostingClassifier	0.904	0.014	max_features="log2", max_depth=7, learning_rate=0.01, n_estimators=300
GradientBoostingClassifier	0.902	0.012	max_features="log2", max_depth=5, learning_rate=0.01, n_estimators=300
GradientBoostingClassifier	0.902	0.011	max_features="log2", max_depth=1, learning_rate=0.1, n_estimators=700
GradientBoostingClassifier	0.902	0.011	max_features="log2", max_depth=1, learning_rate=0.1, n_estimators=500
GradientBoostingClassifier	0.902	0.011	max_features="log2", max_depth=3, learning_rate=0.01, n_estimators=500

Models

LogisticRegression

mean(scores)	std(scores)	params
---------------:	--------------:	:--------------------
0.887	0.011	C=1, penalty="l1"
0.886	0.012	C=10, penalty="l1"
0.881	0.014	C=0.1, penalty="l1"
0.836	0.017	C=1, penalty="l2"
0.829	0.008	C=10, penalty="l2"
0.828	0.017	C=0.1, penalty="l2"

GradientBoostingClassifier

mean(scores)	std(scores)	params
---------------:	--------------:	:-----------------------------------------------------------------------
0.907	0.012	max_features="log2", max_depth=5, learning_rate=0.01, n_estimators=700
0.906	0.014	max_features="log2", max_depth=7, learning_rate=0.01, n_estimators=500
0.905	0.013	max_features="log2", max_depth=5, learning_rate=0.01, n_estimators=500
0.905	0.011	max_features="log2", max_depth=3, learning_rate=0.01, n_estimators=700
0.904	0.015	max_features="log2", max_depth=7, learning_rate=0.01, n_estimators=700
0.904	0.014	max_features="log2", max_depth=7, learning_rate=0.01, n_estimators=300
0.902	0.012	max_features="log2", max_depth=5, learning_rate=0.01, n_estimators=300
0.902	0.011	max_features="log2", max_depth=1, learning_rate=0.1, n_estimators=700
0.902	0.011	max_features="log2", max_depth=1, learning_rate=0.1, n_estimators=500
0.902	0.011	max_features="log2", max_depth=3, learning_rate=0.01, n_estimators=500
0.901	0.014	max_features="log2", max_depth=3, learning_rate=0.1, n_estimators=100
0.9	0.011	max_features="log2", max_depth=3, learning_rate=0.1, n_estimators=300
0.9	0.016	max_features="log2", max_depth=5, learning_rate=0.1, n_estimators=100
0.9	0.01	max_features="log2", max_depth=1, learning_rate=0.1, n_estimators=300
0.899	0.012	max_features="log2", max_depth=1, learning_rate=0.5, n_estimators=500
0.899	0.014	max_features="log2", max_depth=7, learning_rate=0.01, n_estimators=100
0.898	0.015	max_features="log2", max_depth=5, learning_rate=0.1, n_estimators=300
0.897	0.015	max_features="log2", max_depth=3, learning_rate=0.1, n_estimators=500
0.897	0.01	max_features="log2", max_depth=1, learning_rate=0.5, n_estimators=700
0.897	0.009	max_features="log2", max_depth=1, learning_rate=0.5, n_estimators=300
0.897	0.015	max_features="log2", max_depth=7, learning_rate=0.1, n_estimators=100
0.896	0.011	max_features="log2", max_depth=3, learning_rate=0.01, n_estimators=300
0.895	0.014	max_features="log2", max_depth=5, learning_rate=0.01, n_estimators=100
0.894	0.017	max_features="log2", max_depth=3, learning_rate=0.1, n_estimators=700
0.894	0.014	max_features="log2", max_depth=1, learning_rate=0.5, n_estimators=100
0.891	0.02	max_features="log2", max_depth=7, learning_rate=0.1, n_estimators=300
0.891	0.02	max_features="log2", max_depth=5, learning_rate=0.1, n_estimators=500
0.889	0.01	max_features="log2", max_depth=1, learning_rate=0.1, n_estimators=100
0.889	0.018	max_features="log2", max_depth=5, learning_rate=0.1, n_estimators=700
0.887	0.021	max_features="log2", max_depth=7, learning_rate=0.1, n_estimators=700
0.887	0.011	max_features="log2", max_depth=1, learning_rate=0.01, n_estimators=700
0.886	0.015	max_features="log2", max_depth=3, learning_rate=0.01, n_estimators=100
0.885	0.022	max_features="log2", max_depth=7, learning_rate=0.1, n_estimators=500
0.88	0.014	max_features="log2", max_depth=1, learning_rate=0.01, n_estimators=500
0.875	0.014	max_features="log2", max_depth=3, learning_rate=0.5, n_estimators=100
0.869	0.016	max_features="log2", max_depth=1, learning_rate=0.01, n_estimators=300
0.866	0.017	max_features="log2", max_depth=3, learning_rate=0.5, n_estimators=700
0.863	0.018	max_features="log2", max_depth=3, learning_rate=0.5, n_estimators=300
0.861	0.019	max_features="log2", max_depth=3, learning_rate=0.5, n_estimators=500
0.861	0.016	max_features="log2", max_depth=1, learning_rate=0.01, n_estimators=100
0.86	0.015	max_features="log2", max_depth=5, learning_rate=0.5, n_estimators=700
0.856	0.024	max_features="log2", max_depth=1, learning_rate=1, n_estimators=700
0.849	0.025	max_features="log2", max_depth=5, learning_rate=0.5, n_estimators=500
0.847	0.02	max_features="log2", max_depth=7, learning_rate=0.5, n_estimators=500
0.847	0.021	max_features="log2", max_depth=7, learning_rate=0.5, n_estimators=700
0.844	0.027	max_features="log2", max_depth=5, learning_rate=0.5, n_estimators=300
0.841	0.044	max_features="log2", max_depth=1, learning_rate=1, n_estimators=300
0.836	0.05	max_features="log2", max_depth=7, learning_rate=0.5, n_estimators=300
0.836	0.027	max_features="log2", max_depth=7, learning_rate=0.5, n_estimators=100
0.835	0.023	max_features="log2", max_depth=3, learning_rate=1, n_estimators=100
0.832	0.016	max_features="log2", max_depth=5, learning_rate=0.5, n_estimators=100
0.829	0.046	max_features="log2", max_depth=1, learning_rate=1, n_estimators=100
0.823	0.048	max_features="log2", max_depth=1, learning_rate=1, n_estimators=500
0.761	0.048	max_features="log2", max_depth=3, learning_rate=1, n_estimators=300
0.752	0.073	max_features="log2", max_depth=3, learning_rate=1, n_estimators=500
0.745	0.029	max_features="log2", max_depth=3, learning_rate=1, n_estimators=700
0.737	0.05	max_features="log2", max_depth=5, learning_rate=1, n_estimators=100
0.729	0.041	max_features="log2", max_depth=7, learning_rate=1, n_estimators=100
0.668	0.112	max_features="log2", max_depth=5, learning_rate=1, n_estimators=500
0.63	0.129	max_features="log2", max_depth=5, learning_rate=1, n_estimators=300
0.608	0.085	max_features="log2", max_depth=7, learning_rate=1, n_estimators=700
0.596	0.09	max_features="log2", max_depth=7, learning_rate=1, n_estimators=300
0.586	0.079	max_features="log2", max_depth=5, learning_rate=1, n_estimators=700
0.536	0.058	max_features="log2", max_depth=7, learning_rate=1, n_estimators=500

BernoulliNB

mean(scores)	std(scores)	params
---------------:	--------------:	:---------
0.817	0.02

RandomForestClassifier

mean(scores)	std(scores)	params
---------------:	--------------:	:--------------------------------------------------------------------------------
0.9	0.017	min_samples_leaf=13, max_features="log2", n_estimators=640, criterion="entropy"
0.899	0.015	min_samples_leaf=13, max_features="log2", n_estimators=160, criterion="entropy"
0.899	0.016	min_samples_leaf=13, max_features="log2", n_estimators=320, criterion="entropy"
0.899	0.017	min_samples_leaf=5, max_features="log2", n_estimators=640, criterion="entropy"
0.898	0.016	min_samples_leaf=7, max_features="log2", n_estimators=640, criterion="entropy"
0.897	0.016	min_samples_leaf=13, max_features="log2", n_estimators=80, criterion="entropy"
0.897	0.017	min_samples_leaf=5, max_features="log2", n_estimators=320, criterion="entropy"
0.897	0.014	min_samples_leaf=5, max_features="log2", n_estimators=160, criterion="entropy"
0.897	0.018	min_samples_leaf=3, max_features="log2", n_estimators=640, criterion="entropy"
0.896	0.018	min_samples_leaf=7, max_features="log2", n_estimators=320, criterion="entropy"
0.896	0.014	min_samples_leaf=13, max_features="log2", n_estimators=40, criterion="entropy"
0.896	0.014	min_samples_leaf=7, max_features="log2", n_estimators=160, criterion="entropy"
0.896	0.014	min_samples_leaf=13, max_features="log2", n_estimators=320, criterion="gini"
0.895	0.013	min_samples_leaf=13, max_features="log2", n_estimators=640, criterion="gini"
0.894	0.015	min_samples_leaf=3, max_features="log2", n_estimators=80, criterion="entropy"
0.894	0.016	min_samples_leaf=7, max_features="log2", n_estimators=80, criterion="entropy"
0.893	0.014	min_samples_leaf=7, max_features="log2", n_estimators=640, criterion="gini"
0.893	0.014	min_samples_leaf=5, max_features="log2", n_estimators=640, criterion="gini"
0.892	0.016	min_samples_leaf=13, max_features="log2", n_estimators=80, criterion="gini"
0.892	0.014	min_samples_leaf=7, max_features="log2", n_estimators=160, criterion="gini"
0.892	0.018	min_samples_leaf=1, max_features="log2", n_estimators=640, criterion="entropy"
0.892	0.015	min_samples_leaf=3, max_features="log2", n_estimators=160, criterion="entropy"
0.891	0.018	min_samples_leaf=3, max_features="log2", n_estimators=320, criterion="entropy"
0.891	0.015	min_samples_leaf=13, max_features="log2", n_estimators=160, criterion="gini"
0.891	0.015	min_samples_leaf=5, max_features="log2", n_estimators=320, criterion="gini"
0.89	0.018	min_samples_leaf=7, max_features="log2", n_estimators=40, criterion="entropy"
0.89	0.015	min_samples_leaf=7, max_features="log2", n_estimators=320, criterion="gini"
0.89	0.016	min_samples_leaf=5, max_features="log2", n_estimators=80, criterion="entropy"
0.89	0.015	min_samples_leaf=3, max_features="log2", n_estimators=640, criterion="gini"
0.89	0.014	min_samples_leaf=7, max_features="log2", n_estimators=80, criterion="gini"
0.889	0.016	min_samples_leaf=5, max_features="log2", n_estimators=160, criterion="gini"
0.889	0.017	min_samples_leaf=1, max_features="log2", n_estimators=160, criterion="entropy"
0.889	0.013	min_samples_leaf=13, max_features="log2", n_estimators=20, criterion="gini"
0.889	0.02	min_samples_leaf=13, max_features="log2", n_estimators=20, criterion="entropy"
0.889	0.014	min_samples_leaf=3, max_features="log2", n_estimators=320, criterion="gini"
0.888	0.018	min_samples_leaf=3, max_features="log2", n_estimators=40, criterion="entropy"
0.888	0.015	min_samples_leaf=7, max_features="log2", n_estimators=40, criterion="gini"
0.888	0.016	min_samples_leaf=1, max_features="log2", n_estimators=320, criterion="entropy"
0.887	0.015	min_samples_leaf=3, max_features="log2", n_estimators=160, criterion="gini"
0.887	0.016	min_samples_leaf=1, max_features="log2", n_estimators=640, criterion="gini"
0.887	0.018	min_samples_leaf=13, max_features="log2", n_estimators=40, criterion="gini"
0.886	0.016	min_samples_leaf=5, max_features="log2", n_estimators=40, criterion="entropy"
0.885	0.02	min_samples_leaf=5, max_features="log2", n_estimators=40, criterion="gini"
0.885	0.017	min_samples_leaf=3, max_features="log2", n_estimators=80, criterion="gini"
0.884	0.019	min_samples_leaf=5, max_features="log2", n_estimators=80, criterion="gini"
0.883	0.014	min_samples_leaf=1, max_features="log2", n_estimators=320, criterion="gini"
0.882	0.017	min_samples_leaf=7, max_features="log2", n_estimators=20, criterion="entropy"
0.881	0.021	min_samples_leaf=5, max_features="log2", n_estimators=20, criterion="entropy"
0.88	0.019	min_samples_leaf=1, max_features="log2", n_estimators=80, criterion="entropy"
0.876	0.01	min_samples_leaf=3, max_features="log2", n_estimators=40, criterion="gini"
0.876	0.013	min_samples_leaf=1, max_features="log2", n_estimators=160, criterion="gini"
0.875	0.015	min_samples_leaf=7, max_features="log2", n_estimators=20, criterion="gini"
0.874	0.011	min_samples_leaf=13, max_features="log2", n_estimators=10, criterion="entropy"
0.874	0.024	min_samples_leaf=13, max_features="log2", n_estimators=10, criterion="gini"
0.87	0.017	min_samples_leaf=5, max_features="log2", n_estimators=20, criterion="gini"
0.867	0.017	min_samples_leaf=3, max_features="log2", n_estimators=20, criterion="entropy"
0.865	0.013	min_samples_leaf=1, max_features="log2", n_estimators=80, criterion="gini"
0.864	0.015	min_samples_leaf=7, max_features="log2", n_estimators=10, criterion="gini"
0.858	0.012	min_samples_leaf=1, max_features="log2", n_estimators=40, criterion="entropy"
0.858	0.014	min_samples_leaf=3, max_features="log2", n_estimators=20, criterion="gini"
0.857	0.024	min_samples_leaf=7, max_features="log2", n_estimators=10, criterion="entropy"
0.853	0.024	min_samples_leaf=5, max_features="log2", n_estimators=10, criterion="entropy"
0.852	0.015	min_samples_leaf=5, max_features="log2", n_estimators=10, criterion="gini"
0.85	0.014	min_samples_leaf=1, max_features="log2", n_estimators=40, criterion="gini"
0.846	0.025	min_samples_leaf=3, max_features="log2", n_estimators=10, criterion="gini"
0.839	0.006	min_samples_leaf=3, max_features="log2", n_estimators=10, criterion="entropy"
0.836	0.013	min_samples_leaf=1, max_features="log2", n_estimators=20, criterion="entropy"
0.819	0.019	min_samples_leaf=1, max_features="log2", n_estimators=20, criterion="gini"
0.789	0.004	min_samples_leaf=1, max_features="log2", n_estimators=10, criterion="gini"
0.783	0.009	min_samples_leaf=1, max_features="log2", n_estimators=10, criterion="entropy"

GaussianNB

mean(scores)	std(scores)	params
---------------:	--------------:	:---------
0.798	0.032

Halfak mentioned this in T171386: Investigate small loss in fitness with the new data in fawiki.Jul 24 2017, 4:48 PM

After taking Adam's suggestions into account and scoring the sentiment of edits, I still wasn't able to get a decent signal. The 1% rise in accuracy is due to something else, and I think sentiment on edits is not a very good feature. To validate my conclusion further, I took 1000 samples from enwiki, half damaging and half not damaging and here's the result:

Enwiki_1000_damaging_notdamaging_edits_sentiment (640×480 px, 17 KB)

My reasoning for the above result goes like this:
Most of the newer edits on Wikipedia are small or those done with a damaging intent contain random or bad words. Both of these do not fall in the definition of intellectual negative words which might degrade the overall quality of article in a meaningful sense. Hence the sentiment score both positive and negative seems roughly the same for every edit either damaging or not damaging

Sumit moved this task from Parked to Completed on the Machine-Learning-Team (Active Tasks) board.Aug 7 2017, 3:45 AM

Halfak closed this task as Resolved.Sep 22 2017, 9:17 PM

awight mentioned this in Blog Post: Status update (October 6, 2017).Oct 6 2017, 1:26 AM

Test draftquality sentiment feature on EditqualityClosed, ResolvedPublicActions

Description

Related Objects

Event Timeline

Top scoring configurations

Models

GradientBoostingClassifier

GaussianNB

LogisticRegression

BernoulliNB

RandomForestClassifier

Top scoring configurations

Models

LogisticRegression

GradientBoostingClassifier

BernoulliNB

RandomForestClassifier

GaussianNB

Test draftquality sentiment feature on Editquality
Closed, ResolvedPublic
Actions