Page MenuHomePhabricator
Create Task
Maniphest T392148

Run analysis to retrieve thresholds for high impact wikis to deploy recent changes revert risk language agnostic filters to
Closed, ResolvedPublic3 Estimated Story Points
Actions

Description

In T392144: Identify highest impact wikis to deploy recent changes revert risk language agnostic filters to we identified a list of high impact wikis to deploy recent changes revert risk language agnostic filters to.

Now that we have that list we should use it to re-run the analysis to determine appropriate thresholds for each wiki. We should also remove any wikis from the list where we find the model is not performing well enough (has unacceptable rates of false positives).

Acceptance Criteria:

  • Final list of high impact wikis to deploy recent changes revert risk language agnostic filters to
  • Thresholds for each wiki that target an approximate ~15% false positive rate

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
Revert-risk-threshold-analysis from R to Python.repos/machine-learning/exploratory-notebook!3gkyziridisrevert-risk-threshold-analysis-pythonmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

Kgraessle created this task.Apr 16 2025, 7:22 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 16 2025, 7:22 PM
Kgraessle moved this task from To be estimated to Kanban on the Moderator-Tools-Team board.May 5 2025, 3:32 PM
Kgraessle edited projects, added Moderator-Tools-Team (Kanban); removed Moderator-Tools-Team.
Kgraessle moved this task from Kanban to To be estimated on the Moderator-Tools-Team board.May 5 2025, 3:37 PM
Kgraessle edited projects, added Moderator-Tools-Team; removed Moderator-Tools-Team (Kanban).
gkyziridis added subscribers: MNeisler, gkyziridis.May 9 2025, 2:22 PM

Hi @MNeisler, I would like to ask some question about the revert-risk threshold analysis based on this repo.

Based on what I understood from the analysis is the following:

  1. Load revert risk data in a time window for a specific wiki (idwiki in this case)
  2. Clean data (remove duplicates, NaN, etc...)
  3. Calculate the ROC curve for the existing data by assuming the following:
    • Actual -> revert_risk_data$is_reverted (a binary representation of true/false)
    • Predicted -> revert_risk_data$revert_risk_score (considered as the probabilities)
  4. Then we want to find the optimal threshold to apply the binary classification based on the following:
    • Optimal threshold is the one that brings 15% False Positive Rate
  5. So we derive the threshold (upper_threshold <- 0.845 in the idwiki case) from the ROC curve.

Please correct me if I missed something from the above logic.

What I do not understand is that in this notebook at section: "Bucketing Approach #5: Tertiles while meeting 15% False Postive Target Rate" we are using the upper_threshold <- 0.845 but exactly under that block I see that we are applying the classification using different thresholds, and afterwards we are calculate the confusion matrix.

So my question is basically, don't we just need the ROC curve for each wiki, derive the optimal threshold which brings 15% FPR (derived from ROC), then apply the classification rule for that threshold on the revert_risk_score and finally evaluate the classification results with a confusion matrix (which actually will show that we have 15% false positive rate since we used that threshold). So, as a general rule we will need to find that threshold that brings 15% FPR for each wiki right?

Let me know if I am missing something, or if I understood something wrong.

CodeReviewBot added a comment.May 12 2025, 7:24 AM

isaranto updated https://gitlab.wikimedia.org/repos/machine-learning/exploratory-notebook/-/merge_requests/3

Revert-risk-threshold-analysis from R to Python.

CodeReviewBot added a comment.May 12 2025, 9:15 AM

gkyziridis merged https://gitlab.wikimedia.org/repos/machine-learning/exploratory-notebook/-/merge_requests/3

Revert-risk-threshold-analysis from R to Python.

Kgraessle closed this task as Resolved.May 13 2025, 3:13 PM
Kgraessle claimed this task.
Kgraessle set the point value for this task to 5.
Kgraessle changed the point value for this task from 5 to 3.May 13 2025, 3:15 PM
Kgraessle reopened this task as Open.May 13 2025, 3:27 PM
Kgraessle moved this task from To be estimated to Estimated on the Moderator-Tools-Team board.
DMburugu moved this task from Estimated to Kanban on the Moderator-Tools-Team board.May 13 2025, 4:07 PM
DMburugu edited projects, added Moderator-Tools-Team (Kanban); removed Moderator-Tools-Team.
DMburugu triaged this task as High priority.May 13 2025, 4:12 PM
Kgraessle added a comment.May 13 2025, 4:27 PM

@gkyziridis
Now that you have rewritten the analysis in python- do you want me to assign this ticket to you/the machine learning team instead of us?

Note the proposed list of wikis is located here: T392144: Identify highest impact wikis to deploy recent changes revert risk language agnostic filters to

gkyziridis added a comment.May 14 2025, 2:29 PM
In T392148#10817439, @Kgraessle wrote:

@gkyziridis
Now that you have rewritten the analysis in python- do you want me to assign this ticket to you/the machine learning team instead of us?

Note the proposed list of wikis is located here: T392144: Identify highest impact wikis to deploy recent changes revert risk language agnostic filters to

Hi @Kgraessle,
If you could review the proposed-method-notebook and you agree/approve on the logic and the implementation, then the ML team can run it for the wikis and the ticket can be assigned to us.
But we would like to be sure that we are all in the same page regarding the implementation and the refactoring of the code in R into Python, and for the logic behind it as well.

Kgraessle added a project: Machine-Learning-Team.May 15 2025, 6:56 PM
In T392148#10822223, @gkyziridis wrote:
In T392148#10817439, @Kgraessle wrote:

@gkyziridis
Now that you have rewritten the analysis in python- do you want me to assign this ticket to you/the machine learning team instead of us?

Note the proposed list of wikis is located here: T392144: Identify highest impact wikis to deploy recent changes revert risk language agnostic filters to

Hi @Kgraessle,
If you could review the proposed-method-notebook and you agree/approve on the logic and the implementation, then the ML team can run it for the wikis and the ticket can be assigned to us.
But we would like to be sure that we are all in the same page regarding the implementation and the refactoring of the code in R into Python, and for the logic behind it as well.

Hi @gkyziridis, thanks for sharing this. I reviewed and I agree/approve with the logic and implementation, great work!

I did have one question about the query variable you're passing into load_data() is this the same query that is referenced here?

I'm also going to go ahead and tag your team on this ticket.

Kgraessle edited projects, added Moderator-Tools-Team; removed Moderator-Tools-Team (Kanban).May 15 2025, 6:59 PM
Kgraessle moved this task from Inbox to Estimated on the Moderator-Tools-Team board.
Kgraessle moved this task from Estimated to Product backlog on the Moderator-Tools-Team board.
gkyziridis added a comment.May 16 2025, 9:48 AM
In T392148#10827957, @Kgraessle wrote:
In T392148#10822223, @gkyziridis wrote:
In T392148#10817439, @Kgraessle wrote:

@gkyziridis
Now that you have rewritten the analysis in python- do you want me to assign this ticket to you/the machine learning team instead of us?

Note the proposed list of wikis is located here: T392144: Identify highest impact wikis to deploy recent changes revert risk language agnostic filters to

Hi @Kgraessle,
If you could review the proposed-method-notebook and you agree/approve on the logic and the implementation, then the ML team can run it for the wikis and the ticket can be assigned to us.
But we would like to be sure that we are all in the same page regarding the implementation and the refactoring of the code in R into Python, and for the logic behind it as well.

Hi @gkyziridis, thanks for sharing this. I reviewed and I agree/approve with the logic and implementation, great work!

I did have one question about the query variable you're passing into load_data() is this the same query that is referenced here?

I'm also going to go ahead and tag your team on this ticket.

Hey @Kgraessle thank you for reviewing the notebook. Yes it is exactly the same query that you were running.
It would be better to add most of the preprocess logic in the query but this can be done during the next iterations.

• isarantopoulos added a parent task: T391964: [Epic] Recent Changes ORES Enabled Revert Risk Powered Filters Rollout Plan.May 19 2025, 9:51 AM
gkyziridis added a comment.EditedMay 19 2025, 10:40 AM

RevertRisk Thresholds Analysis for all wikis

Using this notebook or this python_script I generated revert risk thresholds for all wikis in one go loading data for each wiki iteratively on memory. You can check the plots under the main section at the bottom of the notebook. The script provides the option to run the analysis on single wiki from user input either on all wikis.
The script ran at eqiad8 on jupyterlab, you can find the results in this paste

P76289 RevertRisk Threshold Analysis for all Wikis
1============ - cywiki - ============
2 - Raw data shape: (200511, 17)
3 - Duplicate rows found and removed: 1265
4 - Clean data shape: (199246, 17)
5 - Unique revision_ids: 199246 | Data Shape: 199246 | Same? : -> True
6 - Removing edits that are reverts from df | New Shape: (190747, 17)
7 - Is any revert_risk_score NA? : False
8 - Is any user_edit_count NA? : False
9 - Is any time_to_revert NA? : False
10 - ROC_cywiki.png saved!
11 - Optimal threshold for 15.0% FPR is: 0.11076630651950836
12 - confusion_matrix_cywiki.png saved!
13 - False Positive Rate is: 0.14999340149691756
14 - CONFUSION MATRIX -
15
16Predicted not reverted reverted
17Actual
18not reverted 135259 23868
19reverted 15169 16451
20
21
22============ - simplewiki - ============
23 - Raw data shape: (312209, 17)
24 - Duplicate rows found and removed: 41914
25 - Clean data shape: (270295, 17)
26 - Number of duplicated revision_ids found: 9
27 - Unique revision_ids: 270289 | Data Shape: 270289 | Same? : -> True
28 - Removing edits that are reverts from df | New Shape: (246893, 17)
29 - Is any revert_risk_score NA? : False
30 - Is any user_edit_count NA? : False
31 - Is any time_to_revert NA? : False
32 - ROC_simplewiki.png saved!
33 - Optimal threshold for 15.0% FPR is: 0.9065559506416321
34 - confusion_matrix_simplewiki.png saved!
35 - False Positive Rate is: 0.1500198513981056
36 - CONFUSION MATRIX -
37
38Predicted not reverted reverted
39Actual
40not reverted 179832 31740
41reverted 13622 21699
42
43
44============ - bewiki - ============
45 - Raw data shape: (80609, 17)
46 - Duplicate rows found and removed: 6221
47 - Clean data shape: (74388, 17)
48 - Unique revision_ids: 74388 | Data Shape: 74388 | Same? : -> True
49 - Removing edits that are reverts from df | New Shape: (73969, 17)
50 - Is any revert_risk_score NA? : False
51 - Is any user_edit_count NA? : False
52 - Is any time_to_revert NA? : False
53 - ROC_bewiki.png saved!
54 - Optimal threshold for 15.0% FPR is: 0.572495698928833
55 - confusion_matrix_bewiki.png saved!
56 - False Positive Rate is: 0.15014515086058638
57 - CONFUSION MATRIX -
58
59Predicted not reverted reverted
60Actual
61not reverted 61770 10913
62reverted 183 1103
63
64
65============ - kkwiki - ============
66 - Raw data shape: (82268, 17)
67 - Duplicate rows found and removed: 16708
68 - Clean data shape: (65560, 17)
69 - Unique revision_ids: 65560 | Data Shape: 65560 | Same? : -> True
70 - Removing edits that are reverts from df | New Shape: (64276, 17)
71 - Is any revert_risk_score NA? : False
72 - Is any user_edit_count NA? : False
73 - Is any time_to_revert NA? : False
74 - ROC_kkwiki.png saved!
75 - Optimal threshold for 15.0% FPR is: 0.6048707962036133
76 - confusion_matrix_kkwiki.png saved!
77 - False Positive Rate is: 0.14999318290272
78 - CONFUSION MATRIX -
79
80Predicted not reverted reverted
81Actual
82not reverted 49875 8801
83reverted 1475 4125
84
85
86============ - nnwiki - ============
87 - Raw data shape: (25248, 17)
88 - Duplicate rows found and removed: 4213
89 - Clean data shape: (21035, 17)
90 - Unique revision_ids: 21035 | Data Shape: 21035 | Same? : -> True
91 - Removing edits that are reverts from df | New Shape: (20392, 17)
92 - Is any revert_risk_score NA? : False
93 - Is any user_edit_count NA? : False
94 - Is any time_to_revert NA? : False
95 - ROC_nnwiki.png saved!
96 - Optimal threshold for 15.0% FPR is: 0.4162459373474121
97 - confusion_matrix_nnwiki.png saved!
98 - False Positive Rate is: 0.15001312680493567
99 - CONFUSION MATRIX -
100
101Predicted not reverted reverted
102Actual
103not reverted 16188 2857
104reverted 48 1299
105
106
107============ - mkwiki - ============
108 - Raw data shape: (54028, 17)
109 - Duplicate rows found and removed: 8215
110 - Clean data shape: (45813, 17)
111 - Unique revision_ids: 45813 | Data Shape: 45813 | Same? : -> True
112 - Removing edits that are reverts from df | New Shape: (44585, 17)
113 - Is any revert_risk_score NA? : False
114 - Is any user_edit_count NA? : False
115 - Is any time_to_revert NA? : False
116 - ROC_mkwiki.png saved!
117 - Optimal threshold for 15.0% FPR is: 0.505830705165863
118 - confusion_matrix_mkwiki.png saved!
119 - False Positive Rate is: 0.15001292809627906
120 - CONFUSION MATRIX -
121
122Predicted not reverted reverted
123Actual
124not reverted 36161 6382
125reverted 507 1535
126
127
128============ - lawiki - ============
129 - Raw data shape: (27151, 17)
130 - Duplicate rows found and removed: 3948
131 - Clean data shape: (23203, 17)
132 - Unique revision_ids: 23203 | Data Shape: 23203 | Same? : -> True
133 - Removing edits that are reverts from df | New Shape: (22893, 17)
134 - Is any revert_risk_score NA? : False
135 - Is any user_edit_count NA? : False
136 - Is any time_to_revert NA? : False
137 - ROC_lawiki.png saved!
138 - Optimal threshold for 15.0% FPR is: 0.6340628266334534
139 - confusion_matrix_lawiki.png saved!
140 - False Positive Rate is: 0.14979973297730306
141 - CONFUSION MATRIX -
142
143Predicted not reverted reverted
144Actual
145not reverted 19104 3366
146reverted 80 343
147
148
149============ - afwiki - ============
150 - Raw data shape: (21996, 17)
151 - Duplicate rows found and removed: 3614
152 - Clean data shape: (18382, 17)
153 - Unique revision_ids: 18382 | Data Shape: 18382 | Same? : -> True
154 - Removing edits that are reverts from df | New Shape: (17768, 17)
155 - Is any revert_risk_score NA? : False
156 - Is any user_edit_count NA? : False
157 - Is any time_to_revert NA? : False
158 - ROC_afwiki.png saved!
159 - Optimal threshold for 15.0% FPR is: 0.7369382977485657
160 - confusion_matrix_afwiki.png saved!
161 - False Positive Rate is: 0.1498459410174181
162 - CONFUSION MATRIX -
163
164Predicted not reverted reverted
165Actual
166not reverted 13520 2383
167reverted 214 1651
168
169============ - tewiki - ============
170 - Raw data shape: (97488, 17)
171 - Duplicate rows found and removed: 5150
172 - Clean data shape: (92338, 17)
173 - Unique revision_ids: 92338 | Data Shape: 92338 | Same? : -> True
174 - Removing edits that are reverts from df | New Shape: (91883, 17)
175 - Is any revert_risk_score NA? : False
176 - Is any user_edit_count NA? : False
177 - Is any time_to_revert NA? : False
178 - ROC_tewiki.png saved!
179 - Optimal threshold for 15.0% FPR is: 0.36725547909736633
180 - confusion_matrix_tewiki.png saved!
181 - False Positive Rate is: 0.1500330323717243
182 - CONFUSION MATRIX -
183
184Predicted not reverted reverted
185Actual
186not reverted 77194 13626
187reverted 242 821
188
189
190============ - mrwiki - ============
191 - Raw data shape: (42535, 17)
192 - Duplicate rows found and removed: 3930
193 - Clean data shape: (38605, 17)
194 - Number of duplicated revision_ids found: 2
195 - Unique revision_ids: 38604 | Data Shape: 38604 | Same? : -> True
196 - Removing edits that are reverts from df | New Shape: (37677, 17)
197 - Is any revert_risk_score NA? : False
198 - Is any user_edit_count NA? : False
199 - Is any time_to_revert NA? : False
200 - ROC_mrwiki.png saved!
201 - Optimal threshold for 15.0% FPR is: 0.8673532605171204
202 - confusion_matrix_mrwiki.png saved!
203 - False Positive Rate is: 0.15002842524161455
204 - CONFUSION MATRIX -
205
206Predicted not reverted reverted
207Actual
208not reverted 29902 5278
209reverted 575 1922
210
211
212============ - swwiki - ============
213 - Raw data shape: (10831, 17)
214 - Duplicate rows found and removed: 682
215 - Clean data shape: (10149, 17)
216 - Unique revision_ids: 10149 | Data Shape: 10149 | Same? : -> True
217 - Removing edits that are reverts from df | New Shape: (9971, 17)
218 - Is any revert_risk_score NA? : False
219 - Is any user_edit_count NA? : False
220 - Is any time_to_revert NA? : False
221 - ROC_swwiki.png saved!
222 - Optimal threshold for 15.0% FPR is: 0.7482092380523682
223 - confusion_matrix_swwiki.png saved!
224 - False Positive Rate is: 0.1493978517955951
225 - CONFUSION MATRIX -
226
227Predicted not reverted reverted
228Actual
229not reverted 7840 1377
230reverted 322 432
231
232
233============ - mlwiki - ============
234 - Raw data shape: (32931, 17)
235 - Duplicate rows found and removed: 5245
236 - Clean data shape: (27686, 17)
237 - Unique revision_ids: 27686 | Data Shape: 27686 | Same? : -> True
238 - Removing edits that are reverts from df | New Shape: (27032, 17)
239 - Is any revert_risk_score NA? : False
240 - Is any user_edit_count NA? : False
241 - Is any time_to_revert NA? : False
242 - ROC_mlwiki.png saved!
243 - Optimal threshold for 15.0% FPR is: 0.9362650513648987
244 - confusion_matrix_mlwiki.png saved!
245 - False Positive Rate is: 0.15019467495182287
246 - CONFUSION MATRIX -
247
248Predicted not reverted reverted
249Actual
250not reverted 21608 3819
251reverted 810 795
252
253
254============ - iswiki - ============
255 - Raw data shape: (17947, 17)
256 - Duplicate rows found and removed: 4129
257 - Clean data shape: (13818, 17)
258 - Unique revision_ids: 13818 | Data Shape: 13818 | Same? : -> True
259 - Removing edits that are reverts from df | New Shape: (13452, 17)
260 - Is any revert_risk_score NA? : False
261 - Is any user_edit_count NA? : False
262 - Is any time_to_revert NA? : False
263 - ROC_iswiki.png saved!
264 - Optimal threshold for 15.0% FPR is: 0.8892603516578674
265 - confusion_matrix_iswiki.png saved!
266 - False Positive Rate is: 0.15050732807215333
267 - CONFUSION MATRIX -
268
269Predicted not reverted reverted
270Actual
271not reverted 10549 1869
272reverted 430 604
273
274
275============ - pawiki - ============
276 - Raw data shape: (20662, 17)
277 - Duplicate rows found and removed: 2817
278 - Clean data shape: (17845, 17)
279 - Unique revision_ids: 17845 | Data Shape: 17845 | Same? : -> True
280 - Removing edits that are reverts from df | New Shape: (17030, 17)
281 - Is any revert_risk_score NA? : False
282 - Is any user_edit_count NA? : False
283 - Is any time_to_revert NA? : False
284 - ROC_pawiki.png saved!
285 - Optimal threshold for 15.0% FPR is: 0.5458896160125732
286 - confusion_matrix_pawiki.png saved!
287 - False Positive Rate is: 0.14929753356228537
288 - CONFUSION MATRIX -
289
290Predicted not reverted reverted
291Actual
292not reverted 13624 2391
293reverted 88 927
294
295
296============ - hawiki - ============
297 - Raw data shape: (142582, 17)
298 - Duplicate rows found and removed: 11926
299 - Clean data shape: (130656, 17)
300 - Unique revision_ids: 130656 | Data Shape: 130656 | Same? : -> True
301 - Removing edits that are reverts from df | New Shape: (130286, 17)
302 - Is any revert_risk_score NA? : False
303 - Is any user_edit_count NA? : False
304 - Is any time_to_revert NA? : False
305 - ROC_hawiki.png saved!
306 - Optimal threshold for 15.0% FPR is: 0.4823181927204132
307 - confusion_matrix_hawiki.png saved!
308 - False Positive Rate is: 0.15009778484218073
309 - CONFUSION MATRIX -
310
311Predicted not reverted reverted
312Actual
313not reverted 109079 19264
314reverted 1329 614
315
316
317============ - tlwiki - ============
318 - Raw data shape: (29823, 17)
319 - Duplicate rows found and removed: 2465
320 - Clean data shape: (27358, 17)
321 - Unique revision_ids: 27358 | Data Shape: 27358 | Same? : -> True
322 - Removing edits that are reverts from df | New Shape: (26356, 17)
323 - Is any revert_risk_score NA? : False
324 - Is any user_edit_count NA? : False
325 - Is any time_to_revert NA? : False
326 - ROC_tlwiki.png saved!
327 - Optimal threshold for 15.0% FPR is: 0.607416570186615
328 - confusion_matrix_tlwiki.png saved!
329 - False Positive Rate is: 0.15018641595072135
330 - CONFUSION MATRIX -
331
332Predicted not reverted reverted
333Actual
334not reverted 20970 3706
335reverted 176 1504
336
337
338============ - bnwiki - ============
339 - Raw data shape: (330764, 17)
340 - Duplicate rows found and removed: 29591
341 - Clean data shape: (301173, 17)
342 - Number of duplicated revision_ids found: 10
343 - Unique revision_ids: 301166 | Data Shape: 301166 | Same? : -> True
344 - Removing edits that are reverts from df | New Shape: (292405, 17)
345 - Is any revert_risk_score NA? : False
346 - Is any user_edit_count NA? : False
347 - Is any time_to_revert NA? : False
348 - ROC_bnwiki.png saved!
349 - Optimal threshold for 15.0% FPR is: 0.6465859413146973
350 - confusion_matrix_bnwiki.png saved!
351 - False Positive Rate is: 0.15002477700693756
352 - CONFUSION MATRIX -
353
354Predicted not reverted reverted
355Actual
356not reverted 233274 41174
357reverted 4019 13938
358
359
360============ - trwiki - ============
361 - Raw data shape: (749190, 17)
362 - Duplicate rows found and removed: 116314
363 - Clean data shape: (632876, 17)
364 - Number of duplicated revision_ids found: 4
365 - Unique revision_ids: 632874 | Data Shape: 632874 | Same? : -> True
366 - Removing edits that are reverts from df | New Shape: (581675, 17)
367 - Is any revert_risk_score NA? : False
368 - Is any user_edit_count NA? : False
369 - Is any time_to_revert NA? : False
370 - ROC_trwiki.png saved!
371 - Optimal threshold for 15.0% FPR is: 0.6082413196563721
372 - confusion_matrix_trwiki.png saved!
373 - False Positive Rate is: 0.14998876387080776
374 - CONFUSION MATRIX -
375
376Predicted not reverted reverted
377Actual
378not reverted 438769 77423
379reverted 10108 55375
380
381
382============ - azwiki - ============
383 - Raw data shape: (224309, 17)
384 - Duplicate rows found and removed: 23643
385 - Clean data shape: (200666, 17)
386 - Number of duplicated revision_ids found: 4
387 - Unique revision_ids: 200664 | Data Shape: 200664 | Same? : -> True
388 - Removing edits that are reverts from df | New Shape: (194127, 17)
389 - Is any revert_risk_score NA? : False
390 - Is any user_edit_count NA? : False
391 - Is any time_to_revert NA? : False
392 - ROC_azwiki.png saved!
393 - Optimal threshold for 15.0% FPR is: 0.5366107821464539
394 - confusion_matrix_azwiki.png saved!
395 - False Positive Rate is: 0.14990706525346845
396 - CONFUSION MATRIX -
397
398Predicted not reverted reverted
399Actual
400not reverted 153673 27099
401reverted 2012 11343
402
403
404Optimal Thresholds calculated at 16-05-2025T19:24:01
405{'cywiki': 0.11076631, 'simplewiki': 0.90655595, 'bewiki': 0.5724957, 'kkwiki': 0.6048708, 'nnwiki': 0.41624594, 'mkwiki': 0.5058307, 'lawiki': 0.6340628, 'afwiki': 0.7369383, 'tewiki': 0.36725548, 'mrwiki': 0.86735326, 'swwiki': 0.74820924, 'mlwiki': 0.93626505, 'iswiki': 0.88926035, 'pawiki': 0.5458896, 'hawiki': 0.4823182, 'tlwiki': 0.60741657, 'bnwiki': 0.64658594, 'trwiki': 0.6082413, 'azwiki': 0.5366108}
406
407Time taken: 4278.109 secs

The execution takes around 70 minutes.

Optimal Thresholds

WikiOptimalThreshold
cywiki0.11076631
simplewiki0.90655595
bewiki0.5724957
kkwiki0.6048708
nnwiki0.41624594
mkwiki0.5058307
lawiki0.6340628
afwiki0.7369383
tewiki0.36725548
mrwiki0.86735326
swwiki0.74820924
mlwiki0.93626505
iswiki0.88926035
pawiki0.545889
hawiki0.4823182
tlwiki0.60741657
bnwiki0.64658594
trwiki0.6082413
azwiki0.5366108
• isarantopoulos reassigned this task from Kgraessle to gkyziridis.May 20 2025, 9:44 AM
• isarantopoulos moved this task from Unsorted to In Progress on the Machine-Learning-Team board.
gkyziridis added a comment.EditedMay 20 2025, 9:58 AM
In T392148#10834187, @gkyziridis wrote:

RevertRisk Thresholds Analysis for all wikis

Using this notebook or this python_script I generated revert risk thresholds for all wikis in one go loading data for each wiki iteratively on memory. You can check the plots under the main section at the bottom of the notebook. The script provides the option to run the analysis on single wiki from user input either on all wikis.
The script ran at eqiad8 on jupyterlab, you can find the results in this paste

P76289 RevertRisk Threshold Analysis for all Wikis
1============ - cywiki - ============
2 - Raw data shape: (200511, 17)
3 - Duplicate rows found and removed: 1265
4 - Clean data shape: (199246, 17)
5 - Unique revision_ids: 199246 | Data Shape: 199246 | Same? : -> True
6 - Removing edits that are reverts from df | New Shape: (190747, 17)
7 - Is any revert_risk_score NA? : False
8 - Is any user_edit_count NA? : False
9 - Is any time_to_revert NA? : False
10 - ROC_cywiki.png saved!
11 - Optimal threshold for 15.0% FPR is: 0.11076630651950836
12 - confusion_matrix_cywiki.png saved!
13 - False Positive Rate is: 0.14999340149691756
14 - CONFUSION MATRIX -
15
16Predicted not reverted reverted
17Actual
18not reverted 135259 23868
19reverted 15169 16451
20
21
22============ - simplewiki - ============
23 - Raw data shape: (312209, 17)
24 - Duplicate rows found and removed: 41914
25 - Clean data shape: (270295, 17)
26 - Number of duplicated revision_ids found: 9
27 - Unique revision_ids: 270289 | Data Shape: 270289 | Same? : -> True
28 - Removing edits that are reverts from df | New Shape: (246893, 17)
29 - Is any revert_risk_score NA? : False
30 - Is any user_edit_count NA? : False
31 - Is any time_to_revert NA? : False
32 - ROC_simplewiki.png saved!
33 - Optimal threshold for 15.0% FPR is: 0.9065559506416321
34 - confusion_matrix_simplewiki.png saved!
35 - False Positive Rate is: 0.1500198513981056
36 - CONFUSION MATRIX -
37
38Predicted not reverted reverted
39Actual
40not reverted 179832 31740
41reverted 13622 21699
42
43
44============ - bewiki - ============
45 - Raw data shape: (80609, 17)
46 - Duplicate rows found and removed: 6221
47 - Clean data shape: (74388, 17)
48 - Unique revision_ids: 74388 | Data Shape: 74388 | Same? : -> True
49 - Removing edits that are reverts from df | New Shape: (73969, 17)
50 - Is any revert_risk_score NA? : False
51 - Is any user_edit_count NA? : False
52 - Is any time_to_revert NA? : False
53 - ROC_bewiki.png saved!
54 - Optimal threshold for 15.0% FPR is: 0.572495698928833
55 - confusion_matrix_bewiki.png saved!
56 - False Positive Rate is: 0.15014515086058638
57 - CONFUSION MATRIX -
58
59Predicted not reverted reverted
60Actual
61not reverted 61770 10913
62reverted 183 1103
63
64
65============ - kkwiki - ============
66 - Raw data shape: (82268, 17)
67 - Duplicate rows found and removed: 16708
68 - Clean data shape: (65560, 17)
69 - Unique revision_ids: 65560 | Data Shape: 65560 | Same? : -> True
70 - Removing edits that are reverts from df | New Shape: (64276, 17)
71 - Is any revert_risk_score NA? : False
72 - Is any user_edit_count NA? : False
73 - Is any time_to_revert NA? : False
74 - ROC_kkwiki.png saved!
75 - Optimal threshold for 15.0% FPR is: 0.6048707962036133
76 - confusion_matrix_kkwiki.png saved!
77 - False Positive Rate is: 0.14999318290272
78 - CONFUSION MATRIX -
79
80Predicted not reverted reverted
81Actual
82not reverted 49875 8801
83reverted 1475 4125
84
85
86============ - nnwiki - ============
87 - Raw data shape: (25248, 17)
88 - Duplicate rows found and removed: 4213
89 - Clean data shape: (21035, 17)
90 - Unique revision_ids: 21035 | Data Shape: 21035 | Same? : -> True
91 - Removing edits that are reverts from df | New Shape: (20392, 17)
92 - Is any revert_risk_score NA? : False
93 - Is any user_edit_count NA? : False
94 - Is any time_to_revert NA? : False
95 - ROC_nnwiki.png saved!
96 - Optimal threshold for 15.0% FPR is: 0.4162459373474121
97 - confusion_matrix_nnwiki.png saved!
98 - False Positive Rate is: 0.15001312680493567
99 - CONFUSION MATRIX -
100
101Predicted not reverted reverted
102Actual
103not reverted 16188 2857
104reverted 48 1299
105
106
107============ - mkwiki - ============
108 - Raw data shape: (54028, 17)
109 - Duplicate rows found and removed: 8215
110 - Clean data shape: (45813, 17)
111 - Unique revision_ids: 45813 | Data Shape: 45813 | Same? : -> True
112 - Removing edits that are reverts from df | New Shape: (44585, 17)
113 - Is any revert_risk_score NA? : False
114 - Is any user_edit_count NA? : False
115 - Is any time_to_revert NA? : False
116 - ROC_mkwiki.png saved!
117 - Optimal threshold for 15.0% FPR is: 0.505830705165863
118 - confusion_matrix_mkwiki.png saved!
119 - False Positive Rate is: 0.15001292809627906
120 - CONFUSION MATRIX -
121
122Predicted not reverted reverted
123Actual
124not reverted 36161 6382
125reverted 507 1535
126
127
128============ - lawiki - ============
129 - Raw data shape: (27151, 17)
130 - Duplicate rows found and removed: 3948
131 - Clean data shape: (23203, 17)
132 - Unique revision_ids: 23203 | Data Shape: 23203 | Same? : -> True
133 - Removing edits that are reverts from df | New Shape: (22893, 17)
134 - Is any revert_risk_score NA? : False
135 - Is any user_edit_count NA? : False
136 - Is any time_to_revert NA? : False
137 - ROC_lawiki.png saved!
138 - Optimal threshold for 15.0% FPR is: 0.6340628266334534
139 - confusion_matrix_lawiki.png saved!
140 - False Positive Rate is: 0.14979973297730306
141 - CONFUSION MATRIX -
142
143Predicted not reverted reverted
144Actual
145not reverted 19104 3366
146reverted 80 343
147
148
149============ - afwiki - ============
150 - Raw data shape: (21996, 17)
151 - Duplicate rows found and removed: 3614
152 - Clean data shape: (18382, 17)
153 - Unique revision_ids: 18382 | Data Shape: 18382 | Same? : -> True
154 - Removing edits that are reverts from df | New Shape: (17768, 17)
155 - Is any revert_risk_score NA? : False
156 - Is any user_edit_count NA? : False
157 - Is any time_to_revert NA? : False
158 - ROC_afwiki.png saved!
159 - Optimal threshold for 15.0% FPR is: 0.7369382977485657
160 - confusion_matrix_afwiki.png saved!
161 - False Positive Rate is: 0.1498459410174181
162 - CONFUSION MATRIX -
163
164Predicted not reverted reverted
165Actual
166not reverted 13520 2383
167reverted 214 1651
168
169============ - tewiki - ============
170 - Raw data shape: (97488, 17)
171 - Duplicate rows found and removed: 5150
172 - Clean data shape: (92338, 17)
173 - Unique revision_ids: 92338 | Data Shape: 92338 | Same? : -> True
174 - Removing edits that are reverts from df | New Shape: (91883, 17)
175 - Is any revert_risk_score NA? : False
176 - Is any user_edit_count NA? : False
177 - Is any time_to_revert NA? : False
178 - ROC_tewiki.png saved!
179 - Optimal threshold for 15.0% FPR is: 0.36725547909736633
180 - confusion_matrix_tewiki.png saved!
181 - False Positive Rate is: 0.1500330323717243
182 - CONFUSION MATRIX -
183
184Predicted not reverted reverted
185Actual
186not reverted 77194 13626
187reverted 242 821
188
189
190============ - mrwiki - ============
191 - Raw data shape: (42535, 17)
192 - Duplicate rows found and removed: 3930
193 - Clean data shape: (38605, 17)
194 - Number of duplicated revision_ids found: 2
195 - Unique revision_ids: 38604 | Data Shape: 38604 | Same? : -> True
196 - Removing edits that are reverts from df | New Shape: (37677, 17)
197 - Is any revert_risk_score NA? : False
198 - Is any user_edit_count NA? : False
199 - Is any time_to_revert NA? : False
200 - ROC_mrwiki.png saved!
201 - Optimal threshold for 15.0% FPR is: 0.8673532605171204
202 - confusion_matrix_mrwiki.png saved!
203 - False Positive Rate is: 0.15002842524161455
204 - CONFUSION MATRIX -
205
206Predicted not reverted reverted
207Actual
208not reverted 29902 5278
209reverted 575 1922
210
211
212============ - swwiki - ============
213 - Raw data shape: (10831, 17)
214 - Duplicate rows found and removed: 682
215 - Clean data shape: (10149, 17)
216 - Unique revision_ids: 10149 | Data Shape: 10149 | Same? : -> True
217 - Removing edits that are reverts from df | New Shape: (9971, 17)
218 - Is any revert_risk_score NA? : False
219 - Is any user_edit_count NA? : False
220 - Is any time_to_revert NA? : False
221 - ROC_swwiki.png saved!
222 - Optimal threshold for 15.0% FPR is: 0.7482092380523682
223 - confusion_matrix_swwiki.png saved!
224 - False Positive Rate is: 0.1493978517955951
225 - CONFUSION MATRIX -
226
227Predicted not reverted reverted
228Actual
229not reverted 7840 1377
230reverted 322 432
231
232
233============ - mlwiki - ============
234 - Raw data shape: (32931, 17)
235 - Duplicate rows found and removed: 5245
236 - Clean data shape: (27686, 17)
237 - Unique revision_ids: 27686 | Data Shape: 27686 | Same? : -> True
238 - Removing edits that are reverts from df | New Shape: (27032, 17)
239 - Is any revert_risk_score NA? : False
240 - Is any user_edit_count NA? : False
241 - Is any time_to_revert NA? : False
242 - ROC_mlwiki.png saved!
243 - Optimal threshold for 15.0% FPR is: 0.9362650513648987
244 - confusion_matrix_mlwiki.png saved!
245 - False Positive Rate is: 0.15019467495182287
246 - CONFUSION MATRIX -
247
248Predicted not reverted reverted
249Actual
250not reverted 21608 3819
251reverted 810 795
252
253
254============ - iswiki - ============
255 - Raw data shape: (17947, 17)
256 - Duplicate rows found and removed: 4129
257 - Clean data shape: (13818, 17)
258 - Unique revision_ids: 13818 | Data Shape: 13818 | Same? : -> True
259 - Removing edits that are reverts from df | New Shape: (13452, 17)
260 - Is any revert_risk_score NA? : False
261 - Is any user_edit_count NA? : False
262 - Is any time_to_revert NA? : False
263 - ROC_iswiki.png saved!
264 - Optimal threshold for 15.0% FPR is: 0.8892603516578674
265 - confusion_matrix_iswiki.png saved!
266 - False Positive Rate is: 0.15050732807215333
267 - CONFUSION MATRIX -
268
269Predicted not reverted reverted
270Actual
271not reverted 10549 1869
272reverted 430 604
273
274
275============ - pawiki - ============
276 - Raw data shape: (20662, 17)
277 - Duplicate rows found and removed: 2817
278 - Clean data shape: (17845, 17)
279 - Unique revision_ids: 17845 | Data Shape: 17845 | Same? : -> True
280 - Removing edits that are reverts from df | New Shape: (17030, 17)
281 - Is any revert_risk_score NA? : False
282 - Is any user_edit_count NA? : False
283 - Is any time_to_revert NA? : False
284 - ROC_pawiki.png saved!
285 - Optimal threshold for 15.0% FPR is: 0.5458896160125732
286 - confusion_matrix_pawiki.png saved!
287 - False Positive Rate is: 0.14929753356228537
288 - CONFUSION MATRIX -
289
290Predicted not reverted reverted
291Actual
292not reverted 13624 2391
293reverted 88 927
294
295
296============ - hawiki - ============
297 - Raw data shape: (142582, 17)
298 - Duplicate rows found and removed: 11926
299 - Clean data shape: (130656, 17)
300 - Unique revision_ids: 130656 | Data Shape: 130656 | Same? : -> True
301 - Removing edits that are reverts from df | New Shape: (130286, 17)
302 - Is any revert_risk_score NA? : False
303 - Is any user_edit_count NA? : False
304 - Is any time_to_revert NA? : False
305 - ROC_hawiki.png saved!
306 - Optimal threshold for 15.0% FPR is: 0.4823181927204132
307 - confusion_matrix_hawiki.png saved!
308 - False Positive Rate is: 0.15009778484218073
309 - CONFUSION MATRIX -
310
311Predicted not reverted reverted
312Actual
313not reverted 109079 19264
314reverted 1329 614
315
316
317============ - tlwiki - ============
318 - Raw data shape: (29823, 17)
319 - Duplicate rows found and removed: 2465
320 - Clean data shape: (27358, 17)
321 - Unique revision_ids: 27358 | Data Shape: 27358 | Same? : -> True
322 - Removing edits that are reverts from df | New Shape: (26356, 17)
323 - Is any revert_risk_score NA? : False
324 - Is any user_edit_count NA? : False
325 - Is any time_to_revert NA? : False
326 - ROC_tlwiki.png saved!
327 - Optimal threshold for 15.0% FPR is: 0.607416570186615
328 - confusion_matrix_tlwiki.png saved!
329 - False Positive Rate is: 0.15018641595072135
330 - CONFUSION MATRIX -
331
332Predicted not reverted reverted
333Actual
334not reverted 20970 3706
335reverted 176 1504
336
337
338============ - bnwiki - ============
339 - Raw data shape: (330764, 17)
340 - Duplicate rows found and removed: 29591
341 - Clean data shape: (301173, 17)
342 - Number of duplicated revision_ids found: 10
343 - Unique revision_ids: 301166 | Data Shape: 301166 | Same? : -> True
344 - Removing edits that are reverts from df | New Shape: (292405, 17)
345 - Is any revert_risk_score NA? : False
346 - Is any user_edit_count NA? : False
347 - Is any time_to_revert NA? : False
348 - ROC_bnwiki.png saved!
349 - Optimal threshold for 15.0% FPR is: 0.6465859413146973
350 - confusion_matrix_bnwiki.png saved!
351 - False Positive Rate is: 0.15002477700693756
352 - CONFUSION MATRIX -
353
354Predicted not reverted reverted
355Actual
356not reverted 233274 41174
357reverted 4019 13938
358
359
360============ - trwiki - ============
361 - Raw data shape: (749190, 17)
362 - Duplicate rows found and removed: 116314
363 - Clean data shape: (632876, 17)
364 - Number of duplicated revision_ids found: 4
365 - Unique revision_ids: 632874 | Data Shape: 632874 | Same? : -> True
366 - Removing edits that are reverts from df | New Shape: (581675, 17)
367 - Is any revert_risk_score NA? : False
368 - Is any user_edit_count NA? : False
369 - Is any time_to_revert NA? : False
370 - ROC_trwiki.png saved!
371 - Optimal threshold for 15.0% FPR is: 0.6082413196563721
372 - confusion_matrix_trwiki.png saved!
373 - False Positive Rate is: 0.14998876387080776
374 - CONFUSION MATRIX -
375
376Predicted not reverted reverted
377Actual
378not reverted 438769 77423
379reverted 10108 55375
380
381
382============ - azwiki - ============
383 - Raw data shape: (224309, 17)
384 - Duplicate rows found and removed: 23643
385 - Clean data shape: (200666, 17)
386 - Number of duplicated revision_ids found: 4
387 - Unique revision_ids: 200664 | Data Shape: 200664 | Same? : -> True
388 - Removing edits that are reverts from df | New Shape: (194127, 17)
389 - Is any revert_risk_score NA? : False
390 - Is any user_edit_count NA? : False
391 - Is any time_to_revert NA? : False
392 - ROC_azwiki.png saved!
393 - Optimal threshold for 15.0% FPR is: 0.5366107821464539
394 - confusion_matrix_azwiki.png saved!
395 - False Positive Rate is: 0.14990706525346845
396 - CONFUSION MATRIX -
397
398Predicted not reverted reverted
399Actual
400not reverted 153673 27099
401reverted 2012 11343
402
403
404Optimal Thresholds calculated at 16-05-2025T19:24:01
405{'cywiki': 0.11076631, 'simplewiki': 0.90655595, 'bewiki': 0.5724957, 'kkwiki': 0.6048708, 'nnwiki': 0.41624594, 'mkwiki': 0.5058307, 'lawiki': 0.6340628, 'afwiki': 0.7369383, 'tewiki': 0.36725548, 'mrwiki': 0.86735326, 'swwiki': 0.74820924, 'mlwiki': 0.93626505, 'iswiki': 0.88926035, 'pawiki': 0.5458896, 'hawiki': 0.4823182, 'tlwiki': 0.60741657, 'bnwiki': 0.64658594, 'trwiki': 0.6082413, 'azwiki': 0.5366108}
406
407Time taken: 4278.109 secs

The execution takes around 70 minutes.

Optimal Thresholds

WikiOptimalThreshold
cywiki0.11076631
simplewiki0.90655595
bewiki0.5724957
kkwiki0.6048708
nnwiki0.41624594
mkwiki0.5058307
lawiki0.6340628
afwiki0.7369383
tewiki0.36725548
mrwiki0.86735326
swwiki0.74820924
mlwiki0.93626505
iswiki0.88926035
pawiki0.545889
hawiki0.4823182
tlwiki0.60741657
bnwiki0.64658594
trwiki0.6082413
azwiki0.5366108

@Kgraessle would it be possible to review/check the above results? Do they seem correct based on your experience ? Do they meet the the expected outcome for each wiki based on our intuition?
For more "in depth" evaluation you can have a look at the plots for RocCurve and ConfusionMatrix at the bottom of this notebook .

Kgraessle added a comment.May 21 2025, 4:25 PM
In T392148#10838253, @gkyziridis wrote:
In T392148#10834187, @gkyziridis wrote:

RevertRisk Thresholds Analysis for all wikis

Using this notebook or this python_script I generated revert risk thresholds for all wikis in one go loading data for each wiki iteratively on memory. You can check the plots under the main section at the bottom of the notebook. The script provides the option to run the analysis on single wiki from user input either on all wikis.
The script ran at eqiad8 on jupyterlab, you can find the results in this paste

P76289 RevertRisk Threshold Analysis for all Wikis
1============ - cywiki - ============
2 - Raw data shape: (200511, 17)
3 - Duplicate rows found and removed: 1265
4 - Clean data shape: (199246, 17)
5 - Unique revision_ids: 199246 | Data Shape: 199246 | Same? : -> True
6 - Removing edits that are reverts from df | New Shape: (190747, 17)
7 - Is any revert_risk_score NA? : False
8 - Is any user_edit_count NA? : False
9 - Is any time_to_revert NA? : False
10 - ROC_cywiki.png saved!
11 - Optimal threshold for 15.0% FPR is: 0.11076630651950836
12 - confusion_matrix_cywiki.png saved!
13 - False Positive Rate is: 0.14999340149691756
14 - CONFUSION MATRIX -
15
16Predicted not reverted reverted
17Actual
18not reverted 135259 23868
19reverted 15169 16451
20
21
22============ - simplewiki - ============
23 - Raw data shape: (312209, 17)
24 - Duplicate rows found and removed: 41914
25 - Clean data shape: (270295, 17)
26 - Number of duplicated revision_ids found: 9
27 - Unique revision_ids: 270289 | Data Shape: 270289 | Same? : -> True
28 - Removing edits that are reverts from df | New Shape: (246893, 17)
29 - Is any revert_risk_score NA? : False
30 - Is any user_edit_count NA? : False
31 - Is any time_to_revert NA? : False
32 - ROC_simplewiki.png saved!
33 - Optimal threshold for 15.0% FPR is: 0.9065559506416321
34 - confusion_matrix_simplewiki.png saved!
35 - False Positive Rate is: 0.1500198513981056
36 - CONFUSION MATRIX -
37
38Predicted not reverted reverted
39Actual
40not reverted 179832 31740
41reverted 13622 21699
42
43
44============ - bewiki - ============
45 - Raw data shape: (80609, 17)
46 - Duplicate rows found and removed: 6221
47 - Clean data shape: (74388, 17)
48 - Unique revision_ids: 74388 | Data Shape: 74388 | Same? : -> True
49 - Removing edits that are reverts from df | New Shape: (73969, 17)
50 - Is any revert_risk_score NA? : False
51 - Is any user_edit_count NA? : False
52 - Is any time_to_revert NA? : False
53 - ROC_bewiki.png saved!
54 - Optimal threshold for 15.0% FPR is: 0.572495698928833
55 - confusion_matrix_bewiki.png saved!
56 - False Positive Rate is: 0.15014515086058638
57 - CONFUSION MATRIX -
58
59Predicted not reverted reverted
60Actual
61not reverted 61770 10913
62reverted 183 1103
63
64
65============ - kkwiki - ============
66 - Raw data shape: (82268, 17)
67 - Duplicate rows found and removed: 16708
68 - Clean data shape: (65560, 17)
69 - Unique revision_ids: 65560 | Data Shape: 65560 | Same? : -> True
70 - Removing edits that are reverts from df | New Shape: (64276, 17)
71 - Is any revert_risk_score NA? : False
72 - Is any user_edit_count NA? : False
73 - Is any time_to_revert NA? : False
74 - ROC_kkwiki.png saved!
75 - Optimal threshold for 15.0% FPR is: 0.6048707962036133
76 - confusion_matrix_kkwiki.png saved!
77 - False Positive Rate is: 0.14999318290272
78 - CONFUSION MATRIX -
79
80Predicted not reverted reverted
81Actual
82not reverted 49875 8801
83reverted 1475 4125
84
85
86============ - nnwiki - ============
87 - Raw data shape: (25248, 17)
88 - Duplicate rows found and removed: 4213
89 - Clean data shape: (21035, 17)
90 - Unique revision_ids: 21035 | Data Shape: 21035 | Same? : -> True
91 - Removing edits that are reverts from df | New Shape: (20392, 17)
92 - Is any revert_risk_score NA? : False
93 - Is any user_edit_count NA? : False
94 - Is any time_to_revert NA? : False
95 - ROC_nnwiki.png saved!
96 - Optimal threshold for 15.0% FPR is: 0.4162459373474121
97 - confusion_matrix_nnwiki.png saved!
98 - False Positive Rate is: 0.15001312680493567
99 - CONFUSION MATRIX -
100
101Predicted not reverted reverted
102Actual
103not reverted 16188 2857
104reverted 48 1299
105
106
107============ - mkwiki - ============
108 - Raw data shape: (54028, 17)
109 - Duplicate rows found and removed: 8215
110 - Clean data shape: (45813, 17)
111 - Unique revision_ids: 45813 | Data Shape: 45813 | Same? : -> True
112 - Removing edits that are reverts from df | New Shape: (44585, 17)
113 - Is any revert_risk_score NA? : False
114 - Is any user_edit_count NA? : False
115 - Is any time_to_revert NA? : False
116 - ROC_mkwiki.png saved!
117 - Optimal threshold for 15.0% FPR is: 0.505830705165863
118 - confusion_matrix_mkwiki.png saved!
119 - False Positive Rate is: 0.15001292809627906
120 - CONFUSION MATRIX -
121
122Predicted not reverted reverted
123Actual
124not reverted 36161 6382
125reverted 507 1535
126
127
128============ - lawiki - ============
129 - Raw data shape: (27151, 17)
130 - Duplicate rows found and removed: 3948
131 - Clean data shape: (23203, 17)
132 - Unique revision_ids: 23203 | Data Shape: 23203 | Same? : -> True
133 - Removing edits that are reverts from df | New Shape: (22893, 17)
134 - Is any revert_risk_score NA? : False
135 - Is any user_edit_count NA? : False
136 - Is any time_to_revert NA? : False
137 - ROC_lawiki.png saved!
138 - Optimal threshold for 15.0% FPR is: 0.6340628266334534
139 - confusion_matrix_lawiki.png saved!
140 - False Positive Rate is: 0.14979973297730306
141 - CONFUSION MATRIX -
142
143Predicted not reverted reverted
144Actual
145not reverted 19104 3366
146reverted 80 343
147
148
149============ - afwiki - ============
150 - Raw data shape: (21996, 17)
151 - Duplicate rows found and removed: 3614
152 - Clean data shape: (18382, 17)
153 - Unique revision_ids: 18382 | Data Shape: 18382 | Same? : -> True
154 - Removing edits that are reverts from df | New Shape: (17768, 17)
155 - Is any revert_risk_score NA? : False
156 - Is any user_edit_count NA? : False
157 - Is any time_to_revert NA? : False
158 - ROC_afwiki.png saved!
159 - Optimal threshold for 15.0% FPR is: 0.7369382977485657
160 - confusion_matrix_afwiki.png saved!
161 - False Positive Rate is: 0.1498459410174181
162 - CONFUSION MATRIX -
163
164Predicted not reverted reverted
165Actual
166not reverted 13520 2383
167reverted 214 1651
168
169============ - tewiki - ============
170 - Raw data shape: (97488, 17)
171 - Duplicate rows found and removed: 5150
172 - Clean data shape: (92338, 17)
173 - Unique revision_ids: 92338 | Data Shape: 92338 | Same? : -> True
174 - Removing edits that are reverts from df | New Shape: (91883, 17)
175 - Is any revert_risk_score NA? : False
176 - Is any user_edit_count NA? : False
177 - Is any time_to_revert NA? : False
178 - ROC_tewiki.png saved!
179 - Optimal threshold for 15.0% FPR is: 0.36725547909736633
180 - confusion_matrix_tewiki.png saved!
181 - False Positive Rate is: 0.1500330323717243
182 - CONFUSION MATRIX -
183
184Predicted not reverted reverted
185Actual
186not reverted 77194 13626
187reverted 242 821
188
189
190============ - mrwiki - ============
191 - Raw data shape: (42535, 17)
192 - Duplicate rows found and removed: 3930
193 - Clean data shape: (38605, 17)
194 - Number of duplicated revision_ids found: 2
195 - Unique revision_ids: 38604 | Data Shape: 38604 | Same? : -> True
196 - Removing edits that are reverts from df | New Shape: (37677, 17)
197 - Is any revert_risk_score NA? : False
198 - Is any user_edit_count NA? : False
199 - Is any time_to_revert NA? : False
200 - ROC_mrwiki.png saved!
201 - Optimal threshold for 15.0% FPR is: 0.8673532605171204
202 - confusion_matrix_mrwiki.png saved!
203 - False Positive Rate is: 0.15002842524161455
204 - CONFUSION MATRIX -
205
206Predicted not reverted reverted
207Actual
208not reverted 29902 5278
209reverted 575 1922
210
211
212============ - swwiki - ============
213 - Raw data shape: (10831, 17)
214 - Duplicate rows found and removed: 682
215 - Clean data shape: (10149, 17)
216 - Unique revision_ids: 10149 | Data Shape: 10149 | Same? : -> True
217 - Removing edits that are reverts from df | New Shape: (9971, 17)
218 - Is any revert_risk_score NA? : False
219 - Is any user_edit_count NA? : False
220 - Is any time_to_revert NA? : False
221 - ROC_swwiki.png saved!
222 - Optimal threshold for 15.0% FPR is: 0.7482092380523682
223 - confusion_matrix_swwiki.png saved!
224 - False Positive Rate is: 0.1493978517955951
225 - CONFUSION MATRIX -
226
227Predicted not reverted reverted
228Actual
229not reverted 7840 1377
230reverted 322 432
231
232
233============ - mlwiki - ============
234 - Raw data shape: (32931, 17)
235 - Duplicate rows found and removed: 5245
236 - Clean data shape: (27686, 17)
237 - Unique revision_ids: 27686 | Data Shape: 27686 | Same? : -> True
238 - Removing edits that are reverts from df | New Shape: (27032, 17)
239 - Is any revert_risk_score NA? : False
240 - Is any user_edit_count NA? : False
241 - Is any time_to_revert NA? : False
242 - ROC_mlwiki.png saved!
243 - Optimal threshold for 15.0% FPR is: 0.9362650513648987
244 - confusion_matrix_mlwiki.png saved!
245 - False Positive Rate is: 0.15019467495182287
246 - CONFUSION MATRIX -
247
248Predicted not reverted reverted
249Actual
250not reverted 21608 3819
251reverted 810 795
252
253
254============ - iswiki - ============
255 - Raw data shape: (17947, 17)
256 - Duplicate rows found and removed: 4129
257 - Clean data shape: (13818, 17)
258 - Unique revision_ids: 13818 | Data Shape: 13818 | Same? : -> True
259 - Removing edits that are reverts from df | New Shape: (13452, 17)
260 - Is any revert_risk_score NA? : False
261 - Is any user_edit_count NA? : False
262 - Is any time_to_revert NA? : False
263 - ROC_iswiki.png saved!
264 - Optimal threshold for 15.0% FPR is: 0.8892603516578674
265 - confusion_matrix_iswiki.png saved!
266 - False Positive Rate is: 0.15050732807215333
267 - CONFUSION MATRIX -
268
269Predicted not reverted reverted
270Actual
271not reverted 10549 1869
272reverted 430 604
273
274
275============ - pawiki - ============
276 - Raw data shape: (20662, 17)
277 - Duplicate rows found and removed: 2817
278 - Clean data shape: (17845, 17)
279 - Unique revision_ids: 17845 | Data Shape: 17845 | Same? : -> True
280 - Removing edits that are reverts from df | New Shape: (17030, 17)
281 - Is any revert_risk_score NA? : False
282 - Is any user_edit_count NA? : False
283 - Is any time_to_revert NA? : False
284 - ROC_pawiki.png saved!
285 - Optimal threshold for 15.0% FPR is: 0.5458896160125732
286 - confusion_matrix_pawiki.png saved!
287 - False Positive Rate is: 0.14929753356228537
288 - CONFUSION MATRIX -
289
290Predicted not reverted reverted
291Actual
292not reverted 13624 2391
293reverted 88 927
294
295
296============ - hawiki - ============
297 - Raw data shape: (142582, 17)
298 - Duplicate rows found and removed: 11926
299 - Clean data shape: (130656, 17)
300 - Unique revision_ids: 130656 | Data Shape: 130656 | Same? : -> True
301 - Removing edits that are reverts from df | New Shape: (130286, 17)
302 - Is any revert_risk_score NA? : False
303 - Is any user_edit_count NA? : False
304 - Is any time_to_revert NA? : False
305 - ROC_hawiki.png saved!
306 - Optimal threshold for 15.0% FPR is: 0.4823181927204132
307 - confusion_matrix_hawiki.png saved!
308 - False Positive Rate is: 0.15009778484218073
309 - CONFUSION MATRIX -
310
311Predicted not reverted reverted
312Actual
313not reverted 109079 19264
314reverted 1329 614
315
316
317============ - tlwiki - ============
318 - Raw data shape: (29823, 17)
319 - Duplicate rows found and removed: 2465
320 - Clean data shape: (27358, 17)
321 - Unique revision_ids: 27358 | Data Shape: 27358 | Same? : -> True
322 - Removing edits that are reverts from df | New Shape: (26356, 17)
323 - Is any revert_risk_score NA? : False
324 - Is any user_edit_count NA? : False
325 - Is any time_to_revert NA? : False
326 - ROC_tlwiki.png saved!
327 - Optimal threshold for 15.0% FPR is: 0.607416570186615
328 - confusion_matrix_tlwiki.png saved!
329 - False Positive Rate is: 0.15018641595072135
330 - CONFUSION MATRIX -
331
332Predicted not reverted reverted
333Actual
334not reverted 20970 3706
335reverted 176 1504
336
337
338============ - bnwiki - ============
339 - Raw data shape: (330764, 17)
340 - Duplicate rows found and removed: 29591
341 - Clean data shape: (301173, 17)
342 - Number of duplicated revision_ids found: 10
343 - Unique revision_ids: 301166 | Data Shape: 301166 | Same? : -> True
344 - Removing edits that are reverts from df | New Shape: (292405, 17)
345 - Is any revert_risk_score NA? : False
346 - Is any user_edit_count NA? : False
347 - Is any time_to_revert NA? : False
348 - ROC_bnwiki.png saved!
349 - Optimal threshold for 15.0% FPR is: 0.6465859413146973
350 - confusion_matrix_bnwiki.png saved!
351 - False Positive Rate is: 0.15002477700693756
352 - CONFUSION MATRIX -
353
354Predicted not reverted reverted
355Actual
356not reverted 233274 41174
357reverted 4019 13938
358
359
360============ - trwiki - ============
361 - Raw data shape: (749190, 17)
362 - Duplicate rows found and removed: 116314
363 - Clean data shape: (632876, 17)
364 - Number of duplicated revision_ids found: 4
365 - Unique revision_ids: 632874 | Data Shape: 632874 | Same? : -> True
366 - Removing edits that are reverts from df | New Shape: (581675, 17)
367 - Is any revert_risk_score NA? : False
368 - Is any user_edit_count NA? : False
369 - Is any time_to_revert NA? : False
370 - ROC_trwiki.png saved!
371 - Optimal threshold for 15.0% FPR is: 0.6082413196563721
372 - confusion_matrix_trwiki.png saved!
373 - False Positive Rate is: 0.14998876387080776
374 - CONFUSION MATRIX -
375
376Predicted not reverted reverted
377Actual
378not reverted 438769 77423
379reverted 10108 55375
380
381
382============ - azwiki - ============
383 - Raw data shape: (224309, 17)
384 - Duplicate rows found and removed: 23643
385 - Clean data shape: (200666, 17)
386 - Number of duplicated revision_ids found: 4
387 - Unique revision_ids: 200664 | Data Shape: 200664 | Same? : -> True
388 - Removing edits that are reverts from df | New Shape: (194127, 17)
389 - Is any revert_risk_score NA? : False
390 - Is any user_edit_count NA? : False
391 - Is any time_to_revert NA? : False
392 - ROC_azwiki.png saved!
393 - Optimal threshold for 15.0% FPR is: 0.5366107821464539
394 - confusion_matrix_azwiki.png saved!
395 - False Positive Rate is: 0.14990706525346845
396 - CONFUSION MATRIX -
397
398Predicted not reverted reverted
399Actual
400not reverted 153673 27099
401reverted 2012 11343
402
403
404Optimal Thresholds calculated at 16-05-2025T19:24:01
405{'cywiki': 0.11076631, 'simplewiki': 0.90655595, 'bewiki': 0.5724957, 'kkwiki': 0.6048708, 'nnwiki': 0.41624594, 'mkwiki': 0.5058307, 'lawiki': 0.6340628, 'afwiki': 0.7369383, 'tewiki': 0.36725548, 'mrwiki': 0.86735326, 'swwiki': 0.74820924, 'mlwiki': 0.93626505, 'iswiki': 0.88926035, 'pawiki': 0.5458896, 'hawiki': 0.4823182, 'tlwiki': 0.60741657, 'bnwiki': 0.64658594, 'trwiki': 0.6082413, 'azwiki': 0.5366108}
406
407Time taken: 4278.109 secs

The execution takes around 70 minutes.

Optimal Thresholds

WikiOptimalThreshold
cywiki0.11076631
simplewiki0.90655595
bewiki0.5724957
kkwiki0.6048708
nnwiki0.41624594
mkwiki0.5058307
lawiki0.6340628
afwiki0.7369383
tewiki0.36725548
mrwiki0.86735326
swwiki0.74820924
mlwiki0.93626505
iswiki0.88926035
pawiki0.545889
hawiki0.4823182
tlwiki0.60741657
bnwiki0.64658594
trwiki0.6082413
azwiki0.5366108

@Kgraessle would it be possible to review/check the above results? Do they seem correct based on your experience ? Do they meet the the expected outcome for each wiki based on our intuition?
For more "in depth" evaluation you can have a look at the plots for RocCurve and ConfusionMatrix at the bottom of this notebook .

Yeah I can take a look, is it ok if I prioritize this for next week?

gkyziridis added a comment.May 22 2025, 10:00 AM
In T392148#10845099, @Kgraessle wrote:
In T392148#10838253, @gkyziridis wrote:
In T392148#10834187, @gkyziridis wrote:

RevertRisk Thresholds Analysis for all wikis

Using this notebook or this python_script I generated revert risk thresholds for all wikis in one go loading data for each wiki iteratively on memory. You can check the plots under the main section at the bottom of the notebook. The script provides the option to run the analysis on single wiki from user input either on all wikis.
The script ran at eqiad8 on jupyterlab, you can find the results in this paste

P76289 RevertRisk Threshold Analysis for all Wikis
1============ - cywiki - ============
2 - Raw data shape: (200511, 17)
3 - Duplicate rows found and removed: 1265
4 - Clean data shape: (199246, 17)
5 - Unique revision_ids: 199246 | Data Shape: 199246 | Same? : -> True
6 - Removing edits that are reverts from df | New Shape: (190747, 17)
7 - Is any revert_risk_score NA? : False
8 - Is any user_edit_count NA? : False
9 - Is any time_to_revert NA? : False
10 - ROC_cywiki.png saved!
11 - Optimal threshold for 15.0% FPR is: 0.11076630651950836
12 - confusion_matrix_cywiki.png saved!
13 - False Positive Rate is: 0.14999340149691756
14 - CONFUSION MATRIX -
15
16Predicted not reverted reverted
17Actual
18not reverted 135259 23868
19reverted 15169 16451
20
21
22============ - simplewiki - ============
23 - Raw data shape: (312209, 17)
24 - Duplicate rows found and removed: 41914
25 - Clean data shape: (270295, 17)
26 - Number of duplicated revision_ids found: 9
27 - Unique revision_ids: 270289 | Data Shape: 270289 | Same? : -> True
28 - Removing edits that are reverts from df | New Shape: (246893, 17)
29 - Is any revert_risk_score NA? : False
30 - Is any user_edit_count NA? : False
31 - Is any time_to_revert NA? : False
32 - ROC_simplewiki.png saved!
33 - Optimal threshold for 15.0% FPR is: 0.9065559506416321
34 - confusion_matrix_simplewiki.png saved!
35 - False Positive Rate is: 0.1500198513981056
36 - CONFUSION MATRIX -
37
38Predicted not reverted reverted
39Actual
40not reverted 179832 31740
41reverted 13622 21699
42
43
44============ - bewiki - ============
45 - Raw data shape: (80609, 17)
46 - Duplicate rows found and removed: 6221
47 - Clean data shape: (74388, 17)
48 - Unique revision_ids: 74388 | Data Shape: 74388 | Same? : -> True
49 - Removing edits that are reverts from df | New Shape: (73969, 17)
50 - Is any revert_risk_score NA? : False
51 - Is any user_edit_count NA? : False
52 - Is any time_to_revert NA? : False
53 - ROC_bewiki.png saved!
54 - Optimal threshold for 15.0% FPR is: 0.572495698928833
55 - confusion_matrix_bewiki.png saved!
56 - False Positive Rate is: 0.15014515086058638
57 - CONFUSION MATRIX -
58
59Predicted not reverted reverted
60Actual
61not reverted 61770 10913
62reverted 183 1103
63
64
65============ - kkwiki - ============
66 - Raw data shape: (82268, 17)
67 - Duplicate rows found and removed: 16708
68 - Clean data shape: (65560, 17)
69 - Unique revision_ids: 65560 | Data Shape: 65560 | Same? : -> True
70 - Removing edits that are reverts from df | New Shape: (64276, 17)
71 - Is any revert_risk_score NA? : False
72 - Is any user_edit_count NA? : False
73 - Is any time_to_revert NA? : False
74 - ROC_kkwiki.png saved!
75 - Optimal threshold for 15.0% FPR is: 0.6048707962036133
76 - confusion_matrix_kkwiki.png saved!
77 - False Positive Rate is: 0.14999318290272
78 - CONFUSION MATRIX -
79
80Predicted not reverted reverted
81Actual
82not reverted 49875 8801
83reverted 1475 4125
84
85
86============ - nnwiki - ============
87 - Raw data shape: (25248, 17)
88 - Duplicate rows found and removed: 4213
89 - Clean data shape: (21035, 17)
90 - Unique revision_ids: 21035 | Data Shape: 21035 | Same? : -> True
91 - Removing edits that are reverts from df | New Shape: (20392, 17)
92 - Is any revert_risk_score NA? : False
93 - Is any user_edit_count NA? : False
94 - Is any time_to_revert NA? : False
95 - ROC_nnwiki.png saved!
96 - Optimal threshold for 15.0% FPR is: 0.4162459373474121
97 - confusion_matrix_nnwiki.png saved!
98 - False Positive Rate is: 0.15001312680493567
99 - CONFUSION MATRIX -
100
101Predicted not reverted reverted
102Actual
103not reverted 16188 2857
104reverted 48 1299
105
106
107============ - mkwiki - ============
108 - Raw data shape: (54028, 17)
109 - Duplicate rows found and removed: 8215
110 - Clean data shape: (45813, 17)
111 - Unique revision_ids: 45813 | Data Shape: 45813 | Same? : -> True
112 - Removing edits that are reverts from df | New Shape: (44585, 17)
113 - Is any revert_risk_score NA? : False
114 - Is any user_edit_count NA? : False
115 - Is any time_to_revert NA? : False
116 - ROC_mkwiki.png saved!
117 - Optimal threshold for 15.0% FPR is: 0.505830705165863
118 - confusion_matrix_mkwiki.png saved!
119 - False Positive Rate is: 0.15001292809627906
120 - CONFUSION MATRIX -
121
122Predicted not reverted reverted
123Actual
124not reverted 36161 6382
125reverted 507 1535
126
127
128============ - lawiki - ============
129 - Raw data shape: (27151, 17)
130 - Duplicate rows found and removed: 3948
131 - Clean data shape: (23203, 17)
132 - Unique revision_ids: 23203 | Data Shape: 23203 | Same? : -> True
133 - Removing edits that are reverts from df | New Shape: (22893, 17)
134 - Is any revert_risk_score NA? : False
135 - Is any user_edit_count NA? : False
136 - Is any time_to_revert NA? : False
137 - ROC_lawiki.png saved!
138 - Optimal threshold for 15.0% FPR is: 0.6340628266334534
139 - confusion_matrix_lawiki.png saved!
140 - False Positive Rate is: 0.14979973297730306
141 - CONFUSION MATRIX -
142
143Predicted not reverted reverted
144Actual
145not reverted 19104 3366
146reverted 80 343
147
148
149============ - afwiki - ============
150 - Raw data shape: (21996, 17)
151 - Duplicate rows found and removed: 3614
152 - Clean data shape: (18382, 17)
153 - Unique revision_ids: 18382 | Data Shape: 18382 | Same? : -> True
154 - Removing edits that are reverts from df | New Shape: (17768, 17)
155 - Is any revert_risk_score NA? : False
156 - Is any user_edit_count NA? : False
157 - Is any time_to_revert NA? : False
158 - ROC_afwiki.png saved!
159 - Optimal threshold for 15.0% FPR is: 0.7369382977485657
160 - confusion_matrix_afwiki.png saved!
161 - False Positive Rate is: 0.1498459410174181
162 - CONFUSION MATRIX -
163
164Predicted not reverted reverted
165Actual
166not reverted 13520 2383
167reverted 214 1651
168
169============ - tewiki - ============
170 - Raw data shape: (97488, 17)
171 - Duplicate rows found and removed: 5150
172 - Clean data shape: (92338, 17)
173 - Unique revision_ids: 92338 | Data Shape: 92338 | Same? : -> True
174 - Removing edits that are reverts from df | New Shape: (91883, 17)
175 - Is any revert_risk_score NA? : False
176 - Is any user_edit_count NA? : False
177 - Is any time_to_revert NA? : False
178 - ROC_tewiki.png saved!
179 - Optimal threshold for 15.0% FPR is: 0.36725547909736633
180 - confusion_matrix_tewiki.png saved!
181 - False Positive Rate is: 0.1500330323717243
182 - CONFUSION MATRIX -
183
184Predicted not reverted reverted
185Actual
186not reverted 77194 13626
187reverted 242 821
188
189
190============ - mrwiki - ============
191 - Raw data shape: (42535, 17)
192 - Duplicate rows found and removed: 3930
193 - Clean data shape: (38605, 17)
194 - Number of duplicated revision_ids found: 2
195 - Unique revision_ids: 38604 | Data Shape: 38604 | Same? : -> True
196 - Removing edits that are reverts from df | New Shape: (37677, 17)
197 - Is any revert_risk_score NA? : False
198 - Is any user_edit_count NA? : False
199 - Is any time_to_revert NA? : False
200 - ROC_mrwiki.png saved!
201 - Optimal threshold for 15.0% FPR is: 0.8673532605171204
202 - confusion_matrix_mrwiki.png saved!
203 - False Positive Rate is: 0.15002842524161455
204 - CONFUSION MATRIX -
205
206Predicted not reverted reverted
207Actual
208not reverted 29902 5278
209reverted 575 1922
210
211
212============ - swwiki - ============
213 - Raw data shape: (10831, 17)
214 - Duplicate rows found and removed: 682
215 - Clean data shape: (10149, 17)
216 - Unique revision_ids: 10149 | Data Shape: 10149 | Same? : -> True
217 - Removing edits that are reverts from df | New Shape: (9971, 17)
218 - Is any revert_risk_score NA? : False
219 - Is any user_edit_count NA? : False
220 - Is any time_to_revert NA? : False
221 - ROC_swwiki.png saved!
222 - Optimal threshold for 15.0% FPR is: 0.7482092380523682
223 - confusion_matrix_swwiki.png saved!
224 - False Positive Rate is: 0.1493978517955951
225 - CONFUSION MATRIX -
226
227Predicted not reverted reverted
228Actual
229not reverted 7840 1377
230reverted 322 432
231
232
233============ - mlwiki - ============
234 - Raw data shape: (32931, 17)
235 - Duplicate rows found and removed: 5245
236 - Clean data shape: (27686, 17)
237 - Unique revision_ids: 27686 | Data Shape: 27686 | Same? : -> True
238 - Removing edits that are reverts from df | New Shape: (27032, 17)
239 - Is any revert_risk_score NA? : False
240 - Is any user_edit_count NA? : False
241 - Is any time_to_revert NA? : False
242 - ROC_mlwiki.png saved!
243 - Optimal threshold for 15.0% FPR is: 0.9362650513648987
244 - confusion_matrix_mlwiki.png saved!
245 - False Positive Rate is: 0.15019467495182287
246 - CONFUSION MATRIX -
247
248Predicted not reverted reverted
249Actual
250not reverted 21608 3819
251reverted 810 795
252
253
254============ - iswiki - ============
255 - Raw data shape: (17947, 17)
256 - Duplicate rows found and removed: 4129
257 - Clean data shape: (13818, 17)
258 - Unique revision_ids: 13818 | Data Shape: 13818 | Same? : -> True
259 - Removing edits that are reverts from df | New Shape: (13452, 17)
260 - Is any revert_risk_score NA? : False
261 - Is any user_edit_count NA? : False
262 - Is any time_to_revert NA? : False
263 - ROC_iswiki.png saved!
264 - Optimal threshold for 15.0% FPR is: 0.8892603516578674
265 - confusion_matrix_iswiki.png saved!
266 - False Positive Rate is: 0.15050732807215333
267 - CONFUSION MATRIX -
268
269Predicted not reverted reverted
270Actual
271not reverted 10549 1869
272reverted 430 604
273
274
275============ - pawiki - ============
276 - Raw data shape: (20662, 17)
277 - Duplicate rows found and removed: 2817
278 - Clean data shape: (17845, 17)
279 - Unique revision_ids: 17845 | Data Shape: 17845 | Same? : -> True
280 - Removing edits that are reverts from df | New Shape: (17030, 17)
281 - Is any revert_risk_score NA? : False
282 - Is any user_edit_count NA? : False
283 - Is any time_to_revert NA? : False
284 - ROC_pawiki.png saved!
285 - Optimal threshold for 15.0% FPR is: 0.5458896160125732
286 - confusion_matrix_pawiki.png saved!
287 - False Positive Rate is: 0.14929753356228537
288 - CONFUSION MATRIX -
289
290Predicted not reverted reverted
291Actual
292not reverted 13624 2391
293reverted 88 927
294
295
296============ - hawiki - ============
297 - Raw data shape: (142582, 17)
298 - Duplicate rows found and removed: 11926
299 - Clean data shape: (130656, 17)
300 - Unique revision_ids: 130656 | Data Shape: 130656 | Same? : -> True
301 - Removing edits that are reverts from df | New Shape: (130286, 17)
302 - Is any revert_risk_score NA? : False
303 - Is any user_edit_count NA? : False
304 - Is any time_to_revert NA? : False
305 - ROC_hawiki.png saved!
306 - Optimal threshold for 15.0% FPR is: 0.4823181927204132
307 - confusion_matrix_hawiki.png saved!
308 - False Positive Rate is: 0.15009778484218073
309 - CONFUSION MATRIX -
310
311Predicted not reverted reverted
312Actual
313not reverted 109079 19264
314reverted 1329 614
315
316
317============ - tlwiki - ============
318 - Raw data shape: (29823, 17)
319 - Duplicate rows found and removed: 2465
320 - Clean data shape: (27358, 17)
321 - Unique revision_ids: 27358 | Data Shape: 27358 | Same? : -> True
322 - Removing edits that are reverts from df | New Shape: (26356, 17)
323 - Is any revert_risk_score NA? : False
324 - Is any user_edit_count NA? : False
325 - Is any time_to_revert NA? : False
326 - ROC_tlwiki.png saved!
327 - Optimal threshold for 15.0% FPR is: 0.607416570186615
328 - confusion_matrix_tlwiki.png saved!
329 - False Positive Rate is: 0.15018641595072135
330 - CONFUSION MATRIX -
331
332Predicted not reverted reverted
333Actual
334not reverted 20970 3706
335reverted 176 1504
336
337
338============ - bnwiki - ============
339 - Raw data shape: (330764, 17)
340 - Duplicate rows found and removed: 29591
341 - Clean data shape: (301173, 17)
342 - Number of duplicated revision_ids found: 10
343 - Unique revision_ids: 301166 | Data Shape: 301166 | Same? : -> True
344 - Removing edits that are reverts from df | New Shape: (292405, 17)
345 - Is any revert_risk_score NA? : False
346 - Is any user_edit_count NA? : False
347 - Is any time_to_revert NA? : False
348 - ROC_bnwiki.png saved!
349 - Optimal threshold for 15.0% FPR is: 0.6465859413146973
350 - confusion_matrix_bnwiki.png saved!
351 - False Positive Rate is: 0.15002477700693756
352 - CONFUSION MATRIX -
353
354Predicted not reverted reverted
355Actual
356not reverted 233274 41174
357reverted 4019 13938
358
359
360============ - trwiki - ============
361 - Raw data shape: (749190, 17)
362 - Duplicate rows found and removed: 116314
363 - Clean data shape: (632876, 17)
364 - Number of duplicated revision_ids found: 4
365 - Unique revision_ids: 632874 | Data Shape: 632874 | Same? : -> True
366 - Removing edits that are reverts from df | New Shape: (581675, 17)
367 - Is any revert_risk_score NA? : False
368 - Is any user_edit_count NA? : False
369 - Is any time_to_revert NA? : False
370 - ROC_trwiki.png saved!
371 - Optimal threshold for 15.0% FPR is: 0.6082413196563721
372 - confusion_matrix_trwiki.png saved!
373 - False Positive Rate is: 0.14998876387080776
374 - CONFUSION MATRIX -
375
376Predicted not reverted reverted
377Actual
378not reverted 438769 77423
379reverted 10108 55375
380
381
382============ - azwiki - ============
383 - Raw data shape: (224309, 17)
384 - Duplicate rows found and removed: 23643
385 - Clean data shape: (200666, 17)
386 - Number of duplicated revision_ids found: 4
387 - Unique revision_ids: 200664 | Data Shape: 200664 | Same? : -> True
388 - Removing edits that are reverts from df | New Shape: (194127, 17)
389 - Is any revert_risk_score NA? : False
390 - Is any user_edit_count NA? : False
391 - Is any time_to_revert NA? : False
392 - ROC_azwiki.png saved!
393 - Optimal threshold for 15.0% FPR is: 0.5366107821464539
394 - confusion_matrix_azwiki.png saved!
395 - False Positive Rate is: 0.14990706525346845
396 - CONFUSION MATRIX -
397
398Predicted not reverted reverted
399Actual
400not reverted 153673 27099
401reverted 2012 11343
402
403
404Optimal Thresholds calculated at 16-05-2025T19:24:01
405{'cywiki': 0.11076631, 'simplewiki': 0.90655595, 'bewiki': 0.5724957, 'kkwiki': 0.6048708, 'nnwiki': 0.41624594, 'mkwiki': 0.5058307, 'lawiki': 0.6340628, 'afwiki': 0.7369383, 'tewiki': 0.36725548, 'mrwiki': 0.86735326, 'swwiki': 0.74820924, 'mlwiki': 0.93626505, 'iswiki': 0.88926035, 'pawiki': 0.5458896, 'hawiki': 0.4823182, 'tlwiki': 0.60741657, 'bnwiki': 0.64658594, 'trwiki': 0.6082413, 'azwiki': 0.5366108}
406
407Time taken: 4278.109 secs

The execution takes around 70 minutes.

Optimal Thresholds

WikiOptimalThreshold
cywiki0.11076631
simplewiki0.90655595
bewiki0.5724957
kkwiki0.6048708
nnwiki0.41624594
mkwiki0.5058307
lawiki0.6340628
afwiki0.7369383
tewiki0.36725548
mrwiki0.86735326
swwiki0.74820924
mlwiki0.93626505
iswiki0.88926035
pawiki0.545889
hawiki0.4823182
tlwiki0.60741657
bnwiki0.64658594
trwiki0.6082413
azwiki0.5366108

@Kgraessle would it be possible to review/check the above results? Do they seem correct based on your experience ? Do they meet the the expected outcome for each wiki based on our intuition?
For more "in depth" evaluation you can have a look at the plots for RocCurve and ConfusionMatrix at the bottom of this notebook .

Yeah I can take a look, is it ok if I prioritize this for next week?

@Kgraessle yeah sure next week is fine, thank you very much.

gkyziridis mentioned this in T395256: [Spike] Investigate why filtering wasn't working on testwiki .May 27 2025, 12:59 PM
gkyziridis closed this task as Resolved.May 28 2025, 8:40 AM
gkyziridis moved this task from In Progress to 2025-2026 Q1 Done on the Machine-Learning-Team board.
• isarantopoulos moved this task from 2025-2026 Q1 Done to 2025-2026 Q2 Done on the Machine-Learning-Team board.Jul 8 2025, 9:12 AM
• isarantopoulos mentioned this in T400590: Investigate revertrisk threshold generation for enwiki.Jul 28 2025, 6:43 AM
• isarantopoulos moved this task from 2025-2026 Q2 Done to Task Archive on the Machine-Learning-Team board.Oct 21 2025, 3:38 PM
• isarantopoulos mentioned this in T408388: WE 1.3.4 Roll out Revert Risk Filters to Wikis that don't have damaging/goodfaith Edit Models.Oct 27 2025, 12:54 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits