Purge old IP data from AbuseFilter on the Beta Cluster
Closed, ResolvedPublic

Description

In order to purge old IP data from the AbuseLog, I'd like to run the purgeOldLogIPData.php. I can do this myself but, does the extension now know which data should be purged? (cf. parent task).

MarcoAurelio triaged this task as Normal priority.
Restricted Application added a project: User-MarcoAurelio. · View Herald TranscriptFeb 9 2018, 10:45 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
MarcoAurelio updated the task description. (Show Details)Feb 9 2018, 9:15 PM

Mentioned in SAL (#wikimedia-releng) [2018-02-10T18:58:24Z] <Hauskatze> maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=eswiki (1695 rows purged - T186870)

I choose beta eswiki since it is closed and I guess I can more safely do that there. Test plan consisted in:

First: query how many rows do we have with private data:

wikiadmin@deployment-db04[eswiki]> select count(afl_ip) from abuse_filter_log;
+---------------+
| count(afl_ip) |
+---------------+
|          1695 |
+---------------+
1 row in set (0.00 sec)

Second: see the oldest and newest abusefilter log entry with wikiadmin@deployment-db04[eswiki]> select afl_id, afl_timestamp, afl_ip from abuse_filter_log order by afl_timestamp desc; Oldest is from 20140718015832 and newest is from 20170425131421. That means all data is older than 90 days so all afl_ip data should go.

Third: run the script:

maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=eswiki
Purging old IP Address data from abuse_filter_log...
200
400
600
800
1000
1200
1400
1600
1695
1695 rows.
Done.

Fourth: check if the data is really gone: wikiadmin@deployment-db04[eswiki]> select afl_id, afl_timestamp, afl_ip from abuse_filter_log order by afl_timestamp desc; shows no data on afl_ip field.

So I guess the script works as expected.

Is that method fine?

Also, I'm not sure if it is possible but it is somewhat kind of affordable to run this manually on all Beta Cluster wikis, but doing so on all WMF Production wikis will be a pain. Apparently foreachwikiindblist 'all-labs.dblist' <script here> expects a maintenance script from the mediawiki core maintenance folder, not from an extension...

MarcoAurelio changed the task status from Stalled to Open.Feb 10 2018, 7:10 PM
Huji added a comment.Feb 10 2018, 7:44 PM

@MarcoAurelio we should make sure it can be run automatically. What was the execution time on beta eswiki?

It took about few seconds to complete.

Huji added a comment.Feb 10 2018, 8:06 PM

That should be fine, I guess? In the production version of enwiki, there are about 8000 abuse logs every day (see graph below). That would translate to a minute or two, I suppose.

I'm not a Wikimedia sysadmin, so I might be wrong. My idea is that for production wikis I assume that the first run should be scheduled with @greg and be run in batches of wikis or so (start with small.dblist and so on, or another method). In the first run we'll have tens of thousands of entries to clear Wikimedia-wide (note that AF is there logging since 2012?), and that will take some time, or maybe disrupt DBs. Further runs should be scheduled with a cron on the puppet IMHO, but they will not be that heavy IMHO even if we run it daily. To be sure, before running the script in Production I'd do a DBA review of the script and co-schedule a deployment window so we don't trap our fingers.

Huji added a comment.EditedFeb 11 2018, 12:15 AM

In the first run we'll have tens of thousands of entries to clear Wikimedia-wide (note that AF is there logging since 2012?), and that will take some time, or maybe disrupt DBs.

Tens of thousands? The number of abuse log entries just in enwiki is around 18 million. I think we are dealing with tens of millions, if not hundreds of millions or rows with IP data, across the board.

T186973 is of utmost importance.

Mentioned in SAL (#wikimedia-releng) [2018-02-12T18:43:58Z] <Hauskatze> Starting to purge old afl_ip data from abuse_filter_log on Beta Cluster - T186870

Mentioned in SAL (#wikimedia-releng) [2018-02-12T18:45:24Z] <Hauskatze> maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged - T186870)

Mentioned in SAL (#wikimedia-releng) [2018-02-12T18:46:15Z] <Hauskatze> maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=arwiki (37 rows purged - T186870)

P6684 - I added some missing rows.

Mentioned in SAL (#wikimedia-releng) [2018-02-12T18:55:06Z] <Hauskatze> Running maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldIPLogData.php for T186870

Done:

1​maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
2​-----------------------------------------------------------------
3​aawiki
4​-----------------------------------------------------------------
5​aawiki: Purging old IP Address data from abuse_filter_log...
6​aawiki: 0 rows.
7​aawiki: Done.
8​-----------------------------------------------------------------
9​arwiki
10​-----------------------------------------------------------------
11​arwiki: Purging old IP Address data from abuse_filter_log...
12​arwiki: 0 rows.
13​arwiki: Done.
14​-----------------------------------------------------------------
15​cawiki
16​-----------------------------------------------------------------
17​cawiki: Purging old IP Address data from abuse_filter_log...
18​cawiki: 40
19​cawiki: 40 rows.
20​cawiki: Done.
21​-----------------------------------------------------------------
22​commonswiki
23​-----------------------------------------------------------------
24​commonswiki: Purging old IP Address data from abuse_filter_log...
25​commonswiki: 200
26​commonswiki: 400
27​commonswiki: 600
28​commonswiki: 800
29​commonswiki: 1000
30​commonswiki: 1200
31​commonswiki: 1400
32​commonswiki: 1600
33​commonswiki: 1800
34​commonswiki: 2000
35​commonswiki: 2200
36​commonswiki: 2400
37​commonswiki: 2600
38​commonswiki: 2800
39​commonswiki: 3000
40​commonswiki: 3200
41​commonswiki: 3400
42​commonswiki: 3600
43​commonswiki: 3800
44​commonswiki: 4000
45​commonswiki: 4200
46​commonswiki: 4400
47​commonswiki: 4600
48​commonswiki: 4800
49​commonswiki: 5000
50​commonswiki: 5200
51​commonswiki: 5400
52​commonswiki: 5600
53​commonswiki: 5800
54​commonswiki: 6000
55​commonswiki: 6200
56​commonswiki: 6400
57​commonswiki: 6600
58​commonswiki: 6800
59​commonswiki: 7000
60​commonswiki: 7200
61​commonswiki: 7400
62​commonswiki: 7600
63​commonswiki: 7800
64​commonswiki: 8000
65​commonswiki: 8200
66​commonswiki: 8400
67​commonswiki: 8600
68​commonswiki: 8800
69​commonswiki: 9000
70​commonswiki: 9200
71​commonswiki: 9400
72​commonswiki: 9600
73​commonswiki: 9800
74​commonswiki: 10000
75​commonswiki: 10200
76​commonswiki: 10400
77​commonswiki: 10600
78​commonswiki: 10800
79​commonswiki: 11000
80​commonswiki: 11200
81​commonswiki: 11400
82​commonswiki: 11600
83​commonswiki: 11800
84​commonswiki: 12000
85​commonswiki: 12200
86​commonswiki: 12400
87​commonswiki: 12600
88​commonswiki: 12800
89​commonswiki: 13000
90​commonswiki: 13200
91​commonswiki: 13400
92​commonswiki: 13600
93​commonswiki: 13800
94​commonswiki: 14000
95​commonswiki: 14200
96​commonswiki: 14400
97​commonswiki: 14600
98​commonswiki: 14800
99​commonswiki: 15000
100​commonswiki: 15200
101​commonswiki: 15400
102​commonswiki: 15600
103​commonswiki: 15800
104​commonswiki: 16000
105​commonswiki: 16200
106​commonswiki: 16400
107​commonswiki: 16600
108​commonswiki: 16800
109​commonswiki: 17000
110​commonswiki: 17200
111​commonswiki: 17400
112​commonswiki: 17600
113​commonswiki: 17800
114​commonswiki: 18000
115​commonswiki: 18200
116​commonswiki: 18400
117​commonswiki: 18600
118​commonswiki: 18800
119​commonswiki: 19000
120​commonswiki: 19200
121​commonswiki: 19400
122​commonswiki: 19600
123​commonswiki: 19800
124​commonswiki: 20000
125​commonswiki: 20200
126​commonswiki: 20400
127​commonswiki: 20600
128​commonswiki: 20800
129​commonswiki: 21000
130​commonswiki: 21200
131​commonswiki: 21400
132​commonswiki: 21600
133​commonswiki: 21800
134​commonswiki: 22000
135​commonswiki: 22200
136​commonswiki: 22400
137​commonswiki: 22600
138​commonswiki: 22800
139​commonswiki: 23000
140​commonswiki: 23200
141​commonswiki: 23400
142​commonswiki: 23600
143​commonswiki: 23800
144​commonswiki: 24000
145​commonswiki: 24200
146​commonswiki: 24400
147​commonswiki: 24600
148​commonswiki: 24800
149​commonswiki: 25000
150​commonswiki: 25200
151​commonswiki: 25400
152​commonswiki: 25600
153​commonswiki: 25800
154​commonswiki: 26000
155​commonswiki: 26200
156​commonswiki: 26373
157​commonswiki: 26373 rows.
158​commonswiki: Done.
159​-----------------------------------------------------------------
160​deploymentwiki
161​-----------------------------------------------------------------
162​deploymentwiki: Purging old IP Address data from abuse_filter_log...
163​deploymentwiki: 200
164​deploymentwiki: 400
165​deploymentwiki: 600
166​deploymentwiki: 800
167​deploymentwiki: 1000
168​deploymentwiki: 1200
169​deploymentwiki: 1400
170​deploymentwiki: 1600
171​deploymentwiki: 1800
172​deploymentwiki: 2000
173​deploymentwiki: 2200
174​deploymentwiki: 2400
175​deploymentwiki: 2600
176​deploymentwiki: 2800
177​deploymentwiki: 3000
178​deploymentwiki: 3200
179​deploymentwiki: 3400
180​deploymentwiki: 3600
181​deploymentwiki: 3800
182​deploymentwiki: 4000
183​deploymentwiki: 4200
184​deploymentwiki: 4400
185​deploymentwiki: 4600
186​deploymentwiki: 4800
187​deploymentwiki: 5000
188​deploymentwiki: 5200
189​deploymentwiki: 5400
190​deploymentwiki: 5600
191​deploymentwiki: 5800
192​deploymentwiki: 6000
193​deploymentwiki: 6200
194​deploymentwiki: 6400
195​deploymentwiki: 6600
196​deploymentwiki: 6800
197​deploymentwiki: 7000
198​deploymentwiki: 7200
199​deploymentwiki: 7400
200​deploymentwiki: 7600
201​deploymentwiki: 7800
202​deploymentwiki: 8000
203​deploymentwiki: 8200
204​deploymentwiki: 8400
205​deploymentwiki: 8600
206​deploymentwiki: 8800
207​deploymentwiki: 9000
208​deploymentwiki: 9200
209​deploymentwiki: 9400
210​deploymentwiki: 9600
211​deploymentwiki: 9800
212​deploymentwiki: 10000
213​deploymentwiki: 10200
214​deploymentwiki: 10400
215​deploymentwiki: 10600
216​deploymentwiki: 10800
217​deploymentwiki: 11000
218​deploymentwiki: 11200
219​deploymentwiki: 11400
220​deploymentwiki: 11600
221​deploymentwiki: 11800
222​deploymentwiki: 12000
223​deploymentwiki: 12200
224​deploymentwiki: 12400
225​deploymentwiki: 12600
226​deploymentwiki: 12800
227​deploymentwiki: 13000
228​deploymentwiki: 13200
229​deploymentwiki: 13400
230​deploymentwiki: 13600
231​deploymentwiki: 13800
232​deploymentwiki: 14000
233​deploymentwiki: 14200
234​deploymentwiki: 14400
235​deploymentwiki: 14600
236​deploymentwiki: 14800
237​deploymentwiki: 15000
238​deploymentwiki: 15200
239​deploymentwiki: 15400
240​deploymentwiki: 15600
241​deploymentwiki: 15800
242​deploymentwiki: 16000
243​deploymentwiki: 16200
244​deploymentwiki: 16400
245​deploymentwiki: 16600
246​deploymentwiki: 16800
247​deploymentwiki: 17000
248​deploymentwiki: 17200
249​deploymentwiki: 17400
250​deploymentwiki: 17600
251​deploymentwiki: 17800
252​deploymentwiki: 18000
253​deploymentwiki: 18200
254​deploymentwiki: 18400
255​deploymentwiki: 18600
256​deploymentwiki: 18800
257​deploymentwiki: 19000
258​deploymentwiki: 19200
259​deploymentwiki: 19400
260​deploymentwiki: 19600
261​deploymentwiki: 19800
262​deploymentwiki: 20000
263​deploymentwiki: 20200
264​deploymentwiki: 20400
265​deploymentwiki: 20600
266​deploymentwiki: 20800
267​deploymentwiki: 21000
268​deploymentwiki: 21200
269​deploymentwiki: 21400
270​deploymentwiki: 21600
271​deploymentwiki: 21800
272​deploymentwiki: 22000
273​deploymentwiki: 22044
274​deploymentwiki: 22044 rows.
275​deploymentwiki: Done.
276​-----------------------------------------------------------------
277​dewiki
278​-----------------------------------------------------------------
279​dewiki: Purging old IP Address data from abuse_filter_log...
280​dewiki: 200
281​dewiki: 400
282​dewiki: 600
283​dewiki: 680
284​dewiki: 680 rows.
285​dewiki: Done.
286​-----------------------------------------------------------------
287​dewiktionary
288​-----------------------------------------------------------------
289​dewiktionary: Purging old IP Address data from abuse_filter_log...
290​dewiktionary: 1
291​dewiktionary: 1 rows.
292​dewiktionary: Done.
293​-----------------------------------------------------------------
294​en_rtlwiki
295​-----------------------------------------------------------------
296​en_rtlwiki: Purging old IP Address data from abuse_filter_log...
297​en_rtlwiki: 6
298​en_rtlwiki: 6 rows.
299​en_rtlwiki: Done.
300​-----------------------------------------------------------------
301​enwiki
302​-----------------------------------------------------------------
303​enwiki: Purging old IP Address data from abuse_filter_log...
304​enwiki: 200
305​enwiki: 400
306​enwiki: 600
307​enwiki: 800
308​enwiki: 1000
309​enwiki: 1200
310​enwiki: 1400
311​enwiki: 1600
312​enwiki: 1800
313​enwiki: 2000
314​enwiki: 2200
315​enwiki: 2400
316​enwiki: 2600
317​enwiki: 2800
318​enwiki: 3000
319​enwiki: 3200
320​enwiki: 3400
321​enwiki: 3600
322​enwiki: 3800
323​enwiki: 4000
324​enwiki: 4200
325​enwiki: 4400
326​enwiki: 4600
327​enwiki: 4800
328​enwiki: 5000
329​enwiki: 5200
330​enwiki: 5400
331​enwiki: 5600
332​enwiki: 5800
333​enwiki: 6000
334​enwiki: 6200
335​enwiki: 6400
336​enwiki: 6600
337​enwiki: 6800
338​enwiki: 7000
339​enwiki: 7200
340​enwiki: 7400
341​enwiki: 7600
342​enwiki: 7800
343​enwiki: 8000
344​enwiki: 8200
345​enwiki: 8400
346​enwiki: 8600
347​enwiki: 8800
348​enwiki: 9000
349​enwiki: 9200
350​enwiki: 9400
351​enwiki: 9600
352​enwiki: 9800
353​enwiki: 10000
354​enwiki: 10200
355​enwiki: 10400
356​enwiki: 10600
357​enwiki: 10800
358​enwiki: 11000
359​enwiki: 11200
360​enwiki: 11400
361​enwiki: 11600
362​enwiki: 11800
363​enwiki: 12000
364​enwiki: 12200
365​enwiki: 12400
366​enwiki: 12600
367​enwiki: 12800
368​enwiki: 13000
369​enwiki: 13200
370​enwiki: 13400
371​enwiki: 13600
372​enwiki: 13800
373​enwiki: 14000
374​enwiki: 14200
375​enwiki: 14400
376​enwiki: 14600
377​enwiki: 14800
378​enwiki: 15000
379​enwiki: 15200
380​enwiki: 15400
381​enwiki: 15600
382​enwiki: 15800
383​enwiki: 16000
384​enwiki: 16200
385​enwiki: 16400
386​enwiki: 16600
387​enwiki: 16800
388​enwiki: 17000
389​enwiki: 17200
390​enwiki: 17400
391​enwiki: 17600
392​enwiki: 17800
393​enwiki: 18000
394​enwiki: 18200
395​enwiki: 18400
396​enwiki: 18600
397​enwiki: 18800
398​enwiki: 19000
399​enwiki: 19200
400​enwiki: 19400
401​enwiki: 19600
402​enwiki: 19800
403​enwiki: 20000
404​enwiki: 20200
405​enwiki: 20400
406​enwiki: 20600
407​enwiki: 20800
408​enwiki: 21000
409​enwiki: 21200
410​enwiki: 21400
411​enwiki: 21600
412​enwiki: 21800
413​enwiki: 22000
414​enwiki: 22200
415​enwiki: 22400
416​enwiki: 22600
417​enwiki: 22800
418​enwiki: 23000
419​enwiki: 23200
420​enwiki: 23400
421​enwiki: 23600
422​enwiki: 23800
423​enwiki: 24000
424​enwiki: 24200
425​enwiki: 24400
426​enwiki: 24600
427​enwiki: 24800
428​enwiki: 25000
429​enwiki: 25200
430​enwiki: 25400
431​enwiki: 25600
432​enwiki: 25800
433​enwiki: 26000
434​enwiki: 26200
435​enwiki: 26400
436​enwiki: 26600
437​enwiki: 26800
438​enwiki: 27000
439​enwiki: 27200
440​enwiki: 27400
441​enwiki: 27600
442​enwiki: 27800
443​enwiki: 28000
444​enwiki: 28200
445​enwiki: 28400
446​enwiki: 28600
447​enwiki: 28800
448​enwiki: 29000
449​enwiki: 29200
450​enwiki: 29400
451​enwiki: 29600
452​enwiki: 29800
453​enwiki: 30000
454​enwiki: 30200
455​enwiki: 30400
456​enwiki: 30600
457​enwiki: 30800
458​enwiki: 31000
459​enwiki: 31200
460​enwiki: 31400
461​enwiki: 31600
462​enwiki: 31800
463​enwiki: 32000
464​enwiki: 32200
465​enwiki: 32400
466​enwiki: 32600
467​enwiki: 32800
468​enwiki: 33000
469​enwiki: 33200
470​enwiki: 33400
471​enwiki: 33600
472​enwiki: 33800
473​enwiki: 34000
474​enwiki: 34200
475​enwiki: 34400
476​enwiki: 34600
477​enwiki: 34800
478​enwiki: 35000
479​enwiki: 35200
480​enwiki: 35400
481​enwiki: 35600
482​enwiki: 35800
483​enwiki: 36000
484​enwiki: 36200
485​enwiki: 36400
486​enwiki: 36600
487​enwiki: 36800
488​enwiki: 37000
489​enwiki: 37200
490​enwiki: 37400
491​enwiki: 37600
492​enwiki: 37800
493​enwiki: 38000
494​enwiki: 38200
495​enwiki: 38400
496​enwiki: 38600
497​enwiki: 38800
498​enwiki: 39000
499​enwiki: 39200
500​enwiki: 39400
501​enwiki: 39600
502​enwiki: 39800
503​enwiki: 40000
504​enwiki: 40200
505​enwiki: 40400
506​enwiki: 40600
507​enwiki: 40800
508​enwiki: 41000
509​enwiki: 41200
510​enwiki: 41400
511​enwiki: 41600
512​enwiki: 41800
513​enwiki: 42000
514​enwiki: 42200
515​enwiki: 42400
516​enwiki: 42600
517​enwiki: 42800
518​enwiki: 43000
519​enwiki: 43200
520​enwiki: 43400
521​enwiki: 43600
522​enwiki: 43800
523​enwiki: 44000
524​enwiki: 44200
525​enwiki: 44400
526​enwiki: 44600
527​enwiki: 44800
528​enwiki: 45000
529​enwiki: 45200
530​enwiki: 45400
531​enwiki: 45600
532​enwiki: 45800
533​enwiki: 46000
534​enwiki: 46200
535​enwiki: 46400
536​enwiki: 46600
537​enwiki: 46800
538​enwiki: 47000
539​enwiki: 47200
540​enwiki: 47400
541​enwiki: 47600
542​enwiki: 47800
543​enwiki: 48000
544​enwiki: 48200
545​enwiki: 48400
546​enwiki: 48600
547​enwiki: 48800
548​enwiki: 49000
549​enwiki: 49200
550​enwiki: 49400
551​enwiki: 49600
552​enwiki: 49800
553​enwiki: 50000
554​enwiki: 50200
555​enwiki: 50400
556​enwiki: 50600
557​enwiki: 50800
558​enwiki: 51000
559​enwiki: 51200
560​enwiki: 51400
561​enwiki: 51600
562​enwiki: 51800
563​enwiki: 52000
564​enwiki: 52200
565​enwiki: 52400
566​enwiki: 52600
567​enwiki: 52800
568​enwiki: 53000
569​enwiki: 53200
570​enwiki: 53400
571​enwiki: 53600
572​enwiki: 53800
573​enwiki: 54000
574​enwiki: 54200
575​enwiki: 54400
576​enwiki: 54600
577​enwiki: 54800
578​enwiki: 55000
579​enwiki: 55200
580​enwiki: 55400
581​enwiki: 55600
582​enwiki: 55800
583​enwiki: 56000
584​enwiki: 56200
585​enwiki: 56400
586​enwiki: 56600
587​enwiki: 56800
588​enwiki: 57000
589​enwiki: 57200
590​enwiki: 57400
591​enwiki: 57600
592​enwiki: 57800
593​enwiki: 58000
594​enwiki: 58200
595​enwiki: 58400
596​enwiki: 58600
597​enwiki: 58800
598​enwiki: 59000
599​enwiki: 59200
600​enwiki: 59400
601​enwiki: 59600
602​enwiki: 59800
603​enwiki: 60000
604​enwiki: 60200
605​enwiki: 60400
606​enwiki: 60600
607​enwiki: 60800
608​enwiki: 61000
609​enwiki: 61200
610​enwiki: 61400
611​enwiki: 61600
612​enwiki: 61800
613​enwiki: 62000
614​enwiki: 62200
615​enwiki: 62400
616​enwiki: 62600
617​enwiki: 62800
618​enwiki: 63000
619​enwiki: 63200
620​enwiki: 63400
621​enwiki: 63600
622​enwiki: 63800
623​enwiki: 64000
624​enwiki: 64200
625​enwiki: 64400
626​enwiki: 64600
627​enwiki: 64800
628​enwiki: 65000
629​enwiki: 65080
630​enwiki: 65080 rows.
631​enwiki: Done.
632​-----------------------------------------------------------------
633​enwikibooks
634​-----------------------------------------------------------------
635​enwikibooks: Purging old IP Address data from abuse_filter_log...
636​enwikibooks: 200
637​enwikibooks: 400
638​enwikibooks: 600
639​enwikibooks: 800
640​enwikibooks: 1000
641​enwikibooks: 1200
642​enwikibooks: 1400
643​enwikibooks: 1600
644​enwikibooks: 1800
645​enwikibooks: 2000
646​enwikibooks: 2200
647​enwikibooks: 2400
648​enwikibooks: 2600
649​enwikibooks: 2800
650​enwikibooks: 3000
651​enwikibooks: 3200
652​enwikibooks: 3400
653​enwikibooks: 3600
654​enwikibooks: 3800
655​enwikibooks: 4000
656​enwikibooks: 4200
657​enwikibooks: 4400
658​enwikibooks: 4600
659​enwikibooks: 4800
660​enwikibooks: 4854
661​enwikibooks: 4854 rows.
662​enwikibooks: Done.
663​-----------------------------------------------------------------
664​enwikinews
665​-----------------------------------------------------------------
666​enwikinews: Purging old IP Address data from abuse_filter_log...
667​enwikinews: 12
668​enwikinews: 12 rows.
669​enwikinews: Done.
670​-----------------------------------------------------------------
671​enwikiquote
672​-----------------------------------------------------------------
673​enwikiquote: Purging old IP Address data from abuse_filter_log...
674​enwikiquote: 11
675​enwikiquote: 11 rows.
676​enwikiquote: Done.
677​-----------------------------------------------------------------
678​enwikisource
679​-----------------------------------------------------------------
680​enwikisource: Purging old IP Address data from abuse_filter_log...
681​enwikisource: 14
682​enwikisource: 14 rows.
683​enwikisource: Done.
684​-----------------------------------------------------------------
685​enwikiversity
686​-----------------------------------------------------------------
687​enwikiversity: Purging old IP Address data from abuse_filter_log...
688​enwikiversity: 15
689​enwikiversity: 15 rows.
690​enwikiversity: Done.
691​-----------------------------------------------------------------
692​enwikivoyage
693​-----------------------------------------------------------------
694​enwikivoyage: Purging old IP Address data from abuse_filter_log...
695​enwikivoyage: 43
696​enwikivoyage: 43 rows.
697​enwikivoyage: Done.
698​-----------------------------------------------------------------
699​enwiktionary
700​-----------------------------------------------------------------
701​enwiktionary: Purging old IP Address data from abuse_filter_log...
702​enwiktionary: 200
703​enwiktionary: 282
704​enwiktionary: 282 rows.
705​enwiktionary: Done.
706​-----------------------------------------------------------------
707​eowiki
708​-----------------------------------------------------------------
709​eowiki: Purging old IP Address data from abuse_filter_log...
710​eowiki: 22
711​eowiki: 22 rows.
712​eowiki: Done.
713​-----------------------------------------------------------------
714​eswiki
715​-----------------------------------------------------------------
716​eswiki: Purging old IP Address data from abuse_filter_log...
717​eswiki: 0 rows.
718​eswiki: Done.
719​-----------------------------------------------------------------
720​fawiki
721​-----------------------------------------------------------------
722​fawiki: Purging old IP Address data from abuse_filter_log...
723​fawiki: 92
724​fawiki: 92 rows.
725​fawiki: Done.
726​-----------------------------------------------------------------
727​hewiki
728​-----------------------------------------------------------------
729​hewiki: Purging old IP Address data from abuse_filter_log...
730​hewiki: 56
731​hewiki: 56 rows.
732​hewiki: Done.
733​-----------------------------------------------------------------
734​hewiktionary
735​-----------------------------------------------------------------
736​hewiktionary: Purging old IP Address data from abuse_filter_log...
737​hewiktionary: 0 rows.
738​hewiktionary: Done.
739​-----------------------------------------------------------------
740​hiwiki
741​-----------------------------------------------------------------
742​hiwiki: Purging old IP Address data from abuse_filter_log...
743​hiwiki: 24
744​hiwiki: 24 rows.
745​hiwiki: Done.
746​-----------------------------------------------------------------
747​jawiki
748​-----------------------------------------------------------------
749​jawiki: Purging old IP Address data from abuse_filter_log...
750​jawiki: 18
751​jawiki: 18 rows.
752​jawiki: Done.
753​-----------------------------------------------------------------
754​kowiki
755​-----------------------------------------------------------------
756​kowiki: Purging old IP Address data from abuse_filter_log...
757​kowiki: 18
758​kowiki: 18 rows.
759​kowiki: Done.
760​-----------------------------------------------------------------
761​loginwiki
762​-----------------------------------------------------------------
763​loginwiki: Purging old IP Address data from abuse_filter_log...
764​loginwiki: 0 rows.
765​loginwiki: Done.
766​-----------------------------------------------------------------
767​metawiki
768​-----------------------------------------------------------------
769​metawiki: Purging old IP Address data from abuse_filter_log...
770​metawiki: 164
771​metawiki: 164 rows.
772​metawiki: Done.
773​-----------------------------------------------------------------
774​nlwiki
775​-----------------------------------------------------------------
776​nlwiki: Purging old IP Address data from abuse_filter_log...
777​nlwiki: 0 rows.
778​nlwiki: Done.
779​-----------------------------------------------------------------
780​ruwiki
781​-----------------------------------------------------------------
782​ruwiki: Purging old IP Address data from abuse_filter_log...
783​ruwiki: 72
784​ruwiki: 72 rows.
785​ruwiki: Done.
786​-----------------------------------------------------------------
787​simplewiki
788​-----------------------------------------------------------------
789​simplewiki: Purging old IP Address data from abuse_filter_log...
790​simplewiki: 200
791​simplewiki: 400
792​simplewiki: 600
793​simplewiki: 800
794​simplewiki: 1000
795​simplewiki: 1200
796​simplewiki: 1400
797​simplewiki: 1600
798​simplewiki: 1800
799​simplewiki: 2000
800​simplewiki: 2200
801​simplewiki: 2400
802​simplewiki: 2600
803​simplewiki: 2800
804​simplewiki: 3000
805​simplewiki: 3200
806​simplewiki: 3400
807​simplewiki: 3600
808​simplewiki: 3800
809​simplewiki: 4000
810​simplewiki: 4200
811​simplewiki: 4400
812​simplewiki: 4600
813​simplewiki: 4800
814​simplewiki: 5000
815​simplewiki: 5200
816​simplewiki: 5400
817​simplewiki: 5600
818​simplewiki: 5800
819​simplewiki: 6000
820​simplewiki: 6200
821​simplewiki: 6400
822​simplewiki: 6600
823​simplewiki: 6800
824​simplewiki: 7000
825​simplewiki: 7200
826​simplewiki: 7400
827​simplewiki: 7600
828​simplewiki: 7800
829​simplewiki: 7948
830​simplewiki: 7948 rows.
831​simplewiki: Done.
832​-----------------------------------------------------------------
833​sqwiki
834​-----------------------------------------------------------------
835​sqwiki: Purging old IP Address data from abuse_filter_log...
836​sqwiki: 26
837​sqwiki: 26 rows.
838​sqwiki: Done.
839​-----------------------------------------------------------------
840​testwiki
841​-----------------------------------------------------------------
842​testwiki: Purging old IP Address data from abuse_filter_log...
843​testwiki: 75
844​testwiki: 75 rows.
845​testwiki: Done.
846​-----------------------------------------------------------------
847​ukwiki
848​-----------------------------------------------------------------
849​ukwiki: Purging old IP Address data from abuse_filter_log...
850​ukwiki: 26
851​ukwiki: 26 rows.
852​ukwiki: Done.
853​-----------------------------------------------------------------
854​wikidatawiki
855​-----------------------------------------------------------------
856​wikidatawiki: Purging old IP Address data from abuse_filter_log...
857​wikidatawiki: 171
858​wikidatawiki: 171 rows.
859​wikidatawiki: Done.
860​-----------------------------------------------------------------
861​zerowiki
862​-----------------------------------------------------------------
863​zerowiki: Purging old IP Address data from abuse_filter_log...
864​zerowiki: 0 rows.
865​zerowiki: Done.
866​-----------------------------------------------------------------
867​zhwiki
868​-----------------------------------------------------------------
869​zhwiki: Purging old IP Address data from abuse_filter_log...
870​zhwiki: 75
871​zhwiki: 75 rows.
872​zhwiki: Done.

However we should either disabling afl_ip logging on Beta Cluster, or restrict who can access that info or both. In any case, a puppet cron should be set there to do this regularly.

Huji added a comment.Feb 12 2018, 7:53 PM

@MarcoAurelio what was the total execution time? Total row count is 128K

@MarcoAurelio what was the total execution time? Total row count is 128K

I don't know. Not more than two minutes probably. It was pretty fast.

That said, on WMF production it has 2 long years of cleaning to do after we fixed the $this->requireExtension thing. If we were to run this manually, I'd say to run it on terbium since it'll be a long-running script. However the cron job should be fixed so it should run today at 01:15 UTC. I'll ask around and see if, at least for this round, they could keep the logs for doublechecking.

Change 410072 had a related patch set uploaded (by MarcoAurelio; owner: MarcoAurelio):
[operations/puppet@production] mediawiki: log next run of purge_abusefilter.pp

https://gerrit.wikimedia.org/r/410072

Change 410072 merged by Dzahn:
[operations/puppet@production] mediawiki: log next run of purge_abusefilter.pp

https://gerrit.wikimedia.org/r/410072

MarcoAurelio closed this task as Resolved.Feb 12 2018, 11:09 PM

Well, the data is purged. I'll create another task so this script can run croned on the beta cluster puppet.