Purge old IP data from AbuseFilter on the Beta Cluster
Closed, ResolvedPublic

Description

In order to purge old IP data from the AbuseLog, I'd like to run the purgeOldLogIPData.php. I can do this myself but, does the extension now know which data should be purged? (cf. parent task).

MarcoAurelio triaged this task as Normal priority.
Restricted Application added a project: User-MarcoAurelio. · View Herald TranscriptFeb 9 2018, 10:45 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Mentioned in SAL (#wikimedia-releng) [2018-02-10T18:58:24Z] <Hauskatze> maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=eswiki (1695 rows purged - T186870)

I choose beta eswiki since it is closed and I guess I can more safely do that there. Test plan consisted in:

First: query how many rows do we have with private data:

wikiadmin@deployment-db04[eswiki]> select count(afl_ip) from abuse_filter_log;
+---------------+
| count(afl_ip) |
+---------------+
|          1695 |
+---------------+
1 row in set (0.00 sec)

Second: see the oldest and newest abusefilter log entry with wikiadmin@deployment-db04[eswiki]> select afl_id, afl_timestamp, afl_ip from abuse_filter_log order by afl_timestamp desc; Oldest is from 20140718015832 and newest is from 20170425131421. That means all data is older than 90 days so all afl_ip data should go.

Third: run the script:

maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=eswiki
Purging old IP Address data from abuse_filter_log...
200
400
600
800
1000
1200
1400
1600
1695
1695 rows.
Done.

Fourth: check if the data is really gone: wikiadmin@deployment-db04[eswiki]> select afl_id, afl_timestamp, afl_ip from abuse_filter_log order by afl_timestamp desc; shows no data on afl_ip field.

So I guess the script works as expected.

Is that method fine?

Also, I'm not sure if it is possible but it is somewhat kind of affordable to run this manually on all Beta Cluster wikis, but doing so on all WMF Production wikis will be a pain. Apparently foreachwikiindblist 'all-labs.dblist' <script here> expects a maintenance script from the mediawiki core maintenance folder, not from an extension...

MarcoAurelio changed the task status from Stalled to Open.Feb 10 2018, 7:10 PM
Huji added a comment.Feb 10 2018, 7:44 PM

@MarcoAurelio we should make sure it can be run automatically. What was the execution time on beta eswiki?

It took about few seconds to complete.

Huji added a comment.Feb 10 2018, 8:06 PM

That should be fine, I guess? In the production version of enwiki, there are about 8000 abuse logs every day (see graph below). That would translate to a minute or two, I suppose.

I'm not a Wikimedia sysadmin, so I might be wrong. My idea is that for production wikis I assume that the first run should be scheduled with @greg and be run in batches of wikis or so (start with small.dblist and so on, or another method). In the first run we'll have tens of thousands of entries to clear Wikimedia-wide (note that AF is there logging since 2012?), and that will take some time, or maybe disrupt DBs. Further runs should be scheduled with a cron on the puppet IMHO, but they will not be that heavy IMHO even if we run it daily. To be sure, before running the script in Production I'd do a DBA review of the script and co-schedule a deployment window so we don't trap our fingers.

Huji added a comment.EditedFeb 11 2018, 12:15 AM

In the first run we'll have tens of thousands of entries to clear Wikimedia-wide (note that AF is there logging since 2012?), and that will take some time, or maybe disrupt DBs.

Tens of thousands? The number of abuse log entries just in enwiki is around 18 million. I think we are dealing with tens of millions, if not hundreds of millions or rows with IP data, across the board.

T186973 is of utmost importance.

Mentioned in SAL (#wikimedia-releng) [2018-02-12T18:43:58Z] <Hauskatze> Starting to purge old afl_ip data from abuse_filter_log on Beta Cluster - T186870

Mentioned in SAL (#wikimedia-releng) [2018-02-12T18:45:24Z] <Hauskatze> maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged - T186870)

Mentioned in SAL (#wikimedia-releng) [2018-02-12T18:46:15Z] <Hauskatze> maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=arwiki (37 rows purged - T186870)

P6684 - I added some missing rows.

Mentioned in SAL (#wikimedia-releng) [2018-02-12T18:55:06Z] <Hauskatze> Running maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldIPLogData.php for T186870

Done:

1maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
2-----------------------------------------------------------------
3aawiki
4-----------------------------------------------------------------
5aawiki: Purging old IP Address data from abuse_filter_log...
6aawiki: 0 rows.
7aawiki: Done.
8-----------------------------------------------------------------
9arwiki
10-----------------------------------------------------------------
11arwiki: Purging old IP Address data from abuse_filter_log...
12arwiki: 0 rows.
13arwiki: Done.
14-----------------------------------------------------------------
15cawiki
16-----------------------------------------------------------------
17cawiki: Purging old IP Address data from abuse_filter_log...
18cawiki: 40
19cawiki: 40 rows.
20cawiki: Done.
21-----------------------------------------------------------------
22commonswiki
23-----------------------------------------------------------------
24commonswiki: Purging old IP Address data from abuse_filter_log...
25commonswiki: 200
26commonswiki: 400
27commonswiki: 600
28commonswiki: 800
29commonswiki: 1000
30commonswiki: 1200
31commonswiki: 1400
32commonswiki: 1600
33commonswiki: 1800
34commonswiki: 2000
35commonswiki: 2200
36commonswiki: 2400
37commonswiki: 2600
38commonswiki: 2800
39commonswiki: 3000
40commonswiki: 3200
41commonswiki: 3400
42commonswiki: 3600
43commonswiki: 3800
44commonswiki: 4000
45commonswiki: 4200
46commonswiki: 4400
47commonswiki: 4600
48commonswiki: 4800
49commonswiki: 5000
50commonswiki: 5200
51commonswiki: 5400
52commonswiki: 5600
53commonswiki: 5800
54commonswiki: 6000
55commonswiki: 6200
56commonswiki: 6400
57commonswiki: 6600
58commonswiki: 6800
59commonswiki: 7000
60commonswiki: 7200
61commonswiki: 7400
62commonswiki: 7600
63commonswiki: 7800
64commonswiki: 8000
65commonswiki: 8200
66commonswiki: 8400
67commonswiki: 8600
68commonswiki: 8800
69commonswiki: 9000
70commonswiki: 9200
71commonswiki: 9400
72commonswiki: 9600
73commonswiki: 9800
74commonswiki: 10000
75commonswiki: 10200
76commonswiki: 10400
77commonswiki: 10600
78commonswiki: 10800
79commonswiki: 11000
80commonswiki: 11200
81commonswiki: 11400
82commonswiki: 11600
83commonswiki: 11800
84commonswiki: 12000
85commonswiki: 12200
86commonswiki: 12400
87commonswiki: 12600
88commonswiki: 12800
89commonswiki: 13000
90commonswiki: 13200
91commonswiki: 13400
92commonswiki: 13600
93commonswiki: 13800
94commonswiki: 14000
95commonswiki: 14200
96commonswiki: 14400
97commonswiki: 14600
98commonswiki: 14800
99commonswiki: 15000
100commonswiki: 15200
101commonswiki: 15400
102commonswiki: 15600
103commonswiki: 15800
104commonswiki: 16000
105commonswiki: 16200
106commonswiki: 16400
107commonswiki: 16600
108commonswiki: 16800
109commonswiki: 17000
110commonswiki: 17200
111commonswiki: 17400
112commonswiki: 17600
113commonswiki: 17800
114commonswiki: 18000
115commonswiki: 18200
116commonswiki: 18400
117commonswiki: 18600
118commonswiki: 18800
119commonswiki: 19000
120commonswiki: 19200
121commonswiki: 19400
122commonswiki: 19600
123commonswiki: 19800
124commonswiki: 20000
125commonswiki: 20200
126commonswiki: 20400
127commonswiki: 20600
128commonswiki: 20800
129commonswiki: 21000
130commonswiki: 21200
131commonswiki: 21400
132commonswiki: 21600
133commonswiki: 21800
134commonswiki: 22000
135commonswiki: 22200
136commonswiki: 22400
137commonswiki: 22600
138commonswiki: 22800
139commonswiki: 23000
140commonswiki: 23200
141commonswiki: 23400
142commonswiki: 23600
143commonswiki: 23800
144commonswiki: 24000
145commonswiki: 24200
146commonswiki: 24400
147commonswiki: 24600
148commonswiki: 24800
149commonswiki: 25000
150commonswiki: 25200
151commonswiki: 25400
152commonswiki: 25600
153commonswiki: 25800
154commonswiki: 26000
155commonswiki: 26200
156commonswiki: 26373
157commonswiki: 26373 rows.
158commonswiki: Done.
159-----------------------------------------------------------------
160deploymentwiki
161-----------------------------------------------------------------
162deploymentwiki: Purging old IP Address data from abuse_filter_log...
163deploymentwiki: 200
164deploymentwiki: 400
165deploymentwiki: 600
166deploymentwiki: 800
167deploymentwiki: 1000
168deploymentwiki: 1200
169deploymentwiki: 1400
170deploymentwiki: 1600
171deploymentwiki: 1800
172deploymentwiki: 2000
173deploymentwiki: 2200
174deploymentwiki: 2400
175deploymentwiki: 2600
176deploymentwiki: 2800
177deploymentwiki: 3000
178deploymentwiki: 3200
179deploymentwiki: 3400
180deploymentwiki: 3600
181deploymentwiki: 3800
182deploymentwiki: 4000
183deploymentwiki: 4200
184deploymentwiki: 4400
185deploymentwiki: 4600
186deploymentwiki: 4800
187deploymentwiki: 5000
188deploymentwiki: 5200
189deploymentwiki: 5400
190deploymentwiki: 5600
191deploymentwiki: 5800
192deploymentwiki: 6000
193deploymentwiki: 6200
194deploymentwiki: 6400
195deploymentwiki: 6600
196deploymentwiki: 6800
197deploymentwiki: 7000
198deploymentwiki: 7200
199deploymentwiki: 7400
200deploymentwiki: 7600
201deploymentwiki: 7800
202deploymentwiki: 8000
203deploymentwiki: 8200
204deploymentwiki: 8400
205deploymentwiki: 8600
206deploymentwiki: 8800
207deploymentwiki: 9000
208deploymentwiki: 9200
209deploymentwiki: 9400
210deploymentwiki: 9600
211deploymentwiki: 9800
212deploymentwiki: 10000
213deploymentwiki: 10200
214deploymentwiki: 10400
215deploymentwiki: 10600
216deploymentwiki: 10800
217deploymentwiki: 11000
218deploymentwiki: 11200
219deploymentwiki: 11400
220deploymentwiki: 11600
221deploymentwiki: 11800
222deploymentwiki: 12000
223deploymentwiki: 12200
224deploymentwiki: 12400
225deploymentwiki: 12600
226deploymentwiki: 12800
227deploymentwiki: 13000
228deploymentwiki: 13200
229deploymentwiki: 13400
230deploymentwiki: 13600
231deploymentwiki: 13800
232deploymentwiki: 14000
233deploymentwiki: 14200
234deploymentwiki: 14400
235deploymentwiki: 14600
236deploymentwiki: 14800
237deploymentwiki: 15000
238deploymentwiki: 15200
239deploymentwiki: 15400
240deploymentwiki: 15600
241deploymentwiki: 15800
242deploymentwiki: 16000
243deploymentwiki: 16200
244deploymentwiki: 16400
245deploymentwiki: 16600
246deploymentwiki: 16800
247deploymentwiki: 17000
248deploymentwiki: 17200
249deploymentwiki: 17400
250deploymentwiki: 17600
251deploymentwiki: 17800
252deploymentwiki: 18000
253deploymentwiki: 18200
254deploymentwiki: 18400
255deploymentwiki: 18600
256deploymentwiki: 18800
257deploymentwiki: 19000
258deploymentwiki: 19200
259deploymentwiki: 19400
260deploymentwiki: 19600
261deploymentwiki: 19800
262deploymentwiki: 20000
263deploymentwiki: 20200
264deploymentwiki: 20400
265deploymentwiki: 20600
266deploymentwiki: 20800
267deploymentwiki: 21000
268deploymentwiki: 21200
269deploymentwiki: 21400
270deploymentwiki: 21600
271deploymentwiki: 21800
272deploymentwiki: 22000
273deploymentwiki: 22044
274deploymentwiki: 22044 rows.
275deploymentwiki: Done.
276-----------------------------------------------------------------
277dewiki
278-----------------------------------------------------------------
279dewiki: Purging old IP Address data from abuse_filter_log...
280dewiki: 200
281dewiki: 400
282dewiki: 600
283dewiki: 680
284dewiki: 680 rows.
285dewiki: Done.
286-----------------------------------------------------------------
287dewiktionary
288-----------------------------------------------------------------
289dewiktionary: Purging old IP Address data from abuse_filter_log...
290dewiktionary: 1
291dewiktionary: 1 rows.
292dewiktionary: Done.
293-----------------------------------------------------------------
294en_rtlwiki
295-----------------------------------------------------------------
296en_rtlwiki: Purging old IP Address data from abuse_filter_log...
297en_rtlwiki: 6
298en_rtlwiki: 6 rows.
299en_rtlwiki: Done.
300-----------------------------------------------------------------
301enwiki
302-----------------------------------------------------------------
303enwiki: Purging old IP Address data from abuse_filter_log...
304enwiki: 200
305enwiki: 400
306enwiki: 600
307enwiki: 800
308enwiki: 1000
309enwiki: 1200
310enwiki: 1400
311enwiki: 1600
312enwiki: 1800
313enwiki: 2000
314enwiki: 2200
315enwiki: 2400
316enwiki: 2600
317enwiki: 2800
318enwiki: 3000
319enwiki: 3200
320enwiki: 3400
321enwiki: 3600
322enwiki: 3800
323enwiki: 4000
324enwiki: 4200
325enwiki: 4400
326enwiki: 4600
327enwiki: 4800
328enwiki: 5000
329enwiki: 5200
330enwiki: 5400
331enwiki: 5600
332enwiki: 5800
333enwiki: 6000
334enwiki: 6200
335enwiki: 6400
336enwiki: 6600
337enwiki: 6800
338enwiki: 7000
339enwiki: 7200
340enwiki: 7400
341enwiki: 7600
342enwiki: 7800
343enwiki: 8000
344enwiki: 8200
345enwiki: 8400
346enwiki: 8600
347enwiki: 8800
348enwiki: 9000
349enwiki: 9200
350enwiki: 9400
351enwiki: 9600
352enwiki: 9800
353enwiki: 10000
354enwiki: 10200
355enwiki: 10400
356enwiki: 10600
357enwiki: 10800
358enwiki: 11000
359enwiki: 11200
360enwiki: 11400
361enwiki: 11600
362enwiki: 11800
363enwiki: 12000
364enwiki: 12200
365enwiki: 12400
366enwiki: 12600
367enwiki: 12800
368enwiki: 13000
369enwiki: 13200
370enwiki: 13400
371enwiki: 13600
372enwiki: 13800
373enwiki: 14000
374enwiki: 14200
375enwiki: 14400
376enwiki: 14600
377enwiki: 14800
378enwiki: 15000
379enwiki: 15200
380enwiki: 15400
381enwiki: 15600
382enwiki: 15800
383enwiki: 16000
384enwiki: 16200
385enwiki: 16400
386enwiki: 16600
387enwiki: 16800
388enwiki: 17000
389enwiki: 17200
390enwiki: 17400
391enwiki: 17600
392enwiki: 17800
393enwiki: 18000
394enwiki: 18200
395enwiki: 18400
396enwiki: 18600
397enwiki: 18800
398enwiki: 19000
399enwiki: 19200
400enwiki: 19400
401enwiki: 19600
402enwiki: 19800
403enwiki: 20000
404enwiki: 20200
405enwiki: 20400
406enwiki: 20600
407enwiki: 20800
408enwiki: 21000
409enwiki: 21200
410enwiki: 21400
411enwiki: 21600
412enwiki: 21800
413enwiki: 22000
414enwiki: 22200
415enwiki: 22400
416enwiki: 22600
417enwiki: 22800
418enwiki: 23000
419enwiki: 23200
420enwiki: 23400
421enwiki: 23600
422enwiki: 23800
423enwiki: 24000
424enwiki: 24200
425enwiki: 24400
426enwiki: 24600
427enwiki: 24800
428enwiki: 25000
429enwiki: 25200
430enwiki: 25400
431enwiki: 25600
432enwiki: 25800
433enwiki: 26000
434enwiki: 26200
435enwiki: 26400
436enwiki: 26600
437enwiki: 26800
438enwiki: 27000
439enwiki: 27200
440enwiki: 27400
441enwiki: 27600
442enwiki: 27800
443enwiki: 28000
444enwiki: 28200
445enwiki: 28400
446enwiki: 28600
447enwiki: 28800
448enwiki: 29000
449enwiki: 29200
450enwiki: 29400
451enwiki: 29600
452enwiki: 29800
453enwiki: 30000
454enwiki: 30200
455enwiki: 30400
456enwiki: 30600
457enwiki: 30800
458enwiki: 31000
459enwiki: 31200
460enwiki: 31400
461enwiki: 31600
462enwiki: 31800
463enwiki: 32000
464enwiki: 32200
465enwiki: 32400
466enwiki: 32600
467enwiki: 32800
468enwiki: 33000
469enwiki: 33200
470enwiki: 33400
471enwiki: 33600
472enwiki: 33800
473enwiki: 34000
474enwiki: 34200
475enwiki: 34400
476enwiki: 34600
477enwiki: 34800
478enwiki: 35000
479enwiki: 35200
480enwiki: 35400
481enwiki: 35600
482enwiki: 35800
483enwiki: 36000
484enwiki: 36200
485enwiki: 36400
486enwiki: 36600
487enwiki: 36800
488enwiki: 37000
489enwiki: 37200
490enwiki: 37400
491enwiki: 37600
492enwiki: 37800
493enwiki: 38000
494enwiki: 38200
495enwiki: 38400
496enwiki: 38600
497enwiki: 38800
498enwiki: 39000
499enwiki: 39200
500enwiki: 39400
501enwiki: 39600
502enwiki: 39800
503enwiki: 40000
504enwiki: 40200
505enwiki: 40400
506enwiki: 40600
507enwiki: 40800
508enwiki: 41000
509enwiki: 41200
510enwiki: 41400
511enwiki: 41600
512enwiki: 41800
513enwiki: 42000
514enwiki: 42200
515enwiki: 42400
516enwiki: 42600
517enwiki: 42800
518enwiki: 43000
519enwiki: 43200
520enwiki: 43400
521enwiki: 43600
522enwiki: 43800
523enwiki: 44000
524enwiki: 44200
525enwiki: 44400
526enwiki: 44600
527enwiki: 44800
528enwiki: 45000
529enwiki: 45200
530enwiki: 45400
531enwiki: 45600
532enwiki: 45800
533enwiki: 46000
534enwiki: 46200
535enwiki: 46400
536enwiki: 46600
537enwiki: 46800
538enwiki: 47000
539enwiki: 47200
540enwiki: 47400
541enwiki: 47600
542enwiki: 47800
543enwiki: 48000
544enwiki: 48200
545enwiki: 48400
546enwiki: 48600
547enwiki: 48800
548enwiki: 49000
549enwiki: 49200
550enwiki: 49400
551enwiki: 49600
552enwiki: 49800
553enwiki: 50000
554enwiki: 50200
555enwiki: 50400
556enwiki: 50600
557enwiki: 50800
558enwiki: 51000
559enwiki: 51200
560enwiki: 51400
561enwiki: 51600
562enwiki: 51800
563enwiki: 52000
564enwiki: 52200
565enwiki: 52400
566enwiki: 52600
567enwiki: 52800
568enwiki: 53000
569enwiki: 53200
570enwiki: 53400
571enwiki: 53600
572enwiki: 53800
573enwiki: 54000
574enwiki: 54200
575enwiki: 54400
576enwiki: 54600
577enwiki: 54800
578enwiki: 55000
579enwiki: 55200
580enwiki: 55400
581enwiki: 55600
582enwiki: 55800
583enwiki: 56000
584enwiki: 56200
585enwiki: 56400
586enwiki: 56600
587enwiki: 56800
588enwiki: 57000
589enwiki: 57200
590enwiki: 57400
591enwiki: 57600
592enwiki: 57800
593enwiki: 58000
594enwiki: 58200
595enwiki: 58400
596enwiki: 58600
597enwiki: 58800
598enwiki: 59000
599enwiki: 59200
600enwiki: 59400
601enwiki: 59600
602enwiki: 59800
603enwiki: 60000
604enwiki: 60200
605enwiki: 60400
606enwiki: 60600
607enwiki: 60800
608enwiki: 61000
609enwiki: 61200
610enwiki: 61400
611enwiki: 61600
612enwiki: 61800
613enwiki: 62000
614enwiki: 62200
615enwiki: 62400
616enwiki: 62600
617enwiki: 62800
618enwiki: 63000
619enwiki: 63200
620enwiki: 63400
621enwiki: 63600
622enwiki: 63800
623enwiki: 64000
624enwiki: 64200
625enwiki: 64400
626enwiki: 64600
627enwiki: 64800
628enwiki: 65000
629enwiki: 65080
630enwiki: 65080 rows.
631enwiki: Done.
632-----------------------------------------------------------------
633enwikibooks
634-----------------------------------------------------------------
635enwikibooks: Purging old IP Address data from abuse_filter_log...
636enwikibooks: 200
637enwikibooks: 400
638enwikibooks: 600
639enwikibooks: 800
640enwikibooks: 1000
641enwikibooks: 1200
642enwikibooks: 1400
643enwikibooks: 1600
644enwikibooks: 1800
645enwikibooks: 2000
646enwikibooks: 2200
647enwikibooks: 2400
648enwikibooks: 2600
649enwikibooks: 2800
650enwikibooks: 3000
651enwikibooks: 3200
652enwikibooks: 3400
653enwikibooks: 3600
654enwikibooks: 3800
655enwikibooks: 4000
656enwikibooks: 4200
657enwikibooks: 4400
658enwikibooks: 4600
659enwikibooks: 4800
660enwikibooks: 4854
661enwikibooks: 4854 rows.
662enwikibooks: Done.
663-----------------------------------------------------------------
664enwikinews
665-----------------------------------------------------------------
666enwikinews: Purging old IP Address data from abuse_filter_log...
667enwikinews: 12
668enwikinews: 12 rows.
669enwikinews: Done.
670-----------------------------------------------------------------
671enwikiquote
672-----------------------------------------------------------------
673enwikiquote: Purging old IP Address data from abuse_filter_log...
674enwikiquote: 11
675enwikiquote: 11 rows.
676enwikiquote: Done.
677-----------------------------------------------------------------
678enwikisource
679-----------------------------------------------------------------
680enwikisource: Purging old IP Address data from abuse_filter_log...
681enwikisource: 14
682enwikisource: 14 rows.
683enwikisource: Done.
684-----------------------------------------------------------------
685enwikiversity
686-----------------------------------------------------------------
687enwikiversity: Purging old IP Address data from abuse_filter_log...
688enwikiversity: 15
689enwikiversity: 15 rows.
690enwikiversity: Done.
691-----------------------------------------------------------------
692enwikivoyage
693-----------------------------------------------------------------
694enwikivoyage: Purging old IP Address data from abuse_filter_log...
695enwikivoyage: 43
696enwikivoyage: 43 rows.
697enwikivoyage: Done.
698-----------------------------------------------------------------
699enwiktionary
700-----------------------------------------------------------------
701enwiktionary: Purging old IP Address data from abuse_filter_log...
702enwiktionary: 200
703enwiktionary: 282
704enwiktionary: 282 rows.
705enwiktionary: Done.
706-----------------------------------------------------------------
707eowiki
708-----------------------------------------------------------------
709eowiki: Purging old IP Address data from abuse_filter_log...
710eowiki: 22
711eowiki: 22 rows.
712eowiki: Done.
713-----------------------------------------------------------------
714eswiki
715-----------------------------------------------------------------
716eswiki: Purging old IP Address data from abuse_filter_log...
717eswiki: 0 rows.
718eswiki: Done.
719-----------------------------------------------------------------
720fawiki
721-----------------------------------------------------------------
722fawiki: Purging old IP Address data from abuse_filter_log...
723fawiki: 92
724fawiki: 92 rows.
725fawiki: Done.
726-----------------------------------------------------------------
727hewiki
728-----------------------------------------------------------------
729hewiki: Purging old IP Address data from abuse_filter_log...
730hewiki: 56
731hewiki: 56 rows.
732hewiki: Done.
733-----------------------------------------------------------------
734hewiktionary
735-----------------------------------------------------------------
736hewiktionary: Purging old IP Address data from abuse_filter_log...
737hewiktionary: 0 rows.
738hewiktionary: Done.
739-----------------------------------------------------------------
740hiwiki
741-----------------------------------------------------------------
742hiwiki: Purging old IP Address data from abuse_filter_log...
743hiwiki: 24
744hiwiki: 24 rows.
745hiwiki: Done.
746-----------------------------------------------------------------
747jawiki
748-----------------------------------------------------------------
749jawiki: Purging old IP Address data from abuse_filter_log...
750jawiki: 18
751jawiki: 18 rows.
752jawiki: Done.
753-----------------------------------------------------------------
754kowiki
755-----------------------------------------------------------------
756kowiki: Purging old IP Address data from abuse_filter_log...
757kowiki: 18
758kowiki: 18 rows.
759kowiki: Done.
760-----------------------------------------------------------------
761loginwiki
762-----------------------------------------------------------------
763loginwiki: Purging old IP Address data from abuse_filter_log...
764loginwiki: 0 rows.
765loginwiki: Done.
766-----------------------------------------------------------------
767metawiki
768-----------------------------------------------------------------
769metawiki: Purging old IP Address data from abuse_filter_log...
770metawiki: 164
771metawiki: 164 rows.
772metawiki: Done.
773-----------------------------------------------------------------
774nlwiki
775-----------------------------------------------------------------
776nlwiki: Purging old IP Address data from abuse_filter_log...
777nlwiki: 0 rows.
778nlwiki: Done.
779-----------------------------------------------------------------
780ruwiki
781-----------------------------------------------------------------
782ruwiki: Purging old IP Address data from abuse_filter_log...
783ruwiki: 72
784ruwiki: 72 rows.
785ruwiki: Done.
786-----------------------------------------------------------------
787simplewiki
788-----------------------------------------------------------------
789simplewiki: Purging old IP Address data from abuse_filter_log...
790simplewiki: 200
791simplewiki: 400
792simplewiki: 600
793simplewiki: 800
794simplewiki: 1000
795simplewiki: 1200
796simplewiki: 1400
797simplewiki: 1600
798simplewiki: 1800
799simplewiki: 2000
800simplewiki: 2200
801simplewiki: 2400
802simplewiki: 2600
803simplewiki: 2800
804simplewiki: 3000
805simplewiki: 3200
806simplewiki: 3400
807simplewiki: 3600
808simplewiki: 3800
809simplewiki: 4000
810simplewiki: 4200
811simplewiki: 4400
812simplewiki: 4600
813simplewiki: 4800
814simplewiki: 5000
815simplewiki: 5200
816simplewiki: 5400
817simplewiki: 5600
818simplewiki: 5800
819simplewiki: 6000
820simplewiki: 6200
821simplewiki: 6400
822simplewiki: 6600
823simplewiki: 6800
824simplewiki: 7000
825simplewiki: 7200
826simplewiki: 7400
827simplewiki: 7600
828simplewiki: 7800
829simplewiki: 7948
830simplewiki: 7948 rows.
831simplewiki: Done.
832-----------------------------------------------------------------
833sqwiki
834-----------------------------------------------------------------
835sqwiki: Purging old IP Address data from abuse_filter_log...
836sqwiki: 26
837sqwiki: 26 rows.
838sqwiki: Done.
839-----------------------------------------------------------------
840testwiki
841-----------------------------------------------------------------
842testwiki: Purging old IP Address data from abuse_filter_log...
843testwiki: 75
844testwiki: 75 rows.
845testwiki: Done.
846-----------------------------------------------------------------
847ukwiki
848-----------------------------------------------------------------
849ukwiki: Purging old IP Address data from abuse_filter_log...
850ukwiki: 26
851ukwiki: 26 rows.
852ukwiki: Done.
853-----------------------------------------------------------------
854wikidatawiki
855-----------------------------------------------------------------
856wikidatawiki: Purging old IP Address data from abuse_filter_log...
857wikidatawiki: 171
858wikidatawiki: 171 rows.
859wikidatawiki: Done.
860-----------------------------------------------------------------
861zerowiki
862-----------------------------------------------------------------
863zerowiki: Purging old IP Address data from abuse_filter_log...
864zerowiki: 0 rows.
865zerowiki: Done.
866-----------------------------------------------------------------
867zhwiki
868-----------------------------------------------------------------
869zhwiki: Purging old IP Address data from abuse_filter_log...
870zhwiki: 75
871zhwiki: 75 rows.
872zhwiki: Done.

However we should either disabling afl_ip logging on Beta Cluster, or restrict who can access that info or both. In any case, a puppet cron should be set there to do this regularly.

Huji added a comment.Feb 12 2018, 7:53 PM

@MarcoAurelio what was the total execution time? Total row count is 128K

@MarcoAurelio what was the total execution time? Total row count is 128K

I don't know. Not more than two minutes probably. It was pretty fast.

That said, on WMF production it has 2 long years of cleaning to do after we fixed the $this->requireExtension thing. If we were to run this manually, I'd say to run it on terbium since it'll be a long-running script. However the cron job should be fixed so it should run today at 01:15 UTC. I'll ask around and see if, at least for this round, they could keep the logs for doublechecking.

Change 410072 had a related patch set uploaded (by MarcoAurelio; owner: MarcoAurelio):
[operations/puppet@production] mediawiki: log next run of purge_abusefilter.pp

https://gerrit.wikimedia.org/r/410072

Change 410072 merged by Dzahn:
[operations/puppet@production] mediawiki: log next run of purge_abusefilter.pp

https://gerrit.wikimedia.org/r/410072

MarcoAurelio closed this task as Resolved.Feb 12 2018, 11:09 PM

Well, the data is purged. I'll create another task so this script can run croned on the beta cluster puppet.