Due to a race condition, production databases have some duplicate block_target rows, mostly with bt_user pointing to the same user.
Rectifying the race condition is tracked by T389028 and will take some time. In the meantime, periodically running cleanupBlocks.php will convert duplicate block_target rows to normal multiblocks, which can be deleted with the API or the Codex Special:Block form.
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| block: Add cleanupBlocks.php | mediawiki/core | master | +455 -0 |
Change #1128063 had a related patch set uploaded (by Tim Starling; author: Tim Starling):
[mediawiki/core@master] block: Add cleanupBlocks.php
Change #1128063 merged by jenkins-bot:
[mediawiki/core@master] block: Add cleanupBlocks.php
This task documents authoring and running a maintenance script, which I don't think is QA'able.
@tstarling @MusikAnimal I am ready to resolve this task, but should I keep it open until we have run the script (if we have not already done so).
I randomly generated various duplicate and non-normalised entries in block_target. I then ran the cleanupBlocks script and compared the data before and after.
The blocks before all remain after the script is ran, with the same parameters and values for bt_address or bt_user and bt_user_text. The bt_count appears to be updated correctly as far as I could calculate.
The script appears to merge together rows in block_target which are exact duplicates for either bt_address (for IPs/ranges) or bt_user/bt_user_text (for accounts).
It does not merge rows which are duplicated due to bt_address not being normalised. So, for example, 100.113.193.178/31 and 100.113.193.179/31 remain as two separate rows, despite them referring to the same IP range. I am guessing we might do cleanup for these cases as part of T389350 (or related tasks).
Test environment: local docker MediaWiki 1.44.0-alpha (af9a1ec) 07:01, 31 March 2025 (latest tested version).
I ran the script.
for w in $(expanddblist all.dblist); do mwscript cleanupBlocks.php --wiki=$w |& sed "s/^/$w /" | tee -a cleanup-2025-04-04.log; done commonswiki Merging bt_id 115214 into 115213: done commonswiki Merging bt_id 122730 into 122728: done dewiki Merging bt_id 193282 into 193281: done dewiki Merging bt_id 193097 into 193096: done dewiki Merging bt_id 198603 into 198601: done dewiki Merging bt_id 198582 into 198581: done dewiki Merging bt_id 198126 into 198125: done dewiki Merging bt_id 198129 into 198128: done dewiki Merging bt_id 202469 into 202468: done dewiki Merging bt_id 202466 into 202465: done dewiki Merging bt_id 202635 into 202634: done dewiki Merging bt_id 218167 into 218166: done enwiki Deleting orphan bt_id=1732978: OK enwiki Merging bt_id 2090152 into 2090149: done enwiki Merging bt_id 2094547 into 2094546: done enwiki Merging bt_id 1962597 into 1962595: done enwiki Merging bt_id 2036243 into 2036238: done enwiki Merging bt_id 2036236 into 2036234: done enwiki Merging bt_id 2031781 into 2031776: done enwiki Merging bt_id 2048729 into 2048728: done enwiki Merging bt_id 2053675 into 2053674: done enwiki Merging bt_id 2049731 into 2049730: done enwiki Merging bt_id 2070290 into 2070289: done enwiki Merging bt_id 2090153 into 2090148: done enwiki Merging bt_id 2062040 into 2062038: done enwiki Merging bt_id 2065414 into 2065412: done enwiki Merging bt_id 2070873 into 2070872: done enwiki Merging bt_id 2088816 into 2088814: done enwiki Merging bt_id 2093178 into 2093175: done enwiki Merging bt_id 2093179 into 2093176: done enwiki Merging bt_id 2108446 into 2108445: done enwiki Merging bt_id 2130690 into 2130686: done enwiki Merging bt_id 2059860 into 2059859: done enwiki Merging bt_id 2141778 into 2141768: done enwikibooks Deleting orphan bt_id=7215: OK enwikiquote Deleting orphan bt_id=8222: OK eswiki Deleting orphan bt_id=337505: OK eswiki Merging bt_id 568025 into 568018: done eswiki Merging bt_id 568021 into 568014: done eswiki Merging bt_id 575183 into 575173: done eswiki Merging bt_id 547204 into 547199: done fawiki Deleting orphan bt_id=72525: OK fiwiki Deleting orphan bt_id=11650: OK frwiki Deleting orphan bt_id=108535: OK hewiki Merging bt_id 24771 into 24769: done iowiki Deleting orphan bt_id=321: OK itwiki Deleting orphan bt_id=93258: OK itwiki Merging bt_id 161972 into 161971: done itwiki Merging bt_id 161980,161981 into 161979: done itwiki Merging bt_id 151211 into 151203: done itwiki Merging bt_id 151212 into 151205: done itwikibooks Deleting orphan bt_id=529: OK itwiktionary Deleting orphan bt_id=351: OK jawiki Deleting orphan bt_id=165772: OK jawiki Merging bt_id 171773,171774 into 171772: done jawiki Merging bt_id 171770 into 171769: done jawiki Merging bt_id 171755,171756 into 171754: done jawiki Merging bt_id 171767,171768 into 171766: done jawiki Merging bt_id 171760 into 171759: done jawiki Merging bt_id 171764,171765 into 171763: done jawiki Merging bt_id 189125 into 189123: done jawikibooks Deleting orphan bt_id=4595: OK jawikinews Deleting orphan bt_id=3898: OK jawikiquote Deleting orphan bt_id=813: OK jawikiversity Deleting orphan bt_id=2685: OK kowiki Deleting orphan bt_id=25389: OK kowiki Merging bt_id 41453 into 41450: done lnwiktionary Deleting orphan bt_id=471: OK mediawikiwiki Deleting orphan bt_id=43286: OK metawiki Deleting orphan bt_id=88787: OK metawiki Merging bt_id 168736,168737,168738,168739,168740,168741,168742 into 168735: done metawiki Merging bt_id 168728,168729,168730,168731,168732,168733,168734 into 168727: done metawiki Merging bt_id 167543,167545,167547,167549,167551,167553 into 167541: done metawiki Merging bt_id 167557,167559,167561,167563,167565,167567,167569 into 167555: done metawiki Merging bt_id 167574,167576,167578,167580,167582,167584,167586 into 167572: done metawiki Merging bt_id 167611,167613,167615,167617,167619,167621,167623 into 167609: done metawiki Merging bt_id 168035,168036,168037,168038,168039,168040,168041 into 168034: done metawiki Merging bt_id 168096,168098,168100,168102,168104,168106,168108 into 168094: done metawiki Merging bt_id 168768,168770,168772,168774,168776,168778,168780 into 168766: done metawiki Merging bt_id 168822,168823,168824,168825,168826,168827,168828 into 168821: done metawiki Merging bt_id 168832,168833,168834,168835,168836,168837,168838 into 168831: done metawiki Merging bt_id 169745,169747,169749,169751,169753,169755 into 169742: done metawiki Merging bt_id 170610,170611,170612,170613,170614,170615,170616 into 170609: done metawiki Merging bt_id 170694,170696,170698,170700,170702,170704,170706 into 170692: done metawiki Merging bt_id 170986,170987,170988,170989,170990,170991 into 170985: done metawiki Merging bt_id 171022,171024,171026,171028,171030,171032,171034 into 171020: done metawiki Merging bt_id 171046,171048,171050,171052,171054,171056,171058 into 171044: done metawiki Merging bt_id 171066,171068,171071,171073,171075,171077,171079 into 171064: done metawiki Merging bt_id 171090,171091,171092,171093,171094,171095,171096 into 171089: done metawiki Merging bt_id 171384,171387,171389,171391,171393,171395 into 171382: done metawiki Merging bt_id 172144,172145,172146,172147,172148,172149,172150 into 172143: done metawiki Merging bt_id 172297,172299,172302,172304,172306,172308,172310,172312 into 172295: done metawiki Merging bt_id 172525,172527,172529,172531,172533,172535,172537 into 172523: done metawiki Merging bt_id 173730,173732,173734,173736,173738,173740,173742 into 173728: done metawiki Merging bt_id 173746,173748,173750,173752,173754,173756,173758 into 173744: done metawiki Merging bt_id 173812,173814,173816,173818,173820,173822,173824 into 173810: done metawiki Merging bt_id 175445,175447,175449,175451,175453 into 175443: done metawiki Merging bt_id 175457,175459,175461,175463,175465,175467,175469 into 175455: done metawiki Merging bt_id 175586,175588,175590,175592,175594,175596,175598 into 175584: done metawiki Merging bt_id 175955,175957,175959,175961,175963,175965,175967 into 175953: done metawiki Merging bt_id 175458,175460,175462,175464,175466,175468,175470 into 175456: done metawiki Merging bt_id 175446,175448,175450,175452,175454 into 175444: done metawiki Merging bt_id 175956,175958,175960,175962,175964,175966,175968 into 175954: done metawiki Merging bt_id 175587,175589,175591,175593,175595,175597,175599 into 175585: done miwiktionary Deleting orphan bt_id=579: OK nycwikimedia Merging bt_id 199,200 into 198: done plwiki Merging bt_id 23060 into 23058: done plwiki Merging bt_id 23332,23334 into 23330: done plwiki Merging bt_id 23447 into 23445: done plwiki Merging bt_id 23973 into 23971: done plwiki Merging bt_id 24009 into 24007: done plwiki Merging bt_id 24052 into 24050: done plwiki Merging bt_id 24680 into 24678: done plwiki Merging bt_id 25321,25323 into 25319: done plwiki Merging bt_id 26178 into 26176: done plwiki Merging bt_id 28090 into 28088: done plwiki Merging bt_id 31479 into 31478: done ptwiki Deleting orphan bt_id=89751: OK ptwiki Merging bt_id 101930 into 101929: done ptwiki Merging bt_id 92773 into 92771: done ptwiki Merging bt_id 92777 into 92775: done ruwiki Deleting orphan bt_id=3368083: OK ruwiki Merging bt_id 4380732 into 4380657: done ruwikiquote Deleting orphan bt_id=3251144: OK ruwiktionary Deleting orphan bt_id=2284: OK rwwiktionary Deleting orphan bt_id=86: OK slwiki Deleting orphan bt_id=1718: OK svwiki Deleting orphan bt_id=29982: OK svwikisource Deleting orphan bt_id=513: OK testwiki Merging bt_id 1021,1023,1025,1027,1029,1031,1033,1035,1037,1039,1041 into 1019: done testwiki Merging bt_id 1022,1024,1026,1028,1030,1032,1034,1036,1038,1040,1042 into 1020: done trwiki Deleting orphan bt_id=30993: OK ukwiki Deleting orphan bt_id=10088: OK viwiki Deleting orphan bt_id=69986: OK wikidatawiki Deleting orphan bt_id=17910: OK zhwiki Deleting orphan bt_id=77215: OK zhwiki Merging bt_id 87169 into 87167: done zhwiki Merging bt_id 95599,95601 into 95597: done zhwiki Merging bt_id 164577 into 164575: done
It will need to be run at least one more time since the bug which causes these duplicate block_target rows is not fixed.
Mentioned in SAL (#wikimedia-operations) [2025-04-22T23:56:41Z] <TimStarling> running cleanupBlocks.php on all wikis T389301
for w in $(expanddblist all.dblist); do mwscript cleanupBlocks.php --wiki=$w |& sed "s/^/$w /" | tee -a cleanup-2025-04-23.log; done metawiki Merging bt_id 177016,177017,177018,177019 into 177015: done metawiki Merging bt_id 177045,177047,177049,177051,177053,177055,177057 into 177043: done metawiki Merging bt_id 177284,177286,177288,177290,177292,177294,177296 into 177282: done metawiki Merging bt_id 179618,179619,179620,179621,179622,179623,179624 into 179617: done metawiki Merging bt_id 179823,179825,179827,179829,179831,179833,179835 into 179821: done metawiki Merging bt_id 183307,183308,183309,183310,183311,183312,183313 into 183306: done metawiki Merging bt_id 183327,183329,183331,183333,183335,183337,183339 into 183325: done metawiki Merging bt_id 183353,183355,183357,183359,183361,183363,183365 into 183351: done
@tstarling Will this script need to be run again? If not, I am happy for you to resolve this task.
Mentioned in SAL (#wikimedia-operations) [2025-06-12T04:27:00Z] <TimStarling> ran cleanupBlocks.php on all wikis for T373847 and T389301