Page MenuHomePhabricator

Run the maintenance script cleanupTitles.php on all wikis to rescue currently-inaccessible pages
Open, Needs TriagePublic

Description

MediaWiki 1.32.0-wmf.5 includes two changes which disallow some new characters in page titles:

Existing pages with titles containing them will become inaccessible (except by page ID, e.g. https://en.wikipedia.org/?curid=8091). I don't know how many pages with directional formatting characters are out there, but for soft hyphens there were 2322 when I last checked (T121979#3923914).

Someone needs to run cleanupTitles.php to fix them. But please do a dry run first (php maintenance/cleanupTitles.php --dry-run) and post the results here. As far as I know the script is not run regularly, so there might be other broken titles in the database already (I think it's likely we have some broken due to Unicode normalization changes after we upgraded ICU recently). It would be good to know what we're dealing with first.

The script moves inaccessible pages to titles with the invalid characters removed, or if that is impossible (such page already exists, or title is still invalid), to page titles beginning with "Broken/". [It doesn't properly move them (with log entries etc.), it just updates some database fields, so there will be no on-wiki record of this having been done.] We should advise users to move them to proper titles afterwards (maybe in Tech News or separate announcement).

Event Timeline

Reedy renamed this task from Run the maintenance script cleanupTItles.php (in dry run mode first) on all wikis after wmf.5 deployment to Run the maintenance script cleanupTitles.php (in dry run mode first) on all wikis after wmf.5 deployment.May 25 2018, 10:54 AM

@matmarex so, where do you want you want your log file? ;)

reedy@terbium:~$ du -h T195546.log 
478M	T195546.log
reedy@terbium:~$ grep Finished T195546.log | grep -v "0 of"
angwiki:  Finished page... 12 of 14260 rows updated
arwiki:  Finished page... 12 of 3567239 rows updated
arwikiversity:  Finished page... 5 of 19923 rows updated
astwiki:  Finished page... 1 of 119811 rows updated
avwiki:  Finished page... 59 of 9031 rows updated
azwikibooks:  Finished page... 1 of 6438 rows updated
azwikiquote:  Finished page... 4 of 4601 rows updated
bewikimedia:  Finished page... 3 of 1839 rows updated
bhwiki:  Finished page... 2 of 54204 rows updated
bnwikisource:  Finished page... 5 of 737660 rows updated
brwiktionary:  Finished page... 1 of 45976 rows updated
cawiki:  Finished page... 15 of 1439661 rows updated
collabwiki:  Finished page... 1 of 10690 rows updated
commonswiki:  Finished page... 74 of 63759547 rows updated
cswikiquote:  Finished page... 3 of 12569 rows updated
cswiktionary:  Finished page... 3 of 127568 rows updated
cywiki:  Finished page... 2 of 215765 rows updated
dawikiquote:  Finished page... 2 of 1966 rows updated
dewikivoyage:  Finished page... 3 of 72360 rows updated
dvwiki:  Finished page... 4 of 10489 rows updated
elwiki:  Finished page... 95 of 448747 rows updated
elwikisource:  Finished page... 3 of 26549 rows updated
enwiki:  Finished page... 6 of 45038995 rows updated
enwikibooks:  Finished page... 57 of 250183 rows updated
enwikisource:  Finished page... 19 of 2361671 rows updated
enwikiversity:  Finished page... 8 of 176349 rows updated
enwiktionary:  Finished page... 2 of 6178600 rows updated
eowikisource:  Finished page... 2 of 7450 rows updated
eowiktionary:  Finished page... 2 of 114914 rows updated
eswiki:  Finished page... 12 of 6333000 rows updated
eswikinews:  Finished page... 1 of 42053 rows updated
eswikiquote:  Finished page... 3 of 28709 rows updated
eswikisource:  Finished page... 4 of 217069 rows updated
eswiktionary:  Finished page... 2 of 902429 rows updated
etwiktionary:  Finished page... 48 of 146404 rows updated
euwiki:  Finished page... 1 of 718810 rows updated
fawikibooks:  Finished page... 1 of 17537 rows updated
fiwikibooks:  Finished page... 1 of 11207 rows updated
fiwiktionary:  Finished page... 27 of 428375 rows updated
fjwiktionary:  Finished page... 1 of 32075 rows updated
frwiki:  Finished page... 48 of 9477314 rows updated
frwikisource:  Finished page... 2 of 2613672 rows updated
frwiktionary:  Finished page... 19 of 3575998 rows updated
ganwiki:  Finished page... 1 of 33200 rows updated
gawikibooks:  Finished page... 1 of 70 rows updated
gawikiquote:  Finished page... 2 of 83 rows updated
gawiktionary:  Finished page... 2 of 5661 rows updated
guwikiquote:  Finished page... 26 of 2269 rows updated
hewiki:  Finished page... 26 of 936381 rows updated
hiwikibooks:  Finished page... 8 of 3148 rows updated
hrwikisource:  Finished page... 1 of 14284 rows updated
huwikibooks:  Finished page... 1 of 71922 rows updated
huwiktionary:  Finished page... 12 of 405988 rows updated
hywiktionary:  Finished page... 2 of 249914 rows updated
idwiki:  Finished page... 7 of 2351997 rows updated
incubatorwiki:  Finished page... 2 of 2797273 rows updated
iowiktionary:  Finished page... 3 of 312779 rows updated
itwiki:  Finished page... 5 of 5801073 rows updated
itwikiquote:  Finished page... 2 of 134562 rows updated
itwikisource:  Finished page... 2 of 489267 rows updated
itwiktionary:  Finished page... 1 of 463015 rows updated
jawikisource:  Finished page... 4 of 21837 rows updated
jbowiki:  Finished page... 1 of 5517 rows updated
kabwiki:  Finished page... 3 of 11088 rows updated
kmwiktionary:  Finished page... 2 of 41191 rows updated
kowikisource:  Finished page... 15 of 36290 rows updated
kowiktionary:  Finished page... 5 of 413034 rows updated
kuwiki:  Finished page... 1 of 59362 rows updated
kuwiktionary:  Finished page... 913 of 642594 rows updated
labswiki:  Finished page... 24 of 18972 rows updated
labtestwiki:  Finished page... 94 of 27962 rows updated
lrcwiki:  Finished page... 1 of 8895 rows updated
ltwiktionary:  Finished page... 1 of 689388 rows updated
lvwiki:  Finished page... 1 of 355248 rows updated
metawiki:  Finished page... 59 of 10440775 rows updated
mgwiktionary:  Finished page... 24 of 4728573 rows updated
mlwiki:  Finished page... 1 of 355147 rows updated
mnwiktionary:  Finished page... 1 of 46705 rows updated
mrwikibooks:  Finished page... 1 of 1020 rows updated
mywiki:  Finished page... 18 of 81108 rows updated
newiki:  Finished page... 21 of 85882 rows updated
newikibooks:  Finished page... 9 of 1964 rows updated
newiktionary:  Finished page... 1 of 15564 rows updated
nlwiktionary:  Finished page... 34 of 701738 rows updated
nostalgiawiki:  Finished page... 1 of 32219 rows updated
nowiktionary:  Finished page... 1 of 153883 rows updated
nrmwiki:  Finished page... 2 of 8798 rows updated
otrs_wikiwiki:  Finished page... 1 of 12689 rows updated
pawiki:  Finished page... 14 of 99400 rows updated
pflwiki:  Finished page... 1 of 5286 rows updated
plwikinews:  Finished page... 1 of 42064 rows updated
plwiktionary:  Finished page... 7 of 647190 rows updated
pnbwiki:  Finished page... 1 of 65939 rows updated
ptwiki:  Finished page... 12 of 4640160 rows updated
ptwikinews:  Finished page... 3 of 42893 rows updated
ptwiktionary:  Finished page... 3 of 350044 rows updated
roa_tarawiki:  Finished page... 14 of 17344 rows updated
rswikimedia:  Finished page... 1 of 1002 rows updated
ruewiki:  Finished page... 1 of 12258 rows updated
ruwikibooks:  Finished page... 9 of 13727 rows updated
ruwikinews:  Finished page... 1 of 153251 rows updated
ruwikisource:  Finished page... 2 of 782732 rows updated
ruwiktionary:  Finished page... 16 of 1332377 rows updated
sawikisource:  Finished page... 13 of 64543 rows updated
sdwiki:  Finished page... 47 of 28147 rows updated
shwiki:  Finished page... 153 of 4596306 rows updated
simplewiki:  Finished page... 2 of 461445 rows updated
skwiktionary:  Finished page... 2 of 26830 rows updated
slwikisource:  Finished page... 1 of 27879 rows updated
sourceswiki:  Finished page... 1 of 103273 rows updated
specieswiki:  Finished page... 4 of 1062244 rows updated
srwikinews:  Finished page... 1 of 124274 rows updated
svwiktionary:  Finished page... 34 of 662130 rows updated
tawiki:  Finished page... 3 of 338786 rows updated
tawikisource:  Finished page... 2 of 415135 rows updated
test2wiki:  Finished page... 46 of 40111 rows updated
tewikibooks:  Finished page... 1 of 773 rows updated
tgwiki:  Finished page... 6 of 175254 rows updated
tgwiktionary:  Finished page... 1 of 34866 rows updated
thwiktionary:  Finished page... 1 of 233314 rows updated
trwiktionary:  Finished page... 81 of 376459 rows updated
ukwiki:  Finished page... 1 of 2386886 rows updated
ukwikiquote:  Finished page... 5 of 27376 rows updated
ukwikisource:  Finished page... 4 of 38850 rows updated
ukwiktionary:  Finished page... 13 of 45323 rows updated
urwiki:  Finished page... 63 of 746927 rows updated
viwiki:  Finished page... 47 of 13285579 rows updated
wawiktionary:  Finished page... 3 of 37721 rows updated
wuuwiki:  Finished page... 2 of 19388 rows updated
yiwiktionary:  Finished page... 1 of 2543 rows updated
zh_classicalwiki:  Finished page... 6 of 75565 rows updated
zhwiki:  Finished page... 3 of 5404774 rows updated
zhwikinews:  Finished page... 4 of 25646 rows updated
zhwiktionary:  Finished page... 7 of 1342202 rows updated
reedy@terbium:~$ grep Finished T195546.log | grep -v " 0 of"
angwiki:  Finished page... 12 of 14260 rows updated
arwiki:  Finished page... 12 of 3567239 rows updated
arwikiversity:  Finished page... 5 of 19923 rows updated
astwiki:  Finished page... 1 of 119811 rows updated
avwiki:  Finished page... 59 of 9031 rows updated
azwikibooks:  Finished page... 1 of 6438 rows updated
azwikiquote:  Finished page... 4 of 4601 rows updated
bewikimedia:  Finished page... 3 of 1839 rows updated
bhwiki:  Finished page... 2 of 54204 rows updated
bnwikisource:  Finished page... 5 of 737660 rows updated
brwiktionary:  Finished page... 1 of 45976 rows updated
cawiki:  Finished page... 15 of 1439661 rows updated
collabwiki:  Finished page... 1 of 10690 rows updated
commonswiki:  Finished page... 74 of 63759547 rows updated
cswikiquote:  Finished page... 3 of 12569 rows updated
cswiktionary:  Finished page... 3 of 127568 rows updated
cywiki:  Finished page... 2 of 215765 rows updated
dawikiquote:  Finished page... 2 of 1966 rows updated
dewikivoyage:  Finished page... 3 of 72360 rows updated
diqwiki:  Finished page... 20 of 26272 rows updated
dvwiki:  Finished page... 4 of 10489 rows updated
elwiki:  Finished page... 95 of 448747 rows updated
elwikisource:  Finished page... 3 of 26549 rows updated
enwiki:  Finished page... 6 of 45038995 rows updated
enwikibooks:  Finished page... 57 of 250183 rows updated
enwikisource:  Finished page... 19 of 2361671 rows updated
enwikiversity:  Finished page... 8 of 176349 rows updated
enwiktionary:  Finished page... 2 of 6178600 rows updated
eowikisource:  Finished page... 2 of 7450 rows updated
eowiktionary:  Finished page... 2 of 114914 rows updated
eswiki:  Finished page... 12 of 6333000 rows updated
eswikinews:  Finished page... 1 of 42053 rows updated
eswikiquote:  Finished page... 3 of 28709 rows updated
eswikisource:  Finished page... 4 of 217069 rows updated
eswiktionary:  Finished page... 2 of 902429 rows updated
etwiktionary:  Finished page... 48 of 146404 rows updated
euwiki:  Finished page... 1 of 718810 rows updated
fawikibooks:  Finished page... 1 of 17537 rows updated
fiwikibooks:  Finished page... 1 of 11207 rows updated
fiwiktionary:  Finished page... 27 of 428375 rows updated
fjwiktionary:  Finished page... 1 of 32075 rows updated
frwiki:  Finished page... 48 of 9477314 rows updated
frwikisource:  Finished page... 2 of 2613672 rows updated
frwiktionary:  Finished page... 19 of 3575998 rows updated
ganwiki:  Finished page... 1 of 33200 rows updated
gawikibooks:  Finished page... 1 of 70 rows updated
gawikiquote:  Finished page... 2 of 83 rows updated
gawiktionary:  Finished page... 2 of 5661 rows updated
guwikiquote:  Finished page... 26 of 2269 rows updated
hewiki:  Finished page... 26 of 936381 rows updated
hiwiki:  Finished page... 10 of 752436 rows updated
hiwikibooks:  Finished page... 8 of 3148 rows updated
hrwikisource:  Finished page... 1 of 14284 rows updated
huwikibooks:  Finished page... 1 of 71922 rows updated
huwiktionary:  Finished page... 12 of 405988 rows updated
hywiktionary:  Finished page... 2 of 249914 rows updated
idwiki:  Finished page... 7 of 2351997 rows updated
incubatorwiki:  Finished page... 2 of 2797273 rows updated
iowiktionary:  Finished page... 3 of 312779 rows updated
itwiki:  Finished page... 5 of 5801073 rows updated
itwikiquote:  Finished page... 2 of 134562 rows updated
itwikisource:  Finished page... 2 of 489267 rows updated
itwiktionary:  Finished page... 1 of 463015 rows updated
jawikisource:  Finished page... 4 of 21837 rows updated
jbowiki:  Finished page... 1 of 5517 rows updated
kabwiki:  Finished page... 3 of 11088 rows updated
kmwiktionary:  Finished page... 2 of 41191 rows updated
kowikisource:  Finished page... 15 of 36290 rows updated
kowiktionary:  Finished page... 5 of 413034 rows updated
kuwiki:  Finished page... 1 of 59362 rows updated
kuwiktionary:  Finished page... 913 of 642594 rows updated
labswiki:  Finished page... 24 of 18972 rows updated
labtestwiki:  Finished page... 94 of 27962 rows updated
lrcwiki:  Finished page... 1 of 8895 rows updated
ltwiktionary:  Finished page... 1 of 689388 rows updated
lvwiki:  Finished page... 1 of 355248 rows updated
metawiki:  Finished page... 59 of 10440775 rows updated
mgwiktionary:  Finished page... 24 of 4728573 rows updated
mlwiki:  Finished page... 1 of 355147 rows updated
mnwiktionary:  Finished page... 1 of 46705 rows updated
mrwikibooks:  Finished page... 1 of 1020 rows updated
mywiki:  Finished page... 18 of 81108 rows updated
newiki:  Finished page... 21 of 85882 rows updated
newikibooks:  Finished page... 9 of 1964 rows updated
newiktionary:  Finished page... 1 of 15564 rows updated
nlwiktionary:  Finished page... 34 of 701738 rows updated
nostalgiawiki:  Finished page... 1 of 32219 rows updated
nowiktionary:  Finished page... 1 of 153883 rows updated
nrmwiki:  Finished page... 2 of 8798 rows updated
otrs_wikiwiki:  Finished page... 1 of 12689 rows updated
pawiki:  Finished page... 14 of 99400 rows updated
pflwiki:  Finished page... 1 of 5286 rows updated
plwikinews:  Finished page... 1 of 42064 rows updated
plwiktionary:  Finished page... 7 of 647190 rows updated
pnbwiki:  Finished page... 1 of 65939 rows updated
pswiki:  Finished page... 80 of 32429 rows updated
ptwiki:  Finished page... 12 of 4640160 rows updated
ptwikinews:  Finished page... 3 of 42893 rows updated
ptwiktionary:  Finished page... 3 of 350044 rows updated
roa_tarawiki:  Finished page... 14 of 17344 rows updated
rswikimedia:  Finished page... 1 of 1002 rows updated
ruewiki:  Finished page... 1 of 12258 rows updated
ruwikibooks:  Finished page... 9 of 13727 rows updated
ruwikinews:  Finished page... 1 of 153251 rows updated
ruwikisource:  Finished page... 2 of 782732 rows updated
ruwiktionary:  Finished page... 16 of 1332377 rows updated
sawikisource:  Finished page... 13 of 64543 rows updated
sdwiki:  Finished page... 47 of 28147 rows updated
shwiki:  Finished page... 153 of 4596306 rows updated
simplewiki:  Finished page... 2 of 461445 rows updated
skwiktionary:  Finished page... 2 of 26830 rows updated
slwikisource:  Finished page... 1 of 27879 rows updated
sourceswiki:  Finished page... 1 of 103273 rows updated
specieswiki:  Finished page... 4 of 1062244 rows updated
srwikinews:  Finished page... 1 of 124274 rows updated
svwiktionary:  Finished page... 34 of 662130 rows updated
tawiki:  Finished page... 3 of 338786 rows updated
tawikisource:  Finished page... 2 of 415135 rows updated
test2wiki:  Finished page... 46 of 40111 rows updated
tewikibooks:  Finished page... 1 of 773 rows updated
tgwiki:  Finished page... 6 of 175254 rows updated
tgwiktionary:  Finished page... 1 of 34866 rows updated
thwiktionary:  Finished page... 1 of 233314 rows updated
trwiktionary:  Finished page... 81 of 376459 rows updated
ukwiki:  Finished page... 1 of 2386886 rows updated
ukwikiquote:  Finished page... 5 of 27376 rows updated
ukwikisource:  Finished page... 4 of 38850 rows updated
ukwiktionary:  Finished page... 13 of 45323 rows updated
urwiki:  Finished page... 63 of 746927 rows updated
viwiki:  Finished page... 47 of 13285579 rows updated
wawiktionary:  Finished page... 3 of 37721 rows updated
wuuwiki:  Finished page... 2 of 19388 rows updated
yiwiktionary:  Finished page... 1 of 2543 rows updated
zh_classicalwiki:  Finished page... 6 of 75565 rows updated
zhwiki:  Finished page... 3 of 5404774 rows updated
zhwikinews:  Finished page... 4 of 25646 rows updated
zhwiktionary:  Finished page... 7 of 1342202 rows updated
reedy@terbium:~$

I think the results are unacceptably bad in some cases, and we should not run the script before it is improved:

frwiki:  DRY RUN: would rename 1547147 (3,'195.175.037.8') to (3,'Broken/195.175.37.8')

Should rename to (0,'Broken/User:195.175.37.8') (or something), in main namespace, so that we can easily produce a list of affected pages (i.e. Special:PrefixIndex/Broken/).

test2wiki:  DRY RUN: would rename 2987 (447,'Columbia_University') to (0,'Columbia_University')

Should rename to (0,'Broken/NS447:Columbia_University') (or something), so that someone can find out the original namespace (in this case, EducationProgram).

bnwikisource:  DRY RUN: would rename 1251 (0,'WS:COPY') to (0,'Broken/COPY')

Where did the WS: prefix go?

avwiki:  DRY RUN: would rename 7541 (1,'Википедия:Requests_for_adminship') to (1,'Broken/\xd0\x92\xd0\xb8\xd0\xba\xd0\xb8\xd0\xbf\xd0\xb5\xd0\xb4\xd0\xb8\xd1\x8f\x3aRequests_for_adminship')

It's not acceptable to mangle perfectly normal characters (but not ASCII) like this.


Also, I think the prefix Broken/ is also unhelpful (and not translateable). The script should take an option to specify a prefix, so that we can e.g. set it to T195546/ or maybe 2018-06-29/ or some other vaguely useful thing that can be used to find out where that page suddenly appeared from.

I don't know when or if I will have the time to work on this. I have other work as a WMF employee and this seems to be a very unfun thing to work on in my free time.

I don't know when or if I will have the time to work on this. I have other work as a WMF employee and this seems to be a very unfun thing to work on in my free time.

Without knowing the back story here: is there someone else you recommend to take this up? This seems like a high priority issue that any PM would OK the work of right now.

In the interim, does this block the rollout of wmf.6? wmf.5 is already rolled out and it doesn't seem like wmf.6 changes the current status of this task. Is that correct?

Two distinct things – the code we rolled out in wmf.5 should probably be reverted, and cleanupTitles.php should be fixed up to not be terrible and run in production for the first time in years, given there are thousands of inaccessible pages right now. I'll make this task about the latter; it has nothing to do with the train, worry not.

Jdforrester-WMF renamed this task from Run the maintenance script cleanupTitles.php (in dry run mode first) on all wikis after wmf.5 deployment to Run the maintenance script cleanupTitles.php on all wikis to rescue currently-inaccessible pages.May 31 2018, 5:31 PM
enwiki:  DRY RUN: would rename 11429643 (3,'195.175.037.008') to (3,'Broken/195.175.37.8')

That is T180996

Two distinct things – the code we rolled out in wmf.5 should probably be reverted, …

Done in https://gerrit.wikimedia.org/r/#/q/project:mediawiki/core+(topic:strip-shy+OR+topic:strip-shy-wmf/1.32.0-wmf.6) (still awaiting wmf.6 deployment).