[17:06:23] <Reedy> andre__: malafaya It's fine on any small wikis
[17:06:39] <Reedy> On "non small" wikis
[17:06:49] <Reedy> It's dying at Mostcategories on frwiki

reedy@fenari:/home/wikipedia/logs/norotate$ ls -al updateSpecialPages*
-rw-r--r-- 1 apache wikidev 433907 Jan 25 16:54 updateSpecialPages.log
-rw-r--r-- 1 apache wikidev 909900 Jan 25 04:54 updateSpecialPages-small.log

frwiki
Statistics completed in 7.76s
Ancientpages disabled
BrokenRedirects got 22 rows in 41.89s
Deadendpages disabled
Disambiguations got 5000 rows in 7m 43.16s
DoubleRedirects got 22 rows in 9.37s
FileDuplicateSearch cheap, skipped
LinkSearch cheap, skipped
Listredirects got 5000 rows in 0.68s
Lonelypages got 1865 rows in 18m 59.30s
Longpages cheap, skipped
MIMEsearch got 0 rows in 0.00s
Mostcategories reedy@fenari:/home/wikipedia/logs/norotate$

Trying to find a pattern, and failing:
Updated today:

https://pt.wikiversity.org/wiki/Especial:Categorias_pedidas

Outdated:

https://fr.wikiversity.org/wiki/Sp%C3%A9cial:Cat%C3%A9gories_demand%C3%A9es
https://fr.wikipedia.org/wiki/Sp%C3%A9cial:Cat%C3%A9gories_demand%C3%A9es
https://fr.wiktionary.org/wiki/Sp%C3%A9cial:Cat%C3%A9gories_demand%C3%A9es
https://pt.wiktionary.org/wiki/Especial:Categorias_pedidas
https://pt.wikipedia.org/wiki/Especial:Categorias_pedidas

Anything after frwiki won't have been updated. Unless said wiki is classed as small. https://noc.wikimedia.org/conf/highlight.php?file=small.dblist

Seems frwiki has got rather large https://gerrit.wikimedia.org/r/45783

(In reply to comment #3)

Seems frwiki has got rather large https://gerrit.wikimedia.org/r/45783

I understand this as a stopgap measure, but e.g. Lonelypages was running nicely and is very useful; when it's confirmed that fr.wiki was the obstacle, can you please re-enable the special pages which are not causing problems?

(In reply to comment #4)

(In reply to comment #3)

Seems frwiki has got rather large https://gerrit.wikimedia.org/r/45783

I understand this as a stopgap measure, but e.g. Lonelypages was running
nicely
and is very useful; when it's confirmed that fr.wiki was the obstacle, can
you
please re-enable the special pages which are not causing problems?

Indeed, it's worth investigating. Like I said on IRC, on a later run, it completed the mostcategories query ran very quickly. I'm not sure if this was because it was mostly cached at the database level.

For reference, the logs suggest the updates ran to completion last night.

I'll change the config back again for frwiki and try to run them all (for frwiki only) and see how things progress.

reedy@fenari:/home/wikipedia/common$ mwscript updateSpecialPages.php frwiki
Statistics completed in 10.26s
Ancientpages disabled
BrokenRedirects got 7 rows in 1m 21.15s
Deadendpages disabled
Disambiguations got 5000 rows in 4m 1.34s
DoubleRedirects got 49 rows in 9.81s
FileDuplicateSearch cheap, skipped
LinkSearch cheap, skipped
Listredirects got 5000 rows in 0.56s
Lonelypages got 1879 rows in 10m 58.88s
Longpages cheap, skipped
MIMEsearch got 0 rows in 0.00s
Mostcategories got 5000 rows in 19m 52.55s
Mostimages got 5000 rows in 2m 30.26s
Mostinterwikis got 5000 rows in 6m 51.26s
Mostlinkedcategories cheap, skipped
Mostlinkedtemplates got 5000 rows in 39m 13.98s
Mostlinked disabled
Mostrevisions disabled
Fewestrevisions disabled
Shortpages cheap, skipped
Uncategorizedcategories got 78 rows in 9.74s
Uncategorizedpages got 57 rows in 15.35s
Uncategorizedimages got 4 rows in 1.23s
Uncategorizedtemplates got 3289 rows in 3.71s
Unusedcategories got 3507 rows in 1m 48.84s
Unusedimages got 5 rows in 0.59s
Wantedcategories got 3980 rows in 3m 52.10s
Wantedfiles got 5000 rows in 41.87s
Wantedpages disabled
Wantedtemplates got 5000 rows in 52.79s
Unwatchedpages got 5000 rows in 38.49s
Unusedtemplates got 4710 rows in 1.75s
Withoutinterwiki got 5000 rows in 8.01s

I'll revert the frwiki change and see how things go. Interesting it broke for like 4 or 5 days in a row

Stopped again last night on frwiki at Mostcategories

Mostcategories disabled for frwiki in https://gerrit.wikimedia.org/r/#/c/46262/

Ran to completion last night. I'm going to close this bug.

If mostcategories being disabled on frwiki is an issue for anyone, please file a separate bug for it, and link to this one.