Special pages that took > 1 hour run script was running on codfw (Note: script is currently at frwiki, so this only includes things alphabetically before frwiki):
bawolff@mwmaint2001:~$ grep 'in.*[0-9]h' /var/log/mediawiki/updateSpecialPages.log | head -50 commonswiki: ListDuplicatedFiles [QueryPage] got 3554 rows in 2h 44m 37.65s commonswiki: MediaStatistics [QueryPage] got 20 rows in 1h 22m 7.48s commonswiki: Mostcategories [QueryPage] got 5000 rows in 7h 34m 9.54s commonswiki: Mostlinkedtemplates [QueryPage] got 5000 rows in 4h 48m 47.34s commonswiki: Uncategorizedcategories [QueryPage] got 154 rows in 3h 51m 43.43s commonswiki: Uncategorizedpages [QueryPage] got 142 rows in 6h 37m 22.52s commonswiki: Unusedimages [QueryPage] got 5 rows in 2h 15m 47.26s commonswiki: Wantedfiles [QueryPage] got 5000 rows in 2h 14m 2.39s commonswiki: Wantedtemplates [QueryPage] got 5000 rows in 3h 5m 47.84s commonswiki: GadgetUsage [QueryPage] got 126 rows in 1h 57m 42.65s commonswiki: MostGloballyLinkedFiles [QueryPage] got 5000 rows in 2h 43m 38.13s commonswiki: GloballyWantedFiles [QueryPage] got 5000 rows in 4h 3m 38.47s commonswiki: GloballyUnusedFiles [QueryPage] got 10 rows in 9h 45.37s enwiki: Uncategorizedpages [QueryPage] got 129 rows in 2h 47.81s
For comparison, from sept 10 log file on mwmaint1001 there are no special pages that took more than an hour. The one's that took more than 10 minutes are:
bawolff@mwmaint1001:~$ grep 'in.*[0-9][0-9]m' /var/log/mediawiki/updateSpecialPages.log | head -50 cebwiki: Mostcategories [QueryPage] got 5000 rows in 12m 55.93s cebwiki: Mostlinkedtemplates [QueryPage] got 4688 rows in 14m 17.10s cebwiki: Wantedtemplates [QueryPage] got 697 rows in 12m 44.98s commonswiki: ListDuplicatedFiles [QueryPage] got 5000 rows in 30m 27.59s commonswiki: Mostlinkedtemplates [QueryPage] got 5000 rows in 37m 12.61s commonswiki: Wantedfiles [QueryPage] got 5000 rows in 11m 0.70s commonswiki: GadgetUsage [QueryPage] got 125 rows in 26m 12.58s commonswiki: MostGloballyLinkedFiles [QueryPage] got 5000 rows in 23m 18.99s commonswiki: GloballyWantedFiles [QueryPage] got 5000 rows in 15m 15.49s dewiki: GadgetUsage [QueryPage] got 56 rows in 14m 0.42s eswiki: GadgetUsage [QueryPage] got 96 rows in 15m 26.02s frwiki: GadgetUsage [QueryPage] got 120 rows in 11m 39.93s ruwiki: GadgetUsage [QueryPage] got 103 rows in 10m 21.76s
As a result, the script seems to be taking > 72 hours, where users expect it to take less than 72 hours (As we run it as a cron job every 3 days).
The obvious difference would be that during switchover the script is hitting db2058 instead of db1121.
WP:VPT thread at https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_(technical)&oldid=863329531#Special:WantedCategories_updates