Page MenuHomePhabricator

ProofreadPage: improve performance of Index stats Lua
Closed, ResolvedPublic

Description

The index stats Lua interface is very slow: ~50 indexes in the enWS Monthly Challenge are hitting the script timeout limits occasionally.

Interestingly, the culprit is not the DB lookups of the stats (which are fast, due to being cached in the pr_index table), it's the template dependency on every page in the index:

69.39% 11334.745    163 - Scribunto_LuaSandboxCallback::__call@1
68.20% 11140.528     13 - ProofreadPage\ProofreadPageLuaLibrary::doGetIndexProgress
68.06% 11118.235     13 - ProofreadPage\ProofreadPageLuaLibrary::addTemplateDependencyOnAllPagesInIndex
65.39% 10680.764   6475 - ProofreadPage\ProofreadPageLuaLibrary::addTemplateDependencyOnPage
62.97% 10286.126  14108 - Title::getArticleID
58.04% 9480.184   9218 - LinkCache::addLinkObj
57.52% 9395.358   9218 - LinkCache::getGoodLinkRow
53.44% 8728.561   9218 - LinkCache::getGoodLinkRowInternal
44.32% 7239.807   6941 - Wikimedia\Rdbms\DBConnRef::__call
43.43% 7094.190   6741 - Wikimedia\Rdbms\Database::select
42.84% 6997.686   6514 - Wikimedia\Rdbms\DBConnRef::selectRow
42.68% 6972.202   6453 - LinkCache::fetchPageRow
42.62% 6961.912   6528 - Wikimedia\Rdbms\Database::selectRow
38.59% 6303.721   6746 - Wikimedia\Rdbms\Database::query
38.09% 6222.709   6746 - Wikimedia\Rdbms\Database::executeQuery
34.17% 5581.356   6746 - Wikimedia\Rdbms\Database::executeQueryAttempt
28.22% 4609.597   6453 - section.query-m: SELECT page_id,page_len,page_is_redirect,page_latest,page_restrictions,page_content_model FROM `page` WHERE page_namespace = N AND page_title = 'X' LIMIT N  [TRX#b3d87ae539]

We should see if we can speed this up a bit (or a lot!)

Event Timeline

Change 740216 had a related patch set uploaded (by Inductiveload; author: Inductiveload):

[mediawiki/extensions/ProofreadPage@master] Lua: use LinkBatch to speed up the template dependencies

https://gerrit.wikimedia.org/r/740216

Since it's occasionally hitting a useful enWS page

Change 740216 merged by jenkins-bot:

[mediawiki/extensions/ProofreadPage@master] Lua: use LinkBatch to speed up the template dependencies

https://gerrit.wikimedia.org/r/740216

Change 740556 had a related patch set uploaded (by Inductiveload; author: Inductiveload):

[mediawiki/extensions/ProofreadPage@wmf/1.38.0-wmf.9] Lua: use LinkBatch to speed up the template dependencies

https://gerrit.wikimedia.org/r/740556

Change 740558 had a related patch set uploaded (by Inductiveload; author: Inductiveload):

[mediawiki/extensions/ProofreadPage@wmf/1.38.0-wmf.7] Lua: use LinkBatch to speed up the template dependencies

https://gerrit.wikimedia.org/r/740558

Change 740556 merged by jenkins-bot:

[mediawiki/extensions/ProofreadPage@wmf/1.38.0-wmf.9] Lua: use LinkBatch to speed up the template dependencies

https://gerrit.wikimedia.org/r/740556

Change 740558 merged by jenkins-bot:

[mediawiki/extensions/ProofreadPage@wmf/1.38.0-wmf.7] Lua: use LinkBatch to speed up the template dependencies

https://gerrit.wikimedia.org/r/740558

Mentioned in SAL (#wikimedia-operations) [2021-11-22T12:45:48Z] <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.38.0-wmf.7/extensions/ProofreadPage/includes/Pagination/Pagination.php: Backport: [[gerrit:740558|Lua: use LinkBatch to speed up the template dependencies (T296092)]] (1/2) (duration: 01m 04s)

Mentioned in SAL (#wikimedia-operations) [2021-11-22T12:47:16Z] <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.38.0-wmf.7/extensions/ProofreadPage/includes/ProofreadPageLuaLibrary.php: Backport: [[gerrit:740558|Lua: use LinkBatch to speed up the template dependencies (T296092)]] (2/2) (duration: 01m 03s)

Mentioned in SAL (#wikimedia-operations) [2021-11-22T12:49:43Z] <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/ProofreadPage/includes/Pagination/Pagination.php: Backport: [[gerrit:740556|Lua: use LinkBatch to speed up the template dependencies (T296092)]] (1/2) (duration: 01m 04s)

Mentioned in SAL (#wikimedia-operations) [2021-11-22T12:51:03Z] <lucaswerkmeister-wmde@deploy1002> Synchronized php-1.38.0-wmf.9/extensions/ProofreadPage/includes/ProofreadPageLuaLibrary.php: Backport: [[gerrit:740556|Lua: use LinkBatch to speed up the template dependencies (T296092)]] (2/2) (duration: 01m 03s)

Inductiveload claimed this task.
Inductiveload moved this task from Top priority to Done: to deploy/check on the ProofreadPage board.

Fix was deployed and affected pages are much faster now.