Description
Details
Related Objects
- Mentioned In
- T293954: 1.38.0-wmf.13 deployment blockers
T297132: DiscussionTools is making duplicate DB requests back to back
T297517: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - Mentioned Here
- T296063: 4x increase in database queries after deploy of 1.38.0-wmf.9 to all wikis
T297132: DiscussionTools is making duplicate DB requests back to back
T297667: mysqli/mysqlnd memory leak
Event Timeline
Change 746920 had a related patch set uploaded (by Ladsgroup; author: Esanders):
[mediawiki/extensions/DiscussionTools@wmf/1.38.0-wmf.12] Cache page properties in memory to avoid extra queries
I'm fast-tracking several changes that reduce the load and due to T297667 any sort of increase in db load would increase in memory usage.
Change 746920 merged by jenkins-bot:
[mediawiki/extensions/DiscussionTools@wmf/1.38.0-wmf.12] Cache page properties in memory to avoid extra queries
We can't leave it like this during Christmas. I'm mostly mitigating it, haven't looked it what has actually increased.
Mentioned in SAL (#wikimedia-operations) [2021-12-14T05:09:05Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.12/extensions/DiscussionTools/includes/Hooks/HookUtils.php: Backport: [[gerrit:746920|Cache page properties in memory to avoid extra queries (T297132 T297669)]] (duration: 00m 57s)
This probably has something to do with parsing because basically every section except wikidata has seen a massive increase:
https://grafana.wikimedia.org/d/000000278/mysql-aggregated?viewPanel=1&orgId=1&from=1639385882027&to=1639450654650
Wikidata being massive actually is hiding the problem by reducing impact of this bug in total numbers.
It is possible that T296063: 4x increase in database queries after deploy of 1.38.0-wmf.9 to all wikis is not fixed properly yet (in wmf.9 we reverted the patch, in wmf.12 the refactor with its fixes went live). cc @daniel
Agree - if we cannot find the issue before the code freeze, this would need to be reverted
I'm looking at the queries in wmf.12 and compare it with an earlier version. It looks like Wikipage::getPageData completely lost any in-process cache it has:
Shouldn't we avoid using this function completely? I dig.
aand a lot of this:
But in previous systems, it was batched:
The funny thing is that the new one also queries batched as well:
(and then again individually)
With this, the problem is this: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/699067 that we reverted in wmf.9.
Aha, Those were diffefrent queries, the ones that are not batched now are actually categories. Which is from OutputPage::addCategoryLinksToLBAndGetResult I think I'm close to fixing this.
Change 747043 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[mediawiki/core@master] Reuse the query result in addCategoryLinks instead of relying on cache
Change 747045 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[mediawiki/core@master] cache: Add two fields to LinkCache::getSelectFields
This patch fixes the biggest reason behind the load and I tested it locally that 1- nothing broke in my localhost 2- It fixed the issue.
I found the underlying problem as well ^^. The patch afterwards fixes that.
Change 747043 merged by jenkins-bot:
[mediawiki/core@master] Reuse the query result in addCategoryLinks instead of relying on cache
Change 747068 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[mediawiki/core@wmf/1.38.0-wmf.12] Reuse the query result in addCategoryLinks instead of relying on cache
Change 747069 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[mediawiki/core@wmf/1.38.0-wmf.13] Reuse the query result in addCategoryLinks instead of relying on cache
Change 747068 merged by jenkins-bot:
[mediawiki/core@wmf/1.38.0-wmf.12] Reuse the query result in addCategoryLinks instead of relying on cache
Change 747069 merged by jenkins-bot:
[mediawiki/core@wmf/1.38.0-wmf.13] Reuse the query result in addCategoryLinks instead of relying on cache
Mentioned in SAL (#wikimedia-operations) [2021-12-14T14:16:31Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.12/includes/OutputPage.php: Backport: [[gerrit:747068|Reuse the query result in addCategoryLinks instead of relying on cache (T297669)]] (duration: 00m 57s)
So this reduced the number of queries but it's still elevated. The fix for underlying issue will possibly improve things much better.
Change 747072 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[mediawiki/core@wmf/1.38.0-wmf.13] cache: Add four fields to LinkCache::getSelectFields
Change 747073 had a related patch set uploaded (by Ladsgroup; author: Amir Sarabadani):
[mediawiki/core@wmf/1.38.0-wmf.12] cache: Add four fields to LinkCache::getSelectFields
Change 747045 merged by jenkins-bot:
[mediawiki/core@master] cache: Add four fields to LinkCache::getSelectFields
Change 747072 merged by jenkins-bot:
[mediawiki/core@wmf/1.38.0-wmf.13] cache: Add four fields to LinkCache::getSelectFields
Change 747073 merged by jenkins-bot:
[mediawiki/core@wmf/1.38.0-wmf.12] cache: Add four fields to LinkCache::getSelectFields
Mentioned in SAL (#wikimedia-operations) [2021-12-14T15:54:00Z] <ladsgroup@deploy1002> Synchronized php-1.38.0-wmf.12/includes/cache/LinkCache.php: Backport: [[gerrit:747073|cache: Add four fields to LinkCache::getSelectFields (T297669)]] (duration: 00m 57s)
The underlying fix reduced the load but not as much as I hoped for which is possibly due to all of WAN caches being cold (and warm cache entries possibly contain corrupted data so I'm not sure the new value gets replaced or we have to wait for them to expire) but it look like it's slowly going down. I expect this to be much better state by 24 hours from now. If not, then I suggest reverting the PageStore patch (but very likely won't be needed)
Change 811282 had a related patch set uploaded (by Zabe; author: Amir Sarabadani):
[mediawiki/core@REL1_37] cache: Add four fields to LinkCache::getSelectFields
Change 811282 merged by jenkins-bot:
[mediawiki/core@REL1_37] cache: Add four fields to LinkCache::getSelectFields