Page MenuHomePhabricator

DBPerformance warning "Query returned XXXX rows: query: SELECT * FROM `translate_metadata`"
Closed, ResolvedPublicPRODUCTION ERROR

Description

Translate extensions uses super simple in-process caching on translate_metadata. Every time a caller wants to read something from that table, the code loads and caches the whole table into process memory. The good side of this is, when the table is small, that it will only cause one extra query instead of a lot of them. This way they callers do not need to care or do preloading.

However, on Meta the table has grown to 22k entries, and MediaWiki has started to warn about big reads from the database. Fixing this properly requires checking all callers and add suitable preloading to those who need it.

Aside, for testing and other purposes, this class shouldn't be static, but a service.

LogStash query

Outcome

Database query performance issues related to translate_metadata table have been resolved.

Event Timeline

Krinkle renamed this task from Query returned 22222 row(s): query: SELECT * FROM `translate_metadata` on Metawiki to DBPerformance warning "Query returned 22186 rows: query: SELECT * FROM `translate_metadata`" on Meta-Wiki.Sep 11 2018, 5:37 PM
Krinkle edited projects, added Wikimedia-production-error; removed Technical-Debt.
jijiki triaged this task as Medium priority.Oct 26 2018, 10:19 AM
mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM
LSobanski subscribed.

Removing the DBA tag and subscribing myself instead. If there are any specific actions for DBA please re-add us or @mention me.

Nikerabbit renamed this task from DBPerformance warning "Query returned 22186 rows: query: SELECT * FROM `translate_metadata`" on Meta-Wiki to DBPerformance warning "Query returned XXXX rows: query: SELECT * FROM `translate_metadata`" on Meta-Wiki.Sep 6 2021, 12:02 PM
Nikerabbit updated the task description. (Show Details)

Metawiki currently has 38332 rows on the table, of which 24207 are for key prioritylangs with empty value. These seem to come mostly from CentralNotice.

This will still leave 12k rows in the table. With some changes in the Translate extension, I believe we can get it down to 432 rows by only reading the priorityforce and prioritylangs key.

Change 719125 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] Optimize queries on Special:LanguageStats

https://gerrit.wikimedia.org/r/719125

Change 719125 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Optimize queries on Special:LanguageStats

https://gerrit.wikimedia.org/r/719125

Change 720743 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/CentralNotice@master] Avoid writing empty prioritylangs rows to translate_metadata

https://gerrit.wikimedia.org/r/720743

Translate patch should reduce the log entries by introducing special in process cache which is filled by tailored query to only load required rows when displaying message group or language statistics. There may be other code paths that trigger the original query.

The patch in CentralNotice prevents creation of new empty content rows. That will stop the unexpected growth of the translate_metadata table. The existing empty content rows should be cleaned up, if Wikimedia-Fundraising approves that, to speed up rest of the queries.

Looking at Logstash, we have gone from ~1000 warnings per day to ~200 warnings per day. Biggest sources now are action=query&meta=messagegroupstats followed by Special:PageTranslation.

Change 722285 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] TranslateMetadata: log caller info in preloadGroups

https://gerrit.wikimedia.org/r/722285

Change 722326 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] Optimize Special:PageTranslation page list metadata query

https://gerrit.wikimedia.org/r/722326

Nikerabbit changed the task status from Open to In Progress.Sep 20 2021, 10:45 AM

Change 722353 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] Optimize metadata query on Special:Translate

https://gerrit.wikimedia.org/r/722353

Change 722285 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] TranslateMetadata: log caller info in preloadGroups

https://gerrit.wikimedia.org/r/722285

Change 722326 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Optimize Special:PageTranslation page list metadata query

https://gerrit.wikimedia.org/r/722326

Change 722353 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Optimize metadata query on Special:Translate

https://gerrit.wikimedia.org/r/722353

Change 724712 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] Optimize metadata query on Special:AggregateGroups

https://gerrit.wikimedia.org/r/724712

Change 724712 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Optimize metadata query on Special:AggregateGroups

https://gerrit.wikimedia.org/r/724712

Change 724995 had a related patch set uploaded (by Nikerabbit; author: Nikerabbit):

[mediawiki/extensions/Translate@master] Optimize metadata query in Recent(Additions)MessageGroup

https://gerrit.wikimedia.org/r/724995

We have gone from ~200 to ~10 per day after last weeks deployment, and there are more fixes coming to address some of the long tail.

Change 724995 merged by jenkins-bot:

[mediawiki/extensions/Translate@master] Optimize metadata query in Recent(Additions)MessageGroup

https://gerrit.wikimedia.org/r/724995

It looks like the number-of-rows-returned issues are fully fixed. There is one warning about sub-optimal transaction, which I filed separately as T292686: Sub-optimal transaction on Special:PageTranslation.

Now waiting for the fix in CentralNotice to reduce the number of rows in the table to improve query performance.

Nikerabbit renamed this task from DBPerformance warning "Query returned XXXX rows: query: SELECT * FROM `translate_metadata`" on Meta-Wiki to DBPerformance warning "Query returned XXXX rows: query: SELECT * FROM `translate_metadata`".Oct 7 2021, 6:50 AM

Change 720743 merged by jenkins-bot:

[mediawiki/extensions/CentralNotice@master] Avoid writing empty prioritylangs rows to translate_metadata

https://gerrit.wikimedia.org/r/720743

Thanks for merging the CentralNotice patch.

Any ETA when it will deployed, and is the proposed database clean-up okay to do?

Change 730262 had a related patch set uploaded (by AndyRussG; author: Nikerabbit):

[mediawiki/extensions/CentralNotice@wmf/1.38.0-wmf.4] Avoid writing empty prioritylangs rows to translate_metadata

https://gerrit.wikimedia.org/r/730262

Change 730262 abandoned by AndyRussG:

[mediawiki/extensions/CentralNotice@wmf/1.38.0-wmf.4] Avoid writing empty prioritylangs rows to translate_metadata

Reason:

https://gerrit.wikimedia.org/r/730262

@Nikerabbit it should be in the deployment process at this moment. Also having taken the request through the team, it's okay to do the database cleanup.

Nikerabbit removed a project: Patch-For-Review.

Thank you! The clean up has been done.