Page MenuHomePhabricator

Wikibase\TermSqlIndex::getMatchingTerms , Wikibase\TermSqlIndex::fetchTerms have bad performance after codfw failover
Closed, ResolvedPublic

Description

these 2 queries were among the most expensive on datacenter failover or after it- while it is normal to have lower performance than usual due to colder caches, most likely they are surfacing existing issues, which could be made better even under normal circunstances. It would be nice to have some mitigation in place before the failback happens on the 3 May to avoid the issues mentioned on the parent ticket- even if it is as silly as reducing query batches or job rates:

https://logstash.wikimedia.org/goto/a94c6c3ffe941fc83616d18adfc372f8

17830	52	7	134,376	db2038, db2045, db2052, db2059, db2066	wikiuser	wikidatawiki
SELECT /* Wikibase\TermSqlIndex::fetchTerms */ term_entity_type, term_type, term_language, term_text, term_entity_id FROM `wb_terms` WHERE term_entity_type = 'item' AND term_entity_id IN ('5682', '11573', '2807', '46857', '119', '96', '34726', '1028', '513', '48278', '235', '8851', '649', '122224', '9077', '8027', '38076', '345', '2831', '9933') AND term_language = 'es' AND term_type = 'description' /* 031a7f86378ef40d4240a44ad46a3cbf db2059 wikidatawiki 11s */

253	325	102	25,836	db2038, db2045, db2052, db2059, db2066	wikiuser	wikidatawiki
SELECT /* Wikibase\TermSqlIndex::getMatchingTerms */ term_entity_type, term_type, term_language, term_text, term_entity_id, term_weight FROM `wb_terms` WHERE ((term_language = 'en' AND (term_search_key LIKE 'aut%' ) AND term_type = 'label' AND term_entity_type = 'item') OR (term_language = 'en' AND (term_search_key LIKE 'aut%' ) AND term_type = 'alias' AND term_entity_type = 'item')) LIMIT 5000 /* c4de2b8173b064fb59493feca2567115 db2052 wikidatawiki 7s */

Event Timeline

I'm looking into this and will see what we can do.

Change 349424 had a related patch set uploaded (by Hoo man):
[mediawiki/extensions/Wikibase@master] Use smaller batches in TermSqlIndex::getTermsOfEntities

https://gerrit.wikimedia.org/r/349424

Change 349429 had a related patch set uploaded (by Hoo man):
[mediawiki/extensions/Wikibase@master] Only retrieve up to 2500 terms in TermSqlIndex

https://gerrit.wikimedia.org/r/349429

Change 349429 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Only retrieve up to 2500 terms in TermSqlIndex

https://gerrit.wikimedia.org/r/349429

From #wikimedia-databases:

<hoo> jynus: For the first query in https://phabricator.wikimedia.org/T163544 how many term_entity_id would be ok?
<hoo> Right now we don't batch there at all, but I can add that rather easily
<hoo> Would 10 be ok?
<jynus> I would say 9
<jynus> I thin 10 is the limit to start getting in some cases bad queries
<jynus> (approximate cardinality)
<hoo> Ok, that's fine

I was thinking on https://dev.mysql.com/doc/refman/5.6/en/server-system-variables.html#sysvar_eq_range_index_dive_limit but that is probably more wishful thinking than practical given we are on MariaDB 10.

Change 349424 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use smaller batches in TermSqlIndex::getTermsOfEntities

https://gerrit.wikimedia.org/r/349424

Both problems have been addressed.