Page MenuHomePhabricator
Create Task
Maniphest T323202

DBPerformance warning "Expectation (readQueryRows <= 10000) not met" in growthexperiments_mentor_mentee
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
normalized_message
Expectation (readQueryRows <= 10000) by MediaWiki::main not met (actual: 14502):
SELECT gemm_mentee_id AS `value` FROM `growthexperiments_mentor_mentee` WHERE gemm_mentor_id = N AND gemm_mentor_role = 'X'
exception.trace
from /srv/mediawiki/php-1.40.0-wmf.8/includes/libs/rdbms/TransactionProfiler.php(467)
#0 /srv/mediawiki/php-1.40.0-wmf.8/includes/libs/rdbms/TransactionProfiler.php(279): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, Wikimedia\Rdbms\GeneralizedSql, integer, string, string)
#1 /srv/mediawiki/php-1.40.0-wmf.8/includes/libs/rdbms/database/TransactionManager.php(612): Wikimedia\Rdbms\TransactionProfiler->recordQueryCompletion(Wikimedia\Rdbms\GeneralizedSql, double, boolean, integer, string, string)
#2 /srv/mediawiki/php-1.40.0-wmf.8/includes/libs/rdbms/database/Database.php(1229): Wikimedia\Rdbms\TransactionManager->recordQueryCompletion(Wikimedia\Rdbms\GeneralizedSql, double, boolean, integer, string)
#3 /srv/mediawiki/php-1.40.0-wmf.8/includes/libs/rdbms/database/Database.php(1043): Wikimedia\Rdbms\Database->attemptQuery(array, array, string, string, boolean, boolean)
#4 /srv/mediawiki/php-1.40.0-wmf.8/includes/libs/rdbms/database/Database.php(880): Wikimedia\Rdbms\Database->executeQuery(string, string, integer, string)
#5 /srv/mediawiki/php-1.40.0-wmf.8/includes/libs/rdbms/database/Database.php(1648): Wikimedia\Rdbms\Database->query(string, string, integer)
#6 /srv/mediawiki/php-1.40.0-wmf.8/includes/libs/rdbms/database/Database.php(1624): Wikimedia\Rdbms\Database->select(array, array, array, string, array, array)
#7 /srv/mediawiki/php-1.40.0-wmf.8/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(678): Wikimedia\Rdbms\Database->selectFieldValues(array, string, array, string, array, array)
#8 /srv/mediawiki/php-1.40.0-wmf.8/extensions/GrowthExperiments/includes/Mentorship/Store/DatabaseMentorStore.php(142): Wikimedia\Rdbms\SelectQueryBuilder->fetchFieldValues()
#9 /srv/mediawiki/php-1.40.0-wmf.8/extensions/GrowthExperiments/includes/Mentorship/Hooks/MentorFilterHooks.php(179): GrowthExperiments\Mentorship\Store\DatabaseMentorStore->getMenteesByMentor(User, string)
#10 /srv/mediawiki/php-1.40.0-wmf.8/extensions/GrowthExperiments/includes/Mentorship/Hooks/MentorFilterHooks.php(139): GrowthExperiments\Mentorship\Hooks\MentorFilterHooks->getUnstarredMenteeIds(User)
#11 /srv/mediawiki/php-1.40.0-wmf.8/includes/changes/ChangesListFilter.php(355): GrowthExperiments\Mentorship\Hooks\MentorFilterHooks->GrowthExperiments\Mentorship\Hooks\{closure}(OldChangesList, RecentChange)
#12 /srv/mediawiki/php-1.40.0-wmf.8/includes/changes/ChangesList.php(263): ChangesListFilter->applyCssClassIfNeeded(OldChangesList, RecentChange, array)
#13 /srv/mediawiki/php-1.40.0-wmf.8/includes/changes/ChangesList.php(236): ChangesList->getHTMLClassesForFilters(RecentChange)
#14 /srv/mediawiki/php-1.40.0-wmf.8/includes/changes/OldChangesList.php(39): ChangesList->getHTMLClasses(RecentChange, boolean)
#15 /srv/mediawiki/php-1.40.0-wmf.8/includes/specials/SpecialWatchlist.php(610): OldChangesList->recentChangesLine(RecentChange, boolean, integer)
#16 /srv/mediawiki/php-1.40.0-wmf.8/includes/specialpage/ChangesListSpecialPage.php(1571): SpecialWatchlist->outputChangesList(Wikimedia\Rdbms\MysqliResultWrapper, FormOptions)
#17 /srv/mediawiki/php-1.40.0-wmf.8/includes/specialpage/ChangesListSpecialPage.php(676): ChangesListSpecialPage->webOutput(Wikimedia\Rdbms\MysqliResultWrapper, FormOptions)
#18 /srv/mediawiki/php-1.40.0-wmf.8/includes/specials/SpecialWatchlist.php(134): ChangesListSpecialPage->execute(NULL)
#19 /srv/mediawiki/php-1.40.0-wmf.8/includes/specialpage/SpecialPage.php(700): SpecialWatchlist->execute(NULL)
#20 /srv/mediawiki/php-1.40.0-wmf.8/includes/specialpage/SpecialPageFactory.php(1451): SpecialPage->run(NULL)
#21 /srv/mediawiki/php-1.40.0-wmf.8/includes/MediaWiki.php(317): MediaWiki\SpecialPage\SpecialPageFactory->executePath(string, RequestContext)
#22 /srv/mediawiki/php-1.40.0-wmf.8/includes/MediaWiki.php(905): MediaWiki->performRequest()
#23 /srv/mediawiki/php-1.40.0-wmf.8/includes/MediaWiki.php(563): MediaWiki->main()
#24 /srv/mediawiki/php-1.40.0-wmf.8/index.php(50): MediaWiki->run()
#25 /srv/mediawiki/php-1.40.0-wmf.8/index.php(46): wfIndexMain()
#26 /srv/mediawiki/w/index.php(3): require(string)
#27 {main}
Impact

3,348 events in the last 30 days

Notes

Might be the same as in T293182 but opening it since the query seems to have changed and affecting different wikis than mentioned in that task.

Details

Request URL
https://pt.wikipedia.org/w/index.php?authorship__hidemyself_color=*&damaging__verylikelybad_color=*&days=*&hideWikibase=*&hidebots=*&hidecategorization=*&hidepreviousrevisions=*&highlight=*&limit=*&title=*&urlversion=*

Related Objects
Search...

Event Timeline

Sgs created this task.Nov 16 2022, 9:56 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2022, 9:56 AM
Sgs updated the task description. (Show Details)Nov 16 2022, 10:00 AM
Sgs moved this task from Inbox to Triaged on the Growth-Team board.Nov 16 2022, 10:22 AM
Maintenance_bot added a project: GrowthExperiments-Mentorship.Nov 16 2022, 10:30 AM
Urbanecm_WMF added subscribers: Tgr, Urbanecm_WMF.EditedNov 16 2022, 10:40 AM

UPDATE: The logic for the is active flag was not working correctly. The comment below is written before figuring that there was a bug in the logic.

Might be the same as in T293182

It is a different instance of the same underlying issue (tracked under T318457), since more wikis reached the threshold of 10k mentees per mentor in the meanwhile (see list of mentees by mentor per wiki).

In T318457#8397052, @Tgr suggested to move the mentor store to the main cluster, which would allow us to JOIN recentchanges and growthexperiments_mentor_mentee directly, avoiding the need for PHP-level join (which causes this warning). But, that's a longer project, as it involves migrating the data from cluster to cluster.

but opening it since the query seems to have changed

That's because we introduced backup mentors in the meanwhile, but the underlying issue is the same.

and affecting different wikis than mentioned in that task

That's because we disabled the broken features for three wikis (arwiki, enwiki, fawiki), and additional wikis reached the 10k threshold in the meanwhile.

We can "fix" this task by deploying the active flag as-is (T318457), which will put us back into stage where only the few biggest wikis have more than 10k active users per mentor. I'll discuss that with Tgr later today.

Urbanecm_WMF added a subtask: T318457: Enable "Your unstarred mentees" at the biggest Growth wikis.Nov 16 2022, 10:44 AM
Urbanecm_WMF triaged this task as Low priority.Nov 16 2022, 10:57 AM
Urbanecm_WMF moved this task from New Tasks to For Later (Triaged tasks) on the GrowthExperiments-Mentorship board.Nov 16 2022, 11:00 AM
Urbanecm_WMF added a subscriber: Etonkovidova.Nov 16 2022, 10:27 PM

@Sgs @Etonkovidova et al: FYI, this should be fixed as-of wmf.10, thanks to the improvements done in T318457. Moving to QA.

Urbanecm_WMF claimed this task.Nov 16 2022, 10:27 PM
Urbanecm_WMF edited projects, added Growth-Team (Sprint 0 (Growth Team)); removed Growth-Team.
Urbanecm_WMF moved this task from Incoming to QA on the Growth-Team (Sprint 0 (Growth Team)) board.
Maintenance_bot moved this task from For Later (Triaged tasks) to In Progress on the GrowthExperiments-Mentorship board.Nov 16 2022, 10:29 PM
Urbanecm_WMF moved this task from In Progress to Ready for QA on the GrowthExperiments-Mentorship board.Nov 17 2022, 12:42 PM
Krinkle renamed this task from Expectation (readQueryRows <= 10000) by MediaWiki::main not met (actual: {actualSeconds}):{query} to DBPerformance warning "Expectation (readQueryRows <= 10000) not met" in growthexperiments_mentor_mentee.Nov 17 2022, 4:33 PM
Krinkle moved this task from Untriaged to Nov 2022 on the Wikimedia-production-error board.
Urbanecm_WMF changed the status of subtask T318457: Enable "Your unstarred mentees" at the biggest Growth wikis from In Progress to Open.Nov 21 2022, 8:35 AM
Etonkovidova closed this task as Resolved.Dec 5 2022, 10:15 PM

The last timestamp (Dec 1, 2022 @ 07:26:14.495) is for wmf.10 (link to logstash-Nov15-Dec05/2022). The total number of the warning for wmf.10 was 2, and for wmf.12 zero.

Etonkovidova closed subtask T318457: Enable "Your unstarred mentees" at the biggest Growth wikis as Resolved.Dec 19 2022, 11:08 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits