Page MenuHomePhabricator

Special:SetSiteLink, other SpecialModifyEntity subclasses open master connection on GET
Closed, ResolvedPublic

Description

Seems to happen for every request of the form /wiki/Special:SetSiteLink/Qnnnnnn. There are also lots of other ones in the logs from other subclasses of SpecialModifyEntity.

Expectation (masterConns <= 0) by MediaWiki::main not met:
[connect to 10.64.16.144 (wikidatawiki)]
#0 /srv/mediawiki/php-1.29.0-wmf.6/includes/libs/rdbms/TransactionProfiler.php(160): TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.29.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(587): TransactionProfiler->recordConnection()
#2 /srv/mediawiki/php-1.29.0-wmf.6/includes/dao/DBAccessBase.php(61): LoadBalancer->getConnection()
#3 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(214): DBAccessBase->getConnection()
#4 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(98): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->selectRevisionInformationMultiple()
#5 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(151): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->loadRevisionInformation()
#6 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(84): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->loadRevisionInformation()
#7 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(197): Wikibase\Lib\Store\WikiPageEntityRevisionLookup->getEntityRevision()
#8 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(180): Wikibase\Repo\Specials\SpecialModifyEntity->loadEntity()
#9 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialSetSiteLink.php(108): Wikibase\Repo\Specials\SpecialModifyEntity->prepareArguments()
#10 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(129): Wikibase\Repo\Specials\SpecialSetSiteLink->prepareArguments()
#11 /srv/mediawiki/php-1.29.0-wmf.6/includes/specialpage/SpecialPage.php(522): Wikibase\Repo\Specials\SpecialModifyEntity->execute()
#12 /srv/mediawiki/php-1.29.0-wmf.6/includes/specialpage/SpecialPageFactory.php(577): SpecialPage->run()
#13 /srv/mediawiki/php-1.29.0-wmf.6/includes/MediaWiki.php(283): SpecialPageFactory::executePath()
#14 /srv/mediawiki/php-1.29.0-wmf.6/includes/MediaWiki.php(851): MediaWiki->performRequest()
#15 /srv/mediawiki/php-1.29.0-wmf.6/includes/MediaWiki.php(512): MediaWiki->main()
#16 /srv/mediawiki/php-1.29.0-wmf.6/index.php(43): MediaWiki->run()
#17 /srv/mediawiki/w/index.php(3): include()
#18 {main}

Event Timeline

Catrope created this task.Jan 4 2017, 1:48 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 4 2017, 1:48 AM

Change 330372 had a related patch set uploaded (by Aude):
Avoid master db connection on special page GET requests

https://gerrit.wikimedia.org/r/330372

Catrope renamed this task from Special:SetSiteLink opens master connection on GET to Special:SetSiteLink, other SpecialModifyEntity subclasses open master connection on GET.Jan 4 2017, 5:43 PM
Catrope updated the task description. (Show Details)
Ladsgroup added subscribers: Lydia_Pintscher, Ladsgroup.

I propose this task for the next sprint for the performance reasons. CC @Lydia_Pintscher.

Ladsgroup claimed this task.Jul 3 2017, 8:52 AM
Ladsgroup moved this task from Proposed to Review on the Wikidata-Former-Sprint-Board board.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJul 3 2017, 8:52 AM
thiemowmde triaged this task as Normal priority.Jul 4 2017, 7:30 AM
thiemowmde moved this task from Review to Proposed on the Wikidata-Former-Sprint-Board board.
thiemowmde added a subscriber: aude.

Change 330372 abandoned by Aude:
Avoid master db connection on special page GET requests

https://gerrit.wikimedia.org/r/330372

Ladsgroup moved this task from incoming to in progress on the Wikidata board.Jul 21 2017, 12:29 PM

I think this is done. It's not easy to test. Let's keep this open until wmf.12 gets deployed in wikidata.

Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptAug 9 2017, 12:46 PM

I still see lots of similar errors in logstash after deploying it https://logstash.wikimedia.org/goto/d9ac61c9e1b1767fa96dc45a511adc9c:

Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 1):
[connect to 10.64.32.228 (wikidatawiki)]
#0 /srv/mediawiki/php-1.30.0-wmf.14/includes/libs/rdbms/TransactionProfiler.php(164): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.30.0-wmf.14/includes/libs/rdbms/loadbalancer/LoadBalancer.php(678): Wikimedia\Rdbms\TransactionProfiler->recordConnection()
#2 /srv/mediawiki/php-1.30.0-wmf.14/includes/dao/DBAccessBase.php(64): Wikimedia\Rdbms\LoadBalancer->getConnection()
#3 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(211): DBAccessBase->getConnection()
#4 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(94): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->selectRevisionInformationMultiple()
#5 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(150): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->loadRevisionInformation()
#6 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(139): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->loadRevisionInformation()
#7 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/EditEntity.php(280): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getLatestRevisionId()
#8 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/EditEntity.php(307): Wikibase\EditEntity->getLatestRevisionId()
#9 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/EditEntity.php(323): Wikibase\EditEntity->getBaseRevisionId()
#10 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(95): Wikibase\EditEntity->getBaseRevision()
#11 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(186): Wikibase\Repo\Specials\SpecialModifyEntity->getBaseRevision()
#12 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialSetLabelDescriptionAliases.php(244): Wikibase\Repo\Specials\SpecialModifyEntity->getEntityForDisplay()
#13 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(209): Wikibase\Repo\Specials\SpecialSetLabelDescriptionAliases->processArguments()
#14 /srv/mediawiki/php-1.30.0-wmf.14/includes/specialpage/SpecialPage.php(522): Wikibase\Repo\Specials\SpecialModifyEntity->execute()
#15 /srv/mediawiki/php-1.30.0-wmf.14/includes/specialpage/SpecialPageFactory.php(578): SpecialPage->run()
#16 /srv/mediawiki/php-1.30.0-wmf.14/includes/MediaWiki.php(287): SpecialPageFactory::executePath()
#17 /srv/mediawiki/php-1.30.0-wmf.14/includes/MediaWiki.php(868): MediaWiki->performRequest()
#18 /srv/mediawiki/php-1.30.0-wmf.14/includes/MediaWiki.php(523): MediaWiki->main()
#19 /srv/mediawiki/php-1.30.0-wmf.14/index.php(43): MediaWiki->run()
#20 /srv/mediawiki/w/index.php(3): include()
#21 {main}
Ladsgroup added a comment.EditedAug 21 2017, 8:42 AM

Okay I checked the codebase and the RfC regarding multi-datacenter and to me it seems the whole concept of "LATEST_FROM_SLAVE_WITH_FALLBACK" is problematic, connection to master should not be opened/requested under any condition on GET mostly because of multi-datacenter scalability. We supposed to be sending GET requests to the replica datacenter where there is no master (the master is actually replica of the master in the master datacenter) but telling mediawiki to open connection to master causes it to gets confused that there might be a change to be written and that can't be done in the replica datacenter. I hereby propose to remove the whole thing and use LATEST_FROM_SLAVE.

I think just switching to LATEST_FROM_SLAVE is not an option - we introduced LATEST_FROM_SLAVE_WITH_FALLBACK for a specific need that was not addressed by LATEST_FROM_SLAVE. If I remember correctly, the scenario was: after an item was created with a sitelink, the client is notified of this, and the linked page gets purged. However, if the notification is too quick, the new item isn't found on the slave database, so no language link is shown. This is especially annoying when the item is created via the client side widget: the user will use the "add link" dialog, and then not see the link until the page gets purged manually. The problem was that the code for fetching entities (or sitelinks) doesn't know about the "urgency" to find the item, nor does it know whether it's called from a POST request or not.

I see two options: introduce getEntityRevisionForUpdate or getEntityRevisionTryHarder or something. Or, make LATEST_FROM_SLAVE_WITH_FALLBACK only actually fall back to master if called from a POST request (or CLI).

Change 373040 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Use replica with fallback only if the request got POST'ed

https://gerrit.wikimedia.org/r/373040

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptAug 30 2017, 7:59 AM

Change 373040 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use replica with fallback only if the request got POST'ed

https://gerrit.wikimedia.org/r/373040

It's still happening in wmf.17 too: https://logstash.wikimedia.org/goto/1f0b8c1062f060ec091c33b8a8e10e6b

I need to investigate this.

Change 376700 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Use master when needed in EditEntity::getLatestRevision()

https://gerrit.wikimedia.org/r/376700

Change 376700 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use master when needed in EditEntity::getLatestRevision()

https://gerrit.wikimedia.org/r/376700

This is plot of the events in the past 12 days:

I LOVE it

Ladsgroup closed this task as Resolved.Sep 20 2017, 11:02 AM