Page MenuHomePhabricator
Create Task
Maniphest T154555

Special:SetSiteLink, other SpecialModifyEntity subclasses open master connection on GET
Closed, ResolvedPublic
Actions

Description

Seems to happen for every request of the form /wiki/Special:SetSiteLink/Qnnnnnn. There are also lots of other ones in the logs from other subclasses of SpecialModifyEntity.

Expectation (masterConns <= 0) by MediaWiki::main not met:
[connect to 10.64.16.144 (wikidatawiki)]
#0 /srv/mediawiki/php-1.29.0-wmf.6/includes/libs/rdbms/TransactionProfiler.php(160): TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.29.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(587): TransactionProfiler->recordConnection()
#2 /srv/mediawiki/php-1.29.0-wmf.6/includes/dao/DBAccessBase.php(61): LoadBalancer->getConnection()
#3 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(214): DBAccessBase->getConnection()
#4 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(98): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->selectRevisionInformationMultiple()
#5 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(151): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->loadRevisionInformation()
#6 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(84): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->loadRevisionInformation()
#7 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(197): Wikibase\Lib\Store\WikiPageEntityRevisionLookup->getEntityRevision()
#8 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(180): Wikibase\Repo\Specials\SpecialModifyEntity->loadEntity()
#9 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialSetSiteLink.php(108): Wikibase\Repo\Specials\SpecialModifyEntity->prepareArguments()
#10 /srv/mediawiki/php-1.29.0-wmf.6/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(129): Wikibase\Repo\Specials\SpecialSetSiteLink->prepareArguments()
#11 /srv/mediawiki/php-1.29.0-wmf.6/includes/specialpage/SpecialPage.php(522): Wikibase\Repo\Specials\SpecialModifyEntity->execute()
#12 /srv/mediawiki/php-1.29.0-wmf.6/includes/specialpage/SpecialPageFactory.php(577): SpecialPage->run()
#13 /srv/mediawiki/php-1.29.0-wmf.6/includes/MediaWiki.php(283): SpecialPageFactory::executePath()
#14 /srv/mediawiki/php-1.29.0-wmf.6/includes/MediaWiki.php(851): MediaWiki->performRequest()
#15 /srv/mediawiki/php-1.29.0-wmf.6/includes/MediaWiki.php(512): MediaWiki->main()
#16 /srv/mediawiki/php-1.29.0-wmf.6/index.php(43): MediaWiki->run()
#17 /srv/mediawiki/w/index.php(3): include()
#18 {main}

Details

SubjectRepoBranchLines +/-
Use master when needed in EditEntity::getLatestRevision()mediawiki/extensions/Wikibasemaster+1 -1
Use replica with fallback only if the request got POST'edmediawiki/extensions/Wikibasemaster+37 -10
Avoid master db connection on special page GET requestsmediawiki/extensions/Wikibasemaster+7 -7
Customize query in gerrit

Related Objects
Search...

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
· · ·
OpenNoneT92357 Fix database master queries from HTTP GET/HEAD before active-active multi-dc
OpenNoneT96040 Wikibase special pages (tracking)
ResolvedLadsgroupT154555 Special:SetSiteLink, other SpecialModifyEntity subclasses open master connection on GET
· · ·

Event Timeline

Catrope created this task.Jan 4 2017, 1:48 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 4 2017, 1:48 AM
Catrope added a parent task: T92357: Fix database master queries from HTTP GET/HEAD before active-active multi-dc.Jan 4 2017, 1:48 AM
• MZMcBride subscribed.Jan 4 2017, 4:01 AM
gerritbot subscribed.Jan 4 2017, 5:32 AM

Change 330372 had a related patch set uploaded (by Aude):
Avoid master db connection on special page GET requests

https://gerrit.wikimedia.org/r/330372

gerritbot added a project: Patch-For-Review.Jan 4 2017, 5:32 AM
Catrope mentioned this in T154596: SpecialModifyEntity master queries on page views.Jan 4 2017, 5:41 PM
Catrope merged a task: T154596: SpecialModifyEntity master queries on page views.
Catrope added subscribers: aaron, Joe, Krenair and 7 others.
Catrope renamed this task from Special:SetSiteLink opens master connection on GET to Special:SetSiteLink, other SpecialModifyEntity subclasses open master connection on GET.Jan 4 2017, 5:43 PM
Catrope updated the task description. (Show Details)
matej_suchanek added a parent task: T96040: Wikibase special pages (tracking).Jan 14 2017, 5:46 PM
Ladsgroup added a project: Wikidata-Former-Sprint-Board.Jun 28 2017, 12:50 AM
Ladsgroup added subscribers: Lydia_Pintscher, Ladsgroup.

I propose this task for the next sprint for the performance reasons. CC @Lydia_Pintscher.

Lydia_Pintscher added a comment.Jun 28 2017, 8:20 AM

Ok.

Ladsgroup claimed this task.Jul 3 2017, 8:52 AM
Ladsgroup moved this task from Proposed to Review on the Wikidata-Former-Sprint-Board board.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJul 3 2017, 8:52 AM
Ladsgroup moved this task from Incoming to Blocked on others on the User-Ladsgroup board.Jul 3 2017, 9:12 PM
thiemowmde triaged this task as Medium priority.Jul 4 2017, 7:30 AM
thiemowmde moved this task from Review to Proposed on the Wikidata-Former-Sprint-Board board.
thiemowmde added a subscriber: aude.
WMDE-leszek moved this task from Proposed to Backlog on the Wikidata-Former-Sprint-Board board.Jul 4 2017, 2:10 PM
WMDE-leszek moved this task from Backlog to Doing on the Wikidata-Former-Sprint-Board board.Jul 5 2017, 9:37 AM
• AnotherLadsgroup moved this task from Doing to Review on the Wikidata-Former-Sprint-Board board.Jul 18 2017, 7:51 AM
Ladsgroup reassigned this task from Ladsgroup to daniel.Jul 20 2017, 6:26 PM
Ladsgroup removed a project: User-Ladsgroup.

It's being done in https://gerrit.wikimedia.org/r/#/c/366606/

gerritbot added a comment.Jul 20 2017, 10:28 PM

Change 330372 abandoned by Aude:
Avoid master db connection on special page GET requests

https://gerrit.wikimedia.org/r/330372

Lucas_Werkmeister_WMDE moved this task from Review to Doing on the Wikidata-Former-Sprint-Board board.Jul 21 2017, 10:04 AM
Ladsgroup moved this task from incoming to in progress on the Wikidata board.Jul 21 2017, 12:29 PM
Ladsgroup added a comment.Aug 9 2017, 12:46 PM

I think this is done. It's not easy to test. Let's keep this open until wmf.12 gets deployed in wikidata.

Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptAug 9 2017, 12:46 PM
Ladsgroup moved this task from Doing to Review on the Wikidata-Former-Sprint-Board board.Aug 9 2017, 12:46 PM
Lucas_Werkmeister_WMDE moved this task from Review to Done on the Wikidata-Former-Sprint-Board board.Aug 15 2017, 10:26 AM
Ladsgroup added a comment.Aug 18 2017, 8:37 AM

I still see lots of similar errors in logstash after deploying it https://logstash.wikimedia.org/goto/d9ac61c9e1b1767fa96dc45a511adc9c:

Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 1):
[connect to 10.64.32.228 (wikidatawiki)]
#0 /srv/mediawiki/php-1.30.0-wmf.14/includes/libs/rdbms/TransactionProfiler.php(164): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated()
#1 /srv/mediawiki/php-1.30.0-wmf.14/includes/libs/rdbms/loadbalancer/LoadBalancer.php(678): Wikimedia\Rdbms\TransactionProfiler->recordConnection()
#2 /srv/mediawiki/php-1.30.0-wmf.14/includes/dao/DBAccessBase.php(64): Wikimedia\Rdbms\LoadBalancer->getConnection()
#3 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(211): DBAccessBase->getConnection()
#4 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php(94): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->selectRevisionInformationMultiple()
#5 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(150): Wikibase\Lib\Store\Sql\WikiPageEntityMetaDataLookup->loadRevisionInformation()
#6 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(139): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->loadRevisionInformation()
#7 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/EditEntity.php(280): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getLatestRevisionId()
#8 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/EditEntity.php(307): Wikibase\EditEntity->getLatestRevisionId()
#9 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/EditEntity.php(323): Wikibase\EditEntity->getBaseRevisionId()
#10 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(95): Wikibase\EditEntity->getBaseRevision()
#11 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(186): Wikibase\Repo\Specials\SpecialModifyEntity->getBaseRevision()
#12 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialSetLabelDescriptionAliases.php(244): Wikibase\Repo\Specials\SpecialModifyEntity->getEntityForDisplay()
#13 /srv/mediawiki/php-1.30.0-wmf.14/extensions/Wikidata/extensions/Wikibase/repo/includes/Specials/SpecialModifyEntity.php(209): Wikibase\Repo\Specials\SpecialSetLabelDescriptionAliases->processArguments()
#14 /srv/mediawiki/php-1.30.0-wmf.14/includes/specialpage/SpecialPage.php(522): Wikibase\Repo\Specials\SpecialModifyEntity->execute()
#15 /srv/mediawiki/php-1.30.0-wmf.14/includes/specialpage/SpecialPageFactory.php(578): SpecialPage->run()
#16 /srv/mediawiki/php-1.30.0-wmf.14/includes/MediaWiki.php(287): SpecialPageFactory::executePath()
#17 /srv/mediawiki/php-1.30.0-wmf.14/includes/MediaWiki.php(868): MediaWiki->performRequest()
#18 /srv/mediawiki/php-1.30.0-wmf.14/includes/MediaWiki.php(523): MediaWiki->main()
#19 /srv/mediawiki/php-1.30.0-wmf.14/index.php(43): MediaWiki->run()
#20 /srv/mediawiki/w/index.php(3): include()
#21 {main}
Ladsgroup moved this task from Done to Doing on the Wikidata-Former-Sprint-Board board.Aug 18 2017, 8:37 AM
Ladsgroup added a comment.EditedAug 21 2017, 8:42 AM

Okay I checked the codebase and the RfC regarding multi-datacenter and to me it seems the whole concept of "LATEST_FROM_SLAVE_WITH_FALLBACK" is problematic, connection to master should not be opened/requested under any condition on GET mostly because of multi-datacenter scalability. We supposed to be sending GET requests to the replica datacenter where there is no master (the master is actually replica of the master in the master datacenter) but telling mediawiki to open connection to master causes it to gets confused that there might be a change to be written and that can't be done in the replica datacenter. I hereby propose to remove the whole thing and use LATEST_FROM_SLAVE.

daniel added a comment.Aug 21 2017, 1:21 PM

I think just switching to LATEST_FROM_SLAVE is not an option - we introduced LATEST_FROM_SLAVE_WITH_FALLBACK for a specific need that was not addressed by LATEST_FROM_SLAVE. If I remember correctly, the scenario was: after an item was created with a sitelink, the client is notified of this, and the linked page gets purged. However, if the notification is too quick, the new item isn't found on the slave database, so no language link is shown. This is especially annoying when the item is created via the client side widget: the user will use the "add link" dialog, and then not see the link until the page gets purged manually. The problem was that the code for fetching entities (or sitelinks) doesn't know about the "urgency" to find the item, nor does it know whether it's called from a POST request or not.

I see two options: introduce getEntityRevisionForUpdate or getEntityRevisionTryHarder or something. Or, make LATEST_FROM_SLAVE_WITH_FALLBACK only actually fall back to master if called from a POST request (or CLI).

gerritbot added a comment.Aug 22 2017, 9:52 AM

Change 373040 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Use replica with fallback only if the request got POST'ed

https://gerrit.wikimedia.org/r/373040

Ladsgroup moved this task from Doing to Review on the Wikidata-Former-Sprint-Board board.Aug 22 2017, 9:54 AM
Lucas_Werkmeister_WMDE moved this task from Review to Doing on the Wikidata-Former-Sprint-Board board.Aug 28 2017, 10:03 AM
Ladsgroup moved this task from Doing to Review on the Wikidata-Former-Sprint-Board board.Aug 29 2017, 8:49 PM
Ladsgroup claimed this task.Aug 30 2017, 7:59 AM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptAug 30 2017, 7:59 AM
gerritbot added a comment.Aug 30 2017, 3:29 PM

Change 373040 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use replica with fallback only if the request got POST'ed

https://gerrit.wikimedia.org/r/373040

Ladsgroup moved this task from Review to Done on the Wikidata-Former-Sprint-Board board.Aug 31 2017, 7:01 AM
Ladsgroup added a comment.Sep 7 2017, 8:15 PM

It's still happening in wmf.17 too: https://logstash.wikimedia.org/goto/1f0b8c1062f060ec091c33b8a8e10e6b

I need to investigate this.

gerritbot added a comment.Sep 8 2017, 10:50 AM

Change 376700 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Use master when needed in EditEntity::getLatestRevision()

https://gerrit.wikimedia.org/r/376700

Ladsgroup moved this task from Done to Review on the Wikidata-Former-Sprint-Board board.Sep 8 2017, 10:50 AM
gerritbot added a comment.Sep 8 2017, 12:03 PM

Change 376700 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use master when needed in EditEntity::getLatestRevision()

https://gerrit.wikimedia.org/r/376700

Ladsgroup moved this task from Review to Done on the Wikidata-Former-Sprint-Board board.Sep 8 2017, 3:45 PM
Ladsgroup added a comment.Sep 20 2017, 11:02 AM

This is plot of the events in the past 12 days:

image.png (280×1 px, 38 KB)

I LOVE it

Ladsgroup closed this task as Resolved.Sep 20 2017, 11:02 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL