Page MenuHomePhabricator
Create Task
Maniphest T268944

update.php ItemTermsRebuilder can fail in master replica situation if replica isn't totally up to date with table creations
Closed, ResolvedPublic
Actions

Description

an update.php run where the new terms tables are created in a situation with master and slave replication can result in the initial set of updates failing (until the replica has caught up with the table creations?

This probably needs an extra wait for replication at the start of the script?

...Failed to save terms of item: Q51
[error] [DBQuery] Error 1146 from Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer::findExistingRecords, Table 'mwdb_xxxx.yyyy_wbt_text' doesn't exist (sql-mariadb-slave.default.svc.cluster.local) SELECT  wbx_text,wbx_id  FROM `mwt_6809576560_wbt_text`    WHERE (wbx_text = 'iNaturalist observation 9642') OR (wbx_text = 'Observation of Bostrychia hagedash hagedash')   sql-mariadb-slave.default.svc.cluster.local
#0 /var/www/html/w/includes/libs/rdbms/database/Database.php(1658): Wikimedia\Rdbms\Database->getQueryExceptionAndLog(string, integer, string, string)
#1 /var/www/html/w/includes/libs/rdbms/database/Database.php(1227): Wikimedia\Rdbms\Database->reportQueryError(string, integer, string, string, boolean)
#2 /var/www/html/w/includes/libs/rdbms/database/Database.php(1907): Wikimedia\Rdbms\Database->query(string, string, integer)
#3 /var/www/html/w/extensions/Wikibase/lib/includes/Store/Sql/Terms/Util/ReplicaMasterAwareRecordIdsAcquirer.php(298): Wikimedia\Rdbms\Database->select(string, array, string, string)
#4 /var/www/html/w/extensions/Wikibase/lib/includes/Store/Sql/Terms/Util/ReplicaMasterAwareRecordIdsAcquirer.php(245): Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer->findExistingRecords(Wikimedia\Rdbms\DatabaseMysqli, array)
#5 /var/www/html/w/extensions/Wikibase/lib/includes/Store/Sql/Terms/Util/ReplicaMasterAwareRecordIdsAcquirer.php(146): Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer->fetchExistingRecordsFromReplica(array)
#6 /var/www/html/w/extensions/Wikibase/lib/includes/Store/Sql/Terms/Util/ReplicaMasterAwareRecordIdsAcquirer.php(128): Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer->fetchExistingRecords(array)
#7 /var/www/html/w/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermInLangIdsAcquirer.php(168): Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer->acquireIds(array)
#8 /var/www/html/w/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermInLangIdsAcquirer.php(135): Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsAcquirer->acquireTextIds(array, Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer)
#9 /var/www/html/w/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermInLangIdsAcquirer.php(416): Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsAcquirer->mapToTextIds(array, Wikibase\Lib\Store\Sql\Terms\Util\ReplicaMasterAwareRecordIdsAcquirer)
#10 /var/www/html/w/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseTermInLangIdsAcquirer.php(62): Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsAcquirer->mapTermsArrayToTermIds(array)
#11 /var/www/html/w/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseItemTermStoreWriter.php(143): Wikibase\Lib\Store\Sql\Terms\DatabaseTermInLangIdsAcquirer->acquireTermInLangIds(array, Closure)
#12 /var/www/html/w/extensions/Wikibase/lib/includes/Store/Sql/Terms/DatabaseItemTermStoreWriter.php(61): Wikibase\Lib\Store\Sql\Terms\DatabaseItemTermStoreWriter->acquireAndInsertTerms(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Term\Fingerprint)
#13 /var/www/html/w/extensions/Wikibase/repo/includes/Store/ItemTermsRebuilder.php(127): Wikibase\Lib\Store\Sql\Terms\DatabaseItemTermStoreWriter->storeTerms(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Term\Fingerprint)
#14 /var/www/html/w/extensions/Wikibase/repo/includes/Store/ItemTermsRebuilder.php(120): Wikibase\Repo\Store\ItemTermsRebuilder->saveTerms(Wikibase\DataModel\Entity\Item)
#15 /var/www/html/w/extensions/Wikibase/repo/includes/Store/ItemTermsRebuilder.php(74): Wikibase\Repo\Store\ItemTermsRebuilder->rebuildTermsForBatch(array)
#16 /var/www/html/w/extensions/Wikibase/repo/includes/Store/Sql/DatabaseSchemaUpdater.php(349): Wikibase\Repo\Store\ItemTermsRebuilder->rebuild()
#17 /var/www/html/w/includes/installer/DatabaseUpdater.php(512): Wikibase\Repo\Store\Sql\DatabaseSchemaUpdater::rebuildItemTerms(MysqlUpdater)
#18 /var/www/html/w/includes/installer/DatabaseUpdater.php(480): DatabaseUpdater->runUpdates(array, boolean)
#19 /var/www/html/w/maintenance/update.php(181): DatabaseUpdater->doUpdates(array)
#20 /var/www/html/w/maintenance/doMaintenance.php(111): UpdateMediaWiki->execute()
#21 /var/www/html/w/maintenance/update.php(253): require_once(string)
#22 {main}

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
wb_terms, wait for replication before building new tables in updatemediawiki/extensions/Wikibasemaster+10 -2
wb_terms, wait for replication before building new tables in updatemediawiki/extensions/WikibaseREL1_35+10 -2
Customize query in gerrit

Event Timeline

Addshore created this task.Nov 29 2020, 3:38 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 29 2020, 3:38 PM
gerritbot added a comment.Nov 29 2020, 3:43 PM

Change 644059 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] wb_terms, wait for replication before building new tables in update

https://gerrit.wikimedia.org/r/644059

gerritbot added a project: Patch-For-Review.Nov 29 2020, 3:43 PM
Addshore added a subscriber: Ladsgroup.Nov 29 2020, 3:59 PM
gerritbot added a comment.Nov 30 2020, 11:29 PM

Change 644217 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@REL1_35] wb_terms, wait for replication before building new tables in update

https://gerrit.wikimedia.org/r/644217

gerritbot added a comment.Dec 1 2020, 2:48 AM

Change 644059 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] wb_terms, wait for replication before building new tables in update

https://gerrit.wikimedia.org/r/644059

gerritbot added a comment.Dec 1 2020, 2:49 AM

Change 644217 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@REL1_35] wb_terms, wait for replication before building new tables in update

https://gerrit.wikimedia.org/r/644217

ReleaseTaggerBot added a project: MW-1.36-notes (1.36.0-wmf.21; 2020-12-08).Dec 1 2020, 3:00 AM
Maintenance_bot removed a project: Patch-For-Review.Dec 1 2020, 3:10 AM
Lucas_Werkmeister_WMDE subscribed.Dec 1 2020, 10:04 AM

I don’t understand why an extra wait for replication should be necessary – DatabaseUpdater::runUpdates() already calls waitForReplication() after each finished update, as far as I can tell.

Addshore closed this task as Resolved.Dec 1 2020, 10:30 AM
Addshore claimed this task.

Indeed it looks as if it should https://gerrit.wikimedia.org/g/mediawiki/core/+/d35735b0d1952b026ed28a2c1bece7cb8ac0d58d/includes/installer/DatabaseUpdater.php#537
Very odd, and per my logs the wait was not timing out..
The additional wait added in Wikibase seemed to make sure that MediaWiki always actually waited until the tables were on the replica.

Restricted Application added a project: User-Addshore. · View Herald TranscriptDec 1 2020, 10:30 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits