Page MenuHomePhabricator
Create Task
Maniphest T229816

RebuildItemTerms (and probably RebuildPropertyTerms) explodes when there's a (double) redirect
Closed, ResolvedPublic
Actions

Description

Problem
Double redirects, if they exist at the moment of rebuilding the new terms store will cause the script to fatal, as those double redirects are not being handled properly (both proeprty and item terms rebuilders).

Solution
Double redirects need to be skipped. Only the final destination item's term need to be moved to the store.

Details

SubjectRepoBranchLines +/-
Ignore ItemLookupException in ItemTermsRebuildermediawiki/extensions/Wikibasemaster+4 -0
Fix the ItemTermsRebuilder exception for redirectsmediawiki/extensions/Wikibasemaster+6 -2
Ignore in case of UnresolvedEntityRedirect in ItemTermsRebuildermediawiki/extensions/Wikibasemaster+7 -2
Customize query in gerrit

Related Objects

Event Timeline

Ladsgroup created this task.Aug 5 2019, 12:44 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 5 2019, 12:44 PM
Ladsgroup added a comment.Aug 5 2019, 1:03 PM

There's literally a TODO for this :)

Ladsgroup claimed this task.Aug 5 2019, 2:32 PM
Ladsgroup moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptAug 5 2019, 2:32 PM
gerritbot added a comment.Aug 5 2019, 2:36 PM

Change 528160 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Ignore in case of UnresolvedEntityRedirect in ItemTermsRebuilder

https://gerrit.wikimedia.org/r/528160

gerritbot added a project: Patch-For-Review.Aug 5 2019, 2:36 PM
• alaa_wmde moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 6 2019, 3:09 PM
gerritbot added a comment.Aug 6 2019, 3:44 PM

Change 528160 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Ignore in case of UnresolvedEntityRedirect in ItemTermsRebuilder

https://gerrit.wikimedia.org/r/528160

Maintenance_bot removed a project: Patch-For-Review.Aug 6 2019, 4:10 PM
gerritbot added a comment.Aug 6 2019, 4:27 PM

Change 528514 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Fix the ItemTermsRebuilder exception for redirects

https://gerrit.wikimedia.org/r/528514

gerritbot added a project: Patch-For-Review.Aug 6 2019, 4:27 PM
Ladsgroup moved this task from Test (Verification) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 6 2019, 4:28 PM

Sorry, I made a mistake :(

gerritbot added a comment.Aug 7 2019, 9:34 PM

Change 528514 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Fix the ItemTermsRebuilder exception for redirects

https://gerrit.wikimedia.org/r/528514

Maintenance_bot removed a project: Patch-For-Review.Aug 7 2019, 10:10 PM
Ladsgroup moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 7 2019, 10:17 PM
• alaa_wmde added a comment.Aug 12 2019, 12:31 PM

This doesn't fix the issue. When testing the updated script with double-redirects, it still failed due to the actual exceptipon being thrown is ItemLookupException and not RevisionedUnresolvedRedirectException.

Wikibase\DataModel\Services\Lookup\ItemLookupException from line 33 of /var/www/mediawiki/vendor/wikibase/data-model-services/src/Lookup/LegacyAdapterItemLookup.php: Unresolved redirect from Q2428 to Q2429
#0 /var/www/mediawiki/extensions/Wikibase/repo/includes/Store/ItemTermsRebuilder.php(109): Wikibase\DataModel\Services\Lookup\LegacyAdapterItemLookup->getItemForId(Object(Wikibase\DataModel\Entity\ItemId))
#1 /var/www/mediawiki/extensions/Wikibase/repo/includes/Store/ItemTermsRebuilder.php(73): Wikibase\Repo\Store\ItemTermsRebuilder->rebuildTermsForBatch(Array)
#2 /var/www/mediawiki/extensions/Wikibase/repo/maintenance/rebuildItemTerms.php(84): Wikibase\Repo\Store\ItemTermsRebuilder->rebuild()
#3 /var/www/mediawiki/maintenance/doMaintenance.php(99): Wikibase\RebuildItemTerms->execute()
#4 /var/www/mediawiki/extensions/Wikibase/repo/maintenance/rebuildItemTerms.php(123): require_once('/var/www/mediaw...')
#5 {main}
Wikibase\Lib\Store\RevisionedUnresolvedRedirectException from line 152 of /var/www/mediawiki/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php: Unresolved redirect from Q2428 to Q2429
#0 /var/www/mediawiki/extensions/Wikibase/lib/includes/Store/DispatchingEntityRevisionLookup.php(59): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId), 0, 'replica')
#1 /var/www/mediawiki/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(54): Wikibase\Lib\Store\DispatchingEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId), 0, 'replica')
#2 /var/www/mediawiki/extensions/Wikibase/lib/includes/Store/CacheRetrievingEntityRevisionLookup.php(75): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId), 0, 'replica')
#3 /var/www/mediawiki/extensions/Wikibase/lib/includes/Store/RevisionBasedEntityLookup.php(39): Wikibase\Lib\Store\CacheRetrievingEntityRevisionLookup->getEntityRevision(Object(Wikibase\DataModel\Entity\ItemId))
#4 /var/www/mediawiki/extensions/WikibaseLexeme/vendor/wikibase/data-model-services/src/Lookup/RedirectResolvingEntityLookup.php(53): Wikibase\Lib\Store\RevisionBasedEntityLookup->getEntity(Object(Wikibase\DataModel\Entity\ItemId))
#5 /var/www/mediawiki/vendor/wikibase/data-model-services/src/Lookup/LegacyAdapterItemLookup.php(31): Wikibase\DataModel\Services\Lookup\RedirectResolvingEntityLookup->getEntity(Object(Wikibase\DataModel\Entity\ItemId))
#6 /var/www/mediawiki/extensions/Wikibase/repo/includes/Store/ItemTermsRebuilder.php(109): Wikibase\DataModel\Services\Lookup\LegacyAdapterItemLookup->getItemForId(Object(Wikibase\DataModel\Entity\ItemId))
#7 /var/www/mediawiki/extensions/Wikibase/repo/includes/Store/ItemTermsRebuilder.php(73): Wikibase\Repo\Store\ItemTermsRebuilder->rebuildTermsForBatch(Array)
#8 /var/www/mediawiki/extensions/Wikibase/repo/maintenance/rebuildItemTerms.php(84): Wikibase\Repo\Store\ItemTermsRebuilder->rebuild()
#9 /var/www/mediawiki/maintenance/doMaintenance.php(99): Wikibase\RebuildItemTerms->execute()
#10 /var/www/mediawiki/extensions/Wikibase/repo/maintenance/rebuildItemTerms.php(123): require_once('/var/www/mediaw...')
#11 {main}

Suggestion here:

  • catch ItemLookupException and report to output those items that fail to rebuild (with exception msg); or
  • fail on all exception so that we fix or decide what to do case-by-case
• alaa_wmde triaged this task as High priority.Aug 12 2019, 12:31 PM
• alaa_wmde moved this task from Test (Verification) to To Do (prioritised from top to bottom) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

This will block migration of item terms to new store

• alaa_wmde updated the task description. (Show Details)Aug 12 2019, 12:35 PM
ReleaseTaggerBot added a project: MW-1.34-notes (1.34.0-wmf.19; 2019-08-20).Aug 20 2019, 11:01 AM
ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.20; 2019-08-27); removed MW-1.34-notes (1.34.0-wmf.19; 2019-08-20).Aug 20 2019, 2:02 PM
Ladsgroup added a comment.Aug 27 2019, 7:54 PM
In T229816#5409434, @alaa_wmde wrote:

This doesn't fix the issue. When testing the updated script with double-redirects, it still failed due to the actual exceptipon being thrown is ItemLookupException and not RevisionedUnresolvedRedirectException.

That's not correct, I got that in beta cluster while doing the migration.

I think we should just ignore anything that can't be read or written.

gerritbot added a comment.Aug 27 2019, 7:55 PM

Change 532777 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/Wikibase@master] Ignore ItemLookupException in ItemTermsRebuilder

https://gerrit.wikimedia.org/r/532777

gerritbot added a project: Patch-For-Review.Aug 27 2019, 7:55 PM
Ladsgroup moved this task from To Do (prioritised from top to bottom) to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 27 2019, 7:55 PM
gerritbot added a comment.Aug 27 2019, 11:56 PM

Change 532777 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Ignore ItemLookupException in ItemTermsRebuilder

https://gerrit.wikimedia.org/r/532777

ReleaseTaggerBot edited projects, added MW-1.34-notes (1.34.0-wmf.21; 2019-09-03); removed MW-1.34-notes (1.34.0-wmf.20; 2019-08-27).Aug 28 2019, 12:00 AM
Ladsgroup moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 28 2019, 12:02 AM
Maintenance_bot removed a project: Patch-For-Review.Aug 28 2019, 12:10 AM
Ladsgroup mentioned this in T225055: Switch `tmpItemTermsMigrationStages` to MIGRATION_WRITE_BOTH.Aug 28 2019, 11:07 AM
WMDE-leszek closed this task as Resolved.Sep 11 2019, 11:25 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL