Page MenuHomePhabricator

wmf.18 - "Failed to load blob from address" while merging entities
Closed, ResolvedPublic

Description

Showing error "Failed to load blob from address tt:737547000". It works when retried.

Details

Related Gerrit Patches:
mediawiki/extensions/Wikibase : masterPass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup
mediawiki/extensions/Wikibase : wmf/1.32.0-wmf.18Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup
mediawiki/extensions/Wikibase : wmf/1.32.0-wmf.18Log Storage Exceptions in ItemMergeInteractor

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 24 2018, 6:16 AM
Addshore added a subscriber: daniel.
Addshore added a subscriber: Addshore.
Addshore claimed this task.Aug 24 2018, 7:35 AM
Restricted Application added a project: User-Addshore. · View Herald TranscriptAug 24 2018, 7:35 AM

Hi @Bugreporter

I just tried reproducing this on test and everything went fine (using the special page) https://test.wikidata.org/w/index.php?title=Q168782&action=history
I also don't appear to be able to find the message you pasted in the logs (that could just be down to my bad searching)

Could you give me some more details?

  • Was this using the API or special page? Or the gadget?
  • Which entities were you trying to merge? (can you link to the diff that was successful after retrying?)

Using Special:MergeItems. (merge.js also does not work).

Error message (raw):

(wikibase-itemmerge-cant-load-entity-content)

(wikibase-error-unexpected: Failed to load blob)

(wikibase-error-unexpected: Failed to load blob from address tt:403615)

@Bugreporter which entities did you try to merge?

Is this still happening today?
I'm trying to determine if this is related to the wmf.16 or wmf.18 branch of code.
It looks like .18 was deployed last night but everything has been rolled back to .16

Nothing is appearing in logstash as the exception it caught and never logged, instead just presented to the user.
This might be something we want to fix in the ItemMergeInteractor.

I try to merge Q123454 -> Q168780 but still does not work.

(You can try to merge Q123455 -> Q168780 via Special:MergeItems

This comment was removed by Addshore.
Addshore added a comment.EditedAug 24 2018, 8:46 AM

(You can try to merge Q123455 -> Q168780 via Special:MergeItems

I guess these are meant for test.wikidata.org?

If so yes I can reproduce this.

This is on the wmf.18 branch then.

Both entities do actually exist:

Is this still reproducible on wikidata.org currently?
Is that were the initial issue was or was it spotted on test?
I assume that because the tt:737547000 ID in the initial description is so large it was from wikidata.org

Change 455101 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.18] Log Storage Exceptions in ItemMergeInteractor

https://gerrit.wikimedia.org/r/455101

I manually hacked the patch above onto mwdebug1001 and got the following error and stacktraces:

https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.08.24/mediawiki?id=AWVrJfY0KHrb1-g-36vw&_g=h@44136fa

[W3-ILgpAIHsAAEBQebMAAAAT] /wiki/Special:MergeItems   Wikibase\Lib\Store\StorageException from line 208 of /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php: Failed to load blob

Merging from Q123455 to Q168780 on testwikidata

exception.previous.trace

#0 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(206): MediaWiki\Storage\SqlBlobStore->getBlob(string)
#1 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(114): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->loadEntity(stdClass)
#2 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/DispatchingEntityRevisionLookup.php(59): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#3 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(55): Wikibase\Lib\Store\DispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#4 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(201): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#5 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(184): Wikibase\Repo\Interactors\ItemMergeInteractor->loadEntity(Wikibase\DataModel\Entity\ItemId)
#6 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(169): Wikibase\Repo\Interactors\ItemMergeInteractor->isEmpty(Wikibase\DataModel\Entity\ItemId)
#7 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Specials/SpecialMergeItems.php(181): Wikibase\Repo\Interactors\ItemMergeInteractor->mergeItems(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Entity\ItemId, array, string)
#8 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Specials/SpecialMergeItems.php(141): Wikibase\Repo\Specials\SpecialMergeItems->mergeItems(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Entity\ItemId, array, string)
#9 /srv/mediawiki/php-1.32.0-wmf.18/includes/specialpage/SpecialPage.php(569): Wikibase\Repo\Specials\SpecialMergeItems->execute(NULL)
#10 /srv/mediawiki/php-1.32.0-wmf.18/includes/specialpage/SpecialPageFactory.php(581): SpecialPage->run(NULL)
#11 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#12 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(868): MediaWiki->performRequest()
#13 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(525): MediaWiki->main()
#14 /srv/mediawiki/php-1.32.0-wmf.18/index.php(42): MediaWiki->run()
#15 /srv/mediawiki/w/index.php(3): include(string)
#16 {main}

exception.trace

#0 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(114): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->loadEntity(stdClass)
#1 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/DispatchingEntityRevisionLookup.php(59): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#2 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(55): Wikibase\Lib\Store\DispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#3 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(201): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#4 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(184): Wikibase\Repo\Interactors\ItemMergeInteractor->loadEntity(Wikibase\DataModel\Entity\ItemId)
#5 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(169): Wikibase\Repo\Interactors\ItemMergeInteractor->isEmpty(Wikibase\DataModel\Entity\ItemId)
#6 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Specials/SpecialMergeItems.php(181): Wikibase\Repo\Interactors\ItemMergeInteractor->mergeItems(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Entity\ItemId, array, string)
#7 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Specials/SpecialMergeItems.php(141): Wikibase\Repo\Specials\SpecialMergeItems->mergeItems(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Entity\ItemId, array, string)
#8 /srv/mediawiki/php-1.32.0-wmf.18/includes/specialpage/SpecialPage.php(569): Wikibase\Repo\Specials\SpecialMergeItems->execute(NULL)
#9 /srv/mediawiki/php-1.32.0-wmf.18/includes/specialpage/SpecialPageFactory.php(581): SpecialPage->run(NULL)
#10 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#11 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(868): MediaWiki->performRequest()
#12 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(525): MediaWiki->main()
#13 /srv/mediawiki/php-1.32.0-wmf.18/index.php(42): MediaWiki->run()
#14 /srv/mediawiki/w/index.php(3): include(string)
#15 {main}

As with before, once hitting submit on the special page a second time the for submits and merged and redirects the item.

It looks like WikiPageEntityRevisionLookup::getEntityRevision is passed LATEST_FROM_MASTER while merging.
However this is not passed in any way into WikiPageEntityRevisionLookup::loadEntity
So when that method interacts with RevisionStore it does not pass any special flags and thus a replica can be used.

The fix is probably to pass information about if a the master or a replica etc should be used into WikiPageEntityRevisionLookup::loadEntity

Addshore renamed this task from Merge can not create redirect to wmf.18 - "Failed to load blob from address" while merging entities.Aug 24 2018, 9:10 AM
Addshore triaged this task as High priority.

Change 455112 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup

https://gerrit.wikimedia.org/r/455112

Change 455101 abandoned by Addshore:
Log Storage Exceptions in ItemMergeInteractor

https://gerrit.wikimedia.org/r/455101

Nikki added a subscriber: Nikki.Aug 24 2018, 10:42 AM
Robby added a subscriber: Robby.Aug 24 2018, 11:04 AM
Sabas88 added a subscriber: Sabas88.EditedAug 25 2018, 7:04 PM

Using the production version, since it's giving this alert consistently, I saw the json answer and the response is this one.
Specifically I merged Q3020989 and Q56027173

{"error":{"code":"cant-load-entity-content","info":"Cannot access content, revision may be deleted.","extradata":["Failed to load blob","Failed to load blob"],"messages":[{"name":"wikibase-api-cant-load-entity-content","parameters":[],"html":{"*":"Cannot access content, revision may be deleted."}}],"*":"See https://www.wikidata.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at <https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce> for notice of API deprecations and breaking changes."},"servedby":"mw1279"}

Adding: when I reload, some days ago it presented the complete entity (no changes), but today seems to have completed the statements move, but not the remaining steps (setting up the redirect?)

but today seems to have completed the statements move, but not the remaining steps (setting up the redirect?)

Could you provide a link?

Change 455542 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.18] Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup

https://gerrit.wikimedia.org/r/455542

Change 455542 merged by Addshore:
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.18] Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup

https://gerrit.wikimedia.org/r/455542

Mentioned in SAL (#wikimedia-operations) [2018-08-27T11:59:58Z] <addshore@deploy1001> Synchronized php-1.32.0-wmf.18/extensions/Wikibase/lib/includes: [[gerrit:455542]] Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup T202706 (duration: 00m 51s)

Fixed in production, but the patch still needs to be reviewed and merged on master so leaving this open.

This still blocks this weeks train as the master patch should be merged before the branch.
If it is not then it should again be backported onto the newly cut branch.

@Addshore thank you very much, I just tried another merge and I didn't get the error

Change 455112 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup

https://gerrit.wikimedia.org/r/455112