Maniphest T202706

wmf.18 - "Failed to load blob from address" while merging entities
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Bugreporter
	Aug 24 2018, 6:16 AM

Description

Showing error "Failed to load blob from address tt:737547000". It works when retried.

Details

Subject	Repo	Branch	Lines +/-
Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup	mediawiki/extensions/Wikibase	master	+10 -4
Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup	mediawiki/extensions/Wikibase	wmf/1.32.0-wmf.18	+10 -4
Log Storage Exceptions in ItemMergeInteractor	mediawiki/extensions/Wikibase	wmf/1.32.0-wmf.18	+1 -0

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved	Release	• dduvall	T191065 1.32.0-wmf.19 deployment blockers
		Resolved		Addshore	T202706 wmf.18 - "Failed to load blob from address" while merging entities

Event Timeline

Bugreporter created this task.Aug 24 2018, 6:16 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 24 2018, 6:16 AM

Addshore added a project: Multi-Content-Revisions.Aug 24 2018, 7:22 AM

Addshore added a subscriber: daniel.

Addshore subscribed.

Addshore added a project: Wikidata-Campsite.Aug 24 2018, 7:25 AM

Addshore claimed this task.Aug 24 2018, 7:35 AM

Restricted Application added a project: User-Addshore. · View Herald TranscriptAug 24 2018, 7:35 AM

Hi @Bugreporter

I just tried reproducing this on test and everything went fine (using the special page) https://test.wikidata.org/w/index.php?title=Q168782&action=history
I also don't appear to be able to find the message you pasted in the logs (that could just be down to my bad searching)

Could you give me some more details?

Was this using the API or special page? Or the gadget?
Which entities were you trying to merge? (can you link to the diff that was successful after retrying?)

Addshore moved this task from Unsorted 💣 to Back Burner 🏛️ on the User-Addshore board.Aug 24 2018, 7:45 AM

Addshore moved this task from incoming to needs discussion or investigation on the Wikidata board.Aug 24 2018, 7:50 AM

Using Special:MergeItems. (merge.js also does not work).

Error message (raw):

(wikibase-itemmerge-cant-load-entity-content)

(wikibase-error-unexpected: Failed to load blob)

(wikibase-error-unexpected: Failed to load blob from address tt:403615)

Diff: https://test.wikidata.org/w/index.php?title=Q123451&diff=prev&oldid=402673

@Bugreporter which entities did you try to merge?

Is this still happening today?
I'm trying to determine if this is related to the wmf.16 or wmf.18 branch of code.
It looks like .18 was deployed last night but everything has been rolled back to .16

Nothing is appearing in logstash as the exception it caught and never logged, instead just presented to the user.
This might be something we want to fix in the ItemMergeInteractor.

I try to merge Q123454 -> Q168780 but still does not work.

(You can try to merge Q123455 -> Q168780 via Special:MergeItems

Addshore added a comment.Aug 24 2018, 8:43 AM

This comment was removed by Addshore.

In T202706#4529270, @Bugreporter wrote:

(You can try to merge Q123455 -> Q168780 via Special:MergeItems

I guess these are meant for test.wikidata.org?

If so yes I can reproduce this.

This is on the wmf.18 branch then.

Both entities do actually exist:

Is this still reproducible on wikidata.org currently?
Is that were the initial issue was or was it spotted on test?
I assume that because the tt:737547000 ID in the initial description is so large it was from wikidata.org

Change 455101 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.18] Log Storage Exceptions in ItemMergeInteractor

https://gerrit.wikimedia.org/r/455101

gerritbot added a project: Patch-For-Review.Aug 24 2018, 8:51 AM

I manually hacked the patch above onto mwdebug1001 and got the following error and stacktraces:

https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2018.08.24/mediawiki?id=AWVrJfY0KHrb1-g-36vw&_g=h@44136fa

[W3-ILgpAIHsAAEBQebMAAAAT] /wiki/Special:MergeItems   Wikibase\Lib\Store\StorageException from line 208 of /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php: Failed to load blob

Merging from Q123455 to Q168780 on testwikidata

exception.previous.trace

#0 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(206): MediaWiki\Storage\SqlBlobStore->getBlob(string)
#1 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(114): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->loadEntity(stdClass)
#2 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/DispatchingEntityRevisionLookup.php(59): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#3 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(55): Wikibase\Lib\Store\DispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#4 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(201): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#5 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(184): Wikibase\Repo\Interactors\ItemMergeInteractor->loadEntity(Wikibase\DataModel\Entity\ItemId)
#6 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(169): Wikibase\Repo\Interactors\ItemMergeInteractor->isEmpty(Wikibase\DataModel\Entity\ItemId)
#7 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Specials/SpecialMergeItems.php(181): Wikibase\Repo\Interactors\ItemMergeInteractor->mergeItems(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Entity\ItemId, array, string)
#8 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Specials/SpecialMergeItems.php(141): Wikibase\Repo\Specials\SpecialMergeItems->mergeItems(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Entity\ItemId, array, string)
#9 /srv/mediawiki/php-1.32.0-wmf.18/includes/specialpage/SpecialPage.php(569): Wikibase\Repo\Specials\SpecialMergeItems->execute(NULL)
#10 /srv/mediawiki/php-1.32.0-wmf.18/includes/specialpage/SpecialPageFactory.php(581): SpecialPage->run(NULL)
#11 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#12 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(868): MediaWiki->performRequest()
#13 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(525): MediaWiki->main()
#14 /srv/mediawiki/php-1.32.0-wmf.18/index.php(42): MediaWiki->run()
#15 /srv/mediawiki/w/index.php(3): include(string)
#16 {main}

exception.trace

#0 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(114): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->loadEntity(stdClass)
#1 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/DispatchingEntityRevisionLookup.php(59): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#2 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(55): Wikibase\Lib\Store\DispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#3 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(201): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#4 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(184): Wikibase\Repo\Interactors\ItemMergeInteractor->loadEntity(Wikibase\DataModel\Entity\ItemId)
#5 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Interactors/ItemMergeInteractor.php(169): Wikibase\Repo\Interactors\ItemMergeInteractor->isEmpty(Wikibase\DataModel\Entity\ItemId)
#6 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Specials/SpecialMergeItems.php(181): Wikibase\Repo\Interactors\ItemMergeInteractor->mergeItems(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Entity\ItemId, array, string)
#7 /srv/mediawiki/php-1.32.0-wmf.18/extensions/Wikibase/repo/includes/Specials/SpecialMergeItems.php(141): Wikibase\Repo\Specials\SpecialMergeItems->mergeItems(Wikibase\DataModel\Entity\ItemId, Wikibase\DataModel\Entity\ItemId, array, string)
#8 /srv/mediawiki/php-1.32.0-wmf.18/includes/specialpage/SpecialPage.php(569): Wikibase\Repo\Specials\SpecialMergeItems->execute(NULL)
#9 /srv/mediawiki/php-1.32.0-wmf.18/includes/specialpage/SpecialPageFactory.php(581): SpecialPage->run(NULL)
#10 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(288): MediaWiki\Special\SpecialPageFactory->executePath(Title, RequestContext)
#11 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(868): MediaWiki->performRequest()
#12 /srv/mediawiki/php-1.32.0-wmf.18/includes/MediaWiki.php(525): MediaWiki->main()
#13 /srv/mediawiki/php-1.32.0-wmf.18/index.php(42): MediaWiki->run()
#14 /srv/mediawiki/w/index.php(3): include(string)
#15 {main}

As with before, once hitting submit on the special page a second time the for submits and merged and redirects the item.

It looks like WikiPageEntityRevisionLookup::getEntityRevision is passed LATEST_FROM_MASTER while merging.
However this is not passed in any way into WikiPageEntityRevisionLookup::loadEntity
So when that method interacts with RevisionStore it does not pass any special flags and thus a replica can be used.

The fix is probably to pass information about if a the master or a replica etc should be used into WikiPageEntityRevisionLookup::loadEntity

Addshore renamed this task from Merge can not create redirect to wmf.18 - "Failed to load blob from address" while merging entities.Aug 24 2018, 9:10 AM

Addshore triaged this task as High priority.

Addshore added a parent task: T191064: 1.32.0-wmf.18 deployment blockers.

Addshore added a project: Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)).Aug 24 2018, 9:16 AM

Addshore moved this task from To Do (prioritised from top to bottom) to Doing on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Addshore moved this task from Incoming to In Progress on the Wikidata-Campsite board.

Addshore moved this task from Inbox to In Progress on the Multi-Content-Revisions board.

Change 455112 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@master] Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup

https://gerrit.wikimedia.org/r/455112

Addshore moved this task from Doing to Peer Review on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.Aug 24 2018, 10:03 AM

Change 455101 abandoned by Addshore:
Log Storage Exceptions in ItemMergeInteractor

https://gerrit.wikimedia.org/r/455101

Nikki subscribed.Aug 24 2018, 10:42 AM

Robby subscribed.Aug 24 2018, 11:04 AM

Liuxinyu970226 subscribed.Aug 24 2018, 12:59 PM

thcipriani added a parent task: T191065: 1.32.0-wmf.19 deployment blockers.Aug 24 2018, 1:51 PM

thcipriani removed a parent task: T191064: 1.32.0-wmf.18 deployment blockers.

thcipriani mentioned this in T191064: 1.32.0-wmf.18 deployment blockers.Aug 24 2018, 2:02 PM

Using the production version, since it's giving this alert consistently, I saw the json answer and the response is this one.
Specifically I merged Q3020989 and Q56027173

{"error":{"code":"cant-load-entity-content","info":"Cannot access content, revision may be deleted.","extradata":["Failed to load blob","Failed to load blob"],"messages":[{"name":"wikibase-api-cant-load-entity-content","parameters":[],"html":{"*":"Cannot access content, revision may be deleted."}}],"*":"See https://www.wikidata.org/w/api.php for API usage. Subscribe to the mediawiki-api-announce mailing list at &lt;https://lists.wikimedia.org/mailman/listinfo/mediawiki-api-announce&gt; for notice of API deprecations and breaking changes."},"servedby":"mw1279"}

Adding: when I reload, some days ago it presented the complete entity (no changes), but today seems to have completed the statements move, but not the remaining steps (setting up the redirect?)

In T202706#4532470, @Sabas88 wrote:

but today seems to have completed the statements move, but not the remaining steps (setting up the redirect?)

Could you provide a link?

Change 455542 had a related patch set uploaded (by Addshore; owner: Addshore):
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.18] Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup

https://gerrit.wikimedia.org/r/455542

Change 455542 merged by Addshore:
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.18] Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup

https://gerrit.wikimedia.org/r/455542

Mentioned in SAL (#wikimedia-operations) [2018-08-27T11:59:58Z] <addshore@deploy1001> Synchronized php-1.32.0-wmf.18/extensions/Wikibase/lib/includes: [[gerrit:455542]] Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup T202706 (duration: 00m 51s)

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)).Aug 27 2018, 12:00 PM

Fixed in production, but the patch still needs to be reviewed and merged on master so leaving this open.

This still blocks this weeks train as the master patch should be merged before the branch.
If it is not then it should again be backported onto the newly cut branch.

@Addshore thank you very much, I just tried another merge and I didn't get the error

Change 455112 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup

https://gerrit.wikimedia.org/r/455112

ReleaseTaggerBot edited projects, added MW-1.32-notes (WMF-deploy-2018-08-28 (1.32.0-wmf.19)); removed MW-1.32-notes (WMF-deploy-2018-08-21 (1.32.0-wmf.18)).Aug 27 2018, 4:00 PM

Addshore moved this task from Back Burner 🏛️ to Closing ✔️ on the User-Addshore board.Aug 28 2018, 5:42 AM

Addshore moved this task from Peer Review to Test (Verification) on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Lydia_Pintscher closed this task as Resolved.Aug 28 2018, 9:24 AM

Lydia_Pintscher moved this task from Test (Verification) to Done on the Wikidata-Campsite (Wikidata-Campsite-Iteration-∞ (On Hold)) board.

Liuxinyu970226 unsubscribed.Aug 30 2018, 11:59 PM

CCicalese_WMF moved this task from In Progress to Done on the Multi-Content-Revisions board.Sep 18 2018, 10:13 PM

Addshore removed a project: Wikidata-Campsite.Nov 1 2018, 11:21 AM