Page MenuHomePhabricator

[SW] [LEX] [TECH] MWContentSerializationException: $entityId (L123) and $targetId can not be the same.
Closed, DeclinedPublicSecurity

Description

Error
labels.normalized_message
[{reqId}] {exception_url}   MWContentSerializationException: $entityId (L123) and $targetId can not be the same.
error.stack_trace
from /srv/mediawiki/php-1.42.0-wmf.1/extensions/Wikibase/lib/includes/Store/EntityContentDataCodec.php(300)
#0 /srv/mediawiki/php-1.42.0-wmf.1/extensions/Wikibase/repo/includes/Content/EntityHandler.php(391): Wikibase\Lib\Store\EntityContentDataCodec->decodeRedirect(string, NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.1/includes/Revision/RevisionStore.php(1237): Wikibase\Repo\Content\EntityHandler->unserializeContent(string, NULL)
#2 /srv/mediawiki/php-1.42.0-wmf.1/includes/Revision/RevisionStore.php(1520): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, NULL, NULL, NULL, integer)
#3 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#4 /srv/mediawiki/php-1.42.0-wmf.1/includes/Revision/SlotRecord.php(324): call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#5 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQueryRevisionsBase.php(567): MediaWiki\Revision\SlotRecord->getContent()
#6 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQueryRevisionsBase.php(464): ApiQueryRevisionsBase->extractSlotInfo(MediaWiki\Revision\SlotRecord, integer, NULL)
#7 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQueryRevisionsBase.php(388): ApiQueryRevisionsBase->extractAllSlotInfo(MediaWiki\Revision\RevisionStoreRecord, integer)
#8 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQueryRevisions.php(430): ApiQueryRevisionsBase->extractRevisionInfo(MediaWiki\Revision\RevisionStoreRecord, stdClass)
#9 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQueryRevisionsBase.php(133): ApiQueryRevisions->run()
#10 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQuery.php(695): ApiQueryRevisionsBase->execute()
#11 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiMain.php(1931): ApiQuery->execute()
#12 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiMain.php(908): ApiMain->executeAction()
#13 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiMain.php(879): ApiMain->executeActionWithErrorHandling()
#14 /srv/mediawiki/php-1.42.0-wmf.1/api.php(95): ApiMain->execute()
#15 /srv/mediawiki/php-1.42.0-wmf.1/api.php(48): wfApiMain()
#16 /srv/mediawiki/w/api.php(3): require(string)
#17 {main}
error.stack.previous_trace
from /srv/mediawiki/php-1.42.0-wmf.1/extensions/Wikibase/lib/packages/wikibase/data-model/src/Entity/EntityRedirect.php(41)
#0 /srv/mediawiki/php-1.42.0-wmf.1/extensions/Wikibase/lib/includes/Store/EntityContentDataCodec.php(297): Wikibase\DataModel\Entity\EntityRedirect->__construct(Wikibase\Lexeme\Domain\Model\LexemeId, Wikibase\Lexeme\Domain\Model\LexemeId)
#1 /srv/mediawiki/php-1.42.0-wmf.1/extensions/Wikibase/repo/includes/Content/EntityHandler.php(391): Wikibase\Lib\Store\EntityContentDataCodec->decodeRedirect(string, NULL)
#2 /srv/mediawiki/php-1.42.0-wmf.1/includes/Revision/RevisionStore.php(1237): Wikibase\Repo\Content\EntityHandler->unserializeContent(string, NULL)
#3 /srv/mediawiki/php-1.42.0-wmf.1/includes/Revision/RevisionStore.php(1520): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, NULL, NULL, NULL, integer)
#4 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#5 /srv/mediawiki/php-1.42.0-wmf.1/includes/Revision/SlotRecord.php(324): call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#6 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQueryRevisionsBase.php(567): MediaWiki\Revision\SlotRecord->getContent()
#7 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQueryRevisionsBase.php(464): ApiQueryRevisionsBase->extractSlotInfo(MediaWiki\Revision\SlotRecord, integer, NULL)
#8 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQueryRevisionsBase.php(388): ApiQueryRevisionsBase->extractAllSlotInfo(MediaWiki\Revision\RevisionStoreRecord, integer)
#9 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQueryRevisions.php(430): ApiQueryRevisionsBase->extractRevisionInfo(MediaWiki\Revision\RevisionStoreRecord, stdClass)
#10 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQueryRevisionsBase.php(133): ApiQueryRevisions->run()
#11 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiQuery.php(695): ApiQueryRevisionsBase->execute()
#12 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiMain.php(1931): ApiQuery->execute()
#13 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiMain.php(908): ApiMain->executeAction()
#14 /srv/mediawiki/php-1.42.0-wmf.1/includes/api/ApiMain.php(879): ApiMain->executeActionWithErrorHandling()
#15 /srv/mediawiki/php-1.42.0-wmf.1/api.php(95): ApiMain->execute()
#16 /srv/mediawiki/php-1.42.0-wmf.1/api.php(48): wfApiMain()
#17 /srv/mediawiki/w/api.php(3): require(string)
#18 {main}
Impact
Notes

Seen after deploy of T348354: 1.42.0-wmf.1 deployment blockers to group1.

Details

Request URL
https://www.wikidata.org/w/api.php?action=query&continue=*&format=*&formatversion=*&prop=*&rvcontinue=*&rvlimit=*&rvprop=*&titles=*

Event Timeline

Michael changed the subtype of this task from "Production Error" to "Security Issue".Oct 18 2023, 7:25 PM
Michael subscribed.

This is being thrown in https://gerrit.wikimedia.org/g/mediawiki/extensions/Wikibase/+/9d617f33075a35e2de369f1ecd82f28ec1686520/lib/packages/wikibase/data-model/src/Entity/EntityRedirect.php#41

Wikibase\DataModel\Entity\EntityRedirect
		if ( $entityId->getSerialization() === $targetId->getSerialization() ) {
			throw new InvalidArgumentException( '$entityId (' . $entityId . ') and $targetId can not be the same.' );
		}

(It would probably be nice to parametrize that message, but not sure how easy it is to inject a logger there.)

Looking at logstash, it does not seem to be common, 6 events in the last hour. However, widening the timeframe to one year, shows entries that seem to have started in August 2023 with 1.41.0-wmf.20.

Suspiciously, all of them are about L123. Do we somehow have a corrupted value/redirect in our database? Or is there somehow a way to create redirects from Lexemes (other Entites too?) to themselves but so far it was only used on L123?

I did not find any tasks that look related to this, so if there is indeed a possibility to create redirects of some Entities to themselves, it seems to be so far unknown.
I'm changing this to a security issue until we can rule that out. The intention is to limit the access to this to prevent it from being abused, if such a problem exists.

Michael set Security to Software security bug.Oct 18 2023, 7:29 PM
Michael added projects: Security, Security-Team.
Michael changed the visibility from "Public (No Login Required)" to "Custom Policy".

This might hint at a possibility to create redirects of (some?) Wikidata Entities to themselves, and thus inject redirects into the DB that should not exist there. Limiting the scope of this task until we can rule that out.

As far as I can tell, there is no ongoing abuse of this potential vulnerability, if it exists.

I hope, I'm not overreacting here.

Ok, maybe I did overreact a bit here. I now found a previous mention of this in T217329: bug in 1.33.0-wmf.18+ breaks abstract dumps on testwikidatawiki, among other things | MWContentSerializationException $entityId and $targetId can not be the same and it also already mentions L123. How do I turn this back into a normal, public production error task?

brennen changed the visibility from "Custom Policy" to "Public (No Login Required)".

@brennen Thank you for fixing the visibility of this task 🙏

Adding Wikibase Reuse Team since the Data Model Lib is in their scope

The bad revision is 843735157, if you try to view it you can also get the error via index.php instead of api.php.

image.png (79×1 px, 63 KB)

What if we just revdelete that revision? It doesn’t sound like there are any others like it, and it’s no longer possible to make a lexeme redirect to itself, so I think we only need to decide what to do with this one weird legacy revision.

What if we just revdelete that revision? It doesn’t sound like there are any others like it, and it’s no longer possible to make a lexeme redirect to itself, so I think we only need to decide what to do with this one weird legacy revision.

Yes, I think this would also be my preferred solution. This should never have happened in the first place, and if it ever happens again, then a hard error is probably what we want, because something is very wrong.
If we can "fix" this particular instance in our database safely by just doing a revdelete, then that seems like a good way forward to me to resolve this particular error.

Making that error "nicer" in some form can still be a follow-up, if we consider that worthwhile.

ItamarWMDE renamed this task from MWContentSerializationException: $entityId (L123) and $targetId can not be the same. to [LEX] [TECH] MWContentSerializationException: $entityId (L123) and $targetId can not be the same..Oct 24 2023, 1:43 PM

Task Review:

  • This is probably a fluke revision, created before the check mentioned in the code example was introduced.
  • Therefore, revdeleting is probably the best course of action

Task Prio Notes:

  • Affects end users /production
  • Does not affect monitoring
  • Does not affect development efforts
  • Does not affect onboarding efforts
  • Does not affect additional stakeholders
ItamarWMDE renamed this task from [LEX] [TECH] MWContentSerializationException: $entityId (L123) and $targetId can not be the same. to [SW] [LEX] [TECH] MWContentSerializationException: $entityId (L123) and $targetId can not be the same..Oct 24 2023, 1:51 PM
ItamarWMDE moved this task from Incoming to [DOT] Prioritized on the wmde-wikidata-tech board.

Revdeleted. Getting all the L123 revisions via the API succeeds now, so I think this is resolved. (It’s probably possible for a sysop to trigger the error by looking at the deleted revision, but let’s not worry about that IMHO.)

Let’s look at logstash again in a few days.

Let’s look at logstash again in a few days.

Note: there will be a bunch of exceptions from today (just now), because I was testing with the URL from T217329#7181299 and trying to figure out why I was still seeing the error – I didn’t realize that, unlike the normal user interface, the API will directly (try to) show you deleted content as long as you have the permission to see it. So if you access that URL while logged into an account with sysop rights, you get the error, but otherwise you get a response indicating hidden content instead.

The revision is indeed deleted. However, searching the error message on Logstash shows there were two more occurrences on the 25th of October and another on 26th.

Both of those exceptions have an api.php URL with action=query, prop=revisions and rvprop=content (and they’re also both for L123 btw). They also have the referrer set to https://www.wikidata.org/wiki/Lexeme:L123, which I find unexpected (I was first thinking of a bot, but that wouldn’t have a referrer)… is there some user script that loads the history (hundreds of revisions back), and therefore makes privileged API requests if a sysop using that script visits the page?

And I think I realized the answer to my own question while writing it: I bet it’s WikidataTrust.js by @Ricordisamoa, which annotates statements with the users who added them (a task that requires going through the whole history). :/

So, quick summary as I see it:

  • L123 has one (1) bad revision which causes server errors when viewed
    • it is now impossible to create more revisions like it, so the only question is what to do about the existing revision
  • we deleted the revision, in the expectation that no sysop would have a reason to view it again, which should effectively eliminate the error
  • however, it turns out the API will directly try to return the content of deleted revisions if the user has the right to see it
    • unlike the normal interface where you have to click through a confirmation dialog to go to Special:Undelete first
  • and, there is a user script used by some users with the right to see deleted revisions which will make such API requests

I’m not sure where to go from here, to be honest. I don’t think the user script could easily skip the bad revision range. Would it be reasonable to ask for the user script to be disabled on L123 entirely? Or should we try to fix the error server-side after all?

brennen moved this task from Backlog to Logs/Train on the User-brennen board.
brennen unsubscribed.

I think it's reasonable to ask that the script should have an exclusion list for high edit volume items / lexemes / properties that are used frequently for testing purposes, such as L123. As we already realized that this is just a case of a particular revision misbehaving since it had the misfortune to be persisted while the code was at a volatile state. Or, am I missing something here? What I'm trying to say is that I don't think we should make backend changes just to support one known edge case on one persisted revision.

Alright. @Ricordisamoa: how would you feel about disabling WikidataTrust on L123? (I’m thinking of something like: if ( mw.config.get( 'wbEntityId' ) === 'L123' ) { return; } )

(In theory, it would only need to be disabled for sysops, but it’s probably less confusing to just disable it for everyone. Perhaps a console.log() would also be nice, so people at least have some way to find out why the annotations aren’t showing up.)

ItamarWMDE changed the task status from Open to Stalled.Nov 7 2023, 9:37 AM

Marking as stalled pending response.

Lucas_Werkmeister_WMDE changed the task status from Stalled to Open.Dec 8 2023, 2:38 PM

The edit request was fulfilled, and the error hasn’t happened again since then:

image.png (395×680 px, 33 KB)

Unstalling, and I think this is good to close then. (Arguably we could close it as Declined instead of Resolved, since we “just” made the error harder to reach and decided against fully fixing it.)

@Lucas_Werkmeister_WMDE Do you think we should still close this as declined?

According to logstash the error hasn’t happened in the past 30 days, so yeah, let’s close it IMHO. If someone complains about excessive logspam in future, we can reconsider it.

ArthurTaylor subscribed.

Last occurrence was in 22. December 2023. Closing this as no longer reproducible.