Page MenuHomePhabricator

BadMethodCallException wbgetentities when getting Lexeme subentities (forms, senses)
Open, HighPublic5 Story Points

Description

The following API call results in a fatal error: https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&origin=*&ids=L11533-F7

API Sandbox: https://www.wikidata.org/wiki/Special:ApiSandbox#action=wbgetentities&format=json&ids=L123-F3&props=labels%7Cdescriptions&languages=de&languagefallback=1

Stacktrace:

BadMethodCallException from line 42 of /srv/mediawiki/php-1.34.0-wmf.5/extensions/WikibaseLexeme/src/DataAccess/Store/MediaWikiPageSubEntityMetaDataAccessor.php: Not Implemented

#0 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TypeDispatchingWikiPageEntityMetaDataAccessor.php(89): Wikibase\Lexeme\DataAccess\Store\MediaWikiPageSubEntityMetaDataAccessor->loadRevisionInformation(array, string)
#1 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(264): Wikibase\Lib\Store\Sql\TypeDispatchingWikiPageEntityMetaDataAccessor->loadRevisionInformation(array, string)
#2 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(186): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->doFetch(string)
#3 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(179): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->loadRevisionInformation(array, string)
#4 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/DispatchingEntityRevisionLookup.php(73): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getLatestRevisionId(Wikibase\Lexeme\Domain\Model\LexemeId, string)
#5 /srv/mediawiki/php-1.34.0-wmf.5/extensions/WikibaseLexeme/src/DataAccess/Store/FormRevisionLookup.php(89): Wikibase\Lib\Store\DispatchingEntityRevisionLookup->getLatestRevisionId(Wikibase\Lexeme\Domain\Model\LexemeId, string)
#6 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(70): Wikibase\Lexeme\DataAccess\Store\FormRevisionLookup->getLatestRevisionId(Wikibase\Lexeme\Domain\Model\FormId, string)
#7 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/CacheRetrievingEntityRevisionLookup.php(108): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getLatestRevisionId(Wikibase\Lexeme\Domain\Model\FormId, string)
#8 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(84): Wikibase\Lib\Store\CacheRetrievingEntityRevisionLookup->getEntityRevisionFromCache(Wikibase\Lexeme\Domain\Model\FormId, integer, string)
#9 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(104): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision(Wikibase\Lexeme\Domain\Model\FormId, integer, string)
#10 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(87): Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision(Wikibase\Lexeme\Domain\Model\FormId, integer, string)
#11 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/repo/includes/Api/GetEntities.php(275): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision(Wikibase\Lexeme\Domain\Model\FormId)
#12 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/repo/includes/Api/GetEntities.php(257): Wikibase\Repo\Api\GetEntities->getEntityRevision(Wikibase\Lexeme\Domain\Model\FormId, boolean)
#13 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/repo/includes/Api/GetEntities.php(136): Wikibase\Repo\Api\GetEntities->getEntityRevisionsFromEntityIds(array, boolean)

Logstash: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2019.05.21/mediawiki?id=AWraLzUKm4XPTDeI3npb&_g=()

Expected behaviour: data of the requested form or sense should be returned.

Event Timeline

Michael created this task.May 21 2019, 12:18 PM
Michael moved this task from Incoming to Ready to estimate on the Wikidata-Campsite board.
hoo renamed this task from BadMethodCallException wbgetentities when getting label + description for Lexeme Form to BadMethodCallException wbgetentities when getting Lexeme Form.May 21 2019, 1:01 PM
hoo added a subscriber: hoo.
hoo renamed this task from BadMethodCallException wbgetentities when getting Lexeme Form to BadMethodCallException wbgetentities when getting Lexemes.May 21 2019, 1:03 PM
hoo renamed this task from BadMethodCallException wbgetentities when getting Lexemes to BadMethodCallException wbgetentities when getting Lexeme subentities (forms, senses).
WMDE-leszek triaged this task as High priority.May 21 2019, 1:05 PM
WMDE-leszek updated the task description. (Show Details)
WMDE-leszek updated the task description. (Show Details)
WMDE-leszek set the point value for this task to 5.
hoo added a comment.May 21 2019, 1:08 PM

This used to work, see T196297 (but we didn't write tests back then :S )

Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptMay 22 2019, 1:31 PM
Ladsgroup removed Ladsgroup as the assignee of this task.Jun 5 2019, 9:58 AM
Ladsgroup added a subscriber: Ladsgroup.

Unlicking the cookie

Unbelievably, it works in beta cluster but not in production and my localhost: https://wikidata.beta.wmflabs.org/w/api.php?action=wbgetentities&ids=L53-S1

Ladsgroup claimed this task.Jun 5 2019, 4:16 PM

Oops!... I licked It Again

Beta cluster lexeme-related configs:

wgLexemePrefixRescoreProfile:"lexeme_prefix"
wgLexemeEnableRepo:true
wgLexemeNamespace:146
wgLexemePrefixSearchProfiles:[]
wgWBQualityConstraintsWikibaseSenseId:"Q505135"
wgLexemeRescoreFunctions:[]
wmgUseWikibaseLexemeCirrusSearch:true
wgLexemeEnableSenses:true
wgLexemeLanguageCodePropertyId:"P218"
wmgUseWikibaseLexeme:true
wgLexemeDisableCirrus:true
wgLexemeUseCirrus:true
wgLexemeTalkNamespace:147
wmgNewWikibaseLexemeCirrusSearch:true
wgLexemeFulltextRescoreProfile:"lexeme_fulltext"
wgLexemePrefixSearchProfile:"lexeme_prefix"
wgWBQualityConstraintsWikibaseLexemeId:"Q505133"

Production:

wgLexemePrefixRescoreProfile:"lexeme_prefix"
wgLexemeEnableRepo:true
wgLexemeNamespace:146
wgLexemePrefixSearchProfiles:[]
wgWBQualityConstraintsWikibaseSenseId:"Q54285715"
wgLexemeRescoreFunctions:[]
wmgUseWikibaseLexemeCirrusSearch:true
wgLexemeEnableSenses:true
wgLexemeLanguageCodePropertyId:"P218"
wmgUseWikibaseLexeme:true
wgLexemeDisableCirrus:true
wgLexemeUseCirrus:true
wgLexemeTalkNamespace:147
wmgNewWikibaseLexemeCirrusSearch:true
wgLexemeFulltextRescoreProfile:"lexeme_fulltext"
wgLexemePrefixSearchProfile:"lexeme_prefix"
wgWBQualityConstraintsWikibaseLexemeId:"Q51885771"

(I'm fascinated that we have "wgLexemeUseCirrus" and "wgLexemeDisableCirrus" both set to true in production and beta cluster)

This is all of wikibase-related configs that differ between production and beta clsuter:

1 wgWBQualityConstraintsWikibaseFormId:"Q505134"
1 wgWBQualityConstraintsWikibaseFormId:"Q54285143"
1 wgWBQualityConstraintsWikibaseItemId:"Q29934200"
1 wgWBQualityConstraintsWikibaseItemId:"Q505131"
1 wgWBQualityConstraintsWikibaseLexemeId:"Q505133"
1 wgWBQualityConstraintsWikibaseLexemeId:"Q51885771"
1 wgWBQualityConstraintsWikibasePropertyId:"Q29934218"
1 wgWBQualityConstraintsWikibasePropertyId:"Q505132"
1 wgWBQualityConstraintsWikibaseSenseId:"Q505135"
1 wgWBQualityConstraintsWikibaseSenseId:"Q54285715"
1 wgWikibaseMultiRepositoryServiceWiringFiles:["\/srv\/mediawiki\/php-1.34.0-wmf.8\/extensions\/Wikibase\/client\/..\/data-access\/src\/MultiRepositoryServiceWiring.php"]
1 wgWikibaseMultiRepositoryServiceWiringFiles:["\/srv\/mediawiki-staging\/php-master\/extensions\/Wikibase\/client\/..\/data-access\/src\/MultiRepositoryServiceWiring.php"]
1 wgWikibasePerRepositoryServiceWiringFiles:["\/srv\/mediawiki\/php-1.34.0-wmf.8\/extensions\/Wikibase\/client\/..\/data-access\/src\/PerRepositoryServiceWiring.php"]
1 wgWikibasePerRepositoryServiceWiringFiles:["\/srv\/mediawiki-staging\/php-master\/extensions\/Wikibase\/client\/..\/data-access\/src\/PerRepositoryServiceWiring.php"]
1 wmgWikibaseClientBadgeClassNames:{"Q17437796":"badge-featuredarticle","Q17437798":"badge-goodarticle","Q17559452":"badge-recommendedarticle","Q17506997":"badge-featuredlist","Q17580674":"badge-featuredportal","Q20748091":"badge-notproofread","Q20748094":"badge-problematic","Q20748092":"badge-proofread","Q20748093":"badge-validated","Q28064618":"badge-digitaldocument","Q51759403":"badge-goodlist"}
1 wmgWikibaseClientBadgeClassNames:{"Q49444":"badge-goodarticle","Q49447":"badge-featuredarticle","Q49448":"badge-recommendedarticle","Q49449":"badge-featuredlist","Q49450":"badge-featuredportal","Q98649":"badge-notproofread","Q98650":"badge-problematic","Q98658":"badge-proofread","Q98651":"badge-validated"}
1 wmgWikibaseClientEchoIcon:{"path":"\/static\/images\/wikibase\/echoIcon.svg"}
1 wmgWikibaseClientEchoIcon:{"url":"\/static\/images\/wikibase\/echoIcon.svg"}
1 wmgWikibaseClientPropertyOrderUrl:"https:\/\/www.wikidata.org\/w\/index.php?title=MediaWiki:Wikibase-SortedProperties&action=raw&sp_ver=1"
1 wmgWikibaseClientRepoConceptBaseUri:"http:\/\/www.wikidata.beta.wmflabs.org\/entity\/"
1 wmgWikibaseClientRepoConceptBaseUri:"http:\/\/www.wikidata.org\/entity\/"
1 wmgWikibaseClientRepositories:{"":{"repoDatabase":"wikidatawiki","baseUri":"http:\/\/www.wikidata.org\/entity\/","entityNamespaces":{"item":0,"property":120,"lexeme":146},"prefixMapping":{"":""}}}
1 wmgWikibaseClientRepositories:{"":{"repoDatabase":"wikidatawiki","entityNamespaces":{"item":0,"property":120,"lexeme":146},"baseUri":"https:\/\/wikidata.beta.wmflabs.org\/entity\/","prefixMapping":{"":""}},"commons":{"repoDatabase":"commonswiki","entityNamespaces":{"mediainfo":"6\/mediainfo"},"baseUri":"https:\/\/commons.wikimedia.beta.wmflabs.org\/entity\/","prefixMapping":[]}}
1 wmgWikibaseClientRepoUrl:"https:\/\/wikidata.beta.wmflabs.org"
1 wmgWikibaseClientRepoUrl:"https:\/\/www.wikidata.org"
1 wmgWikibaseClientUseTermsTableSearchFields:false
1 wmgWikibaseClientUseTermsTableSearchFields:true
1 wmgWikibaseFeatureFlagWbeditentitySetEmptyAliases:false
1 wmgWikibaseFeatureFlagWbeditentitySetEmptyAliases:true
1 wmgWikibaseRepoBadgeItems:{"Q17437798":"wb-badge-goodarticle","Q17437796":"wb-badge-featuredarticle","Q17559452":"wb-badge-recommendedarticle","Q17506997":"wb-badge-featuredlist","Q17580674":"wb-badge-featuredportal","Q20748091":"wb-badge-notproofread","Q20748094":"wb-badge-problematic","Q20748092":"wb-badge-proofread","Q20748093":"wb-badge-validated","Q28064618":"wb-badge-digitaldocument","Q51759403":"wb-badge-goodlist"}
1 wmgWikibaseRepoBadgeItems:{"Q49444":"wb-badge-goodarticle","Q49447":"wb-badge-featuredarticle","Q49448":"wb-badge-recommendedarticle","Q49449":"wb-badge-featuredlist","Q49450":"wb-badge-featuredportal","Q98649":"wb-badge-notproofread","Q98650":"wb-badge-problematic","Q98658":"wb-badge-proofread","Q98651":"wb-badge-validated"}
1 wmgWikibaseSearchIndexProperties:[]
1 wmgWikibaseSearchIndexPropertiesExclude:[]
1 wmgWikibaseSearchIndexPropertiesExclude:["P304","P433","P478","P558","P3903","P3921","P4316","P1433","P2860"]
1 wmgWikibaseSearchIndexProperties:["P31","P279"]
1 wmgWikibaseSearchStatementBoosts:[]
1 wmgWikibaseSearchStatementBoosts:{"P31=Q4167410":-10,"P31=Q13442814":-5,"P31=Q18918145":-5}
1 wmgWikibaseSSRTermboxServerUrl:"https:\/\/ssr-termbox.wmflabs.org\/termbox"
1 wmgWikibaseUseSSRTermbox:true

The only thing that might be at fault is that if we enable federation, it would fix it (very very unlikely). The next step is to compare mediawiki core and similar paths. Will dig deeper tomorrow.

If you set $wgWBRepoSettings['useEntitySourceBasedFederation'] = true; in localhost, it works just fine:

{
    "entities": {
        "L16-F1": {
            "pageid": 513,
            "ns": 146,
            "title": "Lexeme:L16",
            "lastrevid": 515,
            "modified": "2019-05-22T15:06:48Z",
            "id": "L16-F1",
            "representations": {
                "en": {
                    "language": "en",
                    "value": "ff"
                }
            },
            "grammaticalFeatures": [],
            "claims": {}
        }
    },
    "success": 1
}

How about adding a regression (failing) test as a first step? As apparently there has been no tests for this behaviour, no wonder it was (well, I did) broken when changes to federation have been made.

Change 514883 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/WikibaseLexeme@master] Add test for wbgetentities with subentities

https://gerrit.wikimedia.org/r/514883

So this is the backtrace when you set $wgWBRepoSettings['useEntitySourceBasedFederation'] = true;:

/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php:106
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/TypeDispatchingWikiPageEntityMetaDataAccessor.php:88
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php:264
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php:186
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php:125
/var/lib/mediawiki2/extensions/Wikibase/data-access/src/ByTypeDispatchingEntityRevisionLookup.php:54
/var/lib/mediawiki2/extensions/WikibaseLexeme/src/DataAccess/Store/SenseRevisionLookup.php:55
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php:54
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:104
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:87
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:104
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:87
/var/lib/mediawiki2/extensions/Wikibase/repo/includes/Api/GetEntities.php:275
/var/lib/mediawiki2/extensions/Wikibase/repo/includes/Api/GetEntities.php:257
/var/lib/mediawiki2/extensions/Wikibase/repo/includes/Api/GetEntities.php:136
/var/lib/mediawiki2/includes/api/ApiMain.php:1595
/var/lib/mediawiki2/includes/api/ApiMain.php:500
/var/lib/mediawiki2/tests/phpunit/includes/api/ApiTestCase.php:106
/var/lib/mediawiki2/extensions/Wikibase/repo/tests/phpunit/includes/Api/WikibaseApiTestCase.php:156
/var/lib/mediawiki2/extensions/WikibaseLexeme/tests/phpunit/mediawiki/Api/LexemeGetEntitiesTest.php:105
/var/lib/mediawiki2/tests/phpunit/MediaWikiTestCase.php:427
/var/lib/mediawiki2/maintenance/doMaintenance.php:99

This is a big red herring and waste of time, I looked at everything in depth and nothing was responsible. The part that is actually responsible is EntityPrefetcher. GetEntities do prefetch and when you disable EntitySourceBasedFederation, The EntityPrefetcher is actually a federated one which is DispatchingEntityPrefetcher but that doesn't distinguish between different prefetchers of entity types that exist in one federated repo. It's repo-based EntityPrefetcher. while if you set useEntitySourceBasedFederation, it uses MultipleEntitySourceServices to get the EntityPrefetcher (I still don't know how, otherwise I would have made the patch for it already) which that returns ByTypeDispatchingEntityPrefetcher which ignores entity types that don't have EntityPrefetcher defined so it lets the system flow naturally. I need to consolidate and find a better solution than DispatchingEntityPrefetcher but @WMDE-leszek knows way better than I do. Please take a look.

So it seems all of the buggy code will go away soon (TM) according to @WMDE-leszek so I recommend just setting $wgWBRepoSettings['useEntitySourceBasedFederation'] = true; for both commons and wikidata.

Cannot see why the patch is failing, because in zuul's comment on the patch, the link to the failing job gives 404 https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/52351/console (this happened before to me, is there a pattern/rule/reason for that?)

Cannot see why the patch is failing, because in zuul's comment on the patch, the link to the failing job gives 404 https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/52351/console (this happened before to me, is there a pattern/rule/reason for that?)

It means it's too old and jenkins evicted the results page. Do a recheck.

Cool good to know thank you @Ladsgroup

Change 518090 had a related patch set uploaded (by WMDE-leszek; owner: Ladsgroup):
[mediawiki/extensions/WikibaseLexeme@master] Add test for wbgetentities with subentities

https://gerrit.wikimedia.org/r/518090

Change 518091 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/WikibaseLexeme@master] Do not die when Wikibase prefetches Sense or Form Id

https://gerrit.wikimedia.org/r/518091

Addshore moved this task from incoming to in progress on the Wikidata board.Fri, Jun 21, 11:25 PM
Ladsgroup removed Ladsgroup as the assignee of this task.Mon, Jul 8, 11:56 AM