Page MenuHomePhabricator

BadMethodCallException wbgetentities when getting Lexeme subentities (forms, senses)
Closed, ResolvedPublic5 Estimated Story PointsPRODUCTION ERROR

Description

To Do

  • implement MediaWikiPageSubEntityMetaDataAccessor::loadRevisionInformation
  • add tests for the new implementation

The following API call results in a fatal error: https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&origin=*&ids=L11533-F7

API Sandbox: https://www.wikidata.org/wiki/Special:ApiSandbox#action=wbgetentities&format=json&ids=L123-F3&props=labels%7Cdescriptions&languages=de&languagefallback=1

Stacktrace:

BadMethodCallException from line 42 of /srv/mediawiki/php-1.34.0-wmf.5/extensions/WikibaseLexeme/src/DataAccess/Store/MediaWikiPageSubEntityMetaDataAccessor.php: Not Implemented

#0 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TypeDispatchingWikiPageEntityMetaDataAccessor.php(89): Wikibase\Lexeme\DataAccess\Store\MediaWikiPageSubEntityMetaDataAccessor->loadRevisionInformation(array, string)
#1 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(264): Wikibase\Lib\Store\Sql\TypeDispatchingWikiPageEntityMetaDataAccessor->loadRevisionInformation(array, string)
#2 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php(186): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->doFetch(string)
#3 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(179): Wikibase\Lib\Store\Sql\PrefetchingWikiPageEntityMetaDataAccessor->loadRevisionInformation(array, string)
#4 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/DispatchingEntityRevisionLookup.php(73): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getLatestRevisionId(Wikibase\Lexeme\Domain\Model\LexemeId, string)
#5 /srv/mediawiki/php-1.34.0-wmf.5/extensions/WikibaseLexeme/src/DataAccess/Store/FormRevisionLookup.php(89): Wikibase\Lib\Store\DispatchingEntityRevisionLookup->getLatestRevisionId(Wikibase\Lexeme\Domain\Model\LexemeId, string)
#6 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(70): Wikibase\Lexeme\DataAccess\Store\FormRevisionLookup->getLatestRevisionId(Wikibase\Lexeme\Domain\Model\FormId, string)
#7 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/CacheRetrievingEntityRevisionLookup.php(108): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getLatestRevisionId(Wikibase\Lexeme\Domain\Model\FormId, string)
#8 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(84): Wikibase\Lib\Store\CacheRetrievingEntityRevisionLookup->getEntityRevisionFromCache(Wikibase\Lexeme\Domain\Model\FormId, integer, string)
#9 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(104): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision(Wikibase\Lexeme\Domain\Model\FormId, integer, string)
#10 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(87): Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision(Wikibase\Lexeme\Domain\Model\FormId, integer, string)
#11 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/repo/includes/Api/GetEntities.php(275): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision(Wikibase\Lexeme\Domain\Model\FormId)
#12 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/repo/includes/Api/GetEntities.php(257): Wikibase\Repo\Api\GetEntities->getEntityRevision(Wikibase\Lexeme\Domain\Model\FormId, boolean)
#13 /srv/mediawiki/php-1.34.0-wmf.5/extensions/Wikibase/repo/includes/Api/GetEntities.php(136): Wikibase\Repo\Api\GetEntities->getEntityRevisionsFromEntityIds(array, boolean)

Logstash: https://logstash.wikimedia.org/app/kibana#/doc/logstash-*/logstash-2019.05.21/mediawiki?id=AWraLzUKm4XPTDeI3npb&_g=()

Expected behaviour: data of the requested form or sense should be returned.

Event Timeline

hoo renamed this task from BadMethodCallException wbgetentities when getting label + description for Lexeme Form to BadMethodCallException wbgetentities when getting Lexeme Form.May 21 2019, 1:01 PM
hoo subscribed.
hoo renamed this task from BadMethodCallException wbgetentities when getting Lexeme Form to BadMethodCallException wbgetentities when getting Lexemes.May 21 2019, 1:03 PM
hoo renamed this task from BadMethodCallException wbgetentities when getting Lexemes to BadMethodCallException wbgetentities when getting Lexeme subentities (forms, senses).
WMDE-leszek updated the task description. (Show Details)
WMDE-leszek updated the task description. (Show Details)
WMDE-leszek set the point value for this task to 5.

This used to work, see T196297 (but we didn't write tests back then :S )

Beta cluster lexeme-related configs:

wgLexemePrefixRescoreProfile:"lexeme_prefix"
wgLexemeEnableRepo:true
wgLexemeNamespace:146
wgLexemePrefixSearchProfiles:[]
wgWBQualityConstraintsWikibaseSenseId:"Q505135"
wgLexemeRescoreFunctions:[]
wmgUseWikibaseLexemeCirrusSearch:true
wgLexemeEnableSenses:true
wgLexemeLanguageCodePropertyId:"P218"
wmgUseWikibaseLexeme:true
wgLexemeDisableCirrus:true
wgLexemeUseCirrus:true
wgLexemeTalkNamespace:147
wmgNewWikibaseLexemeCirrusSearch:true
wgLexemeFulltextRescoreProfile:"lexeme_fulltext"
wgLexemePrefixSearchProfile:"lexeme_prefix"
wgWBQualityConstraintsWikibaseLexemeId:"Q505133"

Production:

wgLexemePrefixRescoreProfile:"lexeme_prefix"
wgLexemeEnableRepo:true
wgLexemeNamespace:146
wgLexemePrefixSearchProfiles:[]
wgWBQualityConstraintsWikibaseSenseId:"Q54285715"
wgLexemeRescoreFunctions:[]
wmgUseWikibaseLexemeCirrusSearch:true
wgLexemeEnableSenses:true
wgLexemeLanguageCodePropertyId:"P218"
wmgUseWikibaseLexeme:true
wgLexemeDisableCirrus:true
wgLexemeUseCirrus:true
wgLexemeTalkNamespace:147
wmgNewWikibaseLexemeCirrusSearch:true
wgLexemeFulltextRescoreProfile:"lexeme_fulltext"
wgLexemePrefixSearchProfile:"lexeme_prefix"
wgWBQualityConstraintsWikibaseLexemeId:"Q51885771"

(I'm fascinated that we have "wgLexemeUseCirrus" and "wgLexemeDisableCirrus" both set to true in production and beta cluster)

This is all of wikibase-related configs that differ between production and beta clsuter:

1 wgWBQualityConstraintsWikibaseFormId:"Q505134"
1 wgWBQualityConstraintsWikibaseFormId:"Q54285143"
1 wgWBQualityConstraintsWikibaseItemId:"Q29934200"
1 wgWBQualityConstraintsWikibaseItemId:"Q505131"
1 wgWBQualityConstraintsWikibaseLexemeId:"Q505133"
1 wgWBQualityConstraintsWikibaseLexemeId:"Q51885771"
1 wgWBQualityConstraintsWikibasePropertyId:"Q29934218"
1 wgWBQualityConstraintsWikibasePropertyId:"Q505132"
1 wgWBQualityConstraintsWikibaseSenseId:"Q505135"
1 wgWBQualityConstraintsWikibaseSenseId:"Q54285715"
1 wgWikibaseMultiRepositoryServiceWiringFiles:["\/srv\/mediawiki\/php-1.34.0-wmf.8\/extensions\/Wikibase\/client\/..\/data-access\/src\/MultiRepositoryServiceWiring.php"]
1 wgWikibaseMultiRepositoryServiceWiringFiles:["\/srv\/mediawiki-staging\/php-master\/extensions\/Wikibase\/client\/..\/data-access\/src\/MultiRepositoryServiceWiring.php"]
1 wgWikibasePerRepositoryServiceWiringFiles:["\/srv\/mediawiki\/php-1.34.0-wmf.8\/extensions\/Wikibase\/client\/..\/data-access\/src\/PerRepositoryServiceWiring.php"]
1 wgWikibasePerRepositoryServiceWiringFiles:["\/srv\/mediawiki-staging\/php-master\/extensions\/Wikibase\/client\/..\/data-access\/src\/PerRepositoryServiceWiring.php"]
1 wmgWikibaseClientBadgeClassNames:{"Q17437796":"badge-featuredarticle","Q17437798":"badge-goodarticle","Q17559452":"badge-recommendedarticle","Q17506997":"badge-featuredlist","Q17580674":"badge-featuredportal","Q20748091":"badge-notproofread","Q20748094":"badge-problematic","Q20748092":"badge-proofread","Q20748093":"badge-validated","Q28064618":"badge-digitaldocument","Q51759403":"badge-goodlist"}
1 wmgWikibaseClientBadgeClassNames:{"Q49444":"badge-goodarticle","Q49447":"badge-featuredarticle","Q49448":"badge-recommendedarticle","Q49449":"badge-featuredlist","Q49450":"badge-featuredportal","Q98649":"badge-notproofread","Q98650":"badge-problematic","Q98658":"badge-proofread","Q98651":"badge-validated"}
1 wmgWikibaseClientEchoIcon:{"path":"\/static\/images\/wikibase\/echoIcon.svg"}
1 wmgWikibaseClientEchoIcon:{"url":"\/static\/images\/wikibase\/echoIcon.svg"}
1 wmgWikibaseClientPropertyOrderUrl:"https:\/\/www.wikidata.org\/w\/index.php?title=MediaWiki:Wikibase-SortedProperties&action=raw&sp_ver=1"
1 wmgWikibaseClientRepoConceptBaseUri:"http:\/\/www.wikidata.beta.wmflabs.org\/entity\/"
1 wmgWikibaseClientRepoConceptBaseUri:"http:\/\/www.wikidata.org\/entity\/"
1 wmgWikibaseClientRepositories:{"":{"repoDatabase":"wikidatawiki","baseUri":"http:\/\/www.wikidata.org\/entity\/","entityNamespaces":{"item":0,"property":120,"lexeme":146},"prefixMapping":{"":""}}}
1 wmgWikibaseClientRepositories:{"":{"repoDatabase":"wikidatawiki","entityNamespaces":{"item":0,"property":120,"lexeme":146},"baseUri":"https:\/\/wikidata.beta.wmflabs.org\/entity\/","prefixMapping":{"":""}},"commons":{"repoDatabase":"commonswiki","entityNamespaces":{"mediainfo":"6\/mediainfo"},"baseUri":"https:\/\/commons.wikimedia.beta.wmflabs.org\/entity\/","prefixMapping":[]}}
1 wmgWikibaseClientRepoUrl:"https:\/\/wikidata.beta.wmflabs.org"
1 wmgWikibaseClientRepoUrl:"https:\/\/www.wikidata.org"
1 wmgWikibaseClientUseTermsTableSearchFields:false
1 wmgWikibaseClientUseTermsTableSearchFields:true
1 wmgWikibaseFeatureFlagWbeditentitySetEmptyAliases:false
1 wmgWikibaseFeatureFlagWbeditentitySetEmptyAliases:true
1 wmgWikibaseRepoBadgeItems:{"Q17437798":"wb-badge-goodarticle","Q17437796":"wb-badge-featuredarticle","Q17559452":"wb-badge-recommendedarticle","Q17506997":"wb-badge-featuredlist","Q17580674":"wb-badge-featuredportal","Q20748091":"wb-badge-notproofread","Q20748094":"wb-badge-problematic","Q20748092":"wb-badge-proofread","Q20748093":"wb-badge-validated","Q28064618":"wb-badge-digitaldocument","Q51759403":"wb-badge-goodlist"}
1 wmgWikibaseRepoBadgeItems:{"Q49444":"wb-badge-goodarticle","Q49447":"wb-badge-featuredarticle","Q49448":"wb-badge-recommendedarticle","Q49449":"wb-badge-featuredlist","Q49450":"wb-badge-featuredportal","Q98649":"wb-badge-notproofread","Q98650":"wb-badge-problematic","Q98658":"wb-badge-proofread","Q98651":"wb-badge-validated"}
1 wmgWikibaseSearchIndexProperties:[]
1 wmgWikibaseSearchIndexPropertiesExclude:[]
1 wmgWikibaseSearchIndexPropertiesExclude:["P304","P433","P478","P558","P3903","P3921","P4316","P1433","P2860"]
1 wmgWikibaseSearchIndexProperties:["P31","P279"]
1 wmgWikibaseSearchStatementBoosts:[]
1 wmgWikibaseSearchStatementBoosts:{"P31=Q4167410":-10,"P31=Q13442814":-5,"P31=Q18918145":-5}
1 wmgWikibaseSSRTermboxServerUrl:"https:\/\/ssr-termbox.wmflabs.org\/termbox"
1 wmgWikibaseUseSSRTermbox:true

The only thing that might be at fault is that if we enable federation, it would fix it (very very unlikely). The next step is to compare mediawiki core and similar paths. Will dig deeper tomorrow.

If you set $wgWBRepoSettings['useEntitySourceBasedFederation'] = true; in localhost, it works just fine:

{
    "entities": {
        "L16-F1": {
            "pageid": 513,
            "ns": 146,
            "title": "Lexeme:L16",
            "lastrevid": 515,
            "modified": "2019-05-22T15:06:48Z",
            "id": "L16-F1",
            "representations": {
                "en": {
                    "language": "en",
                    "value": "ff"
                }
            },
            "grammaticalFeatures": [],
            "claims": {}
        }
    },
    "success": 1
}

How about adding a regression (failing) test as a first step? As apparently there has been no tests for this behaviour, no wonder it was (well, I did) broken when changes to federation have been made.

Change 514883 had a related patch set uploaded (by Ladsgroup; owner: Ladsgroup):
[mediawiki/extensions/WikibaseLexeme@master] Add test for wbgetentities with subentities

https://gerrit.wikimedia.org/r/514883

So this is the backtrace when you set $wgWBRepoSettings['useEntitySourceBasedFederation'] = true;:

/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityMetaDataLookup.php:106
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/TypeDispatchingWikiPageEntityMetaDataAccessor.php:88
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php:264
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/PrefetchingWikiPageEntityMetaDataAccessor.php:186
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php:125
/var/lib/mediawiki2/extensions/Wikibase/data-access/src/ByTypeDispatchingEntityRevisionLookup.php:54
/var/lib/mediawiki2/extensions/WikibaseLexeme/src/DataAccess/Store/SenseRevisionLookup.php:55
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php:54
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:104
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:87
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:104
/var/lib/mediawiki2/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php:87
/var/lib/mediawiki2/extensions/Wikibase/repo/includes/Api/GetEntities.php:275
/var/lib/mediawiki2/extensions/Wikibase/repo/includes/Api/GetEntities.php:257
/var/lib/mediawiki2/extensions/Wikibase/repo/includes/Api/GetEntities.php:136
/var/lib/mediawiki2/includes/api/ApiMain.php:1595
/var/lib/mediawiki2/includes/api/ApiMain.php:500
/var/lib/mediawiki2/tests/phpunit/includes/api/ApiTestCase.php:106
/var/lib/mediawiki2/extensions/Wikibase/repo/tests/phpunit/includes/Api/WikibaseApiTestCase.php:156
/var/lib/mediawiki2/extensions/WikibaseLexeme/tests/phpunit/mediawiki/Api/LexemeGetEntitiesTest.php:105
/var/lib/mediawiki2/tests/phpunit/MediaWikiTestCase.php:427
/var/lib/mediawiki2/maintenance/doMaintenance.php:99

This is a big red herring and waste of time, I looked at everything in depth and nothing was responsible. The part that is actually responsible is EntityPrefetcher. GetEntities do prefetch and when you disable EntitySourceBasedFederation, The EntityPrefetcher is actually a federated one which is DispatchingEntityPrefetcher but that doesn't distinguish between different prefetchers of entity types that exist in one federated repo. It's repo-based EntityPrefetcher. while if you set useEntitySourceBasedFederation, it uses MultipleEntitySourceServices to get the EntityPrefetcher (I still don't know how, otherwise I would have made the patch for it already) which that returns ByTypeDispatchingEntityPrefetcher which ignores entity types that don't have EntityPrefetcher defined so it lets the system flow naturally. I need to consolidate and find a better solution than DispatchingEntityPrefetcher but @WMDE-leszek knows way better than I do. Please take a look.

So it seems all of the buggy code will go away soon (TM) according to @WMDE-leszek so I recommend just setting $wgWBRepoSettings['useEntitySourceBasedFederation'] = true; for both commons and wikidata.

Cannot see why the patch is failing, because in zuul's comment on the patch, the link to the failing job gives 404 https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/52351/console (this happened before to me, is there a pattern/rule/reason for that?)

Cannot see why the patch is failing, because in zuul's comment on the patch, the link to the failing job gives 404 https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/52351/console (this happened before to me, is there a pattern/rule/reason for that?)

It means it's too old and jenkins evicted the results page. Do a recheck.

Change 518090 had a related patch set uploaded (by WMDE-leszek; owner: Ladsgroup):
[mediawiki/extensions/WikibaseLexeme@master] Add test for wbgetentities with subentities

https://gerrit.wikimedia.org/r/518090

Change 518091 had a related patch set uploaded (by WMDE-leszek; owner: WMDE-leszek):
[mediawiki/extensions/WikibaseLexeme@master] Do not die when Wikibase prefetches Sense or Form Id

https://gerrit.wikimedia.org/r/518091

Change 518090 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] Add test for wbgetentities with subentities

https://gerrit.wikimedia.org/r/518090

Change 514883 abandoned by Alaa Sarhan:
Add test for wbgetentities with subentities

Reason:
done in 518090

https://gerrit.wikimedia.org/r/514883

Change 518091 merged by jenkins-bot:
[mediawiki/extensions/WikibaseLexeme@master] Do not die when Wikibase prefetches Sense or Form Id

https://gerrit.wikimedia.org/r/518091

mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:07 PM