Various `Special:EntityData` calls to `.ttl` or `.rdf` timeout of run out of memory.
The main suspicious code for these paths collects labels from all entities refered to by the entity that RDF is being retrieved from.
This currently does full entity lookups, rather than just retrieving labels from a faster storage mechanism.
See {T243950}
This probably means switching from full EntityStore lookups to using TermLookups.
**Acceptance Cirteria**
[] Features of the RDF endpoints remain the same
[] `/wiki/Special:EntityData/Q30.rdf` in production no longer times out
**Notes & links**
- Docs on the SQL secondary storage https://doc.wikimedia.org/Wikibase/master/php/md_docs_topics_storage.html
- `PrefetchingTermLookup` services would allow batch prefetching of the terms being refered to
----------------------------
**Initial Investigation**
While looking at logstash for some reason @Silvan_WMDE & I noticed there were quite some errors relating to https://www.wikidata.org/wiki/Special:EntityData/Q30.rdf
- Timeouts `[5649a640-5f42-4d22-aaec-36ffdfb053de] /wiki/Special:EntityData/Q30.rdf Wikibase\DataModel\Services\Lookup\EntityLookupException: The maximum execution time of 60 seconds was exceeded`
- OOMs `[0891912f-89b3-4cbc-a675-bf37bff44910] PHP Fatal error: Allowed memory size of 698351616 bytes exhausted (tried to allocate 134217736 bytes) in /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/purtle/src/RdfWriterBase.php:229`
- EntityLookupException `[5649a640-5f42-4d22-aaec-36ffdfb053de] 2021-04-27 13:57:05: Fatal exception of type "Wikibase\DataModel\Services\Lookup\EntityLookupException"` (due to timeout)
Seemingly this has been occurring for the complete time range of existing logstash logs
{F34429833}
Searching phabricator I came across {T62003} **but I do not think this is related** as I also get a 500 error while requesting these pages directly from a mwapp server inside the cluster.
```
addshore@deploy1002:~$ curl -k -L -H "Host: www.wikidata.org" -I https://mw1405.eqiad.wmnet/wiki/Special:EntityData/Q30.rdf
HTTP/1.1 500 Internal Server Error
Date: Tue, 27 Apr 2021 13:57:23 GMT
Server: mw1405.eqiad.wmnet
X-Powered-By: PHP/7.2.31-1+0~20200514.41+debian9~1.gbpe2a56b+wmf1+buster1
X-Request-Id: 5e4f152f-8a45-4d31-a77a-b9a112319b25
Backend-Timing: D=53005048 t=1619531843710982
Content-Type: text/html; charset=UTF-8
X-Envoy-Upstream-Service-Time: 53005
Transfer-Encoding: chunked
```
The stack traces vary.
An example for OOM would be:
```lines=5
#0 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/purtle/src/RdfWriterBase.php(229): unknown()
#1 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/purtle/src/XmlRdfWriter.php(69): Wikimedia\Purtle\RdfWriterBase->write()
#2 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/purtle/src/XmlRdfWriter.php(221): Wikimedia\Purtle\XmlRdfWriter->tag()
#3 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/purtle/src/RdfWriterBase.php(451): Wikimedia\Purtle\XmlRdfWriter->writeText()
#4 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Rdf/TermsRdfBuilder.php(92): Wikimedia\Purtle\RdfWriterBase->text()
#5 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Rdf/TermsRdfBuilder.php(183): Wikibase\Repo\Rdf\TermsRdfBuilder->addLabels()
#6 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(480): Wikibase\Repo\Rdf\TermsRdfBuilder->addEntityStub()
#7 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(444): Wikibase\Repo\Rdf\RdfBuilder->addEntityStub()
#8 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(225): Wikibase\Repo\Rdf\RdfBuilder->resolveMentionedEntities()
#9 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(170): Wikibase\Repo\LinkedData\EntityDataSerializationService->rdfSerialize()
#10 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(529): Wikibase\Repo\LinkedData\EntityDataSerializationService->getSerializedData()
#11 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(275): Wikibase\Repo\LinkedData\EntityDataRequestHandler->showData()
#12 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Specials/SpecialEntityData.php(143): Wikibase\Repo\LinkedData\EntityDataRequestHandler->handleRequest()
#13 /srv/mediawiki/php-1.37.0-wmf.1/includes/specialpage/SpecialPage.php(646): Wikibase\Repo\Specials\SpecialEntityData->execute()
#14 /srv/mediawiki/php-1.37.0-wmf.1/includes/specialpage/SpecialPageFactory.php(1381): SpecialPage->run()
#15 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(313): MediaWiki\SpecialPage\SpecialPageFactory->executePath()
#16 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(916): MediaWiki->performRequest()
#17 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(550): MediaWiki->main()
#18 /srv/mediawiki/php-1.37.0-wmf.1/index.php(53): MediaWiki->run()
#19 /srv/mediawiki/php-1.37.0-wmf.1/index.php(46): wfIndexMain()
#20 /srv/mediawiki/w/index.php(3): require()
```
And an example for timeout could be:
```lines=5
from /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/request-timeout/src/Detail/ExcimerTimerWrapper.php(97)
#0 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikimedia/request-timeout/src/Detail/ExcimerTimerWrapper.php(72): Wikimedia\RequestTimeout\Detail\ExcimerTimerWrapper->onTimeout(integer)
#1 /srv/mediawiki/php-1.37.0-wmf.1/vendor/data-values/serialization/src/Deserializers/DataValueDeserializer.php(115): Wikimedia\RequestTimeout\Detail\ExcimerTimerWrapper->Wikimedia\RequestTimeout\Detail\{closure}(integer)
#2 /srv/mediawiki/php-1.37.0-wmf.1/vendor/data-values/serialization/src/Deserializers/DataValueDeserializer.php(91): DataValues\Deserializers\DataValueDeserializer->getDeserialization(array)
#3 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakDeserializer.php(128): DataValues\Deserializers\DataValueDeserializer->deserialize(array)
#4 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakDeserializer.php(117): Wikibase\DataModel\Deserializers\SnakDeserializer->deserializeDataValue(array)
#5 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakDeserializer.php(100): Wikibase\DataModel\Deserializers\SnakDeserializer->newValueSnak(array)
#6 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakDeserializer.php(82): Wikibase\DataModel\Deserializers\SnakDeserializer->getDeserialized(array)
#7 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakListDeserializer.php(60): Wikibase\DataModel\Deserializers\SnakDeserializer->deserialize(array)
#8 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/SnakListDeserializer.php(41): Wikibase\DataModel\Deserializers\SnakListDeserializer->getDeserialized(array)
#9 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ReferenceDeserializer.php(77): Wikibase\DataModel\Deserializers\SnakListDeserializer->deserialize(array)
#10 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ReferenceDeserializer.php(67): Wikibase\DataModel\Deserializers\ReferenceDeserializer->deserializeSnaks(array)
#11 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ReferenceDeserializer.php(57): Wikibase\DataModel\Deserializers\ReferenceDeserializer->getDeserialized(array)
#12 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ReferenceListDeserializer.php(53): Wikibase\DataModel\Deserializers\ReferenceDeserializer->deserialize(array)
#13 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ReferenceListDeserializer.php(40): Wikibase\DataModel\Deserializers\ReferenceListDeserializer->getDeserialized(array)
#14 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/StatementDeserializer.php(161): Wikibase\DataModel\Deserializers\ReferenceListDeserializer->deserialize(array)
#15 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/StatementDeserializer.php(124): Wikibase\DataModel\Deserializers\StatementDeserializer->setReferencesFromSerialization(array, Wikibase\DataModel\Statement\Statement)
#16 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/StatementDeserializer.php(100): Wikibase\DataModel\Deserializers\StatementDeserializer->getDeserialized(array)
#17 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/StatementListDeserializer.php(60): Wikibase\DataModel\Deserializers\StatementDeserializer->deserialize(array)
#18 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/StatementListDeserializer.php(41): Wikibase\DataModel\Deserializers\StatementListDeserializer->getDeserialized(array)
#19 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ItemDeserializer.php(130): Wikibase\DataModel\Deserializers\StatementListDeserializer->deserialize(array)
#20 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ItemDeserializer.php(85): Wikibase\DataModel\Deserializers\ItemDeserializer->setStatementListFromSerialization(array, Wikibase\DataModel\Entity\Item)
#21 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-serialization/src/Deserializers/ItemDeserializer.php(77): Wikibase\DataModel\Deserializers\ItemDeserializer->getDeserialized(array)
#22 /srv/mediawiki/php-1.37.0-wmf.1/vendor/serialization/serialization/src/Deserializers/DispatchingDeserializer.php(42): Wikibase\DataModel\Deserializers\ItemDeserializer->deserialize(array)
#23 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/internal-serialization/src/Deserializers/EntityDeserializer.php(42): Deserializers\DispatchingDeserializer->deserialize(array)
#24 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/EntityContentDataCodec.php(253): Wikibase\InternalSerialization\Deserializers\EntityDeserializer->deserialize(array)
#25 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityDataLoader.php(82): Wikibase\Lib\Store\EntityContentDataCodec->decodeEntity(string, NULL)
#26 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(236): Wikibase\Lib\Store\Sql\WikiPageEntityDataLoader->loadEntityDataFromWikiPageRevision(MediaWiki\Revision\RevisionStoreRecord, string, integer)
#27 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/Sql/WikiPageEntityRevisionLookup.php(122): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->loadEntity(stdClass, string)
#28 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(54): Wikibase\Lib\Store\Sql\WikiPageEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#29 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/data-access/src/ByTypeDispatchingEntityRevisionLookup.php(55): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#30 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/TypeDispatchingEntityRevisionLookup.php(54): Wikibase\DataAccess\ByTypeDispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#31 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(104): Wikibase\Lib\Store\TypeDispatchingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#32 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(87): Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#33 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(104): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#34 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/CachingEntityRevisionLookup.php(87): Wikibase\Lib\Store\CachingEntityRevisionLookup->fetchEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#35 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/lib/includes/Store/RevisionBasedEntityLookup.php(46): Wikibase\Lib\Store\CachingEntityRevisionLookup->getEntityRevision(Wikibase\DataModel\Entity\ItemId, integer, string)
#36 /srv/mediawiki/php-1.37.0-wmf.1/vendor/wikibase/data-model-services/src/Lookup/RedirectResolvingEntityLookup.php(51): Wikibase\Lib\Store\RevisionBasedEntityLookup->getEntity(Wikibase\DataModel\Entity\ItemId)
#37 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Rdf/RdfBuilder.php(438): Wikibase\DataModel\Services\Lookup\RedirectResolvingEntityLookup->getEntity(Wikibase\DataModel\Entity\ItemId)
#38 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(225): Wikibase\Repo\Rdf\RdfBuilder->resolveMentionedEntities(Wikibase\DataModel\Services\Lookup\RedirectResolvingEntityLookup)
#39 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataSerializationService.php(170): Wikibase\Repo\LinkedData\EntityDataSerializationService->rdfSerialize(Wikibase\Lib\Store\EntityRevision, NULL, array, Wikibase\Repo\Rdf\RdfBuilder, NULL)
#40 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(529): Wikibase\Repo\LinkedData\EntityDataSerializationService->getSerializedData(string, Wikibase\Lib\Store\EntityRevision, NULL, array, NULL)
#41 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/LinkedData/EntityDataRequestHandler.php(275): Wikibase\Repo\LinkedData\EntityDataRequestHandler->showData(WebRequest, OutputPage, string, Wikibase\DataModel\Entity\ItemId, integer)
#42 /srv/mediawiki/php-1.37.0-wmf.1/extensions/Wikibase/repo/includes/Specials/SpecialEntityData.php(143): Wikibase\Repo\LinkedData\EntityDataRequestHandler->handleRequest(string, WebRequest, OutputPage)
#43 /srv/mediawiki/php-1.37.0-wmf.1/includes/specialpage/SpecialPage.php(646): Wikibase\Repo\Specials\SpecialEntityData->execute(string)
#44 /srv/mediawiki/php-1.37.0-wmf.1/includes/specialpage/SpecialPageFactory.php(1381): SpecialPage->run(string)
#45 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(313): MediaWiki\SpecialPage\SpecialPageFactory->executePath(Title, RequestContext)
#46 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(916): MediaWiki->performRequest()
#47 /srv/mediawiki/php-1.37.0-wmf.1/includes/MediaWiki.php(550): MediaWiki->main()
#48 /srv/mediawiki/php-1.37.0-wmf.1/index.php(53): MediaWiki->run()
#49 /srv/mediawiki/php-1.37.0-wmf.1/index.php(46): wfIndexMain()
#50 /srv/mediawiki/w/index.php(3): require(string)
#51 {main}
```
While investigating the situation a little more I discovered that:
- The JSON api is not impacted, and loads quickly https://www.wikidata.org/w/api.php?action=wbgetentities&format=json&ids=Q30
- The JSON Special:EntityData page loads just fine https://www.wikidata.org/wiki/Special:EntityData/Q30.json (including in the cluster `curl -k -L -H "Host: www.wikidata.org" -o -I https://mw1405.eqiad.wmnet/wiki/Special:EntityData/Q30.json`)
- The TTL Special:EntityData page also has this issue for Q30 https://www.wikidata.org/wiki/Special:EntityData/Q30.ttl (including in the cluster `curl -k -L -H "Host: www.wikidata.org" -o -I https://mw1405.eqiad.wmnet/wiki/Special:EntityData/Q30.ttl`)
- The TTL flavour=dump format for Special:EntityData doesn't appear to have this issue? https://www.wikidata.org/wiki/Special:EntityData/Q30.ttl?flavor=dump (BUT IT DOES not going via the edge? `curl -k -L -H "Host: www.wikidata.org" -o -I https://mw1405.eqiad.wmnet/wiki/Special:EntityData/Q30.ttl?flavour=dump`)
- This could end up impacting the WDQS, but right now it appears to have the latest data https://query.wikidata.org/#select%20%2a%20where%20%7B%20wd%3AQ30%20schema%3Aversion%20%3Fv%20%7D
- The item page itself seems to load just fine (even when busting the parser cache) https://www.wikidata.org/wiki/Q30, and infact loads / renders very quickly
Expanding my logstash search it appears this is happening to some other entities too (I'm sure there would be more of these if I looked harder):
- https://www.wikidata.org/wiki/Special:EntityData/Q183.ttl
- https://www.wikidata.org/wiki/Special:EntityData/Q423.ttl
- https://www.wikidata.org/wiki/Special:EntityData/Q81068910.ttl