Page MenuHomePhabricator

Get rough estimates of the sizes of large Wikidata entities and their WF representations
Closed, ResolvedPublic

Description

Description

Q16 and Q30 are blowing up the spot. We need to know some things:

Desired behavior/Acceptance criteria (returned value, expected error, performance expectations, etc.)

  • how big are the Wikidata representations of these objects?
  • how big are the WF transformations?
  • do we need to be more selective about which parts of WD entities we retrieve?
  • do we need to implement streaming pipes through the orchestrator so we can handle arbitrarily large WD entities? (NOOOO we don't)

Completion checklist

Event Timeline

• do we need to be more selective about which parts of WD entities we retrieve?

Absolutely! T382921
Even when large entities are successfully fetched, they often fail to be processed. Chances are high that a function will require only (one or a few) labels or (a selection of) statements, and hardly ever anything else.

We should also consider caching the call to the fetch functions directly, rather than relying on caching the outermost function calls. I think this could be helpful more generally within nested compositions, but the benefits seem most clearcut for the Wikidata fetch functions. As these are both builtin and terminal, the implementation is likely to be most straightforward here too.

Here are some findings:

QIDWikidata Item SizeWikifunctions Transformed SizeEvaluator Request Size
Q525496275500214111545
Q301416056821560178572
Q14419022428170553906

So, the good-ish news is that we aren't massively inflating items with our ZObject normal form transformation (but we are indeed inflating a little bit--roughly, it seems that our format is BIGGER up to 500 KiB, after which point our format is more efficient than the Wikidata representation). The bad-ish news is that there is, therefore, no quick win here. These are genuinely large objects and we have to be smart about which bits of them to process.