@tstarling we would appreciate your thoughts around this as we plan for this quarter - in particular where you would recommend storing a data structure for references, perhaps ParserOutput via setExtensionData ?

Change 264300 had a related patch set uploaded (by Phuedx):
Add script to dump the intermediate representation

https://gerrit.wikimedia.org/r/264300

Change 264300 abandoned by Phuedx:
Add script to dump the intermediate representation

Reason:
I only want to archive the tool for posterity. I do not wish it to be merged.

https://gerrit.wikimedia.org/r/264300

@phuedx @Jdlrobson Is this a spike? An epic?

I expected to see clearer AC since it is in TODO in the current sprint. Should we rename it? Add subtasks with spikes or more concrete tasks?

I used the tool I added in 264300 to determine the size of the serialised IR in KB for a small set of sample articles:

Storing the serialised IR for every page would require storing at least 2 bytes per page (serialize(null); returns "N;").

Sorry @Jhernandez, I added this to the sprint to remind myself to follow up on 264300.

Change 265241 had a related patch set uploaded (by Phuedx):
Toy benchmark for unserializing serialized IR

https://gerrit.wikimedia.org/r/265241

Change 265241 abandoned by Phuedx:
Toy benchmark for unserializing serialized IR

Reason:
I only want to archive the benchmark for posterity. I do not wish it to be merged.

https://gerrit.wikimedia.org/r/265241

Change 266249 had a related patch set uploaded (by Phuedx):
[WIP] Load references only when necessary

https://gerrit.wikimedia.org/r/266249

While the discussion seems to be focused on exporting reference metadata, the description of this task also brings up the more general question of lazy-loading of notes and other page components like navboxes or infoboxes.

We are very interested in providing APIs for selective component retrieval in RESTBase, expanding on the existing section retrieval API. The main thing needed to make that happen is the identification of interesting content elements in Parsoid, possibly using templatedata to categorize templates across languages.

Parsoid HTML also contains very detailed and reliable structured information about references. It might be easier to use this information, rather than adding another custom code path in the Cite extension.

As I just noticed, page_props couldn't be used even for wikitext, since some pages have more than 64 KB of wikitext of references and pp_value is a BLOB. Or the type would have to be changed.

By your talk of "IR" I suppose you want to deliver each reference separately? Why not bundle the whole references section and deliver it to the user in one chunk? Isn't it fair to assume that if the user wants to see one reference, they will want to see others shortly afterwards?

@tstarling: 266249 does exactly that. The entirety of $parser->extCite->mRefs is processed, cached, and only delivered to client when they tap a reference. The "processing" step generates a map of reference key to reference text, as that's all the MobileFrontend requires, i.e.

array(
  'Foo' => array(
    'text' => 'Bar baz',
    'count' => 0,
    'number' => 1
  ),
)

is converted to

array(
  'cite_note_Foo-0' => 'Bar baz'
)

@phuedx
Couldn't the caching be done directly in the cite extension so that it can be used for other purposes, e.g. T124840 ?
(This would need an indefinite cache duration, renewal on reparse, and purging on move or deletion.)

@Cenarium: Yes. Absolutely. 266249 is just me experimenting with the idea, hence the WIP tag. The reason I chose to cache the processed references was that it made the mfreferences API incredibly simple to implement.

@Cenarium: … However, then we're treating a cache more like a store, which is where my "Where should we store this structure?" comes from.

@phuedx: Yes, caches shouldn't be considered reliable for long term storage, but in case of T124840 this isn't too bad if there's a cache miss. So if MobileFrontend doesn't need a long term storage medium, this shouldn't be an issue.

Change 267514 had a related patch set uploaded (by Cenarium):
Store parsed references in cache and page_props

https://gerrit.wikimedia.org/r/267514

Change 271422 had a related patch set uploaded (by Jdlrobson):
Surface references via api query property

https://gerrit.wikimedia.org/r/271422

Change 271422 merged by jenkins-bot:
Surface references via api query property

https://gerrit.wikimedia.org/r/271422

This no longer works. See: https://en.wikipedia.org/w/api.php?action=query&prop=references&titles=Albert%20Einstein
I'm wondering if this is a regression or that is was disabled on purpose. I would like to access the list of references from the API.

@Fako85 The error message is "Cite extension reference storage is not enabled."
To my knowledge this was never enabled in production. You may want to open a new task against Cite extension to investigate that.

In mobile web we've been using the following API to get at references:
https://en.m.wikipedia.org/w/api.php?action=mobileview&format=json&page=Barack%20Obama&sections=references&prop=text&revision=810395720

Thanks a lot @Jdlrobson. I'll try to parse from there what I need.