Page MenuHomePhabricator

Add annotation about offset-type in the data-parsoid blob
Closed, ResolvedPublic0 Estimated Story Points

Description

Proposal by @Tgr:

data-parsoid dsr offsets are language-dependent. When emitted by JS code, the offsets are UCS2 offsets. But, when emitted by Parsoid/PHP code, they are likely going to be byte offsets (or unicode char offsets). This means that for selective serialization purposes, we cannot use offsets generated by Parsoid/JS in Parsoid/PHP code and vice versa. However, if we know the type of offsets stored in the data-parsoid blob, by adding appropriate offset converters to both Parsoid/JS and Parsoid/PHP in the API layer before selective serialization, we can interchangeably use data-parsoid blobs generated by either Parsoid codebase in the other codebase. This means if we need to switch back and forth between Parsoid/JS and Parsoid/PHP in the initial deployment stages, we can do so without having to tightly co-ordinate clearing / versioning of data-parsoid blobs stored in RESTBase to prevent incorrect use of data-parsoid blobs.

Event Timeline

ssastry triaged this task as Medium priority.

I think adding this offsite type annotation is useful for purposes of clarity. So, here is a possibility:

  • Add "offset-type": "byte" in Parsoid/PHP data-parsoid blob
  • Add "offste-type": "ucs2" in Parsoid/JS data-parsoid blob, but absence of any annotation is treated as "offset-type": "ucs2" so that we don't have to version bump or clear restbase storage.

I think that's going to bloat each data-parsoid object quite a bit, with almost entirely redundant information.

If you wanted to add it to the top-level <body> I wouldn't object, but then it's not present when you do section- or sub-section-level editing.

I think it's best to store this but treat it as a 'content type' That is, text/data-parsoid-utf8 or text/data-parsoid-ucs2. Then restbase can do content negotiation and even (in theory) conversion.

I think that's going to bloat each data-parsoid object quite a bit, with almost entirely redundant information.

If you wanted to add it to the top-level <body> I wouldn't object, but then it's not present when you do section- or sub-section-level editing.

I think it's best to store this but treat it as a 'content type' That is, text/data-parsoid-utf8 or text/data-parsoid-ucs2. Then restbase can do content negotiation and even (in theory) conversion.

Yes, I meant in the top-level data-parsoid blob in the pagebundle endpoints.

To clarify, I mean in this blob, add a parsoid.offset-type property. Or even just a top-level offset-type property

{"parsoid":{"counter":1,"ids":{....}},"mw":{"ids":{...}}}

Change 539631 had a related patch set uploaded (by C. Scott Ananian; owner: C. Scott Ananian):
[mediawiki/services/parsoid@master] WIP: Shift offsets in generated HTML

https://gerrit.wikimedia.org/r/539631

Change 539631 merged by jenkins-bot:
[mediawiki/services/parsoid@master] Optionally convert offsets in generated HTML

https://gerrit.wikimedia.org/r/539631

ssastry assigned this task to cscott.