The <head> section of the document produced by Parsoid is only used by the REST APIs, and ought to live in core as part of the REST API implementation rather than Parsoid.
Rendering the <head> based on the metadata in the ParserOutput would allow us to:
- Remove the need to store different ParserOutput contents for legacy (which is just <body> innerHTML) and Parsoid (which is the entire <html> document, reducing maintainer confusing and elimininating unnecessary ParserCache storage for the common case (read views)
- Eventually allow us to implement a more-complete mapping of ParserOutput information into <head> metadata, for those who need that -- for example, REST API users who do want an all-in-one document and who want access to category and other information which is stored in the ParserOutput but only imperfectly reflected in the output HTML. (For example, T418792: Expose MediaWiki Parser render_id as a response header in relevant MW REST API endpoints.)
To make this change without breaking existing endpoint, we're going to start by having ContentHolder track whether it is currently holding full-document content or not, and add a new ContentHolder method, getRawContentHolderText(), which will be used by any endpoint which *wants a full document result*. We'll start with an audit of all codepaths which uses ParserOutput, and shift each one which actually wants a full document to getRawContentHolderText() to document and distinguish this case. Our initial audit of these users:
Sites which should use ::getRawContentHolderText() as they expect full-document output
| Site | Notes |
|---|---|
| Rest/Handler/PageHTMLHandler.php:113 | html + with_html; shared by renderer and shadow helper paths |
| Rest/Handler/RevisionHTMLHandler.php:85 | html output |
| Rest/Handler/RevisionHTMLHandler.php:90 | with_html output |
| Rest/Handler/ParsoidHandler.php:781 | wt2html FORMAT_HTML; trickiest — edit/fragment flavor logic |
Sites which want body-only content; they will continue to use ::getContentHolderText()
| Site | Why body-only is correct |
|---|---|
| Status/StatusFormatter.php:347 | message HTML → text |
| Language/Message/Message.php:1075,1078 | message HTML, stripOuterParagraph |
| Page/ParserOutputAccess.php:908 | post-pipeline; appends <!--debug--> comment to body-only text (also a write site) |
| Content/WikiTextStructure.php:158 | search indexing; parses body to DOM, extracts text |
| Revision/RevisionRenderer.php:278 | concatenates per-slot body HTML |
| Api/ApiParse.php:502,513 | action API returns body HTML |
| Output/OutputPage.php:2573,2590,2676,2708,2732 | page view; skin wraps body content |
| Api/ApiQueryRevisionsBase.php:688 | runOutputPipeline(...)->… (pipeline ran) |
| EditPage/EditPage.php:3458,3464 | diff of body content (3464 runs pipeline) |
| Specials/SpecialExpandTemplates.php:113 | ->run(...)->… (pipeline ran) |
| Specials/SpecialRecentChanges.php:473 | RC body content |
| JobQueue/Jobs/RefreshLinksJob.php:459 | equality compare of cached vs fresh output; both normalize identically |
| Installer/Installer.php:766 | ->getContentHolderText() after pipeline |
| Actions/McrUndoAction.php:343 | pipeline output |
| Actions/InfoAction.php:197 | info page body |
| FileRepo/File/LocalFile.php:2622 | runOutputPipeline(...)->… |
| Parser/ParserOutput.php:1111 | setText() deprecated back-compat (returns previous value) |
We also want to maintain the output of these external API endpoints:
| Endpoint | Expected shape |
|---|---|
| GET /v1/page/Main_Page/html | full document |
| GET /v1/page/Main_Page/with_html | JSON, html = full document |
| GET /v1/revision/{id}/html | full document |
| GET /v1/revision/{id}/with_html | JSON, html = full document |
| POST /v1/transform/wikitext/to/html/Main_Page | full document (edit flavor) |
| POST …/to/html/Main_Page?body_only=true | body-only fragment |
| GET /api.php?action=parse&page=Main_Page&parser=parsoid&formatversion=2 | body-only in .parse.text |
| GET /api.php?action=parse&page=Main_Page&parser=legacy&formatversion=2 | body-only in .parse.text |
| GET /api.php?action=visualeditor&page=Main_Page&paction=parse&formatversion=2 | full document in .visualeditor.content |
| POST /localhost/v3/transform/wikitext/to/pagebundle/Main_Page (Parsoid extension) | full document in .html.body |
| POST /localhost/v3/transform/wikitext/to/html/Main_Page (Parsoid extension) | full document |
| POST /v1/transform/wikitext/to/html/Main_Page?stash=true (requires auth) | full document + stash-key ETag |
| POST /v1/transform/html/to/wikitext/Main_Page w/ If-Match: <stash ETag> (requires auth) | recovers original wikitext (selser) |
We expect that passing an Accept-Language header to invoke language conversion will maintain the same output shape (full document or body-only):
| Endpoint (+ Accept-Language: en-x-piglatin) | Expected shape |
|---|---|
| GET page/html, page/with_html, revision/html, revision/with_html | full document |
| POST transform wikitext→html (edit) | full document |
| POST transform wikitext→html ?body_only=true | body-only (T428485) |
| POST v3 pagebundle, v3 wikitext→html | full document |
Implementation plan
The transition will be made in several steps. At the end, the ContentHolder will only hold body-only content, and any client which wants a full document will access it via an HtmlPageBundle. The conversion from ParserOutput to HtmlPageBundle will add the <head> and <body> wrappers.
- Track full-document status in ContentHolder; introduction of ContentHolder::getAsRawHtmlString() to mark users who expect a full document. No behavior change yet, but we can add assertions to test that our classifications are correct.
- ContentHolder::getAsHtmlString(BODY_FRAGMENT) will now strip the result to always return body-only content (regardless of whether the raw html is full-document or not). Anything which needed the full document should be using ::getAsRawHtmlString().
- Deprecate ::getAsRawHtmlString() and incrementally switch clients who are using it to HtmlPageBundle using htmlPageBundleFromParserOutput with the asFullDocument flag.
- Once there are no more users of ::getAsRawHtmlString(), we can flip Parsoid to start storing body_only content in the ParserCache. We still need the on-demand strip in step #2 because old cache contents will still be body_only.
- Once the ParserCache expiration time has expired, we can remove the lazy strip and/or strip during deserialization.