Store HTML and page properties with multi-part content handler
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• GWicke
	Jun 4 2013, 5:37 PM

Description

MediaWiki currently stores the entire page content as WikiText. In addition to WikiText, we would like to store

The fully expanded HTML DOM
Page properties: categories, magic word flags (notoc etc), DISPLAYTITLE, bug 48812, etc
Parsoid-internal information: Basically data-parsoid moved out of the main page DOM

Eventually we'd also like to be able to drop WikiText storage without having to rework the storage architecture.

In the current MediaWiki external storage and ContentHandler architecture this can be achieved by adding a multi-part content type with a corresponding ContentHandler. This could be a JSON object or some other serialization.

A possible downside of the compound document approach stems from the need to update transclusion or image expansions for a given revision. With append-only and immutable external storage this can be implemented by storing a new compound document and then updating the revision to point to it. Without garbage collection this will result in several copies of unmodified WikiText and page properties in external storage. However, this issue should probably be addressed in the storage layer.

Note: This is now being addressed with RESTBase.

Version: unspecified
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=35066
https://bugzilla.wikimedia.org/show_bug.cgi?id=50882
https://bugzilla.wikimedia.org/show_bug.cgi?id=53508

Details

Reference: bz49143

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	Release	None	T84936 Release VisualEditor-MediaWiki as "1.0"
Open		None	T50429 [Epic] Support editing parts of a page in VisualEditor-MediaWiki
Open		None	T54365 Explore performance gains from progressive (JIT?) de-alienation in VisualEditor
Open		None	T174303 Copy-pasting linked ISBN numbers from view mode HTML into VisualEditor inserts wikitext links to Special:BookSources (it should turn them into magic links?)
Open	Feature	None	T54091 The read HTML should have hinting to allow full DOM copying (as opposed to just rich copying) from read mode into VE surfaces
Open		None	T55784 [EPIC] Use Parsoid HTML for all page views
Open	Feature	None	T73838 Allow saved books to be loaded with the revisions the articles had when the book was stored
Open	Feature	None	T7877 View all articles as they were at a specified point in time
Open	Feature	None	T2851 when viewing an old version of a page, use old version of templates
Resolved		ssastry	T50812 Get ParserOutput options from expandtemplates or similar action, and encode them as properties
Open		None	T55508 Move invisible page properties from the DOM to dedicated metadata
Resolved		• GWicke	T51143 Store HTML and page properties with multi-part content handler
Resolved		• GWicke	T89481 RESTBase beta release (revision storage / content API)
Resolved		• GWicke	T1228 RESTbase deployment
Resolved		fgiunchedi	T76986 RESTBase production hardware
Resolved		fgiunchedi	T88805 rack and setup restbase production cluster in eqiad
Resolved		• Cmjohnson	T83551 Split memcached in eqiad across multiple racks/rows
Resolved		• Cmjohnson	T82259 mc1016 mgmt not working
Resolved		Joe	T89345 Missing memcached logs
Resolved		fgiunchedi	T89639 restbase1006 faulty disk controller
Resolved		• GWicke	T75955 RESTBase / Parsoid integration - waiting for parsoid deploy
Resolved		• Jdouglas	T88456 On-demand generation of HTML and data-parsoid
Resolved		• GWicke	T88457 html2wt conversion support
Resolved		• GWicke	T88588 Formalize the RESTBase v1 specification
Resolved		Arlolra	T89411 Parsoid v2 API selser might not be working correctly
Resolved		• GWicke	T76370 RESTBase performance testing
Resolved		• mobrovac	T76165 Handle revision deletion and suppression in RESTBase
Resolved		• mobrovac	T87520 Set up update jobs for RESTBase: code done and deployed, to be configured
Resolved		• mobrovac	T88652 Integrate revisions into /page structure. Was: Create a /page/revision/{revision} entry point
Resolved		• mobrovac	T87392 Add property & index on namespace in page_revisions
Duplicate		• Jdouglas	T78214 Documentation location and deployment process
Resolved		Andrew	T88850 Import Cassandra packages for Jessie
Resolved		• GWicke	T89461 trebuchet puppet provider broken on systems without upstart
Resolved		fgiunchedi	T89636 setup internal LVS for restbase eqiad servers
Resolved		fgiunchedi	T78514 Detailed cassandra monitoring: metrics and dashboards done, need to set up alerts
Resolved		fgiunchedi	T93886 Cassandra/CQL query interface monitoring
Resolved		Eevans	T93884 graphs for Cassandra metrics
Resolved		fgiunchedi	T97024 some cassandra metrics sent with invalid values
Resolved		• GWicke	T78194 Public entry point for RESTBase
Resolved		• GWicke	T89567 Load test production cluster
Resolved		fgiunchedi	T89366 Access to restbase / cassandra cluster
Resolved		• mobrovac	T91102 Update / maintain Beta Cluster restbase cluster: Up & working with VE
Resolved		• GWicke	T91188 Need to change data model for >~300 wikis in single cassandra cluster

Event Timeline

• bzimport raised the priority of this task from to Low.Nov 22 2014, 1:48 AM

• bzimport added a project: Parsoid.

• bzimport set Reference to bz49143.

• GWicke created this task.Jun 4 2013, 5:37 PM

Flow needs something very similar: https://www.mediawiki.org/wiki/Flow_Portal/Architecture/Discussion_Storage

We should provide an abstract interface to retrieve parts of multi-part content, so that the storage implementation in the backend can be optimized independently. Possibly something like this:

$html = $rev->getPart( "html" );
$wikitext = $rev->getPart( "wikitext" );
$pageProps = json_decode( $rev->getPart( "pageprops" ) );

Parts can be set / updated with $rev->setPart( "key", "value" );

The backend is free to store each part independently or concatenate parts with some efficient segmentation mechanism. This part interface can be used by higher-level content handlers to implement a consistent ContentHandler interface.

gwicke and i seem to disagree whether this plan involves eventually removing page properties (ie, #REDIRECT, NOTOC) from the wikitext/DOM or not.

(In reply to comment #3)

gwicke and i seem to disagree whether this plan involves eventually removing
page properties (ie, #REDIRECT, NOTOC) from the wikitext/DOM or not.

This bug is primarily about the multi-part storage, but I'll reply nevertheless:

The VE page property dialog makes page-global properties easier to discover and modify. This dialog (or something very close to it) can also be used in combination with wikitext editing.

With a page property UI and diffing support for properties in place I don't see a good reason for keeping page properties both in a versioned page property structure *and* inline in the longer term.

Just a clarification for those that have not been following the discussions in the last year or so:

We plan to store fully expanded snapshots of HTML for each revision. This makes HTML retrieval fast, but also lets us provide a view of the page as it looked like in the past including all transclusion/extension/file dependencies. There are some storage volume trade-offs (which can likely be addressed with compression), but if we decide to store a snapshot on each re-render after a template/file change, we can provide a copy of each page at any point in the past. Retrieving 'yesterday's Main Page' becomes possible. So far that has only been possible with the flagged revision extension at considerable expense.

Other data structures that change with transclusion/extension/file updates will be snapshotted similarly. This applies to dynamic page properties for example.

See bug 851 (from 2004) for the "Yesterday's Main Page" problematic.

Moving page properties to page metadata is tracked in bug 53508.