Page MenuHomePhabricator

Bump content type version in RESTBase
Closed, DuplicatePublic

Description

Parsoid is bumping up the content type schema version to 1.1.0, cf. this patch. We need to bump the version in our spec and tests once that is deployed.

Moreover, we need to ensure consistency between the spec and what we hand out. That means that when we retrieve the content from storage, we need to check its version, and, if it does not conform to the latest one, convert the retrieved document into the new version. The conversion can be done in two ways:

  • @ssatry will look into providing an html2html endpoint in Parsoid, which RESTBase could then use as a direct conversion mechanism
  • retrieve the wikitext form the MW API and then send it to Parsoid along with the revision's data-parsoid blob.

Open question: this might introduce inconsistent historical views of a page. (TODO: expand this).

Event Timeline

• mobrovac raised the priority of this task from to High.
• mobrovac updated the task description. (Show Details)
• mobrovac subscribed.

As per IRC, the Parsoid folks plan to deploy it on Monday, 2015-07-06

• mobrovac set Security to None.
• mobrovac edited subscribers, added: • Pchelolo, ssastry; removed: • Aklapper.

Okay, so, there are 2 considerations here. Longer-term version upgrade protocol, and short-term (for Monday) version upgrade solution.

So, two possibilities for getting HTML upgrade (a) use the regular wt -> html endpoint (b) use a html -> html endpoint if Parsoid supports it for that version pair.

The question is how does RB know which solution to use. Longer-term, one thought is to let the version string tell RESTBase how to get an upgraded HTML if it finds a stale version in storage .. i.e. "text/html;profile=mediawiki.org/specs/html/1.1.0;charset=utf-8;html2html=true" tells RB that it can get from the previous version to this version by asking Parsoid's html2html endpoint. If "html2html=true" is absent or if it is set to false, RB can use the regular wt -> html pagebundle endpoint.

Anyway, this proposal needs to be thought through, refined, and implemented.

For Monday, RB can assume html2html is false and use the wt -> html pagebundle endpoint to upgrade. The impact is that for pages that have the older version in storage, the first VE load for some of those pages will have a higher latency.

The question is how does RB know which solution to use. Longer-term, one thought is to let the version string tell RESTBase how to get an upgraded HTML if it finds a stale version in storage .. i.e. "text/html;profile=mediawiki.org/specs/html/1.1.0;charset=utf-8;html2html=true" tells RB that it can get from the previous version to this version by asking Parsoid's html2html endpoint. If "html2html=true" is absent or if it is set to false, RB can use the regular wt -> html pagebundle endpoint.

It is true that this is most straightforward way. However, that means that:

  • Parsoid would attach that to each response while RESTBase would need to remove it each time
  • There is no guarantee that, given enough version bumps, it would be possible to convert previous generations, as in, if Parsoid sends text/html;profile=mediawiki.org/specs/html/3.0.0;charset=utf-8;html2html=true but RESTBase retrieves a document with version 1.0.0, how can it be deduced whether a conversion is possible or not?

I see a couple of other possibilities:

  • Parsoid could provide the conversion matrix in its _version endpoint, something like a hash of from_version => to_version key/value pairs. RESTBase could then read that (and cache it) and use that info as a general rule of thumb for using the html2html endpoint.
  • Since it's not that we bump versions on a daily basis, we could even set the same matrix as a configuration parameter for both Parsoid and RESTBase

(when I started writing this reply I had other ideas which now escape me)

For Monday, RB can assume html2html is false and use the wt -> html pagebundle endpoint to upgrade.

Correct. When RESTBase retrieves a document from storage, it could check the version, and if it doesn't match what is specified in the Swagger spec, it behaves as if a storage miss occurred.

The impact is that for pages that have the older version in storage, the first VE load for some of those pages will have a higher latency.

The impact should be exactly the same as when introducing new wikis into RESTBase, so ... acceptable ? @Jdforrester-WMF , @Esanders ?

The question is how does RB know which solution to use. Longer-term, one thought is to let the version string tell RESTBase how to get an upgraded HTML if it finds a stale version in storage .. i.e. "text/html;profile=mediawiki.org/specs/html/1.1.0;charset=utf-8;html2html=true" tells RB that it can get from the previous version to this version by asking Parsoid's html2html endpoint. If "html2html=true" is absent or if it is set to false, RB can use the regular wt -> html pagebundle endpoint.

It is true that this is most straightforward way. However, that means that:

  • Parsoid would attach that to each response while RESTBase would need to remove it each time

We do this already now. content-type is part of the response headers for the pagebundle API endpoint. So, there is no change there.

  • There is no guarantee that, given enough version bumps, it would be possible to convert previous generations, as in, if Parsoid sends text/html;profile=mediawiki.org/specs/html/3.0.0;charset=utf-8;html2html=true but RESTBase retrieves a document with version 1.0.0, how can it be deduced whether a conversion is possible or not?

I see no real use case for rapid version bumps. We want to keep HTML stable as much as possible. Most bumps will be minor tweaks as we have done in the past without bumping version numbers. We also don't plan to support html version upgrades beyond the most recent bump for that reason. We don't want to carry around cruft in our codebase for long periods.

I see a couple of other possibilities:

  • Parsoid could provide the conversion matrix in its _version endpoint, something like a hash of from_version => to_version key/value pairs. RESTBase could then read that (and cache it) and use that info as a general rule of thumb for using the html2html endpoint.
  • Since it's not that we bump versions on a daily basis, we could even set the same matrix as a configuration parameter for both Parsoid and RESTBase

This could work as well and this also addresses the issue of how RB becomes aware of the version changes.

(when I started writing this reply I had other ideas which now escape me)

The question is how does RB know which solution to use. Longer-term, one thought is to let the version string tell RESTBase how to get an upgraded HTML if it finds a stale version in storage .. i.e. "text/html;profile=mediawiki.org/specs/html/1.1.0;charset=utf-8;html2html=true" tells RB that it can get from the previous version to this version by asking Parsoid's html2html endpoint. If "html2html=true" is absent or if it is set to false, RB can use the regular wt -> html pagebundle endpoint.

It is true that this is most straightforward way. However, that means that:

  • Parsoid would attach that to each response while RESTBase would need to remove it each time

We do this already now. content-type is part of the response headers for the pagebundle API endpoint. So, there is no change there.

Right, I was referring to the html2html=true part - we don't want to serve it necessarily to our clients as this represents an SLA between Parsoid and RESTBase.

  • There is no guarantee that, given enough version bumps, it would be possible to convert previous generations, as in, if Parsoid sends text/html;profile=mediawiki.org/specs/html/3.0.0;charset=utf-8;html2html=true but RESTBase retrieves a document with version 1.0.0, how can it be deduced whether a conversion is possible or not?

I see no real use case for rapid version bumps. We want to keep HTML stable as much as possible. Most bumps will be minor tweaks as we have done in the past without bumping version numbers. We also don't plan to support html version upgrades beyond the most recent bump for that reason. We don't want to carry around cruft in our codebase for long periods.

Oh, I agree completely - we don't want to introduce changes often in DOM spec. However, it is conceivable that in some point in time there might come a version which for whichever reason is not backwards-compatible. I'm just thinking out loud here and try to cover our basis and reduce future maintenance overhead.

I see a couple of other possibilities:

  • Parsoid could provide the conversion matrix in its _version endpoint, something like a hash of from_version => to_version key/value pairs. RESTBase could then read that (and cache it) and use that info as a general rule of thumb for using the html2html endpoint.
  • Since it's not that we bump versions on a daily basis, we could even set the same matrix as a configuration parameter for both Parsoid and RESTBase

This could work as well and this also addresses the issue of how RB becomes aware of the version changes.

Yup. It kind of makes the most sense to me, since we have to manually change the spec version and will both be aware of the upcoming change (kind of like we are now)

(when I started writing this reply I had other ideas which now escape me)

Partial fix was merged as a PR https://github.com/wikimedia/restbase/pull/275 which only fixed tests. The second part would be to actually bump the content type, but right now we can't afford additional cassandra load that would produce.

Right. For now, RESTBase will keep serving old, 1.0.0-based content from storage. Next step would probably be for Parsoid to supply an html2html endpoint which RESTBase could use to convert the old content.

Note that CXServer encounters sporadic failures because of older-version HTML (cf. T104848)

We determined yesterday that VE can handle both 1.0.0 and 1.1.0 content, so the Parsoid deploy went ahead.

Compared to template re-renders the load created by VE requests should be low (it peaks at about 2/s), but the issue is that latency for clients will suck as all of those requests would be generated from scratch. We need to

a) either have a fast way to update existing content (html2html end point in Parsoid, as you say), or
b) a background job that performs upgrades ideally before the user requests them. This could simply be a HTML dump that requests current versions of all articles.

For changes like the cite one a) would be much more efficient. It is also the only realistic way to deal with older, stored content in the longer term, as the cost of re-generating all that from scratch would be prohibitive.

The content-type was bumped, and the second part (creating automatic content-regeneration machinery) is tracked by T116333 , so merging this