Investigate what to do with WikiLinkFixer now that Parsoid supports red links
Open, Needs TriagePublic
Actions

Assigned To

None

Authored By

	• Mattflaschen-WMF
	Jun 21 2017, 6:08 PM

Description

Verify Parsoid link support works for us, then see if we can remove WikiLinkFixer.

We might need to start tracking HTML versions, since old HTML still needs to be fixed. Also, we need to request https://www.mediawiki.org/wiki/Specs/HTML/1.5.0 or later (currently 1.2.1).

For the relative path issue, we could start using BaseHrefFixer for all content.

Related Objects
Search...

Status	Subtype	Assigned	Task
Open	Feature	None	T40891 MediaWiki API 2.0 (tracking)
Resolved		Jdlrobson	T95741 API: Inconsistent response for empty collections
Open		None	T168555 Investigate what to do with WikiLinkFixer now that Parsoid supports red links
Resolved		Arlolra	T39902 RFC: Implement rendering of redlinks in Parsoid HTML as post-processor
Resolved		Esanders	T64803 VisualEditor: Extract red link API code from VE and put into Parsoid
Resolved		Catrope	T39901 VisualEditor: Links are not shown as redlinks when their target is blank (or as 'stub-alternate-colour' if appropriate)
Declined		None	T101046 [EPIC] Use MCS as parser for main content in mobile web
Declined		None	T78734 Support for mobile's section use cases in Parsoid HTML
Resolved		ssastry	T114072 <section> tags for MediaWiki sections
Resolved		• bearND	T178705 Prepare MCS for Parsoid section tags
Resolved		• bearND	T178706 Improve section parsing in mobile-sections endpoint
Resolved		• bearND	T178707 Create script to compare old and new sectioning code
Resolved		• bearND	T178708 Update Wiktionary definition parsing to account for new (possibly nested) section tags
Open		None	T114194 MediaWiki should be able to support multiple parser engines
Resolved		• Jhernandez	T114279 Parsoid API responses contain unnecessary meta data bloating HTML response
Open		None	T78676 Store & load data-mw separately
Resolved		• marcoil	T88290 Update data-mw encoding for <ref> tags to point to the HTML content in the <references /> output rather than duplicating it
Resolved		Catrope	T88650 Support data-mw.body.id for reference contents
Resolved		• marcoil	T88660 Parsoid Cite: Render missing reflists
Resolved		Arlolra	T76518 Errors in Parsoid v2 entry point
Resolved		Arlolra	T54936 Move data-parsoid and possibly data-mw out of the DOM, add uids
Resolved		• GWicke	T50483 Expose a parsoid public web API end point in the MediaWiki API
Resolved		• GWicke	T89481 RESTBase beta release (revision storage / content API)
Resolved		• GWicke	T1228 RESTbase deployment
Resolved		fgiunchedi	T76986 RESTBase production hardware
Resolved		fgiunchedi	T88805 rack and setup restbase production cluster in eqiad
Resolved		• Cmjohnson	T83551 Split memcached in eqiad across multiple racks/rows
Resolved		• Cmjohnson	T82259 mc1016 mgmt not working
Resolved		Joe	T89345 Missing memcached logs
Resolved		fgiunchedi	T89639 restbase1006 faulty disk controller
Resolved		• GWicke	T75955 RESTBase / Parsoid integration - waiting for parsoid deploy
Resolved		• Jdouglas	T88456 On-demand generation of HTML and data-parsoid
Resolved		• GWicke	T88457 html2wt conversion support
Resolved		• GWicke	T88588 Formalize the RESTBase v1 specification
Resolved		Arlolra	T89411 Parsoid v2 API selser might not be working correctly
Resolved		• GWicke	T76370 RESTBase performance testing
Resolved		• mobrovac	T76165 Handle revision deletion and suppression in RESTBase
Resolved		• mobrovac	T87520 Set up update jobs for RESTBase: code done and deployed, to be configured
Resolved		• mobrovac	T88652 Integrate revisions into /page structure. Was: Create a /page/revision/{revision} entry point
Resolved		• mobrovac	T87392 Add property & index on namespace in page_revisions
Duplicate		• Jdouglas	T78214 Documentation location and deployment process
Resolved		Andrew	T88850 Import Cassandra packages for Jessie
Resolved		• GWicke	T89461 trebuchet puppet provider broken on systems without upstart
Resolved		fgiunchedi	T89636 setup internal LVS for restbase eqiad servers
Resolved		fgiunchedi	T78514 Detailed cassandra monitoring: metrics and dashboards done, need to set up alerts
Resolved		fgiunchedi	T93886 Cassandra/CQL query interface monitoring
Resolved		Eevans	T93884 graphs for Cassandra metrics
Resolved		fgiunchedi	T97024 some cassandra metrics sent with invalid values
Resolved		• GWicke	T78194 Public entry point for RESTBase
Resolved		• GWicke	T89567 Load test production cluster
Resolved		fgiunchedi	T89366 Access to restbase / cassandra cluster
Resolved		• mobrovac	T91102 Update / maintain Beta Cluster restbase cluster: Up & working with VE
Resolved		• GWicke	T91188 Need to change data model for >~300 wikis in single cassandra cluster
Resolved		• GWicke	T128231 All known clients of Parsoid HTML (either directly hitting Parsoid or via RESTBase) should provide Accept: headers in their requests
Resolved		• AlexMonk-WMF	T128233 VisualEditor: Add Accept: header to your HTML requests to RESTBase/Parsoid as per T124365
Resolved		• santhosh	T128235 CX Server: Add Accept: header to your HTML requests to RESTBase/Parsoid as per T124365
Resolved		ssastry	T128236 Flow: Add Accept: header to your HTML requests to RESTBase/Parsoid as per T124365
Resolved		• Mholloway	T128237 Mobile Content Service: Add Accept: header to your HTML requests to RESTBase/Parsoid as per T124365
Resolved		Jackmcbarn	T128238 EPH Gadget: Add Accept: header to your HTML requests to RESTBase/Parsoid as per T124365
Resolved		Renxiaoyi	T128239 Google: Add Accept: header to your HTML requests to RESTBase/Parsoid as per T124365
Resolved		cscott	T130685 OCG: Add Accept: header to your HTML requests to RESTBase/Parsoid as per T124365
Resolved		Kelson	T130686 Kiwix: Add Accept: header to your HTML requests to RESTBase/Parsoid as per T124365
Open		None	T114413 Support various conversions in Parsoid's pb2pb endpoint
Open		None	T130639 All known clients of Parsoid HTML that require data-mw should fetch data-mw separately (if using RESTBase) or process the data-mw blob in Parsoid's pagebundle API response (if using Parsoid directly)
Open		None	T88623 On Parsoid HTML read pages, VisualEditor should be able to load data-mw from a separate API call and zip it into the DOM
Resolved		Arlolra	T130638 Add data-mw as a separate JSON blob in the pagebundle output of Parsoid's API
Resolved		Arlolra	T91700 Store and unpack data-* attributes when serializing/parsing document fragments throughout the pipeline
Open		None	T130643 Content Translation should load data-mw from a separate API call alongside the body content
Declined		None	T130687 OCG should load data-mw from a separate API call alongside the body content
Open		None	T130689 Google's services should load data-mw from a separate API call to RESTBase
Open		None	T124837 Update Flow for Parsoid changes re data-mw
Declined		None	T125885 Switch Flow storage to store a page bundle blob instead of HTML
Declined		None	T125886 Run migration script (HTML => wikitext => page bundle) in production
Declined		None	T125888 Write migration script (HTML => wikitext => page bundle) in production
Open		None	T125889 Update code (extractors, converters) that uses data-mw to deal with separate components
Open		None	T125890 Figure out how to get page bundle from Flow VE
Invalid		None	T194980 Triage
Invalid		None	T130641 Allocate a storage bucket for Parsoid's data-mw attribute
Duplicate		None	T114282 Red links via Parsoid
Duplicate		None	T114286 Resolving styling inconsistencies between HTML returned by Parsoid and HTML returned by PHP parser
Resolved		• Jhernandez	T113769 Spike: Scope the first version of JavaScript-based prototype on existing Parsoid service
Resolved		Jdlrobson	T113649 Demonstrate impact of loading just lead sections of a group of static pages
Duplicate		Peter	T114216 Support CSV output from the WPT wrapper script
Resolved		• bmansurov	T115073 [6 hours] Test a subset of pages with only a lead section [take 2]
Declined		None	T145219 [EPIC] Cleanup MobileFormatter and MobileViewAPI where they overlap
Duplicate		None	T109461 [EPIC]: Improvements to mobileview API
Resolved		• nray	T86210 MobileFrontend throws away warnings and other data from action=parse
Resolved		phuedx	T69054 API mobileview "protection" sometimes returns object, sometimes returns empty array in JSON output
Resolved		Anomie	T12887 API JSON formatter returns [] as an empty return value - inconsistent with {} for non-empty values
Resolved		Anomie	T76728 Clean up ApiResult and ApiFormatXml, create new formatversion
Resolved		Anomie	T85236 Better unit testing for ApiFormat*
Resolved		EBernhardson	T90745 Flow ApiFlowModerateTopicTest::testModerateTopic() assumes ApiResult will pass through arbitrary objects
Resolved		Anomie	T98083 add checkbox to ApiSandbox to use formatversion=2
Resolved		Anomie	T98086 improve RELEASE-NOTES-1.25 for formatversion=2 change
Open		None	T212069 API action=wbgetentities does not handle formatversion=2
Duplicate		None	T78479 API error messages should be localised
Resolved		None	T98528 mobileview "thumbwidth" parameter doesn't always result in return of desired image url for SVG files
Resolved		phuedx	T156408 Create interface between Skin and MobileFormatter
Declined		None	T146396 Improve performance of mobile references API
Resolved		• Pchelolo	T133221 Support red links updates in change-propagation
Resolved		Arlolra	T169975 Missing images render as broken img tags, not redlinks

Event Timeline

• Mattflaschen-WMF created this task.Jun 21 2017, 6:08 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 21 2017, 6:08 PM

• Mattflaschen-WMF added a project: StructuredDiscussions.Jun 21 2017, 6:08 PM

Restricted Application added a project: Collaboration-Team-Triage. · View Herald TranscriptJun 21 2017, 6:08 PM

• Mattflaschen-WMF added a subtask: T39902: RFC: Implement rendering of redlinks in Parsoid HTML as post-processor.Jun 21 2017, 6:09 PM

• Mattflaschen-WMF moved this task from Untriaged to Flow Holding Area on the Collaboration-Team-Triage board.

What is the relative path issue? And, is there a bug for that?

• Mattflaschen-WMF updated the task description. (Show Details)Jun 21 2017, 6:28 PM

@ssastry It's not a bug per se, but basically Parsoid uses relative links starting with .:

E.g.

<!DOCTYPE html>
<html prefix="dc: http://purl.org/dc/terms/ mw: http://mediawiki.org/rdf/">
<head prefix="mwr: http://en.wikipedia.org/wiki/Special:Redirect/">
	<meta charset="utf-8">
	<meta content="0" property="mw:pageNamespace">
	<meta content="1.4.0" property="mw:html:version">
	<link href="//en.wikipedia.org/wiki/Earth/Foo" rel="dc:isVersionOf">
	<title></title>
	<base href="//en.wikipedia.org/wiki/">
	<link href="//en.wikipedia.org/w/load.php?modules=mediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.skinning.content.parsoid%7Cmediawiki.skinning.interface%7Cskins.vector.styles%7Csite.styles%7Cext.cite.style%7Cmediawiki.page.gallery.styles&amp;only=styles&amp;skin=vector" rel="stylesheet">
</head>
<body class="mw-content-ltr sitedir-ltr ltr mw-body-content parsoid-body mediawiki mw-parser-output" dir="ltr" id="mwAA" lang="en">
	<p id="mwAQ"><a href="./Earth/Bar" id="mwAg" rel="mw:WikiLink" title="Earth/Bar">Earth/Bar</a></p>
</body>
</html>

This is correct, since there is a valid <base href>. But to use that, we'd have to either:

Inject it into the head, above the normal Flow scope (which might cause issues with unrelated content e.g. header/sidebar/footer). Don't know if we tried this, though, it might work.
Do our own base href fix (what we currently do).

If Parsoid used URLs starting with / we wouldn't need to do anything here.

Yes, hrefs in Parsoid HTML have to be resolved (absolutized) using the base href before you can use them in other contexts. We need to do this in VE too, to prevent URLs from breaking when the path is /w/index.php?... instead of /wiki/Pagename.

• Mattflaschen-WMF updated the task description. (Show Details)Jun 26 2017, 8:37 PM

Trizek-WMF subscribed.Jul 11 2017, 5:07 PM