Page MenuHomePhabricator

Extract URLs from reference tags
Open, Needs TriagePublic

Description

Inside VisualEditor it would sometimes be useful to be able to extract all relevant URLs from a reference to enable e.g. checking for blocked URLs, analyzing reliability, verifying sources, etc. Currently this would have to be done manually.

Because references are just freeform wikitext, they could contain any number of URLs in any format. References generated through Citoid will use known templates (MediaWiki:Citoid-template-type-map.json) and their contents could be backtracked through said templates templatedata which is required to contain a "maps":{"citoid":{...}} structure that we could use to determine the fields that Citoid would put URLs into. However, a reference that just contains a bare URL is valid, as is one that contains non-template wikitext including links. (Or, for that matter, references that use non-citoid templates.)

Example references with URLs:

  • <ref>{{cite_web|website=https://google.com|title=Foo|archiveurl=https://...}}</ref>
  • <ref>[https://... Article title] at [https://... Daily Mail]</ref>

A loose heuristic of "check all external URLs inside the reference, and then also find all templates in the reference and check all their parameters for things that look like a URL" would probably work. It's just pretty inelegant.

Event Timeline

Incidentally, I wrote a very hacky version of this in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/VisualEditor/+/1301464 -- in the onDocumentChange method it grabs URLs from all existing references.