Page MenuHomePhabricator

RFC: Future of magic links
Open, MediumPublic

Description

  • Affected components: Parser (MediaWiki core).
  • Engineer for initial implementation: Parsing Team (WMF).
  • Code steward: Parsing Team (WMF).

Motivation

Background

Magic links are a feature of MediaWiki core that create automatic links for 3 hardcoded external identifiers:

See also: https://en.wikipedia.org/wiki/Help:Magic_links.

For the purposes of this RFC, we are considering free external links (e.g. typing just "https://www.example.org") to not be a "magic link".

Problem

These three magic links are hardcoded, inflexible, un-localizable (T15335), and generally unexpected. If this feature were proposed today, it would be rejected in favor of using templates or interwiki links. There have been long standing requests to make them disable-able (T47942), move them to an extension (T28207), or remove them outright (T136342).

In many cases, local templates are preferable and more advanced than magic links. For example, on the English Wikipedia, Template:ISBN checks for invalid ISBNs and adds them to a tracking category for editors to fix up.

Requirements

Plain text that does not involve some kind of syntatical grammar (such as {{ or <), must not have rich text side-effects.


Exploration

The RFC proposes three steps:

  1. Disable the magic link functionality by default for the MediaWiki 1.28 release, and mark it as deprecated. (approved in E287)
  2. Deprecate magic links on Wikimedia wikis (e.g. Wikipedia), providing alternatives for this functionality and tools to aid the migration. We agreed to start building these tools in E287.
  3. Disable magic links functionality a year or so after the MediaWiki 1.28 release (in time for the next MediaWiki LTS release)

Related Objects

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

No longer as a 1.28 blocker as those parts were completed.

I am not a defender of magic sources.

This magic syntax breaks syntax understanding, since it is an unexplainable exception, also for newbies and scripts. Nobody knows why just these three, no other keyword support. It was an accident in the first years of Wikipedia, when no-one knew how the seed will grow, and should be remedied in the long run.

Easiest replacement for fast ISBN magic would be an interwiki /Special:Booksources/$1.

In German Wikipedia I would like to see replacement by templates.

They give opportunity to check validity of input value (PMID and RFC numerical, RFC perhaps with #fragment of section, no RFC number beyond 9999 yet). If invalid, throw project defined maintenance category and show appropriate error message.

Same for ISBN, but things are much worse than a simple parser function woud help. See T148274#2788217 for details. The cite book pendant already checks PMID and ISBN and files warnings and cats.

Some people started to replace the magic links by some other syntax (templates) on some wikis, and a strange case was also found : it seems that magic links is broken when there are several consecutive filling characters in the text.

For example, this version of Ahmad Shah Durrani has a broken magic link at the end of the Bibliography. I experimented a bit and found that the magic doesn't work when there are several consecutive filling characters in the ISBN (2 whitespace, 1 whitespace and a dash...). None of the following work for example : ISBN 978- 1-4907 - 1441-7 ; ISBN 978- 1-4907-1441-7 ; ISBN 978 14907 14417 ; ISBN 97814907 14417

I think we should fix that before removing the magic links because the articles using this broken syntax are probably not listed in the tracking categories, so the bot won't go over them...

Could someone say what the status of this is, please? There has been very little discussion about it on enwiki. Editors who don't use citation templates use magic links for ISBN and PMID. There are around 340,000 pages using the former and 6,000 the latter. They are easy and fast to use, and it seems a pity to get rid of them.

Could someone say what the status of this is, please?

The status of the MediaWiki RfC (https://www.mediawiki.org/wiki/Requests_for_comment/Future_of_magic_links) is "in draft", and it has not yet been listed as "Ready for discussion" at https://www.mediawiki.org/wiki/Requests_for_comment. Unless the plan is to open it for discussion only after it is a fait accompli, phabricating an implementation seems premature..

Could someone say what the status of this is, please?

See T145604#2695311 and https://lists.wikimedia.org/pipermail/wikitech-ambassadors/2016-November/001500.html

There has been very little discussion about it on enwiki. Editors who don't use citation templates use magic links for ISBN and PMID. There are around 340,000 pages using the former and 6,000 the latter. They are easy and fast to use, and it seems a pity to get rid of them.

Right, if you don't use the standard citation templates, adding ISBN/PMID links will get a little harder. Interestingly, out of the 6.4k pages that use PMID magic links, less than half are in articles (https://quarry.wmflabs.org/query/15293). ISBN is ~317k out of ~350k in mainspace (which I think we expected, hence T148274).

The status of the MediaWiki RfC (https://www.mediawiki.org/wiki/Requests_for_comment/Future_of_magic_links) is "in draft", and it has not yet been listed as "Ready for discussion" at https://www.mediawiki.org/wiki/Requests_for_comment. Unless the plan is to open it for discussion only after it is a fait accompli, phabricating an implementation seems premature..

That was out of date sorry, I've changed it to "accepted" now. But this has been open for discussion since mid-September when I announced it on wikitech-l (https://lists.wikimedia.org/pipermail/wikitech-l/2016-September/086515.html), there was a formal RfC meeting (announcement: https://lists.wikimedia.org/pipermail/wikitech-l/2016-October/086713.html), and then there was more discussion when I announced it on wikitech-ambassadors in November (https://lists.wikimedia.org/pipermail/wikitech-ambassadors/2016-November/001500.html). So saying that it wasn't open for discussion is wrong.

The parts that have been agreed to is that magic links should be considered deprecated on Wikimedia wikis, and we (MediaWiki core developers) should work with communities to provide alternatives for this functionality and tools to aid the migration. How the migration happens and what alternatives are available are still open to discussion. T148274 has seen some pretty good discussion for ISBN links.

Hi Legoktm, you wrote that it was open for discussion because several places were notified, but none of these are where Wikimedia writers tend to hang out, and it's the writers who rely on these links. Can we hold a more central discussion about whether the ISBN and PMID links should be deprecated, rather than assuming it's a done deal?

Hi Legoktm, you wrote that it was open for discussion because several places were notified, but none of these are where Wikimedia writers tend to hang out, and it's the writers who rely on these links. Can we hold a more central discussion about whether the ISBN and PMID links should be deprecated, rather than assuming it's a done deal?

Discussion is always ongoing on https://www.mediawiki.org/wiki/Talk:Requests_for_comment/Future_of_magic_links and I'm more than willing to continue discussing use cases and figuring out alternatives. However I'm not "assuming it's a done deal" - it's been approved by the architecture committee (see "AGREE" points in T145604#2695311) that at some point in the future magic links are going to go away on Wikimedia wikis with specifics about timelines and alternatives still TBD. If you want to re-open that for discussion you should contact an ArchCom member to figure out what the process is.

Scott added a subscriber: Scott.Jan 7 2017, 2:01 AM

I agree with SV above. This is a major (and somewhat foolish) change. It should not be imposed by a technocracy.

jeblad added a subscriber: jeblad.Apr 15 2017, 1:37 PM

At first I thought this was a good idea, but not anymore. This type of magic formatting is a good thing. The problem isn't that it happen, but that it isn't smart enough. We should actively limit the number of weird templates the editors must remember. If we can add a little smartness to wikitext, then we should do it.

Whether architecture committee agree on something has virtually no bearing in this case, this is at its core about editing not about boxes. If the editing communities want a feature, then the devs should deliver that feature. It is the editors that is the customer in this case, not the architecture committee or any other dev.

At first I thought this was a good idea, but not anymore. This type of magic formatting is a good thing. The problem isn't that it happen, but that it isn't smart enough. We should actively limit the number of weird templates the editors must remember. If we can add a little smartness to wikitext, then we should do it.

I think this argument conflates the ease with which editors can insert formatted markup and whether we should support magic formatting. As noted repeatedly, these three magic links are anomalies. Editors expect to be able to use templates for formatting and it often causes confusion when editors notice that plain syntax has become a link. Templates can also be localized and have a built-in and supported means of tracking uses. Templates are what we use for nearly every other kind of similar inline formatting. Adding a little smartness to wikitext is historically and demonstrably where we get into trouble.

Adding a little smartness to wikitext is historically and demonstrably where we get into trouble.

Which is a unsubstantiated claim.

If I should make an unsubstantiated claim I would say that the current template solutions is what creates havoc, as it is very as they invites an editing style that creates extremely messy and unreadable markup. Don't replace simple markup with more messy and difficult to read and understand markup.

Not sure that the template markup "{{PMID|somenumber}}" is in any way messier and unreadabler than "PMID somenumber". The very existence of this task is kind of proof that the magic links can make problems. As well as a number of "RFC somenumber" making false positive links on enwiki.

Ghilt added a subscriber: Ghilt.May 23 2017, 12:11 PM

Since the vast majority of editors are non-coders, as well as potential new authors (especially older people), increasing the amount of special characters in the source text (e.g. for templates) is imho a step in the wrong direction. I wish more of mediawiki would work like magic words...

I think between all bad ideas which were discussed the MediaWiki development, removing magic links and especially ISBN is one of the worst. I do not consent with MZMcBride when he talks about confusion under editors because of the ISBN is suddenly linked. Editors and especially new editors are happy that this happens.

I do also contest PerfektesChaos' claim he would like to see the issue resolved by templates in the German WP. I am part of the German WP a good time longer than he and I can assure that that community is absolutely template-hostile, and they explicitely decided against Vorlage:ISBN and Vorlage:PMID in 2007. The main reason is that the German WP community dislikes, and part of it hates, template within "Fließtext" as they call the part of source text outside infoboxes and categories.

If you'll remove the magic links the following will happen (off course only if the community agrees to reinstate the formerly deleted template):

  • some editors will use the ISBN template and some will not
  • some editors will add templates which other editors had omitted
  • they will start editwars about the necessarity to link a specific ISBN (using template)
  • regular bot runs needed to fix where editors had been too lazy/scatty to use template

Or the outcome will be that what happened with doi: Most of doi's are standing in articles unformatted (unlinked) because of editors don't care. Jepp, I think that will happen to ISBN links. Most editors won't care and Wikipedia again goes one step user unfriendly.

Thus I call to revise the decision to remove magic links, I also call to reintroduce doi magic link and to introduce yet more useful magic links.

I do also contest PerfektesChaos' claim he would like to see the issue resolved by templates in the German WP. I am part of the German WP a good time longer than he and I can assure that that community is absolutely template-hostile, and they explicitely decided against Vorlage:ISBN and Vorlage:PMID in 2007. The main reason is that the German WP community dislikes, and part of it hates, template within "Fließtext" as they call the part of source text outside infoboxes and categories.

What about references? ISBNs are typically used as part of references, correct?

Does the German Wikipedia use <ref></ref> tags with templates? On the English Wikipedia, it's common to use references with templates such as <ref>{{cite book|isbn=1234}}</ref>, which can then accept the ISBN as an argument. This means no change to the markup of many pages, since only the underlying "Template:Cite book" needs to be potentially updated.

How do you explain to users that almost every citation identifier, including DOI, needs extra markup, except ISBN, PMID, and RFC? These three are the exception to otherwise consistent behavior. Why not remove the magical behavior and be explicit instead? If you look at https://en.wikipedia.org/wiki/Template:Citation/identifier#Usage, you can see many other kinds of citation identifiers that use wrapper templates such as {{ASIN}} or {{OCLC}}? Why is this bad? Why does it make sense to have the software try to parse the page and guess what is and is not a valid citation identifier?

You could also just write [[Special:BookSources/1234]] in the page instead of using a template such as {{ISBN|1234}}. Do you and other German Wikipedians dislike all wiki markup, including link markup? If so, perhaps VisualEditor is the answer?

Or the outcome will be that what happened with doi: Most of doi's are standing in articles unformatted (unlinked) because of editors don't care. Jepp, I think that will happen to ISBN links. Most editors won't care and Wikipedia again goes one step user unfriendly.

Your argument is that nobody will care enough to make links? This seems to point to the questionable value of having the links at all, if nobody is using them or interested in supporting them.

Ghilt added a comment.EditedMay 25 2017, 12:18 AM

Hi MZMcBride,
yes, ISBN are part of references. We generally use ref-tags, but citing templates are much less used than in en.wp. For example, in natural sciences afaik mostly a PMID- and DOI-converter is used (http://www.hbz-da.de/wikipedia/PMID2reference.php) instead of templates. This also reduces imho unnecessary code and increases readability of the source text. I wish magic links were extended to reduce the amount of special characters in the source text. And due to the necessity of mouse, trackpad or trackpoint for all operations besides typing words in the visual editor, it is much slower than writing wikitext. Which could be simplified with more magic links. So VE is only the answer for some. Most wikipedians i know personally (mainly active and very active editors) write wikitext. And i don't think his argument was that nobody would care to make links but most would prefer not to have to and some won't care to.

By the way, how would a bot do the job?

By the way, how would a bot do the job?

The idea is to have a bot do a one-time conversion. Then we can stop supporting these magic links and this magical linking behavior in MediaWiki core forever. RFC in particular has been very problematic since the wiki universe has its own concept of requests for comment that is distinct from the Internet Engineering Task Force's concept of RFCs.

Another nice advantage to templates is that we can then translate/localize these identifiers, instead of relying on the English "ISBN" or "PMID". For example, perhaps the abbreviation would be different in German or French or Spanish. By using templates, we also can track usage more easily.

I completely agree that {{template syntax}} and [[link syntax]] is ugly, but it's what we're already using everywhere, so I'm struggling to see a reason to continue supporting magic links or expanding their use.

A bot is currently running on the English Wikipedia to do this one-time conversion: https://en.wikipedia.org/wiki/Special:Contributions/Magic_links_bot. For wikis that do not want to do this conversion, the ISBNs and PMIDs will likely become unlinked at some point in the future. The options then will be to use link or template markup to make ISBN and PMID clickable, the same way we do with everything else currently, or to have the markup be unlinked. A third option might be a JavaScript hack, if the local wiki really were interested in setting that up and maintaining it.

Ghilt added a comment.EditedMay 25 2017, 12:53 AM

what we're already using everywhere

Tradition by itself is imho not a very strong argument. And a one-time conversion, does it mean we have to manually correct missing ISBN- and PMID-templates in future edits? I guess, the RfC isn't used in de.wp, as we have polls and requests for decisions instead. And, scnr, do i have a chance of convincing you of the good that is in our child?

patilise added a subscriber: patilise.EditedJun 19 2017, 4:14 PM

Hope I am not too late to chip in a reply -

! In T145604#3290707, @MZMcBride wrote:
How do you explain to users that almost every citation identifier, including DOI, needs extra markup, except ISBN, PMID, and RFC?

Why is an explanation needed? For average users they probably only want to see them linked, and that's it.

Your argument is that nobody will care enough to make links? This seems to point to the questionable value of having the links at all, if nobody is using them or interested in supporting them.

We don't have unlimited manpower. I am from the Japanese Wikipedia, where the user base is broad but the number of people able to (let alone interested to) support is really thin. Simply being "different and inconsistent" (or other technical reasons) is not a valid argument to cut a working feature and increase the already-too-heavy workload. If it is not like MediaWiki will break down tomorrow, can you please revise this decision to drop magic links?
(Adding one footnote: using a bot increases the number of revisions and uses up some manpower so that doesn't change my argument.)

In T145604#3290707, @MZMcBride wrote:

How do you explain to users that almost every citation identifier, including DOI, needs extra markup, except ISBN, PMID, and RFC?

Why should there be a need to explain that? That is, how it works, period.

Do you and other German Wikipedians dislike all wiki markup, including link markup?

It's not me, but some people want a clean, easy readable source text, and they dislike the VE either. However as Ghilt wrote, the usage of citation templates in DE:WP is not common, less than 10 percent of all articles use them, and there even exist users which remove them and convert them in plain link citations. What is much more needed is making en:Template:Citation a global template so it works in each and every language version. It are the same people blocking each development of citation templates (de:Vorlage:Citation is full protected for years!)

Your argument is that nobody will care enough to make links?

Not really. My argument is that they won't care enough to make links which formerly had been generated automatically, i.e. they will create the link only if it has a value for their own interest as in an article, for instance. They won't if it would be only courtesy, f. ex. in a discussion. My argument is also that people won't care to add such links if other editors did not add them.

In T145604#3290882, @MZMcBride wrote:

Another nice advantage to templates is that we can then translate/localize these identifiers, instead of relying on the English "ISBN" or "PMID".

Well, ISBN is ISBN everywhere on the world. Besides, localization has been the worst idea WP developpers implemented so far. Why?

  1. Users who want to use localized syntax still need to learn English syntax when they want to work on commons – [[Datei:Irgendwas.jpg|mini|hochkant|Irgendeinbild]] won't work on Commons.
  2. There are thousands of useless edits each and every day because of useless users convert File: to Datei: and thumb to mini, and those users spam watchlists and make it more difficult for authors to detect vandalism.

If it was me I would remove all syntax localizations. However, I like the usage of centralized templates with localized output, e.g. en:Template:Infobox UK settlement and de:Vorlage:Infobox Ort im Vereinigten Königreich use the same syntax of course the appearance is in the respective language. (That, basically, is the believe, that for example the Italian Wikipedia is most comprehensive for Italien settlement so DE:WP uses their (the Italian) syntax, and it uses the French WP syntax for French settlements infoboxes. Sorry I made a disgression but I felt it was needed to advert the impression I was some kind of fundamentalist ;-)

The explanation is needed because otherwise the feature is unused or people make bad markup for things that don't have the feature.

The explanation is needed because otherwise the feature is unused or people make bad markup for things that don't have the feature.

That's not a valid argument. If the feature is removed it will be unused as well. Or the other way around, so far it isn't possible to not-use the feature as ISBN <numbers> is marking up the number as ISBN link. Also "people making bad mark-up" is no argument. There are two possibilities: the ISBN a user typed is correct ( -> good markup) or the ISBN is wrong ( -> still good mark-up but the number does not exist). "ISBN 1234" is creating a bad ISBN-Link, as "1234" is not a valid ISBN. However, {{ISBN|1234}} still creates the same bad link. There is no difference between them. And the last part of your argument isn't valid as well. Actually thats kind of [[:en:WP:POINT]]; and most major language versions won't accept it. There is no reason to remove a feature because of another feature does not exist. The question shoudl be wether that other feature link should be a magic link.

For make it clear: IMHO we need much more magic links, not the other way around.

I was just contesting the claim that people will figure out by themselves how these things work.

"ISBN 1234" is creating a bad ISBN-Link, as "1234" is not a valid ISBN. However, {{ISBN|1234}} still creates the same bad link. There is no difference between them.

On the English Wikipedia, https://en.wikipedia.org/wiki/Template:ISBN uses https://en.wikipedia.org/wiki/Module:Check_isxn for input validation. If a user adds {{ISBN|1234}} to a page, the page will be automatically added to https://en.wikipedia.org/wiki/Category:Pages_with_ISBN_errors and the user is informed with red error text.

An "intentionally" invalid ISBN is handled by a separate template, as noted at https://en.wikipedia.org/wiki/Template_talk:ISBN#Allow_hiding_.22Invalid_ISBN.22_error.

Besides, localization has been the worst idea WP developpers implemented so far.

This is wrong, period. I suggest you read https://www.mediawiki.org/wiki/Principles, where #2 is "internationalized, with equal support for all languages;"

Just to let you know that I've wrapped wsd's patch to https://www.mediawiki.org/wiki/Extension:MagicalLinkers
It has been developed against MW 1.31. I think that with this extension, the core functionality can be removed.

jeblad added a comment.EditedAug 7 2018, 2:05 PM

Strange, it seems like mw:Requests for comment/Future of magic links is marked as accepted, still there are a lot of serious opposing comments that are not addressed. It the points to phabricator, but this thread neither does address the raised concerns.

Quite frankly, I wonder if this RfC is accepted at all.

<rant>What a few developers do in this case is a quite good example of reimplementing a markup language A with another markup language B, without having a real discussion of why B would be a better language than A. Because of this "I have a much better idea then you" we have several partially implemented languages. It has turned into a complete mess, and it will not be better by adding more templates. The editors already has to memorize to many weird templates, and that is not a good path forward!</rant>

Saimongoltinio closed this task as Declined.Jul 21 2019, 9:52 PM

А я попробую , час и будет готово

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptJul 21 2019, 9:52 PM
JJMC89 reopened this task as Open.Jul 21 2019, 9:53 PM
Elitre removed a subscriber: Elitre.Aug 29 2019, 9:08 AM

What is the status of this ticket? As far as I can tell, it has been more than three years since the release of MW 1.28 (https://www.mediawiki.org/wiki/MediaWiki_1.28), but magic links are still functioning, at least on en.WP.

Is it up to individual WP instances to turn off magic links locally? Did we at en.WP miss a memo at some point?

Krinkle renamed this task from [RfC] Future of magic links to RFC: Future of magic links.Apr 3 2020, 11:43 PM
Krinkle added a project: Parsing-Team.
Krinkle updated the task description. (Show Details)
Krinkle updated the task description. (Show Details)
Krinkle moved this task from Under discussion to P4: Tune on the TechCom-RFC board.

FWIW, we do not believe this task is blocked on Parsing-Team and we don't currently have this on our roadmap. Magic links are implemented and work in both the legacy parser and Parsoid as well as Visual Editor, so this is not blocking any feature development. It seems likely that magic links wouldn't be included in any notional "wikitext 2.0" proposal, but that's a long way off (if we ever get there).

If there is someone who wants to drive magic link removal, we are happy to support, but at this point that seems like it would be limited to a 2-ish line patch to add either a tracking category or a linter category to track "legacy" magic link usage on pages. We wouldn't write or deploy such a patch until we knew someone had an active effort to use this, though.

I don't understand the comment about tracking. The MW software has performed this tracking for more than three years, per Part 2 of the Exploration section of the original ticket description above.

See https://en.wikipedia.org/wiki/Category:Pages_using_ISBN_magic_links and https://en.wikipedia.org/wiki/Special:TrackingCategories

The categories, at least on en.WP, have been the subject of an active effort to convert article-space instances of ISBN magic links to ISBN templates for over three years, because MWF developers told us that magic links were going away.

At this point, this ticket is about Part 3 of the "Exploration" section of this ticket's original description, disabling magic links functionality. My understanding is that implementation of Part 3 is wholly in the hands of WMF developers. If I am wrong about that, and we should be implementing Part 3 (disabling magic links) locally on a per-wiki basis, please let us know.

ssastry added a subscriber: ssastry.EditedApr 20 2020, 11:40 PM

I think @cscott was trying to say: we don't want to pursue removal at this time if it needs us to ensure that all wikis fix up their links first. But, if wikis have already done the work of fixing up their pages, then it becomes easier for us to remove this. This is mostly a matter of work priorities. We haven't researched where different wikis are at this point. We are currently focused on getting Parsoid to replace the current parser and if taking this RFC to completion requires us to spend energies to ensure all wikis to fix their magic links first, then, it is lower priority for us at this point.

That said, over the next year, we are likely going to discover new linting categories for making Parsoid the default wikitext engine for wikimedia wikis. At that point, we could potentially fold magic link removal effort with it if there are still wikis out there that are using magic links.

Feedback welcome.

Thanks for the response.

Can en.WP turn off magic links locally?

Yes, magic links are a per-wiki feature flag: https://www.mediawiki.org/wiki/Manual:$wgEnableMagicLinks

That said, I'm not certain that Parsoid and VE can turn these off quite that easily yet: https://codesearch.wmflabs.org/search/?q=EnableMagicLinks&i=nope&files=&repos= doesn't show anything checking this configuration variable in the VIsualEditor or Parsoid codebases. So if a wiki (enwiki?) is actually ready to turn off magic links in core, perhaps I was wrong and this is a task the Editing-team and Parsing-Team need to schedule. That's T145590: Update Parsoid to be compatible with magic links being disabled and T145589: Update VisualEditor to be compatible with magic links being disabled, which were created in 2016 but not linked up (until I just did now) as subtasks of this RFC.

cscott added a comment.May 6 2020, 6:30 PM

Created T252054: Implement magic link functionality as an extension for possible future follow-up work that would let us pull the magic link functionality out of the core codebase. Not a blocker here; in fact pulling magic links out of the core codebase could be done even if we got blocked on actually turning off magic links on one of the WMF wikis.

@cscott Are there unanswered questions here and/or other outreach you'd like to happen before this proceeds? Note that closing of this RFC does not mean that it has to be disabled/removed from prod the next day, so if this plan is accurate and up-to-date and there are no unknowns, I'd recommend you propose this for Last Call :)

Krinkle triaged this task as Medium priority.Jul 29 2020, 8:35 PM