Page MenuHomePhabricator

Copy-pasting linked ISBN numbers from view mode HTML into VisualEditor inserts wikitext links to Special:BookSources (it should turn them into magic links?)
Open, LowPublic

Description

I was unable to find an existing bug for this, so I am filing a new one. This is either a VisualEditor or Citoid bug. Very occasionally, edits on en.WP tagged with "VisualEditor" are inserting a linked "ISBN" text followed by the ISBN value itself liked to Special:BookSources. This redundant linking is unnecessary. Here's a diff showing it happening:

https://en.wikipedia.org/w/index.php?title=Emperor_Jimmu&diff=next&oldid=795599385

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript
• Deskana changed the task status from Open to Stalled.Aug 29 2017, 12:23 PM
• Deskana triaged this task as Low priority.
• Deskana subscribed.

What are the steps to reproduce this? I tried:

  • writing an ISBN by hand
  • copying and pasting an ISBN from another site
  • copying and pasting an ISBN in read mode from another article on-wiki
  • copying and pasting an ISBN in edit mode in VisualEditor from another article on-wiki

I thought one of the last two might be the culprit, since VisualEditor might try to be "smart" and explicitly copy the link to Special:BookSources, but actually it just did the right thing.

If this can't be reproduced, diagnosis will be hard.

I don't know how it's happening. Maybe the editor who created the diff above will know.

I don't think this is a Citoid bug. These are bare references and citoid adds templates. And one of the ISBNs isn't even in worldcat which means citoid can't be used to add it anyway... maybe these were copy-pasted from somewhere?

This seems to have been copy-pasted from https://en.wikipedia.org/wiki/Nara_period#cite_note-2 (from the view mode, not from VE). I can reproduce the issue when doing this (I get wikitext like [[International Standard Book Number|ISBN]] [[Special:BookSources/978-1-59884-162-6|978-1-59884-162-6]]).

This is still happening, and gnomes need to clean up every time it happens. Can you please fix it? Here's one of the latest additions:

https://en.wikipedia.org/w/index.php?title=Redback_spider&type=revision&diff=806760513&oldid=806760142

You can see more of them here:

https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Check_Wikipedia/ISBN_errors

This is still happening, and gnomes need to clean up every time it happens. Can you please fix it? Here's one of the latest additions:

As noted above, I've been unable to reproduce this behaviour, and if it can't be reproduced then fixing it will probably be impossible.

• matmarex renamed this task from VisualEditor (Citoid?) inserts Special:BookSources links when ISBN is added to Copy-pasting linked ISBN numbers from view mode HTML into VisualEditor inserts wikitext links to Special:BookSources (it should turn them into magic links?).Nov 7 2017, 11:06 PM
• matmarex changed the task status from Stalled to Open.

Per my reproduction steps earlier (T174303#3562738).

Can someone please fix this Visual Editor bug? It is causing gnomes to have to do unnecessary work. In an extreme example, this VE-using editor unwittingly added Special:BookSources links to 170+ articles on a single day:

https://en.wikipedia.org/w/index.php?limit=500&title=Special%3AContributions&contribs=user&target=Recensore256&namespace=&tagfilter=&start=2018-08-01&end=2018-08-01

Now all of those edits need to be fixed by a gnome. It sure would be better if this bug in VE were fixed. You can see many more examples of recent additions of Special:BookSources links here:

https://en.wikipedia.org/w/index.php?title=Wikipedia:WikiProject_Check_Wikipedia/ISBN_errors&curid=42611293&diff=857871061&oldid=855596578

This is not a rare occurrence.

I tested this again today, with this test page: https://en.wikipedia.org/w/index.php?title=User:Matma_Rex/sandbox&oldid=858044048. It seems @Deskana has already noticed and described this behavior above, but to re-state clearly:

  • Copy-pasting of ISBN links from view mode already works if they are generated using the magic link syntax, like ISBN 0-7475-3269-9
    • A bit surprisingly, it also works for manual links that mimic the output of magic links, like [[Special:BookSources/0747532699|ISBN 0-7475-3269-9]]
    • It does not work when the link target includes the extra separators, like [[Special:BookSources/0-7475-3269-9|ISBN 0-7475-3269-9]]
  • Copy-pasting of ISBN links from view mode does not work for the ISBN template, like {{ISBN|0-7475-3269-9}}
CaseInput wikitextOutput wikitext (after copy-pasting from view mode to VE)
Template{{ISBN|0-7475-3269-9}}[[International Standard Book Number|ISBN]] [[Special:BookSources/0-7475-3269-9|0-7475-3269-9]]
Template output[[International Standard Book Number|ISBN]] [[Special:BookSources/0-7475-3269-9 |0-7475-3269-9]][[International Standard Book Number|ISBN]] [[Special:BookSources/0-7475-3269-9|0-7475-3269-9]]
Magic linkISBN 0-7475-3269-9ISBN 0-7475-3269-9
Manual link 1[[Special:BookSources/0747532699|ISBN 0-7475-3269-9]]ISBN 0-7475-3269-9
Manual link 2[[Special:BookSources/0-7475-3269-9|ISBN 0-7475-3269-9]][[Special:BookSources/0-7475-3269-9|ISBN 0-7475-3269-9]]

@Jonesey95 If you wish to avoid the issue, convince the English Wikipedia community to change https://en.wikipedia.org/wiki/Template:ISBN so that it produces output similar to the "Manual link 1" case, rather than the current output ("Template output"). I'm actually surprised that it doesn't do that; all the links to https://en.wikipedia.org/wiki/International_Standard_Book_Number that it produces seem like a bad case of overlinking, anyway. Was that already proposed before and rejected?

Doing so will at least produce magic links on copy-paste from view mode to VisualEditor, rather than the messy syntax.

Producing the actual template when copy-pasted from view mode is currently impossible, because there is no data as to what template generated the text in the PHP parser's output. It will become possible in the future when we use Parsoid's output for view mode (T54091, T55784). You can see that output for my test page at https://en.wikipedia.org/api/rest_v1/page/html/User%3AMatma_Rex%2Fsandbox/858044048, and copy-pasting from there to VE works perfectly for all cases.


In short: We can't improve this behavior in VisualEditor, or not least not until far future. Instead, you can change Template:ISBN so that it copy-pastes better, but that has to be done by English Wikipedians rather than us.

@Jonesey95 If you wish to avoid the issue, convince the English Wikipedia community to change https://en.wikipedia.org/wiki/Template:ISBN so that it produces output similar to the "Manual link 1" case, rather than the current output ("Template output"). I'm actually surprised that it doesn't do that; all the links to https://en.wikipedia.org/wiki/International_Standard_Book_Number that it produces seem like a bad case of overlinking, anyway. Was that already proposed before and rejected?

There's been some discussion of this here and there from an OLINK perspective, but what I've seen it would be replacing the wiki link with an external link + plainlink class applied, which would fix the WLH for the ISBN page but which isn't better for this particular task (I'd say worse, in fact).

On the point of whether that link should be there, the reason the link is desirable is so that people who don't know what an ISBN is can go and find out. This is the same for all of our identifier templates, in fact, so I would guess this issue can be observed with e.g. {{DOI}}.

The link to ISBN exists for a reason, as @Izno explains.

Meanwhile, this problem continues and copy-pasting (or something) appears to be generating superfluous span tags as well. See https://en.wikipedia.org/w/index.php?title=Virgil_Thomson&type=revision&diff=884836275&oldid=877453735

I wish the developers luck with fixing this annoying bug. In the meantime, it keeps us gnomes employed.

Per @Izno, this also appears to be happening with DOI and ISSN links. More junk for gnomes to clean up. See https://en.wikipedia.org/w/index.php?title=Purges_of_the_Communist_Party_of_the_Soviet_Union&diff=prev&oldid=884801720 for a sample diff.

And now bdi tags are being added inside the Special:Booksources link:

https://en.wikipedia.org/w/index.php?title=E._Pauline_Johnson&type=revision&diff=931250434&oldid=930478428

Please just treat ISBNs as plain text.

Still happening (edit from March 29):

https://en.wikipedia.org/w/index.php?title=Queen%27s_Royal_Regiment_%28West_Surrey%29&type=revision&diff=947998273&oldid=946711376

I just fixed about 40 articles with this problem. In addition, VE appears to be turning some cite and OCLC and DOI templates into plain text with links, which is a violation of CITEVAR and undesirable from a maintenance standpoint. See, for example:

https://en.wikipedia.org/w/index.php?title=Symbolic_interactionism&type=revision&diff=947587694&oldid=943355082

This is still happening as of March 2021:

https://en.wikipedia.org/w/index.php?title=Joana_Vasconcelos&type=revision&diff=1013657448&oldid=1008275436

That link shows a BookSources manual link, bdi tags, and undesirable OCLC linking instead of the OCLC template.

Is it possible to get this bug fixed before the fourth anniversary of it being reported? That would be wonderful.

As I have explained soon after the first anniversary of this bug being reported (T174303#4557445), this is caused by users copy-pasting the ISBN template from view mode into edit mode.

As far as I know, it can still be fixed, at any moment, by changing that template in the way that I suggested back then.

Failing that, the solution is using Parsoid HTML, with its associated metadata, for read mode views, as I also explained in that comment. The Parsing team's work on that is still progressing but it's not the highest priority, but perhaps we can celebrate that happening around fifth or sixth anniversary of this bug.

The above workaround (displaying ISBNs without surrounding template code) is not viable. Magic links were deprecated by MediaWiki developers in 2016, a year before this bug was reported:

https://www.mediawiki.org/wiki/Requests_for_comment/Future_of_magic_links#Proposal

They were deprecated on the English Wikipedia in 2017, four months before this bug was reported:

https://en.wikipedia.org/w/index.php?oldid=772743896#Future_of_magic_links

All ISBN magic links were replaced on en.WP, and after much delay, MediaWiki developers were persuaded to follow through with their commitment to disable magic links on en.WP in early 2021 (see T275951).

The fifth anniversary of this bug is coming up in less than a year. Let me know if I can help with troubleshooting or QA of any proposed fix for it.

Once or twice a month, this bug results in tens of thousands of bytes being added to the error report at https://en.wikipedia.org/w/index.php?title=Wikipedia:WikiProject_Check_Wikipedia/ISBN_errors&action=history

It would be nice for this bug, and related bugs that cause this error report to expand, to be fixed so that cleaning up that report would not seem like cleaning out the Augean stables.

Please fix this bug. Here's the latest diff showing many pages with errors being added to the report. It is pointless to try to fix these pages when this bug is still generating more errors every day.

A bot just cleaned up hundreds of edits caused by this bug on the English Wikipedia, and they are happening again at a rate of at least one per day. Here's one from teh last 24 hours: https://en.wikipedia.org/w/index.php?title=History_of_the_metre&diff=1274677738&oldid=1274351949

The ongoing work on Parsoid read views should make this feasible to resolve. This is much closer to happening now than it was when the task was created.

@Jonesey95, the task itself does not include any details on how to reproduce the error, but skimming the discussion above, it seems that you all have since worked out how it is happening. Updating the task description might help push this toward resolution.

DLynch subscribed.

I looked into this a bit.

  1. Visit https://en.wikipedia.org/w/index.php?title=Sea_of_Faith&diff=next&oldid=1375340109
  2. Go to Bibliography and find the list item Odyssey on the Sea of Faith: The Life and Writings of Don Cupitt, Nigel Leaves, Polebridge Press, 2004, ISBN 0-944344-62-3
  3. Inspect it with the developer tools:
<li id="mwAek"><i id="mwAeo">Odyssey on the Sea of Faith: The Life and Writings of Don Cupitt</i>, Nigel Leaves, Polebridge Press, 2004, <link rel="mw-deduplicated-inline-style" href="mw-data:TemplateStyles:r1333433106" about="#mwt90" typeof="mw:Extension/templatestyles mw:Transclusion" id="mwAes" data-mw="{&quot;name&quot;:&quot;templatestyles&quot;,&quot;attrs&quot;:{&quot;src&quot;:&quot;Module:Citation/CS1/styles.css&quot;},&quot;body&quot;:{&quot;extsrc&quot;:&quot;&quot;},&quot;parts&quot;:[{&quot;template&quot;:{&quot;target&quot;:{&quot;wt&quot;:&quot;ISBN&quot;,&quot;href&quot;:&quot;./Template:ISBN&quot;},&quot;params&quot;:{&quot;1&quot;:{&quot;wt&quot;:&quot;0-944344-62-3&quot;}},&quot;i&quot;:0}}]}"><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/ISBN_(identifier)" title="ISBN (identifier)" about="#mwt90" class="mw-redirect">ISBN</a><span typeof="mw:Entity" about="#mwt90">&nbsp;</span><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Special:BookSources/0-944344-62-3" title="Special:BookSources/0-944344-62-3" about="#mwt90" id="mwAew"><bdi>0-944344-62-3</bdi></a></li>
  1. Select the entire list item and copy it, then inspect your system clipboard in some way (I use Pasteboard Viewer):
<meta charset='utf-8'><ul id="mwAdI" style="margin: 0.3em 0px 0px 1.6em; margin-inline: 1.6em 0px; padding: 0px; list-style-type: disc; color: rgb(234, 236, 240); font-family: sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; white-space: normal; background-color: rgb(16, 20, 24); text-decoration-thickness: initial; text-decoration-style: initial; text-decoration-color: initial;"><li id="mwAek" style="margin-bottom: 0.1em;"><i id="mwAeo">Odyssey on the Sea of Faith: The Life and Writings of Don Cupitt</i>, Nigel Leaves, Polebridge Press, 2004,<span> </span><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/ISBN_(identifier)" title="ISBN (identifier)" about="#mwt90" class="mw-redirect" style="text-decoration: none; color: rgb(136, 163, 232); background: none; border-radius: 2px; overflow-wrap: break-word;">ISBN</a><span typeof="mw:Entity" about="#mwt90"> </span><a rel="mw:WikiLink" href="https://en.wikipedia.org/wiki/Special:BookSources/0-944344-62-3" title="Special:BookSources/0-944344-62-3" about="#mwt90" id="mwAew" style="text-decoration: none; color: rgb(136, 163, 232); background: none; border-radius: 2px; overflow-wrap: break-word;"><bdi>0-944344-62-3</bdi></a></li></ul>

The first is what's in the read view. The second is what VE has to work with when we import. Note that they are actually quite different, because (in this case) Chrome does a lot to bake the appearance of the copied text into it.

The problem is obvious: Chrome stripped the <link> that contains every bit of information about what template was used. Presumably as a combination of security and removing invisible elements. It leaves us with an <a> that doesn't know anything about the template it was made from, and so it's actually-impossible for VE to reconstruct the pasted markup correctly.

Why is there a <link> there? TemplateStyles in the template. Why does it contain all the information? Parsoid puts an id and data-mw on the first element a template outputs, and then associates the rest of the template's contents with it via about attributes.

I double-checked copy/paste of templates from parsoid read-views that don't open with a templatestyle, and they paste just fine.

Conclusion: VE cannot fix this specific {{ISBN}} template alone, though many other templates are already greatly improved compared to when this ticket was first filed.

What could be done?

  • The template could move its <templatestyles> to the end of its output. Parsoid would then attach its data to an earlier element that doesn't get stripped by the browser, and we'd be able to reconstruct the template. This is not ideal, because it relies on edits to specific templates and so we'd have to fix it case-by-case whenever it crops up... and also we've already gone 8 years with @matmarex's "just edit the template" suggestion not having been acted on. (Though this one is an invisible-to-the-reader change, unlike the earlier suggestion!)
  • Parsoid could change how it attaches the data-mw attribute. Instead of putting it on the first element in an aboutgroup, it could put it on the first rendered element. This might need some adjustments in VE, because I'm certain we assume the current behavior, and there's probably cases where it wouldn't get stripped before reaching us.
  • We could add client-side JS to all mediawiki article views that watches for a copy event and rewrites the HTML so that the information will be preserved in this case. (You've seen this more-invasively on e.g. news sites that sneak a link to themselves into whatever you copy from them.)

Probably what we should be doing is using the same clipboard handler we use for copy-and-pasting from VE content, which stores the actual "canonical parsoid" content in a separate mediatype. Now that Parsoid Read Views are effectively "canonical parsoid content", we should be passing them the same way.

There's probably some interesting corner cases around section tags and maybe some other things which show up in read view, but it should be manageable. Probably worth testing copying a full section (including its heading), which should ideally *not* paste the section edit link, the extra <div> wrappers, the <section> tag, etc.

We're assuming that Editing is going to write the first version of this (read views clipboard handler) and Content-Platform-Team can consult if there are architectural issues (ie section tag handling, etc).

Change #1345235 had a related patch set uploaded (by DLynch; author: DLynch):

[mediawiki/extensions/VisualEditor@master] ve.init.mw.ArticleCopyHandler adjusts copied HTML to keep parsoid data

https://gerrit.wikimedia.org/r/1345235

Change #1345235 merged by jenkins-bot:

[mediawiki/extensions/VisualEditor@master] ve.init.mw.ArticleCopyHandler adjusts copied HTML to keep parsoid data

https://gerrit.wikimedia.org/r/1345235