ArchCom RFC Meeting W40: Future of magic links. (2016-10-05, #wikimedia-office)

Hosted by daniel on Oct 5 2016, 9:00 PM - 9:00 PM.



Meeting summary

Meeting ended at 22:02:45 UTC.

People present (lines said)

  • cscott (66)
  • TimStarling (52)
  • robla (39)
  • subbu (22)
  • bd808 (22)
  • brion (9)
  • James_F (8)
  • Scott_WUaS (4)
  • marktraceur (3)
  • kaldari (3)
  • wm-labs-meetbot (3)
  • stashbot (2)


121:01:02 <TimStarling> #startmeeting RFC meeting
221:01:02 <wm-labs-meetbot> Meeting started Wed Oct 5 21:01:02 2016 UTC and is due to finish in 60 minutes. The chair is TimStarling. Information about MeetBot at http://wiki.debian.org/MeetBot.
321:01:02 <wm-labs-meetbot> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
421:01:02 <wm-labs-meetbot> The meeting name has been set to 'rfc_meeting'
521:01:05 <cscott> good afternoon
621:01:18 <TimStarling> #topic RFC: Future of magic links | Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) |​ Logs: https://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/
721:01:54 <robla> #link https://phabricator.wikimedia.org/T145604
821:02:08 <robla> #link https://www.mediawiki.org/wiki/Requests_for_comment/Future_of_magic_links
921:02:49 <robla> hi everyone! hi legokt^H^H^H^H^H^Hcscott :-)
1021:03:32 <cscott> i will be serving as your legokt today
1121:04:28 <robla> there are three steps we discussed that are summarized in the description of T145604
1221:04:29 <stashbot> T145604: [RfC] Future of magic links - https://phabricator.wikimedia.org/T145604
1321:05:04 <Scott_WUaS> Hi
1421:05:12 <robla> step 1: Disable the magic link functionality by default for the MediaWiki 1.28 release, and mark it as deprecated.
1521:05:30 <robla> 2. Deprecate magic links on Wikimedia wikis (e.g. Wikipedia), providing alternatives for this functionality and tools to aid the migration
1621:05:30 <robla> 3. Disable magic links functionality a year or so after the MediaWiki 1.28 release (in time for the next MediaWiki LTS release)
1721:05:50 <robla> is there anyone that objects to step 1?
1821:06:34 <TimStarling> is legoktm online?
1921:06:43 <cscott> maybe we should find out who's here? (that is, actively participating in this discussion.)
2021:07:03 <cscott> TimStarling: no, legoktm is in class, so I'm being his surrogate for this meeting.
2121:07:16 <robla> legoktm is at a class right now, so if there are objections or things he needs to clarify, we'll need to plan a followup
2221:08:02 <Scott_WUaS> (Hi CScott - I'm reading along, typing occasionally)
2321:08:05 <robla> it seems, though, that this is a "yeah, we should just do this" kind of thing for step 1
2421:08:08 * subbu is around and is generally happy with the direction, modulo details that might need tweaking
2521:08:31 <TimStarling> is there going to be an extension to replace it?
2621:08:39 <bd808> step 1 is normal deprecation procedure I think
2721:09:02 <brion> a workalike extension could be created if someone wants one but i don't know if anyone does :)
2821:09:08 <cscott> i think there's still a semi-open question w/ what the replacement should be: 1) ordinary template, 2) parser function, 3) extension, 4) interwiki link, 5) something else?
2921:09:13 <TimStarling> if there's no migration path for external users then I don't think it should be deprecated
3021:09:13 <bd808> Kunal wrote something about that in email/phab today...
3121:09:21 <TimStarling> just disabled by default
3221:09:50 <brion> my inclination is ordinary template is the right thing for wikipedia
3321:09:54 <cscott> 1) and 2) look like {{ISBN|xxx}} and require content edits, 3) would be a workalike using a parser hook, 4) would look like [[isbn:xxx]] and require content edits.
3421:09:58 <robla> legoktm's position is that disabling just disables the hyperlinks, but it's still readable
3521:10:27 <brion> it is a relatively clean 'failure mode' yes
3621:10:38 <brion> no explosions/php fatals :)
3721:10:40 <cscott> i think we already have a config option for disabling it in the parser? legoktm wrote that patch. so it's just a matter of turning that config option off on WMF wikis.
3821:11:27 <cscott> we can discuss removing the feature from the codebase and/or moving it to a parser extension as a separate thing.
3921:11:31 <subbu> I don't like option (3)
4021:11:38 <bd808> TimStarling: "We would move the Booksources code and ISBN parser function to an
4121:11:39 <bd808> a extension." -- https://lists.wikimedia.org/pipermail/wikitech-l/2016-October/086716.html
4221:12:11 <subbu> i.e. option (3) for parsing "ISBN ....." as a magic link with a parser hook
4321:14:15 <cscott> i don't like it for WMF wikis, but it would be a reasonable thing to allow 3rd parties to continue to support magic links on their external wikis if they felt strongly about it
4421:14:21 <TimStarling> using square brackets seems elegant to me
4521:14:27 <cscott> re TimStarling's "if there's no migration path for external users then I don't think it should be deprecated"
4621:14:32 <TimStarling> considering it is an actual link
4721:15:03 <subbu> I can go with either {{isbn|..}} or [[isbn:..]] personally
4821:15:04 <bd808> pmid and rfc are proposed to become default interwiki links
4921:15:24 <bd808> isbn is the only oddball right?
5021:15:27 <cscott> [[rfc:1234]] would work fine
5121:15:37 <cscott> isbn is the oddball because it goes to *the wiki itself*
5221:15:42 <bd808> *nod*
5321:15:54 <TimStarling> yeah, it would be like a namespace alias
5421:15:55 <cscott> although i don't think there's anything preventing interwiki links from being relative URLs?
5521:15:58 <TimStarling> like media
5621:16:29 <TimStarling> it could even be a negative namespace ID rather than an interwiki prefix
5721:17:15 <brion> interesting idea
5821:17:37 <brion> i think i prefer an interwiki, with special:booksources as a special service that happens to be the link target
5921:18:27 <robla> is there a precedent for magic link functionality in other wiki markups? e.g. does anyone else automatically turn "ISBN \d+" into a hyperlink automatically?
6021:18:36 <brion> but negative (magic) namespace has a certain elegance to it too
6121:19:11 <cscott> robla: external link hyperlinking is the precedent
6221:19:16 <cscott> ie, http://foo.bar/bat
6321:19:34 <cscott> most other markup languages have that
6421:19:43 <brion> bugzilla autolinks textual things like 'bug 1234'
6521:19:49 <cscott> when given a bare url
6621:19:57 <brion> and of course many mobile browsers autolink phone numbers (which i hate ;)
6721:20:00 <cscott> github markdown autolinks hashes and bug references
6821:20:25 <TimStarling> $RFCPattern = "RFC\\s?(\\d+)";
6921:20:25 <TimStarling> $ISBNPattern = "ISBN:?([0-9- xX]{10,})";
7021:20:32 <TimStarling> s/$RFCPattern/&StoreRFC($1)/geo;
7121:20:38 <TimStarling> s/$ISBNPattern/&StoreISBN($1)/geo;
7221:20:39 <cscott> https://help.github.com/articles/autolinked-references-and-urls/
7321:20:40 <TimStarling> says usemod
7421:20:49 <TimStarling> so this feature was in fact carried over from usemodwiki
7521:21:08 <cscott> PMID was a more recent addition
7621:21:47 <robla> that would imply a precedent that we're continuing rather than merely an oddball legacy feature of our own
7721:22:17 <cscott> weeelll.
7821:22:30 <robla> that said, I think I agree with brion's hatred of autolinked phone numbers
7921:22:38 <cscott> precent becomes oddball legacy given enough time (and lack of external adoption)
8021:22:45 <TimStarling> usemod is also responsible for the "free link" terminology
8121:23:25 <robla> it's quite likely just an oddball legacy feature that we inherited from grandpa usemod :-)
8221:24:45 <TimStarling> the argument for deprecation would be simpler parser code?
8321:25:27 <cscott> simple parser spec & removal of an english-specific bit of markup
8421:25:42 <TimStarling> but as long as we magically link free URLs, we will still have Parser::doMagicLinks()
8521:25:48 <cscott> RFCs in particular are english-only and not really relevant for most of our projects
8621:26:15 <TimStarling> that's the argument for step 2
8721:26:16 <James_F> TimStarling: And the magic of image links -> remote transclusion when that evil flag is enabled.
8821:26:20 <TimStarling> I mean the argument for step 1
8921:26:23 <cscott> sure, but doMagicLinks() could be greatly simplified and only match https?: prefixes
9021:26:30 <cscott> for example
9121:26:39 <cscott> instead of the pretty complicated regexp for ISBNs
9221:26:40 <TimStarling> I am fine with step 2 and 3 but not so much with step 1
9321:27:02 <cscott> and our free URLs autolinking is pretty crazy complicated because of all the different protocol schemes we support in theory.
9421:27:30 <subbu> I don't like "magic links" because they suddenly add special meaning to some text substrings ...
9521:27:59 <cscott> it complicates WTS as well, you have to watch for all sorts of boundary conditions when serializing
9621:28:01 <TimStarling> if it's disabled on WMF wikis then parsoid doesn't need to support it
9721:28:19 <James_F> Parsoid is for more than just WMF users, eventually.
9821:28:22 <subbu> instead of using uniform syntax .. and as someone argued, RFC xyz is ocfusing since mediawiki has RFCs.
9921:28:25 <bd808> o_0 parsoid is Wikimedia only TimStarling?
10021:28:29 <subbu> *confusing
10121:28:44 <cscott> and if we're going to continue to do round-trips for content migration, whether to wikitext 2.0 or markdown or <fill in the blank>, then simplifying WTS (wikitext serialization) will continue to be desirable
10221:29:02 <TimStarling> parsoid has a reduced feature set compared to MW
10321:29:04 <robla> it seems like protocol links (like mailto:.*, http?s:.*) have ample precedent in both email and wikis. Magic words as arbitrary barewords are relatively rare
10421:29:05 <subbu> and all the nowiki crap.
10521:29:09 <TimStarling> it doesn't even support wikipedia properly
10621:29:24 <bd808> well that I can't argue against
10721:29:38 <cscott> robla: yes, but have you looked at the full list of protocols we support?
10821:29:54 <TimStarling> there are no plans to make parsoid support every single MW parser extension
10921:30:01 <robla> cscott: not in a while
11021:30:05 * robla goes to look
11121:30:22 <cscott> $wgUrlProtocols = [ 'bitcoin:', 'ftp://', 'ftps://', 'geo:', 'git://', 'gopher://', 'http:// ', 'https://', 'irc://', 'ircs://', 'magnet:', 'mailto:', 'mms://', 'news:' , 'nntp://', 'redis://', 'sftp://', 'sip:', 'sips:', 'sms:', 'ssh://', 'svn://', 'tel:', 'telnet://', 'urn:', 'worldwind://', 'xmpp:', '//' ];
11221:30:27 <subbu> TimStarling, yes .. i've proposed that parser hooks that rely on parser internals should instead be deprecated.
11321:30:46 <subbu> parsoid and php parser have different internals.
11421:30:47 <cscott> i'm pretty sure we could drop gopher:// and worldwind://
11521:31:05 <bd808> I hear gopher is going to make a comeback ;)
11621:31:25 <TimStarling> free external links could support a reduced set of protocols compared to bracketed external links
11721:31:27 <cscott> but in general, I like {{bitcoin|<hash>}} or [[bitcoin:hash]] better than autolinking bitcoin:cafebabe in text
11821:31:42 <subbu> but, anyway ... we do want to get to a unified model .. and that parsoid doesn't support wikipedias is neither here nor there .. whatever parser it is that supports wikipedias will need to grapple with the roundtripping and html2wt issue.
11921:31:45 <TimStarling> that's why we have so many protocols, because non-WMF users want to link to those things
12021:32:25 * robla doesn't think the list cscott provided seems so crazy
12121:32:29 <cscott> TimStarling: perhaps, but I think they could manage to use slightly different markup to accomplish the same task.
12221:32:55 <TimStarling> like bracketed external links?
12321:32:59 <cscott> exactly
12421:33:23 <cscott> or double-bracketed links for some things perhaps
12521:33:26 <TimStarling> [[Gopher (protocol)]] does of course have many gopher links on it
12621:33:29 <Scott_WUaS> robla: agreed - cscott's list could make sense
12721:33:40 <TimStarling> but all bracketed as far as I can see
12821:33:50 <marktraceur> cscott: Pretty sure gopher:// is officially deprecated now, from what I vaguely recall
12921:35:55 <cscott> so i'm going to be a poor advocate for legoktm and say I'm not particularly chuffed by magic links. i'd get rid of them for wikitext 2.0 as part of a broader simplification, but now that they are implemented and working it's not really helping us much to get rid of them.
13021:36:24 <robla> so...perhaps we can agree to make the linking aspect off by default, without necessarily declaring them deprecated in 1.28
13121:36:35 <cscott> i guess the question is whether you think we can slim down wikitext incrementally by turning off these weird crufty bits one by one, or whether we should treat it as a completely new markup language
13221:36:48 <cscott> and use something like "parsoid2" to do round-trips between "wikitext 1" and "wikitext 2.0"
13321:37:03 <subbu> what i think of wikitext 2.0 is probably different from what cscott thinks of wikitext 2.0 probably :)
13421:37:15 * marktraceur may have been wrong, the minnpost article he must have seen was just a random summary of Gopher
13521:37:27 <cscott> i imagine a language with a formal grammar, small enough to fit on a single sheet of paper.
13621:37:36 * robla doesn't want to rabbithole on that topic, when we might actually be able to make a decision about Mediawiki 1.28
13721:37:36 <cscott> but otherwise looking as much as possible like wikitext 1.0
13821:37:36 <subbu> marktraceur, ah ... now it makes sense gopher came form UMn ... and that is why gopher
13921:37:53 <marktraceur> cscott: How big is the piece of paper? Implementation details...
14021:38:13 <robla> can we turn magic words off by default in 1.28 without deprecating?
14121:38:27 <cscott> and so, listing 28 different external link prefixes and 12 or so separate productions for ISBN/RFC/PMID is not going to help wikitext 2.0 fit on a single sheet of paper
14221:39:16 <robla> er...I suppose I should have said "magic links"
14321:39:28 <robla> can we turn magic links off by default in 1.28 without deprecating?
14421:40:01 <subbu> sure .. i guess the discussion is more about whether deprecation is required / desirable.
14521:40:12 <cscott> https://github.com/wikimedia/parsoid/blob/master/lib/wt2html/pegTokenizer.pegjs.txt#L449 is the grammar for RFC/PMID/ISBN in parsoid, for reference.
14621:40:46 <cscott> as far as i'm concerned the real question is: can we get the *wiki communities to start rewriting content to use whatever our preferred replacement markup is?
14721:40:46 <robla> subbu: yes, but that seems to be sending us down the rabbithole of talking about Wikitext 2.0, which we aren't going to finish in this hour
14821:41:06 <subbu> personally, i don't think we need to talk about wikitext 2.0 there.
14921:41:20 <cscott> https://en.wikipedia.org/wiki/Template:ISBN was created by MZMcBride
15021:42:01 <cscott> and it seems to already be used by quite a number of pages
15121:42:15 <TimStarling> as a compromise, how about not deprecating it immediately, but revisit after a year or two and see how many people are turning on the option?
15221:42:31 <subbu> i think it is a worthwhile discussion in its own merit. but, like cscott i don't have strong feelings ... but yes, it will simplify some code in parsoid .. but i won't be heartbroken if it is kept as is.
15321:42:37 <cscott> one intermediate step might be to hack the parser to add a parser warning if you use magic link syntax, suggesting an appropriate rewrite
15421:42:46 * robla likes TimStarling's suggestion
15521:43:08 <cscott> we can do that w/o committing to deprecation, just encouraging people not to use that syntax.
15621:43:14 <bd808> THen we'd just need a way of collecting that data
15721:43:19 <cscott> and then, as TimStarling suggests, give it a year or so and see what our trends are.
15821:43:30 <subbu> ya what bd808 says .. do we have a mechanism for collecting that data?
15921:43:45 <cscott> we could probably hack together a dumpGrepper script that would collect repeatable numbers for long-term comparison purposes
16021:43:47 <TimStarling> there's that pingback feature ori introduced...
16121:44:06 <cscott> we could also add [[Category:Uses Magic Link Syntax]]
16221:44:38 * robla looks for the link to the aforementioned pingback feature
16321:44:44 <subbu> i know (or am i imagining it?) kaldari had strong feelings about magic links. curious if he is around.
16421:44:49 <TimStarling> currently it doesn't send any configuration
16521:45:31 <TimStarling> #link https://www.mediawiki.org/wiki/Manual:$wgPingback
16621:46:03 <kaldari> My suggestion was to just kill the RFC magic linking entirely, as it's usefulness is very marginal
16721:46:22 <kaldari> otherwise, I support making it configurable
16821:46:42 <kaldari> and depricating
16921:46:46 <bd808> I think that part has been done now. there's a feature flag for it
17021:47:14 <bd808> and this discussion is about deprecation and removal from core
17121:47:26 <TimStarling> the dynamic dates was removed, there is that precedent, but dynamic dates was never on by default and was rarely used by non-WMF wikis as far as we know
17221:47:57 <robla> we could conceivably deprecate in 1.29 if the 1.28 release goes well
17321:48:03 <TimStarling> whereas we've been told that magic links are regularly used
17421:48:22 <subbu> ISBN in citations only from what I can tell.
17521:48:31 <TimStarling> I mean outside WMF
17621:48:35 <subbu> ah ..
17721:48:40 <TimStarling> for WMF we can run bots, outside WMF not so much
17821:48:57 <TimStarling> outside WMF we don't know if people even want to run bots, or if they like their magic links the way they are
17921:49:04 <robla> the pingback feature would give us more certainty, but I think even just "did anyone complain?" might be a good enough test in this case
18021:49:35 <bd808> or would anyone who would complain actually upgrade anyway...
18121:50:35 <cscott> it would be nice if WMF published a bot framework and scripts for it w/ every major upgrade
18221:50:53 <cscott> like how we upgrade database schemas, we could provide tools for external wikis to update their content
18321:51:02 <bd808> I guess I'm not sure why it needs to die if there is a feature flag other than hypothetical future parser world that would like to not support the feature
18421:51:18 <bd808> cscott: I think we call that pywikibot
18521:51:22 <robla> I suppose deprecating in 1.28 would be the "be bold" way of doing it. we could be prepared to "undeprecate" if people hate that we turned it off
18621:51:43 <cscott> bd808: sure, it's the "official" part and "with scripts for every major upgrade" which is missing.
18721:51:47 <James_F> Do "off by default" changes need to go through the one-version-notice process?
18821:52:07 <TimStarling> heh
18921:52:14 <TimStarling> disable by default then silently remove in 1.29?
19021:52:20 <James_F> Tut. :-)
19121:53:05 <robla> James_F: I don't *think* we have a policy like that, but I suppose that may be what legoktm's other rfc covers in more depth
19221:53:27 <bd808> cscott: I'm not sure I agree on "official" and WMF being intrinsically intertwined, but some suggested wikitext cleanup scripts would be an interesting thing to add when changing the syntax
19321:53:38 <James_F> Sorry, yes, I more meant "would it be covered by the semi-in-practice policy?".
19421:53:46 <robla> the deprecation rfc: T146965
19521:53:47 <stashbot> T146965: [RfC] Deprecation policy for PHP code in MediaWiki - https://phabricator.wikimedia.org/T146965
19621:53:52 <cscott> i'm always interested in process improvements to make it easier for us to evolve wikitext syntax
19721:54:57 <TimStarling> so should we call the non-controversial parts of this approved?
19821:55:13 * subbu cares less about syntax and more about underlying processing model / semantics except where the syntax gets in the way
19921:56:09 <subbu> wfm .. but, explicitly listing out the non-controversial parts would be useful.
20021:56:26 <TimStarling> #agreed 1. Disable the magic link functionality by default for the MediaWiki 1.28 release
20121:56:58 <TimStarling> #agreed 2. Deprecate magic links on Wikimedia wikis (e.g. Wikipedia), providing alternatives for this functionality and tools to aid the migration
20221:57:26 <robla> hmmm....I'd feel more comfortable with broader input on #2
20321:57:42 <cscott> sure. and the controversial step 3 would be removing the magic link code from core (perhaps moving it to an extension)?
20421:57:52 <robla> I suppose "start step #2" isn't yet controversial
20521:58:01 <James_F> Yes.
20621:58:10 <cscott> i'd propose 2(a) -- add a category and parser warning for magic links on wikimedia wikis, without officially deprecating the feature.
20721:58:22 <James_F> cscott: To move to an extension it'd have to keep the horrible hooks into the PHP parser, though?
20821:58:28 <cscott> then sit back and see if folks are getting the magic links cleaned up or not.
20921:58:35 <TimStarling> #info no consensus on removing the functionality from MW or flagging its pending removal via deprecation
21021:59:21 <TimStarling> no, step 3 is disabling magic links on WMF wikis
21121:59:23 <cscott> James_F: yes, although in my big picture worldview we're adding a pluggable parser API, and gradually moving from the "old" PHP parser to a newer one.
21221:59:32 * James_F nods.
21321:59:34 <TimStarling> removing from MW core was 1b
21421:59:45 <cscott> so the hooks would only stay in the legacy PHP parser. which, of course, you could keep running if you like and are willing to keep it maintained.
21522:00:05 <bd808> that's one possible universe, yes cscott
21622:00:06 <Scott_WUaS> (Thanks, All)
21722:00:27 <cscott> bd808: i keep pushing the universe towards my platonic ideal of it. ;)
21822:00:35 * cscott has to turn into a pumpkin
21922:00:54 <bd808> I like the "everything in the core php app" universe better
22022:01:03 <bd808> but that's why we have these chats
22122:01:10 * robla plans to turn into a different type of squash
22222:01:29 <bd808> its a dangerous time of year to be a pumpkin
22322:01:37 <robla> srsly
22422:01:46 <bd808> could be gutted and filled with candles at any point
22522:01:47 <TimStarling> what was next week's RFC again robla?
22622:01:57 <robla> next week we'll plan on talking about CREDITs file
22722:01:59 * robla finds link
22822:02:18 <robla> https://phabricator.wikimedia.org/T139300
22922:02:20 <cscott> bd808: pluggable doesn't mean not monolithic or not php, btw
23022:02:25 <cscott> just that things are decoupled
23122:02:39 <cscott> so you can have a pure-PHP single-process markdown wiki if you like.
23222:02:42 <TimStarling> ok, all done
23322:02:45 <TimStarling> #endmeeting

Other meetings

Architecture meetings
13:00 PT ArchCom Planning Meetingsupcomingall since 2016-03-30
14:00 PT ArchCom-RFC Meetingsupcomingall since 2015-09-09

Recurring Event

Event Series
This event is an instance of E66: ArchCom RFC Meeting Wxx: <topic TBD> (<see "Starts" field>, #wikimedia-office), and repeats every week.

Event Timeline

RobLa-WMF renamed this event from ArchCom RFC Meeting Wxx: <topic TBD> (<see "Starts" field>, #wikimedia-office) to ArchCom RFC Meeting W40: Future of magic links. (2016-10-05, #wikimedia-office).Oct 3 2016, 11:49 PM
RobLa-WMF updated the event description. (Show Details)
daniel renamed this event from ArchCom RFC Meeting W40: Future of magic links. (2016-10-05, #wikimedia-office) to ArchCom RFC Meeting Wxx: <topic TBD> (<see "Starts" field>, #wikimedia-office).Nov 21 2016, 6:11 PM
daniel changed the host of this event from RobLa-WMF to daniel.
daniel invited: ; uninvited: .
daniel updated the event description. (Show Details)
daniel renamed this event from ArchCom RFC Meeting Wxx: <topic TBD> (<see "Starts" field>, #wikimedia-office) to ArchCom RFC Meeting W40: Future of magic links. (2016-10-05, #wikimedia-office).