Agenda
- Location: #wikimedia-office IRC channel
- Meeting type: Problem definition
- Time: 2016-08-10, Wednesday 21:00 UTC (2pm PDT, 23:00 CEST)
- https://www.mediawiki.org/wiki/Wikitext (and whether the movement should invest in a spec)
- https://www.mediawiki.org/wiki/Parsing/Notes/A_Spec_For_Wikitext and in this thread https://lists.wikimedia.org/pipermail/wikitech-l/2016-August/086200.html
This week's office hour: Wikitext! This discussion is intended to be
a continuation of the "Loosing the history of our projects to bitrot."
thread.
Coren stated the work in front of us very well at the start of it.
You know, this is actually quite troublesome: as the platform evolves
the older data becomes increasingly hard to use at all - making it
effectively lost even if we kept the bits around. This is a rather
widespread issue in computing as a rule; but I now find myself distressed
at its unavoidable effect on what we've always intended to be a permanent
contribution to humanity.
The thread he started had pretty robust participation on a really
important, which seemed to us in ArchCom worth continuing the
discussion in one of our weekly office hours. So, after checking with
Subbu (in my list message) that's what ended up as the top candidate.
Subbu did some work to structure the conversation ([A Spec For
Wikitext] and I did some cleanup of the [Wikitext] page on
mw.org as a possible hub for information on this topic, with
[Talk:Wikitext] providing a durable conversation venue.
Meeting summary
- LINK: https://phabricator.wikimedia.org/E259 (Krinkle, 21:00:34)
- Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/ (Krinkle, 21:00:48)
- LINK: https://phabricator.wikimedia.org/E259 (Krinkle, 21:01:06)
- LINK: https://www.mediawiki.org/wiki/Parsing/Notes/A_Spec_For_Wikitext (Krinkle, 21:03:17)
- LINK: https://lists.wikimedia.org/pipermail/wikitech-l/2016-August/086200.html (Krinkle, 21:03:23)
- subbu quotes from essay "whether we want a spec cannot be separated from the question of what the goals are of wanting a spec" (Krinkle, 21:06:33)
- <tgr> would the spec include stuff like "call the Lua compiler version X with parameters Y"? (Krinkle, 21:08:43)
- <gwicke> I think it's pretty clear that a spec that would fully address the archival use case would be extremely expensive & probably harder to read than an actual implementation (Krinkle, 21:11:42)
- <brion> But the data model of a classic <gallery> tag is a distinct domain specific language (Krinkle, 21:12:56)
- <subbu> I propose we first address the question: why a spec, i.e. what are the goals for writing a spec. (Krinkle, 21:15:13)
- <DanielK_WMDE> just an accepting grammar would be useless (Krinkle, 21:20:04)
- <brion> Are we specing Wikitext the character sequence <brion> Or the document model? (Krinkle, 21:20:55)
- <subbu> let me reframe my goal: an executable spec for "old / legacy wikitext" + clean wikitext processing model as a spec for wikitext 2.0. the former lets you deal with old content. (Krinkle, 21:22:41)
- <subbu> the latter lets you clean up wikitext and move forward. (Krinkle, 21:22:48)
- <gwicke> many have made the case that that wikitext should be treated as a textual UI & not as a storage format. <gwicke> I tend to agree, it's not a good storage format (Krinkle, 21:23:43)
- <brion> in general HTML has too much low level detail: images list several distinct URLs, you have lots of presentation markup, etc. (Krinkle, 21:26:38)
- <SMalyshev> I think there are degrees of it. Even right now wikitext on one wiki may be not reproducible on other wiki because of missing modules/templates. (Krinkle, 21:28:14)
- <subbu> * is personally not interested in the grammar as a spec direction (Krinkle, 21:29:50)
- LINK: https://www.mediawiki.org/wiki/Markup_spec/ANTLR/draft (Krinkle, 21:30:21)
- <subbu> syntax spec is not useful for actually understanding wikitext semantics or generaitng html from it. (Krinkle, 21:31:45)
- LINK: https://github.com/wikimedia/parsoid/blob/master/lib/wt2html/pegTokenizer.pegjs.txt (DanielK_WMDE, 21:32:00)
- LINK: https://www.mediawiki.org/wiki/Markup_spec (DanielK_WMDE, 21:32:35)
- <TimStarling> spec for tokenization in the MW preprocessor: https://www.mediawiki.org/wiki/Preprocessor_ABNF (Krinkle, 21:33:10)
- <subbu> so, for example, red links, etc. may not be essential in the executable spec. (Krinkle, 21:41:31)
- <robla> the thing that's nice about a natural language version of a spec (as opposed to an executable one) is that it's possible to have an "incomplete" spec that's still useful (Krinkle, 21:41:57)
- Â <DanielK_WMDE>Â perhaps an improved PEG grammar as proposed by Tim with lots of semi-formal comments would be a decent compromize (robla, 21:47:06)
- <TimStarling> for purposes of archiving, I think HTML+CSS+images, like kiwix, is good enough for most things (Krinkle, 21:48:04)
- <TimStarling>Â although storing wikitext is still essential, to keep a record of user intentions with each edit (robla, 21:48:40)
- <gwicke> we already store Parsoid HTML for each edit <DanielK_WMDE> gwicke: can we publish dumps of that? <gwicke> DanielK_WMDE: yes, subject to some attention from ops (Krinkle, 21:51:11)
- the idea that stroing (and possibly publishing) parsoid HTML for each revision for achieval seems to have some support (DanielK_WMDE, 21:51:36)
- <subbu> now that we covered one part of the picture ... anyone have thoughts on moving to a future wikitext spec with an improved processing model? :) (Krinkle, 21:54:22)
- ACTION: subbu file a Phab task to track https://www.mediawiki.org/wiki/Parsing/Notes/A_Spec_For_Wikitext for possible conversion to RFC (robla, 21:56:44)
- subbu to create RFC (Krinkle, 21:57:58)
- <gwicke> specs are hard (Krinkle, 21:58:24)
- <brion> Specs and models for everyooooooone (Krinkle, 21:58:36)
Meeting ended at 21:58:38 UTC.
People present (lines said)
- Krinkle (61)
- brion (59)
- subbu (56)
- gwicke (56)
- DanielK_WMDE (36)
- robla (35)
- TimStarling (27)
- Platonides (13)
- tgr (10)
- SMalyshev (10)
- James_F (5)
- Scott_WUaS (5)
- wm-labs-meetbot` (3)
- Debra (3)
- legoktm (3)
- arlolra (2)
Full log
1 | 21:00:09 <Krinkle> #startmeeting ArchCom RFC Meeting W32: A spec for Wikitext |
---|---|
2 | 21:00:11 <wm-labs-meetbot`> Meeting started Wed Aug 10 21:00:09 2016 UTC and is due to finish in 60 minutes. The chair is Krinkle. Information about MeetBot at http://wiki.debian.org/MeetBot. |
3 | 21:00:11 <wm-labs-meetbot`> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. |
4 | 21:00:11 <wm-labs-meetbot`> The meeting name has been set to 'archcom_rfc_meeting_w32__a_spec_for_wikitext' |
5 | 21:00:34 <Krinkle> https://phabricator.wikimedia.org/E259 |
6 | 21:00:48 <Krinkle> #topic Please note: Channel is logged and publicly posted (DO NOT REMOVE THIS NOTE) | Logs: http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-office/ |
7 | 21:01:01 <legoktm> o/ |
8 | 21:01:06 <Krinkle> #link https://phabricator.wikimedia.org/E259 |
9 | 21:01:09 <Krinkle> Hey all |
10 | 21:01:24 <robla> o/ |
11 | 21:01:42 <SMalyshev> hey |
12 | 21:01:44 * Krinkle has his first time meetbot experience |
13 | 21:01:46 <robla> thanks for chairing Krinkle! |
14 | 21:02:21 <subbu> o/ |
15 | 21:03:09 <Scott_WUaS> (Yay Krinkle :) |
16 | 21:03:15 <Krinkle> The main topic for today will be about whether and how we'll proceed with the specification of wikitext. This follows after an essay Subbu wrote on mediawiki.org and a wikitech-l thread. |
17 | 21:03:17 <Krinkle> #link https://www.mediawiki.org/wiki/Parsing/Notes/A_Spec_For_Wikitext |
18 | 21:03:23 <Krinkle> #link https://lists.wikimedia.org/pipermail/wikitech-l/2016-August/086200.html |
19 | 21:04:27 <robla> so, should we have a spec? I say "yes" |
20 | 21:05:06 <subbu> as i have argued in that essay, "whether we want a spec cannot be separated from the question of what the goals are of wanting a spec". |
21 | 21:05:14 <subbu> and there are different specs for different needs / audiences. |
22 | 21:05:15 <SMalyshev> surely we should. It'd be quite time-consuming task though |
23 | 21:05:23 <brion> A spec for what, in what context :) |
24 | 21:05:55 <Krinkle> One of the problems people would like to see solved in this area is to be able to confidently interact with older content. The status quo is that things change with time, and that rendering is somewhat unpredictable for older revisions (expansion of templates and external links was mentioned). |
25 | 21:06:25 <gwicke> specifically, older content stored in wikitext |
26 | 21:06:29 <Krinkle> # subbu quotes from essay "whether we want a spec cannot be separated from the question of what the goals are of wanting a spec" |
27 | 21:06:33 <Krinkle> #info subbu quotes from essay "whether we want a spec cannot be separated from the question of what the goals are of wanting a spec" |
28 | 21:06:39 <brion> We could for instance specify "legacy" Wikitext as it exists, but that doesn't make it an easy system. It'll still be complex |
29 | 21:06:49 <robla> a spec is also helpful to make sure that the markup means the same thing to humans and computers |
30 | 21:06:58 <SMalyshev> I'd say at least a narrative spec from which a reasonably knowledgeable programmer would be able to implement a parser which renders at least 90% (arbitrary high percentage here) of wikitext properly |
31 | 21:07:08 <brion> I suspect we need that for the context of interacting with old archival content, even if what we use for future content changes |
32 | 21:07:33 <gwicke> SMalyshev: we pretty much have that already, in the form of a PEG grammar |
33 | 21:07:47 <subbu> gwicke, well ... tokenizer, you mean. |
34 | 21:08:14 <TimStarling> specifying "legacy" wikitext does not necesarily meet subbu's goals, e.g. "Ease implementation of tools and libraries that need to operate with wikitext directly" |
35 | 21:08:15 <tgr> would the spec include stuff like "call the Lua compiler version X with parameters Y"? |
36 | 21:08:19 <James_F> Also the concept of "what is wikitext" differs by installation. Are the core parser functions part of the spec? What about the others? What about <gallery> syntax? Etc. Each change we make is generally breaking for people that use it. |
37 | 21:08:23 <gwicke> it covers the "easy" parts SMalyshev mentioned |
38 | 21:08:24 <SMalyshev> gwicke: PEG grammar alone doesn't seem enough for me... it specifies what can be said, but not what it means? |
39 | 21:08:25 <subbu> gwicke, there is no guarantee those tokens will render as they are tokenized. |
40 | 21:08:26 <robla> a spec can also help us comment those areas where computers will do counterintuitive things with the markup |
41 | 21:08:37 <tgr> if it would, how is that different from "call the MediaWiki parser version X with parameters Y"? |
42 | 21:08:43 <Krinkle> #info <tgr> would the spec include stuff like "call the Lua compiler version X with parameters Y"? |
43 | 21:08:59 <subbu> gwicke, right. it is useful for sure. |
44 | 21:09:11 <Krinkle> This is an interesting end use-case. How to deal with underlying dependencies. Same goes for syntax inside extension tags. |
45 | 21:09:33 <brion> I think for future facing needs its more useful to have a clean document model |
46 | 21:09:35 <SMalyshev> tgr: I think "call the Lua compiler" is not what most people mean by spec... |
47 | 21:09:49 <legoktm> would extensions that add parser functions / tags have their own specs? |
48 | 21:10:01 <brion> For instance if layouts for tables etc were mor structured and separate from the tokens for content |
49 | 21:10:11 <tgr> SMalyshev: we do rely heavily on Lua scripts, how would you specify around that then? |
50 | 21:10:17 <gwicke> I think it's pretty clear that a spec that would fully address the archival use case would be extremely expensive & probably harder to read than an actual implementation |
51 | 21:10:19 <subbu> brion, yup .. to me, as well, the most useful argument for a spec is for future needs, i.e. clean up the processing / document model. |
52 | 21:10:22 <brion> Cleaner interfaces between components such as templates and Lua modules |
53 | 21:10:27 <robla> SMalyshev: it's a reasonable spec-writing crutch to refer to other specs, even if those specs aren't very well specified |
54 | 21:10:34 <DanielK_WMDE> legoktm: ideally, yes. at least, they can't really be covered by the main spec... |
55 | 21:10:53 <SMalyshev> tgr: have docs for lua scripts... but I think we need to be declarative, not procedural, here |
56 | 21:10:57 <Krinkle> We could consider extension behaviour outside the scope of the spec. Extensions would need to either maintain indefinite compatibility, or somehow have required attributes for versioning (e.g. <foo version="">) and then decide to keep older parsers or to transform it somehow internally. |
57 | 21:11:00 <subbu> gwicke, +1 |
58 | 21:11:35 <brion> One thing to consider with extensions is having a clean enough interface that we can actually do that :) |
59 | 21:11:42 <Krinkle> #info <gwicke> I think it's pretty clear that a spec that would fully address the archival use case would be extremely expensive & probably harder to read than an actual implementation |
60 | 21:11:45 <subbu> i think it might be useful to address the first question: why a spec? before going into discussions about what kind of spec maybe. |
61 | 21:11:46 <robla> gwicke: subbu : do you think the implementation should be the spec, then? |
62 | 21:11:50 <James_F> Krinkle: Including inside-core 'extension' tags like <gallery>? |
63 | 21:11:54 <James_F> Yeah. |
64 | 21:11:59 <tgr> SMalyshev: wikinews uses a Lisp interpreter written in Lua for some of its templates |
65 | 21:12:00 <brion> For instance the data model of the text passed into a ref tag should be known |
66 | 21:12:05 <tgr> have fun documenting that |
67 | 21:12:06 <Krinkle> Essentially yes. |
68 | 21:12:14 <tgr> as gwicke said, it's just not a realistic goal |
69 | 21:12:15 <brion> But the data model of a classic gallery tag is a distinct domain specific language |
70 | 21:12:35 <robla> tgr: it's not realistic to invest in if we don't believe the data is very important |
71 | 21:12:38 <Krinkle> But I think it's more important for the spec to detail what the impact is of returned html from <gallery> with the rest of the content, more important than the syntax of the text inside <gallery> |
72 | 21:12:40 <subbu> robla, for legacy wikitext, i think that is the unfortunate reality. |
73 | 21:12:56 <Krinkle> #info <brion> But the data model of a classic <gallery> tag is a distinct domain specific language |
74 | 21:12:57 <gwicke> robla: that would imply that a spec would be the only way to preserve data |
75 | 21:12:57 <brion> That's something our current system doesn't grok, so we lose the ability to refer to the contents of the ref |
76 | 21:13:43 <robla> gwicke: no it doesn't. we would still have our implementation of the spec |
77 | 21:13:52 <gwicke> it's far from clear that a spec is a viable way to do that at all, and even less so that it is the only viable way |
78 | 21:14:05 <legoktm> I don't think focusing on a spec as the way to solve the old wikitext no longer works properly is a good way to frame this discussion - we should focus on the reasons we need a spec and that it could possibly help with that problem |
79 | 21:14:31 <Krinkle> One radical idea would be consider our shortcuts (extensions, templates) merely a way to provide input and create a revision, rather than being the revision itself. Essentially producing something in between that is still minimal and canonical but not end-user/localised/skinned. |
80 | 21:14:48 <subbu> I propose we first address the question: why a spec, i.e. what are the goals for writing a spec. |
81 | 21:14:50 <Krinkle> that would move expansion out of scope somewhat. (e.g. does ~~~~ need a spec?) |
82 | 21:15:13 <Krinkle> #info <subbu> I propose we first address the question: why a spec, i.e. what are the goals for writing a spec. |
83 | 21:15:39 * gwicke is curious |
84 | 21:15:41 <robla> goal for writing a spec: have a human readable description of how wikitext will be interpreted by computers |
85 | 21:15:57 <brion> Goal: to have a consistent way to interpret, edit, and display Wikitext from any era after the spec epoch consistently (? Sample) |
86 | 21:16:17 <DanielK_WMDE> some reasons to have a spec: allows a canonical test suit, allows transformation into other markup formats, enables multiple alternative parsers/processors (and editors) |
87 | 21:16:35 <robla> brion: I suppose the example I keep trotting out is ANSI C. Very complicated, but very worthwhile |
88 | 21:16:35 * tgr finds the MediaWiki parser more human-readable than twenty pages of ABNF |
89 | 21:16:36 <DanielK_WMDE> a spect would need to define the semantics of each syntactical element |
90 | 21:16:37 <gwicke> DanielK_WMDE: we already have a test suite |
91 | 21:16:42 <DanielK_WMDE> just an accepting grammar would be useless |
92 | 21:16:58 <gwicke> brion: does your proposal require a spec, or an implementation? |
93 | 21:17:00 <TimStarling> subbu's idea of an executable (HTML5-like) spec is interesting to me |
94 | 21:17:01 <DanielK_WMDE> gwicke: sure. so? |
95 | 21:17:09 <arlolra> maybe there's a parallel to be drawn between wikitext and js's don't break the web model |
96 | 21:17:27 <brion> gwicke: depends on what were specing |
97 | 21:17:29 <subbu> goal: write a future-looking spec to evolve the wikitext language and processing model .. which doesn't help with old wikitext of course. |
98 | 21:17:31 <DanielK_WMDE> gwicke: is it canonical? in thw way that we say if the software breaks a test, this MUST be because the software is wrong? |
99 | 21:17:43 <brion> Are we specing Wikitext the character sequence |
100 | 21:17:44 <robla> tgr: for C programming, do you think that compilers are easier to read than the ANSI C spec? |
101 | 21:17:48 <brion> Or the document model? |
102 | 21:17:49 <DanielK_WMDE> i often end up fixing parser tests, not code... because the tests make assumptions, or rely on unspecified behavior |
103 | 21:17:52 <gwicke> DanielK_WMDE: pretty much, yes |
104 | 21:17:59 <gwicke> considering the amount of stored, existing content |
105 | 21:18:07 <subbu> TimStarling, yes .. that is one way html authors probably addressed the problem of old html out there and html compatibility. |
106 | 21:18:12 <SMalyshev> brion: both I'd say. Character sequence is probably the easier part :) |
107 | 21:18:15 <brion> A document model spec gives us what we and others need to transform our parsed documents into other formats etc |
108 | 21:18:19 <TimStarling> for some purposes, it would be nice to be reductionist, write grammars etc., but for archiving and ease of reimplementation it makes more sense to be complete |
109 | 21:18:27 <subbu> which is probably not dissimilar to the current problem we have. |
110 | 21:18:43 <TimStarling> and that really means specifying algorithms |
111 | 21:18:43 <brion> Even if we only have one implementation of the tokenizer/parser |
112 | 21:18:43 <DanielK_WMDE> gwicke: the best spec is the stored, existing content, then. if you break it, you are doing it wrong... |
113 | 21:18:47 <Krinkle> TimStarling: subbu: an executable spec, would that mean expansion is part of the model, or left to producers of content (e.g. notion of a "template" could be in the data atrributes, but not required for consumers to understand) |
114 | 21:18:55 <gwicke> that's how tests are set up, and parsoid was tested |
115 | 21:19:08 <robla> another goal: interoperability between multiple implementations |
116 | 21:19:08 <tgr> robla: no clue about that, but there are several magnitudes in difference between the resources behind C and wikitext, and also the potential userbase, so I don't think it's a useful comparison |
117 | 21:19:22 <SMalyshev> DanielK_WMDE: the problem it's not a constructive spec :) It doesn't giv you a way to do it right, only tells you when it's wrong |
118 | 21:19:25 <DanielK_WMDE> robla: hell yea |
119 | 21:19:32 <subbu> so, let me reframe my goal: an executable spec for "old / legacy wikitext" + clean wikitext processing model as a spec for wikitext 2.0 |
120 | 21:19:41 <subbu> the former lets you deal with old content. |
121 | 21:19:43 <brion> :) |
122 | 21:19:50 <subbu> the latter lets you clean up wikitext and move forward. :) |
123 | 21:19:58 <DanielK_WMDE> SMalyshev: many things in life are like that ;) |
124 | 21:19:58 <brion> I like |
125 | 21:20:04 <Krinkle> #info <DanielK_WMDE> just an accepting grammar would be useless |
126 | 21:20:14 <robla> subbu: is there an example of a really good executable spec? |
127 | 21:20:14 <subbu> DanielK_WMDE, i agree reg. an accept grammar. |
128 | 21:20:17 <subbu> html5 |
129 | 21:20:24 <TimStarling> Krinkle: I think expansion would have to be fully part of the executable spec, including what version of Lua to use etc. |
130 | 21:20:37 <brion> Subbu do you imagine a common document model attached to both old and new grammars? |
131 | 21:20:40 <DanielK_WMDE> well, "yes" is an accepting grammar |
132 | 21:20:47 <DanielK_WMDE> since all text is valid wikitext |
133 | 21:20:50 <brion> Heh |
134 | 21:20:55 <Krinkle> #info <brion> Are we specing Wikitext the character sequence <brion> Or the document model? |
135 | 21:21:03 <TimStarling> this is what subbu means by an executable spec: https://www.w3.org/TR/html5/syntax.html |
136 | 21:21:06 * subbu is trying to find a link to the html5 tree building algo. |
137 | 21:21:12 <subbu> oh, there TimStarling posted it |
138 | 21:21:16 <TimStarling> it's a natural language description of an algorithm |
139 | 21:21:25 * robla looks at the link |
140 | 21:21:33 <gwicke> many have made the case that that wikitext should be treated as a textual UI & not as a storage format |
141 | 21:21:52 * robla sees a lot of English in that ;-) |
142 | 21:21:53 <gwicke> I tend to agree, it's not a good storage format |
143 | 21:21:57 <DanielK_WMDE> i thin we need both: characters -> AST -> semantics. We could rely on the HTML spec for some parts of the AST and semantics. |
144 | 21:22:18 <subbu> brion, i didn't understand your qn. reg. common document model |
145 | 21:22:22 <tgr> and then the spec would throw up its hands any time an extension is involved? what is the use case for that? |
146 | 21:22:29 <brion> gwicke: yes, though our main alternative now is HTML which is at the wrong abstraction level :) |
147 | 21:22:41 <Krinkle> #info <subbu> let me reframe my goal: an executable spec for "old / legacy wikitext" + clean wikitext processing model as a spec for wikitext 2.0. the former lets you deal with old content. |
148 | 21:22:41 <gwicke> brion: is it? |
149 | 21:22:46 <tgr> to be able to write an alternative parser that absolutely cannot handle real wikitext as it appears on Wikipedia? |
150 | 21:22:48 <Krinkle> #info <subbu> the latter lets you clean up wikitext and move forward. |
151 | 21:23:02 <gwicke> brion: which issues do you see in the Parsoid DOM spec? |
152 | 21:23:10 <Scott_WUaS> (If there's a choice in spec development between archiving a lot/all and archiving some, I'd be for the former, with some limitations). |
153 | 21:23:13 <brion> subbu: I'm thinking like, would old and new parser rules end up creating compatible in memory object representations that could be transformed to one another |
154 | 21:23:20 <DanielK_WMDE> robla: "Script data double escaped less-than sign state" <-- do we really want to go there? |
155 | 21:23:43 <Krinkle> #info <gwicke> many have made the case that that wikitext should be treated as a textual UI & not as a storage format. <gwicke> I tend to agree, it's not a good storage format |
156 | 21:23:48 <subbu> brion, ah .. i see. well, parsoid is an example implementation that can bridge between the two. |
157 | 21:23:52 <Platonides> I don't think separating old/new wikitext format is workable |
158 | 21:24:07 <subbu> Platonides, content-handler. |
159 | 21:24:07 <brion> gwicke: in general HTML has too much low level detail: images list several distinct URLs, you have lots of presentation markup, etc. nothing wrong with Parsoid but it makes it inefficient and easy to change in ways that will be weird |
160 | 21:24:13 <Platonides> lots of old-page calling new-template |
161 | 21:24:24 <brion> I'd love a model that's slightly higher level than HTML |
162 | 21:24:25 <Platonides> or in other order |
163 | 21:24:28 <SMalyshev> tgr: I think there are degrees of it. Even right now wikitext on one wiki may be not reproducible on other wiki because of missing modules/templates. But we can get at least basic syntax? |
164 | 21:24:40 <Platonides> wiktitext fragments being passed as parameters... |
165 | 21:24:48 <TimStarling> DanielK_WMDE: it's simpler to write that than to try to reduce the existing algorithm to a formal description |
166 | 21:24:54 <gwicke> brion: it's trivial to simplify to XML, but then you lose the rendering |
167 | 21:25:22 <brion> Yes, you also lose the rendering if your details change like URL structure |
168 | 21:25:23 <gwicke> in any case, the purpose of the DOM model is to clearly define what is semantically important, and what is just one way to format it |
169 | 21:25:35 <brion> Yep |
170 | 21:25:39 <brion> Dom is good :) |
171 | 21:25:52 <Scott_WUaS> (If MediaWiki Content Translation became part of this spec writing, or similar, in what ways would it be best to write a spec to facilitate much translation of past resources?) |
172 | 21:26:00 <gwicke> otherwise, Parsoid could just have said "here, it's HTML5" |
173 | 21:26:13 <gwicke> & not bothered with a DOM sepc |
174 | 21:26:17 <Krinkle> brion: We can express a higher model in HTML potentially. Parsoid does that to some extent already when an expansion stage exists (e.g. we could have <mw-image> instead of <img>, the model doens't need to be renderable in browsers as-is per se) |
175 | 21:26:17 <gwicke> *spec |
176 | 21:26:22 <brion> Scott_WUaS: persistent ids for content snippets that don't interfere with human readability |
177 | 21:26:38 <Krinkle> #info <brion> in general HTML has too much low level detail: images list several distinct URLs, you have lots of presentation markup, etc. |
178 | 21:26:42 <Debra> subbu: What were you hoping to accomplish in this session? |
179 | 21:26:46 <Scott_WUaS> brion: thanks |
180 | 21:26:55 <brion> Krinkle: yes, that's about what I'm thinking :) |
181 | 21:26:58 <DanielK_WMDE> hm, has anyone looked at the attempt to build an ANTLR grammar for wikitext? If I remember corectly, the effort came quite far |
182 | 21:26:59 <subbu> brion, but to answer your qn. i think using that is an implementation question ... but, the compatibility between old & new would be bridged via the output spec perhaps .. i am thinking aloud here. |
183 | 21:27:15 <Krinkle> brion: OKay, so you didn't mean that the higher model should be something that isn't HTML syntax |
184 | 21:27:28 <Platonides> DanielK_WMDE: but failed as always |
185 | 21:27:30 <gwicke> DanielK_WMDE: there is not much benefit over PEG |
186 | 21:27:32 * subbu is personally not interested in the grammar as a spec direction |
187 | 21:27:33 <brion> Kringle yeah I'm agnostic to syntax really |
188 | 21:27:34 <Platonides> if I remember right |
189 | 21:27:36 <brion> Heh autocorrect |
190 | 21:27:38 <Krinkle> (it could be browser renderable too, with custom elements nowadays) |
191 | 21:27:45 <Platonides> the first 90% is easy |
192 | 21:27:49 <DanielK_WMDE> Platonides: sure, it failed to cover the critical last 5%, but perhaps it's a good start. or a good lesson. |
193 | 21:27:54 <Platonides> but the last 10% makes you mad |
194 | 21:27:59 <DanielK_WMDE> for reference: https://www.mediawiki.org/wiki/Markup_spec/ANTLR/draft |
195 | 21:28:01 <TimStarling> note that we don't actually have a full PEG spec |
196 | 21:28:14 <Krinkle> #info <SMalyshev> I think there are degrees of it. Even right now wikitext on one wiki may be not reproducible on other wiki because of missing modules/templates. |
197 | 21:28:18 <subbu> Debra, I was responding to robla's call which I felt was a good question to delve into about a spec ... to understand where perspectives are wrt to wikitext, spec, old / new wikitext, evolving wikitext, etc. |
198 | 21:28:25 <gwicke> a full grammar spec is impossible anyway |
199 | 21:28:35 <subbu> Debra, so far, i think it is being met well. as far as I am concerned, not sure what robla thinks. :) |
200 | 21:28:41 <Debra> All right. |
201 | 21:28:44 <gwicke> unless the grammar is fully turing complete |
202 | 21:28:52 <robla> subbu: I think I mostly agree with you (not interested in grammar being the sole focus), but I do think syntax is important |
203 | 21:28:52 <Debra> I think general discussion is fine, but sometimes people want more concrete action items from these meetings. |
204 | 21:29:04 <TimStarling> we have PEG+stops, and I think it would be interesting to replace stops with an extension to the PEG formalism, instead of being implemented in JS |
205 | 21:29:21 <gwicke> stops are just a compression technique |
206 | 21:29:26 <TimStarling> since we are probably forking PEG.js anyway |
207 | 21:29:30 <gwicke> you can unroll that into a larger grammar |
208 | 21:29:47 <TimStarling> yeah, but unrolling defeats the purpose of having the grammar, the purpose is reduction |
209 | 21:29:50 <Krinkle> #info <subbu> * is personally not interested in the grammar as a spec direction |
210 | 21:29:51 <gwicke> very tedious, but possible |
211 | 21:30:21 <Krinkle> #link https://www.mediawiki.org/wiki/Markup_spec/ANTLR/draft |
212 | 21:30:32 <subbu> robla, i think wikitext syntax parsing is a solved problem ... the peg grammer with stops is good enough .. TimStarling also pointed me to an ebnf grammar for the php preprocessor .. i think between the two, we have wikitext syntax tokenization covered. |
213 | 21:30:56 <DanielK_WMDE> do we have a link to the PEG thing? |
214 | 21:30:58 <subbu> but, that syntax spec is not useful for actually understanding wikitext semantics or generaitng html from it. |
215 | 21:31:06 <brion> Full data model is more complex potentially yes! |
216 | 21:31:08 <TimStarling> DanielK_WMDE: you won't like it |
217 | 21:31:11 <subbu> :) |
218 | 21:31:20 <gwicke> DanielK_WMDE: https://github.com/wikimedia/parsoid/blob/master/lib/wt2html/pegTokenizer.pegjs.txt |
219 | 21:31:35 <robla> subbu: yup, the latter is way more interesting (understanding wikitext semantics or generaitng html from it) |
220 | 21:31:45 <Krinkle> #info <subbu> syntax spec is not useful for actually understanding wikitext semantics or generaitng html from it. |
221 | 21:31:49 <DanielK_WMDE> gwicke: thanks |
222 | 21:31:56 <brion> subbu: does the Parsoid Dom model help in terms of defining some elements? |
223 | 21:31:59 <Krinkle> subbu: Exactly, we need to specify not the syntax, but what the tokens mean in relation to each other. |
224 | 21:32:00 <DanielK_WMDE> #link https://github.com/wikimedia/parsoid/blob/master/lib/wt2html/pegTokenizer.pegjs.txt |
225 | 21:32:12 <brion> Or do we need to go farther with semantic info |
226 | 21:32:18 <TimStarling> spec for tokenization in the MW preprocessor: https://www.mediawiki.org/wiki/Preprocessor_ABNF |
227 | 21:32:19 <subbu> brion, https://www.mediawiki.org/wiki/Parsing/Notes/A_Spec_For_Wikitext#What_kind_of_specs_can_we_develop.3F addresses your qn i think |
228 | 21:32:20 <brion> Eg, here's how a section is ordered |
229 | 21:32:35 <DanielK_WMDE> #link https://www.mediawiki.org/wiki/Markup_spec |
230 | 21:32:41 <gwicke> I'm personally most interested in developing the DOM spec further, as well as cleaning up the messy semantics around transclusion & templating |
231 | 21:33:00 <TimStarling> the awkward thing about ABNF is that precedence is unspecified |
232 | 21:33:07 <TimStarling> in PEG, there is a specified precedence |
233 | 21:33:10 <Krinkle> #info <TimStarling> spec for tokenization in the MW preprocessor: https://www.mediawiki.org/wiki/Preprocessor_ABNF |
234 | 21:33:24 <brion> Subbu: Yes good :) |
235 | 21:33:26 <subbu> robla, for semantics ... i think TimStarling is on the right track about an executable spec .. since we have html5 tree builder as a successful model for dealing with legacy formats. |
236 | 21:33:33 <SMalyshev> gwicke: wow that file breaks syntax highlighter :) |
237 | 21:33:42 <subbu> but, i think if we stuck just there, that would be unfortunate. |
238 | 21:33:47 <subbu> if we *were* .. |
239 | 21:33:55 <gwicke> SMalyshev: I think the .txt suffix throws it off |
240 | 21:33:56 <robla> (a side note for us to cover in the last few minutes: should we have a Phab task to track the state of Parsing/Notes/A_Spec_For_Wikitext and turn the latter into an RFC) |
241 | 21:34:00 <brion> The idea that extensions may need some more description is on point I think. |
242 | 21:34:32 <brion> Params for modules/templates/extensions may themselves be Wikitext, or may be just little string tokens |
243 | 21:34:47 <Krinkle> subbu: TimStarling: So the executable spec would dictate that when encountering <mw-template> (or {{template) that it include that target in a certain way? |
244 | 21:34:52 <brion> If you want a library that greps or search replaces in text, that matters to you |
245 | 21:35:25 <subbu> Krinkle, it would define that that token be preprocessed to generate expanded wikitext, for example. |
246 | 21:35:35 <subbu> a reference implementation does not have to be a high-performance implementation. |
247 | 21:35:37 <brion> Similar to needing to know that some HTML tags are self closing or etc |
248 | 21:35:51 <subbu> it can be very slow, but built with the goal of being understandable and easy to grok. |
249 | 21:35:57 <DanielK_WMDE> TimStarling: hm... much of the problematic parts are related to tag extensions, parser functions, and other transclusion mechanisms. it seems to be the preprocessor is a tool to separate these from the "wikitext proper" parts. that would perhaps make it easier to write a spec just for these parts. |
250 | 21:36:20 <Krinkle> subbu: Is the goal for the spec to allow third-parties to do what MediaWiki does now when viewing an old revision? (e.g. current version of templates) - or do we intend to improve that behaviour as part of this? |
251 | 21:36:25 <Platonides> the preprocessor is relatively easy |
252 | 21:36:30 <Platonides> "relatively" |
253 | 21:36:44 <brion> Heh |
254 | 21:36:45 <gwicke> "just do it as this code does" |
255 | 21:36:46 <subbu> Krinkle, that is a reasonable goal, yes. |
256 | 21:36:47 <DanielK_WMDE> Krinkle: i think an executable speci is called a "parser". |
257 | 21:36:55 <Krinkle> (or perhaps even move it out of the spec, by as gwicke mentioned, to consider that only an input method to the model, and never a storage format) |
258 | 21:37:33 <robla> the model would still have a canonical disk representation |
259 | 21:37:49 <subbu> DanielK_WMDE, maybe .. but, I think it would also extract the most important semantics out of the guts of mediawiki. |
260 | 21:38:09 <DanielK_WMDE> a really well written, nicely readable parser could server as a spec. probably a recursive descend parser. |
261 | 21:38:25 <gwicke> DanielK_WMDE: PEG is recursive descent |
262 | 21:38:28 <subbu> i.e. how far can you pull the parser out and see how much spaghetti links back into the mediawiki guts you can cut out without breaking the essential interpretation of content. |
263 | 21:38:44 <subbu> so, for example, red links, etc. may not be essential in the executable spec. |
264 | 21:39:00 <subbu> they are all best viewed as post-parser transformations even for old revisions. |
265 | 21:39:07 <subbu> and need not be part of the spec. |
266 | 21:39:15 <DanielK_WMDE> gwicke: so is the PEG code sufficiently readable that others could reasonably use it to build their own grammaer or parser? |
267 | 21:39:27 <robla> the thing that's nice about a natural language version of a spec (as opposed to an executable one) is that it's possible to have an "incomplete" spec that's still useful |
268 | 21:39:35 <gwicke> DanielK_WMDE: yes, it even compiles to a tokenizer out of the box |
269 | 21:39:55 <gwicke> it's still not easy, but there is only so much complexity you can pretend to not be there |
270 | 21:40:24 <robla> focusing on making the spec executable seems like a "nice to have", not a hard and fast requirement. It seems like overengineering |
271 | 21:40:48 <DanielK_WMDE> gwicke: can we package the complexity into nice bundles, that can be covered or ignored? a modular spec? |
272 | 21:40:57 <Krinkle> In other words, if a third party has an xml dump of all page titles and revisions, can they use this to figure out how to render them? Or do we involve other stuff that wouldn't be in there. References to other pages is doable I guess, but references to other stuff gets more complicated ({{int:}}, {{gender:}}, including special pages, extension tags) |
273 | 21:41:08 <DanielK_WMDE> (to an extent, all specs are modular, since they all build on top of pre-established conventions) |
274 | 21:41:17 <gwicke> like wikitext-without-italic-and-bold? |
275 | 21:41:19 <brion> Mediawiki standard library ;) |
276 | 21:41:22 <TimStarling> the PEG grammar would be more readable if the JS event code was separated from the PEG recognizer |
277 | 21:41:25 <robla> DanielK_WMDE: yes, exactuly (builidng on other specs) |
278 | 21:41:31 <Krinkle> #info <subbu> so, for example, red links, etc. may not be essential in the executable spec. |
279 | 21:41:33 <TimStarling> many PEG libraries actually do that |
280 | 21:41:57 <Krinkle> #info <robla> the thing that's nice about a natural language version of a spec (as opposed to an executable one) is that it's possible to have an "incomplete" spec that's still useful |
281 | 21:42:16 <Krinkle> I think the executable spec would be written in natural language. The HTML5 executable spec is that way. |
282 | 21:42:17 <gwicke> an incomplete spec isn't useful for the archival problem |
283 | 21:42:30 <brion> Krinkle: I like the idea of a layered spec. Things like extensions and parser functions are an additional layer needed for some uses but not all |
284 | 21:42:33 <arlolra> TimStarling: we should do that |
285 | 21:43:02 <James_F> gwicke: Well, a parser/etc. for the log wikitext sub-type (bold/italics/links and nothing else) is probably needed. |
286 | 21:43:17 <robla> gwicke: I think usefulness a matter of degrees, not binary. A really complete spec is most useful, but an incomplete spec can be useful. |
287 | 21:43:19 <gwicke> also, no links / images, I guess |
288 | 21:43:21 <brion> Eg if mediawiki burns in a fire and we have all this data, what do we need to separate out the various levels of data in its place |
289 | 21:43:27 <gwicke> as those depend on the target of the link |
290 | 21:43:39 <brion> And how can we rearrange and refactor internally to represent those layers more maintainable |
291 | 21:43:43 * subbu is trying to imagine software bits burning in a fire ... |
292 | 21:43:46 <brion> Hehe |
293 | 21:43:46 <Krinkle> If I understand correctly, Parsoid currently considers extension tags as instructions to make a request for generated content (aside from the few ones it implements natively). So it would depend on the availability of an HTTP service. |
294 | 21:44:05 <Krinkle> this is extendable, but probably not desireable for the archival use case. |
295 | 21:44:21 <gwicke> hey, you can make each extension its own spec |
296 | 21:44:36 <subbu> Krinkle, wikitext-native extensions need to have a parsoid-native equivalent. |
297 | 21:44:49 <subbu> i mean .. extensions that process wikitext. |
298 | 21:44:57 <subbu> ex: ref, gallery |
299 | 21:45:18 <James_F> And then there's extensions which don't process wikitext but do depend on it, like <timeline> |
300 | 21:45:43 <gwicke> the old cobol folks would have smugly argued that their language was so close to human readable that they wouldn't need to bother writing separate prose |
301 | 21:46:10 <robla> gwicke: :-) |
302 | 21:46:12 <DanielK_WMDE> perhaps an improved PEG grammar as proposed by Tim with lots of semi-formal comments would be a decent compromize |
303 | 21:46:15 <brion> Hehe |
304 | 21:46:24 <DanielK_WMDE> it has the advantage that we already have half of it. |
305 | 21:46:48 <DanielK_WMDE> it would not cover all layers. the missine layers should have well defined interfaces. |
306 | 21:46:51 <subbu> DanielK_WMDE, it only tokenizes right now. |
307 | 21:47:05 <DanielK_WMDE> well, that's at least the first layer |
308 | 21:47:06 <robla> #info <DanielK_WMDE> perhaps an improved PEG grammar as proposed by Tim with lots of semi-formal comments would be a decent compromize |
309 | 21:47:17 <TimStarling> my position on project planning is that a reductionist spec, which does not necessarily precisely reflect legacy wikitext, would be more useful than a complete spec |
310 | 21:47:43 <TimStarling> for purposes of archiving, I think HTML+CSS+images, like kiwix, is good enough for most things |
311 | 21:47:49 <gwicke> it's more likely to actually happen |
312 | 21:47:58 <gwicke> but for archival purposes, it doesn't seem to have much value |
313 | 21:48:04 <Krinkle> #info <TimStarling> for purposes of archiving, I think HTML+CSS+images, like kiwix, is good enough for most things |
314 | 21:48:09 <TimStarling> although storing wikitext is still essential, to keep a record of user intentions with each edit |
315 | 21:48:26 <TimStarling> maybe we should be storing parsoid HTML before and after each VE edit for the same reason |
316 | 21:48:29 <Krinkle> For historical review and annotation/blame |
317 | 21:48:40 <robla> #info <TimStarling> although storing wikitext is still essential, to keep a record of user intentions with each edit |
318 | 21:49:05 <gwicke> we already store Parsoid HTML for each edit |
319 | 21:49:10 <gwicke> but only going forward, not for the past |
320 | 21:49:19 <DanielK_WMDE> gwicke: can we publish dumps of that? |
321 | 21:49:24 <subbu> TimStarling, given that parsoid converts that back to equivalent wikitext, why would you need to store parsoid html before/after each ve edit? |
322 | 21:49:40 <gwicke> DanielK_WMDE: yes, subject to some attention from ops |
323 | 21:49:47 <brion> Well there's the whole template update issue |
324 | 21:49:48 <DanielK_WMDE> subbu: because parsoid changes |
325 | 21:49:52 <brion> That too |
326 | 21:50:05 <DanielK_WMDE> brion: yes, indeed |
327 | 21:50:07 <Krinkle> one consideration with storing expanded canonical form is oversight/deletion. |
328 | 21:50:19 <subbu> brion, but, that is a generic wikitext problem, not is not restricted to ve edits. |
329 | 21:50:25 <TimStarling> yeah, because parsoid changes, but I'm not going to try to sell you this idea right now |
330 | 21:50:26 <brion> Yup |
331 | 21:50:49 <gwicke> any spec needs versioning & format upgrades |
332 | 21:51:09 <robla> so....this seems worthy of being an RFC going forward, no? |
333 | 21:51:11 <Krinkle> #info <gwicke> we already store Parsoid HTML for each edit <DanielK_WMDE> gwicke: can we publish dumps of that? <gwicke> DanielK_WMDE: yes, subject to some attention from ops |
334 | 21:51:22 <TimStarling> maybe it should be noted what an abysmally bad job we are doing with historical preservation of edits, for reasons unrelated to a spec |
335 | 21:51:36 <DanielK_WMDE> #info the idea that stroing (and possibly publishing) parsoid HTML for each revision for achieval seems to have some support |
336 | 21:51:41 <TimStarling> for example, most of the first 12 months of the history of the project are still missing |
337 | 21:51:46 <gwicke> DanielK_WMDE: https://phabricator.wikimedia.org/T133547 |
338 | 21:52:01 <gwicke> an old dump at https://dumps.wikimedia.org/htmldumps/dumps/ |
339 | 21:52:37 <robla> TimStarling: I wouldn't call that abysmally bad, but I agree it'd be really fantastic to make the first 12 months more easily accessible |
340 | 21:52:41 <TimStarling> 4 of those 12 months only exist in a landfill probably somewhere in Orange County |
341 | 21:53:19 <gwicke> faithfully rendering early history would require changes similar to what the Memento project did |
342 | 21:53:27 <TimStarling> the other 8 just nobody could be bothered importing |
343 | 21:53:28 <subbu> now that we covered one part of the picture ... anyone have thoughts on moving to a future wikitext spec with an improved processing model? :) |
344 | 21:53:31 * robla is now depressed after Tim's landfill remark :-( (because he imagines Tim's not wrong) |
345 | 21:53:41 <gwicke> and even that would only go half way at best |
346 | 21:53:53 <brion> subbu: :) |
347 | 21:54:02 <gwicke> but, fortunately early content is also a lot simpler, so in practice not much actual content should be lost |
348 | 21:54:18 <brion> I love the idea of specing things like extension input models better |
349 | 21:54:22 <Krinkle> #info <subbu> now that we covered one part of the picture ... anyone have thoughts on moving to a future wikitext spec with an improved processing model? :) |
350 | 21:54:26 <Platonides> maybe we could start importing that early data |
351 | 21:54:46 <Platonides> then we could more optimistacally go ahead with the future wikitext |
352 | 21:54:47 <gwicke> subbu: I wouldn't call it wikitext spec |
353 | 21:54:51 <robla> subbu, it'd be great for you to file a stub Phab task for us to track state of the wiki page, could you do that? |
354 | 21:55:08 <gwicke> "wiki content processing spec" or "wiki content model spec"? |
355 | 21:55:16 <subbu> gwicke, sure .. <insert-name-here> spec |
356 | 21:55:26 * DanielK_WMDE whispers "hooks" @brion |
357 | 21:55:30 <brion> subbu: yeah, content model or document model perhaps is the angle |
358 | 21:55:41 <gwicke> the page component / composition stuff is aiming in that direction as well |
359 | 21:55:56 <subbu> robla, i didn't follow reg. "track state of the wiki page" part .. can you say more? |
360 | 21:55:56 <Krinkle> brion: If we go with the route of promoting (expanded, but annotated) html as archival format, that would remove dependencies like extensions. We'd store wikitext for review only (as direct input, no intent to re-parse). |
361 | 21:55:56 <robla> we can morph https://www.mediawiki.org/wiki/Parsing/Notes/A_Spec_For_Wikitext into an RFC |
362 | 21:55:56 * brion is a pirate and has hooks for hands. Arrrrr! |
363 | 21:55:56 <subbu> ah, morphing that into a rfc .. task for that? |
364 | 21:56:16 <subbu> if yes, sure. i can. |
365 | 21:56:18 <brion> Krinkle: mmmmm, depends on the extension. If it needs js, your life is harder |
366 | 21:56:22 <robla> subbu, yup |
367 | 21:56:44 <robla> #action subbu file a Phab task to track https://www.mediawiki.org/wiki/Parsing/Notes/A_Spec_For_Wikitext for possible conversion to RFC |
368 | 21:56:58 <gwicke> the fun part is that anything new will have to consider existing content |
369 | 21:57:01 <Krinkle> brion: enhancement (same for CSS). Presumably the core content wouldn't depend on JS. Interaction and styling are up to the consumer to decide on. |
370 | 21:57:11 <Scott_WUaS> (Cheers:) |
371 | 21:57:12 <brion> *nod* |
372 | 21:57:15 <Krinkle> If not, that's a bug in the extension :) |
373 | 21:57:17 <Platonides> "fun" |
374 | 21:57:18 <brion> :) |
375 | 21:57:20 <Krinkle> (and we have some) |
376 | 21:57:33 <Krinkle> Any last thoughts? |
377 | 21:57:56 <gwicke> specs are hard |
378 | 21:57:58 <Krinkle> #info subbu to create RFC |
379 | 21:58:05 <brion> Specs and models for everyooooooone |
380 | 21:58:24 <Krinkle> #info <gwicke> specs are hard |
381 | 21:58:29 <robla> :-) |
382 | 21:58:30 <brion> True :) |
383 | 21:58:36 <Krinkle> #info <brion> Specs and models for everyooooooone |
384 | 21:58:38 <Krinkle> #endmeeting |
Other meetings
Architecture meetings | ||
---|---|---|
13:00 PT ArchCom Planning Meetings | upcoming | all since 2016-03-30 |
14:00 PT ArchCom-RFC Meetings | upcoming | all since 2015-09-09 |