Page MenuHomePhabricator

[RFC] Visual Templates: Authoring templates with Visual Editor
Open, Stalled, LowPublic

Description

Type of activity: Pre-scheduled session.
Main topic: Handling wiki content beyond plaintext

The problem

VisualEditor is a friendly means to edit *content*, but it presently provides no means to author *templates*, which are an important component of our projects.

We can use Visual Editor to edit templates, and in the process better separate code, content, and presentation. I'm going to call these "Visual Templates".

Here are the key ideas:

  1. A minimal "logic-less" template language, like the Spacebars or htmlbars variants of Handlebars/mustache templates. There are only three basic template constructs (variable interpolation, block tags, and partials), which can all be visualized in the editor. There are no hidden directives.
  2. All logic is written in a real programming language, via Scribunto (or something very much like it). In the examples below I'll be using a version of Scribunto that supports ECMAScript 2015 as well as Lua; forgive my eccentricities.
  3. All data is escaped and hygienic by default. String data is automatically escaped. The Scribunto code manipulates and returns DocumentFragments, not wikitext strings.
  4. Similarly, arguments are provided as JSON types (strings/numerical data), DocumentFragments, or wikidata queries, not wikitext.
  5. Composition of content is structural, not string concatenation. You can't "accidentally" create a list item if your template result happens to start with a : or *.

So: every Visual Template is composed of two parts: a Scribunto module (edited with a code editor), and a "layout" (edited with Visual Editor).

Here's what the layout of a particular template might look like:


Forgive the terrible yellow; a designer could make this look much nicer. Note that there are two "block helper" invocations, each and caption. The each helper is a built-in, and it is given row from the template data context object. The caption helper is defined by the Scribunto module. Each row of the table gets "variable" invocations to fill in the table cells.

There are only three new functions added to the VE toolbar in template mode, for inserting "variable", "block helper", and "partial" elements. Context-sensitive help will be given to choose from available variable, block helper, and partial names.

The invocation of this template would, for this example, take structured data of the following form, perhaps from a wikidata query:

eventgoldsilverbronze
ShotputUSAGERMEX
DiscusMEXUSAGER
JavelinGERMEXUSA

The Scribunto module transforms this raw data by computing medal counts for each country. Here's what it might look like, using JavaScript 2015 (ES6) module syntax,

// The main module entry point is a transformation of a "data context".
// In this example the data context comes from a wikidata query.
export default function(events) {
  // `events` is an array: each item has `event`, `gold`, `silver`, and `bronze` props
  // (the latter three holding country names)
  let totals = new Map(), countries = new Set(), allColors = ['gold','silver','bronze'];
  function mget(m, key, defaultval) {
    if (!m.has(key)) { m.set(key, defaultval); }
    return m.get(key);
  }
  function inc(country, color) {
    let old = mget(mget(totals, country, new Map()), color, 0);
    totals.get(country).set(color, old + 1);
    countries.add(country);
  }
  // count up how many medals of each color a given country has.
  events.forEach( (e) => {
    for (let color of allColors) { inc(e[color], color); }
  }):
  // Now sort the countries by medal count
  let rows = Array.from(countries).sort( (a,b) => {
    let aa = totals.get(a), bb = totals.get(b);
    for (let color of allColors) {
      let c = aa[color] - bb[color];
      if (c!==0) { return c; }
    }
    return c;
  }).map( (c) => totals.get(c) );
  // our resulting data context object has just one
  // property, named 'rows', which is an array.
  return { row: rows };
}
// Additional block helper.  Helpers must return a `DocumentFragment`,
// though for brevity we allow `Node` or `Array<Node>` as well.
// In fact, if you returned a string we'd make a text node for you,
// but that would make this example even more trivial.
export function caption() {
  return document.createTextNode('caption');
}

The default entry point transforms the template arguments to set up the template data context. It does no text manipulation. There is a second caption entry point, which defines the caption block helper. The exact API used in block helpers definitions might be tweaked: one obvious improvement would be to allow jquery methods for DOM construction, instead of using the DOM API directly. The key point is that the result is conceptually a chunk of structural DOM, not a wikitext string.

Visual Template invocation in Visual Editor
The primary authoring mechanism for Visual Templates is expected to be Visual Editor. The UX is expected to be roughly the same as exists now for templates: a mechanism like TemplateData exposes the expected types of arguments and descriptive text, so that Visual Editor can expose appropriate editing widgets. Initially the expected types are: "markup" (edited with VE recursively), JSON (edited with a JSON editor, as with TemplateData), and "wikidata" (a SPARQL wikidata query).

Visual Template Invocation in wikitext

This is a strawman for the wikitext serialization of a template invocation:

{{#visual
   template_name
   param1=<markup>'''bold'''</markup>
   param2=<json>{"foo":"bar"}</json>
   param3=<wikidata>
SELECT ?spaceProbeLabel ?date ?picture
 WHERE {
   ?spaceProbe wdt:P31 wd:Q26529 ;
                      wdt:P18 ?picture ;
                      wdt:P619 ?date .
  SERVICE wikibase:label {
    bd:serviceParam wikibase:language "fr,en" .
   }
 }
ORDER BY ?date
</wikidata>
}}

The parameter name entity is reserved; the data context object for the template always includes the wikidata entity object (equivalent to mw.wikibase.getEntityObject()) as an implicit parameter named entity. This makes it easy to generate infoboxes (for example), since you can just insert {{#formatproperty entity.P17}} in your template (equivalent to entity.formatPropertyValues( 'P17' ).value). Hopefully there will be a friendly shortcut for entity.<something>, which will include mapping wikidata identifier <something> to a label in your language.

Since arguments are delimited by html-style tags, the usual &amp; and &lt; escapes are all that are required to include arbitrary data inside a parameter, and whitespace can be used to separate arguments. The parameter names can be omitted for positional arguments.

Perhaps we can allow for an abbreivated format for certain arguments: "..." -> literal string, \d+ -> number, {...} -> JSON value. Then unit conversion (for example) can still be simply {{#visual convertToMiles 5 km}}

*Syntax details can be tweaked: your input wanted!*

Expected outcome

  • The primary goal for the summit would be to stimulate discussion of "visual template editing" in general, and agree on a strategy. Is template authorship always going to be a function of wikitext? Or do we want to build a specialized IDE with little resemblance to VE? What's the 10-year view for templates?
  • From this discussion, hopefully a subset of this proposal could be identified which can achieve consensus as a reasonable "first step" for experimental implementation.
  • A secondary goal would be to stimulate interest in/discussion of/development of related technologies, for example alternate templating engines or Scribunto/Js.

Current status of the discussion

Mailing list discussion: https://lists.wikimedia.org/pipermail/wikitech-l/2015-October/083467.html
Discussion below as well.

Links

*See also:* T114251: [RFC] Magic Infobox implementation

Related Objects

Event Timeline

cscott created this task.Oct 1 2015, 11:52 PM
cscott raised the priority of this task from to Needs Triage.
cscott updated the task description. (Show Details)
cscott added a subscriber: cscott.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 1 2015, 11:52 PM
cscott updated the task description. (Show Details)Oct 2 2015, 10:04 PM
cscott updated the task description. (Show Details)Oct 2 2015, 11:23 PM
cscott updated the task description. (Show Details)
cscott updated the task description. (Show Details)Oct 2 2015, 11:25 PM
cscott updated the task description. (Show Details)Oct 2 2015, 11:39 PM
cscott updated the task description. (Show Details)Oct 2 2015, 11:56 PM
cscott updated the task description. (Show Details)Oct 3 2015, 3:06 AM
cscott updated the task description. (Show Details)Oct 3 2015, 3:21 AM
cscott updated the task description. (Show Details)
Anomie added a subscriber: Anomie.Oct 3 2015, 1:44 PM

In the examples below I'll be using a version of Scribunto that supports JavaScript 2015 as well as Lua; forgive my eccentricities.

As long as you realize that's not terribly likely to happen in real Scribunto; various JavaScript engines were considered but had some major shortcomings.

This is a strawman for the wikitext serialization of a template invocation:

That's pretty awful, especially since the syntax looks almost like normal wikitext syntax but has subtle differences that are likely to confuse anyone using it directly.

cscott added a comment.EditedOct 3 2015, 2:08 PM

@Anomie the problems with v8 have since been resolved; you might notice that I'm one of the maintainers of v8js.

Re the syntax: yes, I agree it's not beautiful: improvements welcome! I added the strawman just to clearly illustrate the main feature, which was that the arguments weren't raw strings but rather typed data. I also wanted to avoid some of the argument-escape pitfalls, and I deliberately made it different from existing syntax to clearly distinguish it.

But those issues could be addressed in different ways: using template data to move the data types away from the call site (at the risk of obscuring errors); and perhaps using my hygenic arguments proposal to deal with the escaping issues, rather than inventing new syntax. Ideas welcome!

Qgil added a subscriber: Qgil.Oct 3 2015, 8:56 PM

Congratulations! This is one of the 52 proposals that made it through the first deadline of the Wikimedia-Developer-Summit-2016 selection process. Please pay attention to the next one: > By 6 Nov 2015, all Summit proposals must have active discussions and a Summit plan documented in the description. Proposals not reaching this critical mass can continue at their own path out of the Summit.

tstarling moved this task from Inbox to Under discussion on the TechCom-RFC board.Oct 14 2015, 8:45 PM
daniel added a subscriber: daniel.Oct 14 2015, 8:47 PM

This should be discussed together with T114251: [RFC] Magic Infobox implementation

Qgil added a comment.Oct 28 2015, 10:44 AM

This should be discussed together with T114251: [RFC] Magic Infobox implementation

Does this mean that we should remove either this task or the T114251 from the list of Summit proposals, in order to focus the online and Summit discussion in one task?

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptNov 6 2015, 10:54 AM

November 6, and this proposal doesn't seem to have much traction, it is not on track. Unless there is a sudden change, I will leave the ultimate decision of pre-scheduling it for the Wikimedia-Developer-Summit-2016 to @RobLa-WMF and the Architecture Committee.

cscott updated the task description. (Show Details)Nov 6 2015, 6:45 PM
Krinkle updated the task description. (Show Details)Dec 8 2015, 12:22 AM

Wikimedia Developer Summit 2016 ended two weeks ago. This task is still open. If the session in this task took place, please make sure 1) that the session Etherpad notes are linked from this task, 2) that followup tasks for any actions identified have been created and linked from this task, 3) to change the status of this task to "resolved". If this session did not take place, change the task status to "declined". If this task itself has become a well-defined action which is not finished yet, drag and drop this task into the "Work continues after Summit" column on the project workboard. Thank you for your help!

DStrine moved this task from (unused) to Backlog on the TechCom-RFC board.Mar 2 2016, 10:48 PM
RobLa-WMF changed the task status from Open to Stalled.Mar 2 2016, 10:49 PM

Per E146

Qgil removed a subscriber: Qgil.Mar 3 2016, 9:58 AM
RobLa-WMF mentioned this in Unknown Object (Event).May 4 2016, 7:33 PM
RobLa-WMF triaged this task as Low priority.Jun 8 2016, 7:07 PM

Belated priority update discussed in E187: RFC Meeting: triage meeting (2016-05-25, #wikimedia-office) (see log at P3179). In particular:

21:34:29 <robla> priority of T114454 , "low", plus a comment "please file a bigger picture RFC so that we can block this RFC with that RFC?

This proposal didn't actually get a session at the 2016 dev summit. It was elected as a topic for the all hands unconference, but when everyone showed up it turned out that all of the attendees (other than me) thought this was going to be a tutorial on authoring templates, not some crazy idea about how you *would* write them in the future. So we redirected the unconference session.

At any rate, re-nominating for the 2017 dev summit, in the "Handling wiki content beyond plaintext" track.

Can you apply the template for Wikimedia-Developer-Summit (2017) proposals to the description, please?

Qgil added a comment.Nov 10 2016, 11:35 AM

Please remember the next deadline in the selection process:

2016-11-14: Deadline for defining a good problem statement, expectations, and links to relevant resources.

That is next Monday.

Qgil added a comment.Nov 11 2016, 12:10 PM

Who would be the person facilitating this session? Please assign this task to that person if you are aiming to have this session pre-scheduled. Thank you!

cscott claimed this task.Nov 11 2016, 10:12 PM
cscott added a subscriber: Qgil.Nov 11 2016, 10:16 PM

Who would be the person facilitating this session? Please assign this task to that person if you are aiming to have this session pre-scheduled. Thank you!

That would be me. Assigned. A reminder that TPG needs to help out in the in-person facilitation of the "beyond wikitext" topic sessions, since I can't really facilitate and present at once.

(I'm also relying on the token-voting scheme for proposals, or something similar, to provide a bit of objective prioritizations between proposals, so i'm not tempted to schedule the "beyond wikitext" entirely with my mediawiki wishlist.)

This proposal hasn't registered much interest or renewed discussion in this current Summit round. Applying the same criteria that we are applying to all the proposals, this fits well with the Unconference.

@cscott Hey! As developer summit is less than four weeks from now, we are working on a plan to incorporate the ‘unconference sessions’ that have been proposed so far and would be generated on the spot. Thus, could you confirm if you plan to facilitate this session at the summit? Also, if your answer is 'YES,' I would like to encourage you to update/ arrange the task description fields to appear in the following format:

Session title
Main topic
Type of activity
Description Move ‘The Problem,' ‘Expected Outcome,' ‘Current status of the discussion’ and ‘Links’ to this section
Proposed by Your name linked to your MediaWiki URL, or profile elsewhere on the internet
Preferred group size
Any supplies that you would need to run the session e.g. post-its
Interested attendees (sign up below)

  1. Add your name here

We will be reaching out to the summit participants next week asking them to express their interest in unconference sessions by signing up.

To maintain the consistency, please consider referring to the template of the following task description: https://phabricator.wikimedia.org/T149564.

There has been no activity on this RfC since E146 . @cscott are you still interesting in pursuing this idea? Are the relevant actors aware of this proposal?

kchapman added a subscriber: kchapman.

product strategy decision not a RFC at this time.

Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptJul 22 2019, 5:14 PM
Qgil removed a subscriber: Qgil.Jul 22 2019, 9:14 PM