November 6, and this proposal doesn't seem to have much traction, it is not on track. Unless there is a sudden change, I will leave the ultimate decision of pre-scheduling it for the Wikimedia-Developer-Summit-2016 to @RobLa-WMF and the Architecture Committee.

Currently it is written as if the individual steps will only happen one at a time. Should something be added to indicate that for performance every step of it needs to support batching?

@JanZerebecki: The current intention is to keep change propagation relatively simple and efficient. Most services can be implemented with very small relative per-request overheads, and services with high per-request overheads can consider applying opportunistic batching transparently to all requests, for example using a batching proxy.

Wikimedia Developer Summit 2016 ended two weeks ago. This task is still open. If the session in this task took place, please make sure 1) that the session Etherpad notes are linked from this task, 2) that followup tasks for any actions identified have been created and linked from this task, 3) to change the status of this task to "resolved". If this session did not take place, change the task status to "declined". If this task itself has become a well-defined action which is not finished yet, drag and drop this task into the "Work continues after Summit" column on the project workboard. Thank you for your help!

TechCom plans to discuss this next week in E184: RFC Meeting: RFC: Requirements for change propagation (2016-05-18, #wikimedia-office)

In order to have a productive conversation, we need to figure out which things are done and decided, and which items need broader consensus. Which questions should we focus on next week? Which questions need to be answered before this is approved/or rejected? What does "approved" mean in this context?

@RobLa-WMF @daniel, I updated the task summary with a status summary & a sketch of next steps & open questions. Please have a look & tweak as necessary.

@GWicke - I made a conversion of the description to wikitext at mw:User:RobLa-WMF/T102476. The next step will be to convert it into mw.org RFC format. RFCs with long prose seem better to have over on mw.org, with a short description + link here on Phab; MediaWiki's editing and diffing tools are better (yay you folks!)

On E184, @GWicke said "I'm not opposed to moving the general portion especially to mw.org, lets just make sure we don't end up with multiple copies" My edits to the description reflect my understanding of that.

I moved the current status / next steps section back here, so that only the general background section is now on MediaWiki.org. Status & next steps is more volatile & links directly to ongoing work, so benefits from being in this task.

This was discussed on #wikimedia-office on 2016-05-18.
Minutes: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-05-18-21.00.html
Full log: https://tools.wmflabs.org/meetbot/wikimedia-office/2016/wikimedia-office.2016-05-18-21.00.log.html

Summary:

• ''LINK:'' https://phabricator.wikimedia.org/T102476 (gwicke, 21:01:03)
• robla asks for response on T102476#2296335. gwicke answers "I'm not opposed to moving the general portion especially to mw.org, lets just make sure we don't end up with multiple copies" (robla, 21:12:37)
• ''LINK:'' https://commons.wikimedia.org/wiki/Template:LangSwitch (matt_flaschen, 21:24:59)
• significant use: track dependencies when rendering pages with {{int}} and <translate> so that they can be purged when conditional dependencies change (TimStarling, 21:33:47)
• what needs purging is *gernerated* content, so we need to track what it depends on (DanielK_WMDE, 21:37:47)
• ''LINK:'' https://phabricator.wikimedia.org/T130528 (DanielK_WMDE, 21:50:39)
• ''ACTION:'' update the RFC to clarify the anticipated concrete dependency relations (TimStarling, 22:01:38)

During yesterday's meeting, it became apparent that it would be useful for me to summarize the requirements that Wikibase has for dependency tracking. I will try to do that below. The current system that Wikibase uses for this purpose is documented in docs/usagetracking.wiki.

Our assumptions, questions, and requirements:

• in order to propagate changes, we need to track which resources or other artifacts a generated artifact depends on.
• The tracking is effectively a DAG, with artifact URIs as the nodes. We may want to associated additional information though, like the tough date.
• generated artifacts may depend on other generated artifacts, or on human edited resources.
• a user edited resource, such as wikitext on a wiki page, never depends on anything (though it may reference other things).
  • MediaWiki uses link tables to track what wikitext references what; currently, we also use this information to infer when to purge the parser cache of which page. This system fails if different renderings have different dependencies.
• a rendering of a resource depends at least on that resource, and possibly on other resources (typically the ones referenced in the resource) or other artifacts.
• artifacts may be created on the fly during a GET request; their dependency then needs to be stored somewhere/somehow, causing the need to write during a GET.
  • example: a multilingual page (e.g. an image description page on commons) is requested in a language for which there exists no rendering yet. Technically, this is due to things like {{langswitch}} and <translate>.
  • another thing the rendering could vary on is desktop vs. mobile.
  • ...not to speak of the skin, date format, stub threshold....
  • note: it is not computationally feasible to create all possible renderings in advance, to avoid generating them on the fly.
• When purging caches (e.g. CDN cache or ParserCache), we need artifacts to be associated with multiple buckets. E.g. we need to be able to purge all renderings of X, or all renderings that depends on Y.
• Artifacts may expire and get purged. Any tracked dependencies are then redundant, and could be pruned (garbage collection).
• Conversely, artifacts that are no longer needed by any other may be pruned / garbage collected
• How reliable do we need the dependency tracking to be? What happens if we lose this info?
• How do we scale this beyond the wikimedia cluster?
  • use case: InstantCommons and, in the future, InstantWikidata
  • this essentially needs a PubSub mechanism, with fairly high granularity. How to make it scale for a large subscriber base, and high update frequency?
• Rough sketch of a dependency storage interface: put(x,y), drop(x,y), get(x), rget(y), purge(x), rpurge(y); batch operations: replace(x,y1,y2,y3,....), add(x,y1,y2,y3,...), remove(x,y1,y2,y3,...) plus possible the inverse

I'm going to make a note here mainly for myself as shepherd. In E202, I noted I was going to take this off of our weekly action items. There's probably some work that needs to happen to incorporate Daniel's thinking expressed in T102476#2309963 into the document at https://www.mediawiki.org/wiki/Requests_for_comment/Requirements_for_change_propagation . @daniel and @GWicke seemed to believe that this RfC is on track, and would be a good example of an RFC where we might need the "on track" state we discussed (see T137860)

This seems obsolete. Is there any interest in keeping this open and continue the RFC process here?

Change propagation is an important topic, but it seems we need a fresh start.

In other languages? Wikimedia's Director Katherine Maher in Wikimania 2017 mentioned potentially 7k languages in Wikipedia by 2030