Page MenuHomePhabricator

Support red links updates in change-propagation
Closed, ResolvedPublic


As a part of T39902 we need to support rerendering pages which link to the new page after a new page is created. I've been battling with supporting this in change-propagation for some time, and here's a plan I've came up with:

  1. There will be a rule subscribed to revision_create topic, with a test for rev_parent_id being null - this means new created pages.
  2. The executor for this rule will post a request to MW API getting all the backlinks, the executor should be a full-blown request handler, not just a template.
    • NEW FEATURE: Export handlerTemplate from HyperSwitch
    • NEW FEATURE: Add support for handlerTemplate in change propagation.
  3. Upon receiving a list of items, we need to do a foreach over items and post a resource_change event for each item.
    • NEW FEATURE: Add support of a foreach keyword in handler template. Syntactically it could look like [1]
    • NEW FEATURE: Add support for posting events within change propagation from a rule exec template. It's questionable whether we want to post an event directly or if we better go through the eventlogging-service. Pros of the former is that it's faster and creates way less load on the eventlogging-service (and the load here may be quite high as the same mechanism will likely be applied for template updates). Also with direct producing we can post events to change-prop internal topics like retry queues etc. Pros for the latter is that we get event validation.
  4. Each resource_change is processed normally by a rule that would be set up for rerenders on MW purge and null_edit
  5. Last required piece is to support continue. Several options are possible:
    1. Set up an endpoint that supports continuation within a kafka module. Then a final step of the handler template would issue a request to the continuation service with details of the message and continue property received from MW API. The module would emit an event to a special internal continuation topics (it could be combined with a retry topic, just a message would have a different envelope).
    2. Add general support for 100 HTTP status response. If the handler returned 100 with some body identifying the continue - post a message to the continuation topic.
    3. NEW FEATURE: Add support for continuation.

[1] Example syntax for foreach in handler templates:

- process_all_items:
    foreach: '{{result.body.items}}'
      uri: '/{{}}'

@GWicke @mobrovac @Eevans What do you think about this approach?

Related Objects

Resolved GWicke
Resolved bearND
Resolved Mholloway
Resolved bearND
Resolved Pchelolo

Event Timeline

We discussed this a bit in the office. Here is a sketch of a possible solution based on that discussion:

  • Set up a 'backlink' module that, provided a title & a destination topic (resource_change most likely), iteratively produces individual change events. In addition to the URL, these events can provide some hints on the reason of the change (ex: page creation for title X, template edit for template Y). (See T114413#2240381 for an updates property strawman for this.)
    • The iterative expansion process will need a separate internal topic to keep track of state, plus its own retry topic. These topics are managed by the module.
  • Process individual resource change events as usual, using the regular per-event retry logic.
  • Defer dealing with templated event production until it's really needed, and just assemble specific resource change events in the respective modules for now.

Change 285558 had a related patch set uploaded (by Ppchelko):
Set up schema for a continuation topic

Change 285558 merged by jenkins-bot:
Set up schema for a continuation topic

Change 285678 had a related patch set uploaded (by Ppchelko):
Set up redlinks processing in change propagation

Change 285678 abandoned by Ppchelko:
Set up redlinks processing in change propagation

GWicke triaged this task as Medium priority.Oct 12 2016, 5:58 PM

The task descriptionis brutally outdated, we've implemented a very different approach to dependent pages updates. However, this still needs to be supported an PR for it is awaiting mediawiki train deployment:

Change 362278 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/change-propagation/deploy@master] Config: Support red links in ChangeProp

Change 362278 merged by Mobrovac:
[mediawiki/services/change-propagation/deploy@master] Config: Support red links in ChangeProp

Mentioned in SAL (#wikimedia-operations) [2017-06-29T20:38:17Z] <ppchelko@tin> Started deploy [changeprop/deploy@350076c]: Config: Enable red links processing. T133221

Mentioned in SAL (#wikimedia-operations) [2017-06-29T20:39:19Z] <ppchelko@tin> Finished deploy [changeprop/deploy@350076c]: Config: Enable red links processing. T133221 (duration: 01m 01s)

Pchelolo edited projects, added Services (done); removed Patch-For-Review.

Deployed and confirmed to work. Resolving the ticket.