Page MenuHomePhabricator

Support red links updates in change-propagation
Closed, ResolvedPublic

Description

As a part of T39902 we need to support rerendering pages which link to the new page after a new page is created. I've been battling with supporting this in change-propagation for some time, and here's a plan I've came up with:

  1. There will be a rule subscribed to revision_create topic, with a test for rev_parent_id being null - this means new created pages.
  2. The executor for this rule will post a request to MW API getting all the backlinks, the executor should be a full-blown request handler, not just a template.
    • NEW FEATURE: Export handlerTemplate from HyperSwitch
    • NEW FEATURE: Add support for handlerTemplate in change propagation.
  3. Upon receiving a list of items, we need to do a foreach over items and post a resource_change event for each item.
    • NEW FEATURE: Add support of a foreach keyword in handler template. Syntactically it could look like [1]
    • NEW FEATURE: Add support for posting events within change propagation from a rule exec template. It's questionable whether we want to post an event directly or if we better go through the eventlogging-service. Pros of the former is that it's faster and creates way less load on the eventlogging-service (and the load here may be quite high as the same mechanism will likely be applied for template updates). Also with direct producing we can post events to change-prop internal topics like retry queues etc. Pros for the latter is that we get event validation.
  4. Each resource_change is processed normally by a rule that would be set up for rerenders on MW purge and null_edit
  5. Last required piece is to support continue. Several options are possible:
    1. Set up an endpoint that supports continuation within a kafka module. Then a final step of the handler template would issue a request to the continuation service with details of the message and continue property received from MW API. The module would emit an event to a special internal continuation topics (it could be combined with a retry topic, just a message would have a different envelope).
    2. Add general support for 100 HTTP status response. If the handler returned 100 with some body identifying the continue - post a message to the continuation topic.
    3. NEW FEATURE: Add support for continuation.

[1] Example syntax for foreach in handler templates:

- process_all_items:
    foreach: '{{result.body.items}}'
    request:
      uri: '/{{item.name}}'

@GWicke @mobrovac @Eevans What do you think about this approach?

Related Objects

StatusAssignedTask
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
DeclinedNone
ResolvedCatrope
ResolvedSbailey
DeclinedNone
OpenNone
Resolved GWicke
ResolvedNone
Resolvedssastry
OpenNone
ResolvedDbrant
ResolvedbearND
ResolvedMholloway
ResolvedNone
DuplicateNone
ResolvedJdforrester-WMF
ResolvedbearND
OpenNone
OpenNone
ResolvedArlolra
ResolvedPchelolo

Event Timeline

Pchelolo created this task.Apr 20 2016, 9:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 20 2016, 9:24 PM
GWicke added a comment.EditedApr 26 2016, 12:02 AM

We discussed this a bit in the office. Here is a sketch of a possible solution based on that discussion:

  • Set up a 'backlink' module that, provided a title & a destination topic (resource_change most likely), iteratively produces individual change events. In addition to the URL, these events can provide some hints on the reason of the change (ex: page creation for title X, template edit for template Y). (See T114413#2240381 for an updates property strawman for this.)
    • The iterative expansion process will need a separate internal topic to keep track of state, plus its own retry topic. These topics are managed by the module.
  • Process individual resource change events as usual, using the regular per-event retry logic.
  • Defer dealing with templated event production until it's really needed, and just assemble specific resource change events in the respective modules for now.

Change 285558 had a related patch set uploaded (by Ppchelko):
Set up schema for a continuation topic

https://gerrit.wikimedia.org/r/285558

Change 285558 merged by jenkins-bot:
Set up schema for a continuation topic

https://gerrit.wikimedia.org/r/285558

Change 285678 had a related patch set uploaded (by Ppchelko):
Set up redlinks processing in change propagation

https://gerrit.wikimedia.org/r/285678

Change 285678 abandoned by Ppchelko:
Set up redlinks processing in change propagation

https://gerrit.wikimedia.org/r/285678

Pchelolo moved this task from Backlog to blocked on the Services board.Oct 12 2016, 5:28 PM
Pchelolo edited projects, added Services (blocked); removed Services.
GWicke triaged this task as Normal priority.Oct 12 2016, 5:58 PM
Pchelolo moved this task from blocked to doing on the Services board.Jun 15 2017, 12:13 AM
Pchelolo edited projects, added Services (doing); removed Services (blocked).

The task descriptionis brutally outdated, we've implemented a very different approach to dependent pages updates. However, this still needs to be supported an PR for it is awaiting mediawiki train deployment: https://github.com/wikimedia/change-propagation/pull/191

Change 362278 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/change-propagation/deploy@master] Config: Support red links in ChangeProp

https://gerrit.wikimedia.org/r/362278

Change 362278 merged by Mobrovac:
[mediawiki/services/change-propagation/deploy@master] Config: Support red links in ChangeProp

https://gerrit.wikimedia.org/r/362278

Mentioned in SAL (#wikimedia-operations) [2017-06-29T20:38:17Z] <ppchelko@tin> Started deploy [changeprop/deploy@350076c]: Config: Enable red links processing. T133221

Mentioned in SAL (#wikimedia-operations) [2017-06-29T20:39:19Z] <ppchelko@tin> Finished deploy [changeprop/deploy@350076c]: Config: Enable red links processing. T133221 (duration: 01m 01s)

Pchelolo closed this task as Resolved.Jun 29 2017, 8:48 PM
Pchelolo edited projects, added Services (done); removed Patch-For-Review.

Deployed and confirmed to work. Resolving the ticket.