Page MenuHomePhabricator

Support red links updates in change-propagation
Closed, ResolvedPublic

Description

As a part of T39902 we need to support rerendering pages which link to the new page after a new page is created. I've been battling with supporting this in change-propagation for some time, and here's a plan I've came up with:

  1. There will be a rule subscribed to revision_create topic, with a test for rev_parent_id being null - this means new created pages.
  2. The executor for this rule will post a request to MW API getting all the backlinks, the executor should be a full-blown request handler, not just a template.
    • NEW FEATURE: Export handlerTemplate from HyperSwitch
    • NEW FEATURE: Add support for handlerTemplate in change propagation.
  3. Upon receiving a list of items, we need to do a foreach over items and post a resource_change event for each item.
    • NEW FEATURE: Add support of a foreach keyword in handler template. Syntactically it could look like [1]
    • NEW FEATURE: Add support for posting events within change propagation from a rule exec template. It's questionable whether we want to post an event directly or if we better go through the eventlogging-service. Pros of the former is that it's faster and creates way less load on the eventlogging-service (and the load here may be quite high as the same mechanism will likely be applied for template updates). Also with direct producing we can post events to change-prop internal topics like retry queues etc. Pros for the latter is that we get event validation.
  4. Each resource_change is processed normally by a rule that would be set up for rerenders on MW purge and null_edit
  5. Last required piece is to support continue. Several options are possible:
    1. Set up an endpoint that supports continuation within a kafka module. Then a final step of the handler template would issue a request to the continuation service with details of the message and continue property received from MW API. The module would emit an event to a special internal continuation topics (it could be combined with a retry topic, just a message would have a different envelope).
    2. Add general support for 100 HTTP status response. If the handler returned 100 with some body identifying the continue - post a message to the continuation topic.
    3. NEW FEATURE: Add support for continuation.

[1] Example syntax for foreach in handler templates:

- process_all_items:
    foreach: '{{result.body.items}}'
    request:
      uri: '/{{item.name}}'

@GWicke @mobrovac @Eevans What do you think about this approach?

Details

Related Gerrit Patches:
mediawiki/services/change-propagation/deploy : masterConfig: Support red links in ChangeProp
operations/puppet : productionSet up redlinks processing in change propagation
mediawiki/event-schemas : masterSet up schema for a continuation topic

Related Objects

StatusAssignedTask
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
DeclinedNone
ResolvedCatrope
ResolvedSbailey
DeclinedNone
OpenNone
Resolved GWicke
ResolvedNone
Resolvedssastry
OpenNone
ResolvedDbrant
ResolvedbearND
ResolvedMholloway
ResolvedNone
DuplicateNone
ResolvedJdforrester-WMF
ResolvedbearND
OpenNone
OpenNone
ResolvedArlolra
ResolvedPchelolo

Event Timeline

Pchelolo created this task.Apr 20 2016, 9:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 20 2016, 9:24 PM
GWicke added a comment.EditedApr 26 2016, 12:02 AM

We discussed this a bit in the office. Here is a sketch of a possible solution based on that discussion:

  • Set up a 'backlink' module that, provided a title & a destination topic (resource_change most likely), iteratively produces individual change events. In addition to the URL, these events can provide some hints on the reason of the change (ex: page creation for title X, template edit for template Y). (See T114413#2240381 for an updates property strawman for this.)
    • The iterative expansion process will need a separate internal topic to keep track of state, plus its own retry topic. These topics are managed by the module.
  • Process individual resource change events as usual, using the regular per-event retry logic.
  • Defer dealing with templated event production until it's really needed, and just assemble specific resource change events in the respective modules for now.

Change 285558 had a related patch set uploaded (by Ppchelko):
Set up schema for a continuation topic

https://gerrit.wikimedia.org/r/285558

Change 285558 merged by jenkins-bot:
Set up schema for a continuation topic

https://gerrit.wikimedia.org/r/285558

Change 285678 had a related patch set uploaded (by Ppchelko):
Set up redlinks processing in change propagation

https://gerrit.wikimedia.org/r/285678

Change 285678 abandoned by Ppchelko:
Set up redlinks processing in change propagation

https://gerrit.wikimedia.org/r/285678

Pchelolo moved this task from Backlog to blocked on the Services board.Oct 12 2016, 5:28 PM
Pchelolo edited projects, added Services (blocked); removed Services.
GWicke triaged this task as Medium priority.Oct 12 2016, 5:58 PM
Pchelolo moved this task from blocked to doing on the Services board.Jun 15 2017, 12:13 AM
Pchelolo edited projects, added Services (doing); removed Services (blocked).

The task descriptionis brutally outdated, we've implemented a very different approach to dependent pages updates. However, this still needs to be supported an PR for it is awaiting mediawiki train deployment: https://github.com/wikimedia/change-propagation/pull/191

Change 362278 had a related patch set uploaded (by Ppchelko; owner: Ppchelko):
[mediawiki/services/change-propagation/deploy@master] Config: Support red links in ChangeProp

https://gerrit.wikimedia.org/r/362278

Change 362278 merged by Mobrovac:
[mediawiki/services/change-propagation/deploy@master] Config: Support red links in ChangeProp

https://gerrit.wikimedia.org/r/362278

Mentioned in SAL (#wikimedia-operations) [2017-06-29T20:38:17Z] <ppchelko@tin> Started deploy [changeprop/deploy@350076c]: Config: Enable red links processing. T133221

Mentioned in SAL (#wikimedia-operations) [2017-06-29T20:39:19Z] <ppchelko@tin> Finished deploy [changeprop/deploy@350076c]: Config: Enable red links processing. T133221 (duration: 01m 01s)

Pchelolo closed this task as Resolved.Jun 29 2017, 8:48 PM
Pchelolo edited projects, added Services (done); removed Patch-For-Review.

Deployed and confirmed to work. Resolving the ticket.