Page MenuHomePhabricator

Emit change events from RESTBase
Closed, ResolvedPublic

Description

As a general change propagation mechanism, we plan to emit general resource change events from RESTBase. These will contain a URL of the changed resource, and possibly a flag about the nature of the change (creation, deletion, modification).

These events are then used to propagate changes:

  • Purging of caches (Varnish).
  • Updates of dependent resources.

In the longer term, this is intended to be aggregated in a Kafka topic. Until that & the associated processing pipeline is available, it might be useful to process those events locally instead. This will let us define and exercise internal interface for such change events without depending on a full change propagation pipeline to be available right away. A simple implementation would emit purge events for all 'modified' or 'deleted' urls. This is over UDP, so won't hold up general request processing.

See also:

Event Timeline

GWicke raised the priority of this task from to Needs Triage.
GWicke updated the task description. (Show Details)
GWicke subscribed.

Defining a JS interface for this looks rather straightforward, but we'll also need to consider how to support event emission from spec-only entry points. Something declarative would be nice, but means that we'd have to add this to hyperswitch.

GWicke set Security to None.
GWicke updated the task description. (Show Details)

We don't necessarily need new JS interface for that or any changes in HyperSwitch As emitting the event is just an http call to the Event-Platform service, this could be achieved by adding one more sys module to RESTBase and calling it when necessary. However, ideally we want to have support for the following features:

  • We should be able to make requests to event emitting service asynchronous, without waiting for a result and without failing the original request if event emitting failed. In JS modules it's straightforward, however in spec modules the syntax wouldn't be very neat, so we could consider support for async steps in handler template.
  • Ideally, we should be able to check if the content actually changed before emitting a change event. Again, in JS it's obvious, but in spec modules it's not currently possible. Options for spec modules are adding new syntax, or passing old and new content to the change event emitting module, and letting it compare the content.
  • For even more accurate change detection we might want to apply some transformations to the content, like we do for Parsoid html to cleanup non-significant changes. Again, for JS modules it's not a problem, for spec-based modules it could be achieved by adding a spall JS addition to the module and exporting a cleanup function to module's globals.

So, I propose to add a new module /sys/event.js with the following interface:

POST /change
body:
 uri: [Array of URIs]
 old_content: optional old content
 new_content: optional new content. In case any of new/old content are omitted, the event is emitted unconditionally.

I like @Pchelolo's idea.

Here is a slightly different interface variant:

uri: /{domain}/sys/events
method: post
body:
  - uri: //{domain}/api/rest_v1/page/mobile-sections/{title}
  - uri: //{domain}/api/rest_v1/page/mobile-sections-lead/{title}
  - uri: //{domain}/api/rest_v1/page/mobile-sections-remaining/{title}

For this to work with proper escaping, we'll need to stop treating {foo} the same way as {{foo}}, which I think is a good clean-up in any case, and should at this point not affect much code any more.

As a follow-up, @mobrovac pointed out that we should wrap the uri into a meta block for consistency with the event schemas.

Change 281740 had a related patch set uploaded (by Ppchelko):
Emit resource_change events from RESTBase.

https://gerrit.wikimedia.org/r/281740

Change 281740 merged by Filippo Giunchedi:
Emit resource_change events from RESTBase.

https://gerrit.wikimedia.org/r/281740

Pchelolo claimed this task.

Enabled in production, resolving.