We are getting serious about implementing the first phase of the change propagation service, as generally discussed in T102476. In this first phase, the change propagation service should take care of tasks like re-rendering and purging content in response to Event-Platform events. In this first iteration, we focusing on use cases currently covered by the RESTBaseUpdate extension. Specifically, we are leaving elaborate dependency tracking to a later phase.
Requirements
- Manages subscriptions, with several end points possibly listening for the same kinds of events.
- Listens to Kafka events in all topics with subscriptions.
- For each matching event and handler, constructs and dispatches requests (typically HTTP). With services like RESTBase, Cache-Control headers are typically used to trigger a re-render.
- Possibly, emit events back to the Event-Platform to
- signal a resource having changed (possibly triggering other updates)
- continue processing for large jobs (ex: invalidating millions of pages after a template edit)
- retry a failed request a limited number of times
- finally, keep permanently failed requests in a dead-letter queue for later inspection
Implementation
Phase 0: MVP
- /sys/queue module
Phase 1: Simple local route subscriptions
- POST /sys/queue/subscribe registration end point
- Dynamic registration via x-setup-handler POST on startup
- Event description (topic / other criteria)
- Optionally, a request template. Default: Send full event to calling route.
Phase 2: Remote route subscriptions
- Expose a public subscription API mapping to /sys/queue/subscribe.
- Share these dynamic subscriptions with other change propagation nodes, for example by updating table storage table with TTL retention policy & periodic reloading.
- Consider basing this API on https://en.wikipedia.org/wiki/PubSubHubbub (spec).
- In the API RESTBase cluster, set up /sys/queue/subscribe to register subscriptions with the remote change propagation service using a lease system (ex: renew once per hour).
Phase 3: Dependency storage, streaming support
- Add dependency graph storage & a public API for dependency updates.
- Possibly, add streaming subscription support.