Change Details

We are getting serious about implementing the first phase of the change propagation service, as generally discussed in T102476. In this first phase, the change propagation service should take care of tasks like re-rendering and purging content in response to #eventbus events. In this first iteration, we focusing on use cases currently covered by the RESTBaseUpdate extension. Specifically, we are leaving elaborate dependency tracking to a later phase. ## Requirements - Manages subscriptions, with several end points possibly listening for the same kinds of events. - Listens to Kafka events in all topics with subscriptions. - For each matching event and handler, constructs and dispatches requests (typically HTTP). With services like RESTBase, `Cache-Control` headers are typically used to trigger a re-render. - Possibly, emit events back to the #eventbus to - signal a resource having changed (possibly triggering other updates) - continue processing for large jobs (ex: invalidating millions of pages after a template edit) - retry a failed request a limited number of times - finally, keep permanently failed requests in a dead-letter queue for later inspection ## Implementation options @mobrovac has created a [config strawman](https://gist.github.com/d00rman/217560aa9c73551db216), which (not entirely accidentally) looks very similar to RESTBase event handlers. In a first discussion today we felt that this kind of declarative handler is a good starting point. From there, we explored two main ideas: ### Separate service Build a new service backing a config like this, reusing some of the request templating code from RESTBase, but not using any of RESTBase's routing logic. Pros: - Minimal implementation, only pulls in what's strictly needed for the MVP. - Easy to set up a single instance MVP. Cons: - Less simple to add dependency storage later. - For HA / load balancing / multiple workers, need to figure out which instance processes which topics (same as for RB-based service). ### Event subscription support for RESTBase entry points Leverage RESTBase-the-framework. Add a way to subscribe RESTBase entry points to events, and add a Kafka listener to RESTBase taking those subscriptions from the spec, subscribing to Kafka & then dispatching pseudo requests to each matching handler. Pros: - Declarative way to add subscriptions directly in RESTBase entry points. - Option of deploying as separate services (change propagation vs. REST API), or as a single service, depending on config. - Can leverage RESTBase modules / functionality (table storage, API wrapper) as needed. Table storage might come in handy for dependency storage. - Queue abtraction in RB might be useful more generally. Cons: - For HA / multi-worker setups, need to figure out which instance processes which topics / events. We want to process each event only once. - Potential for name confusion between RESTBase-the-API vs. RESTBase-the-framework. While change propagation can be deployed as a separate service, people might perceive it as a monolith. Could we address this with different names for the framework vs. the REST API service? - Pulls in some functionality that's not strictly needed for basic change propagation.

We are getting serious about implementing the first phase of the change propagation service, as generally discussed in T102476. In this first phase, the change propagation service should take care of tasks like re-rendering and purging content in response to #eventbus events. In this first iteration, we focusing on use cases currently covered by the RESTBaseUpdate extension. Specifically, we are leaving elaborate dependency tracking to a later phase. ## Requirements - Manages subscriptions, with several end points possibly listening for the same kinds of events. - Listens to Kafka events in all topics with subscriptions. - For each matching event and handler, constructs and dispatches requests (typically HTTP). With services like RESTBase, `Cache-Control` headers are typically used to trigger a re-render. - Possibly, emit events back to the #eventbus to - signal a resource having changed (possibly triggering other updates) - continue processing for large jobs (ex: invalidating millions of pages after a template edit) - retry a failed request a limited number of times - finally, keep permanently failed requests in a dead-letter queue for later inspection ## Implementation options @mobrovac has created a [config strawman](https://gist.github.com/d00rman/217560aa9c73551db216), which (not entirely accidentally) looks very similar to RESTBase event handlers. In a first discussion today we felt that this kind of declarative handler is a good starting point. From there, we explored two main ideas: ### Separate service Build a new service backing a config like this, reusing some of the request templating code from RESTBase, but not using any of RESTBase's routing logic. Pros: - Minimal implementation, only pulls in what's strictly needed for the MVP. Cons: - Less simple to add dependency storage later. ### Event subscription support for RESTBase entry points Leverage RESTBase-the-framework. Add a way to subscribe RESTBase entry points to events, and add a Kafka listener to RESTBase taking those subscriptions from the spec, subscribing to Kafka & then dispatching pseudo requests to each matching handler. Pros: - Declarative way to add subscriptions directly in RESTBase entry points. - Option of deploying as separate services (change propagation vs. REST API), or as a single service, depending on config. - Can leverage RESTBase modules / functionality (table storage, API wrapper) as needed. Table storage might come in handy for dependency storage. - Queue abtraction in RB might be useful more generally. Cons: - Potential for name confusion between RESTBase-the-API vs. RESTBase-the-framework. While change propagation can be deployed as a separate service, people might perceive it as a monolith. Could we address this with different names for the framework vs. the REST API service? - Pulls in some functionality that's not strictly needed for basic change propagation.