The topic / partition -> node mapping problem should actually be fairly easy to address by using the Kafka HighlevelConsumer. This consumer interface takes care of balancing topic / partition assignment across members of a consumer group.

Edit: Removed the 'cons' entries in the task description to reflect this.

Less simple to add dependency storage later.

I have looked a bit at hooking up restbase-mod-table-* modules directly into a separate service, and there are no show-stoppers for that: we can load the module just like we do in RESTBase and then use the exposed operations directly. So I'd say adding storage later is as complicated as adding it for the RESTBase-as-a-framework approach.

Declarative way to add subscriptions directly in RESTBase entry points.

If there are two RB instances (the other one being the change-propagation system), then how is this relevant given that subscriptions would need to be declared in the change-prop config, not the REST API one ?

Option of deploying as separate services (change propagation vs. REST API), or as a single service, depending on config.

This is a pro wrt. the current RESTBase deployment, but there is no difference between this and having a separate service for change propagation as both do exactly that - have two service instances for two different things.

@mobrovac, I added easy & quick setup to either option, as we have puppet roles for both.

In T117933#1799186, @GWicke wrote:

@mobrovac, I added easy & quick setup to either option, as we have puppet roles for both.

Correct, but I am not convinced that deploying a second RB instance on sc[ab] would be as easy as deploying a separate service based on the service template.

We discussed the deployment question on IRC. service::node should handle any service-runner based service, including services based on RESTBase.

T118401 tracks using service::node for the REST API service (commonly called 'RESTBase').

We discussed this further this morning. Summary:

• We are leaning towards a consumer group per subscription model (see T118429 for background).
• The first version should support a static config similar to @mobrovac's proposal, subscribe to Kafka & emit templated events. The implementation can use RESTBase's existing request templating infrastructure.
• We mostly agreed on splitting out the framework portion of restbase to a separate package (T118404), and likely use it for future REST API needs in the change propagation service. The name of the new package is to be discussed / decided.

Action items:

• Marko to prototype a fairly stand-alone node module similar to 'first version' above.
• Group to decide on a name for the framework package (T118404), and Gabriel to implement.

A first-version prototype is available at https://github.com/d00rman/restbase-mod-queue-kafka . As discussed, this version creates a new consumer group per topic subscription. I've tested it and, as stated in the the Kafka docs, only one consumer process gets the message. Unfortunately, the first subscribed client gets all of the messages for the consumer group. Moreover, that holds true for all of the consumer groups, resulting in one process getting all of the messages, while all of the other processes idling.

We will have to resort to a more complex round-robin message-assignment algorithm. An idea could be to have a dispatcher process per topic, which would subscribe to the topic and then dispatch it to worker processes. This would require, however, a coordination mechanism amongst all service processes involved in the process, which isn't trivial.

For phase one, we might even go with a simple one process per topic approach. Having this behaviour replicated over multiple nodes coupled with Kafka's rebalancing would ensure protection against failures. However, it is clear that this approach is not scalable.