We need to add concurrency capabilities to the general/subscriber package that will allow subscriber to use multiple go routines for subscription, this will allow us to use compute resources more efficiently and decline number of instances.
Acceptance criteria
Ability to provide number of workers to subscriber package.
ToDo
- update the subscriber package
- update structured-data services to use new version of the package
- update on-demand service to use the new version of the package
- deploy to dev and test that it's working properly
Notes
Need to make sure that messages with the same key endup being processed by the same worker, this requires internal partitioning, we have that code in commit history.