Page MenuHomePhabricator

Support regexes in topic names in ChangeProp
Closed, ResolvedPublic

Description

In JobQueue we have one queue per job type, so we need a separate Kafka topic per job type. For better load distribution we not only should be able to connect a worker to all topics that match some regex (like job_queue.*), but also create a separate RuleExecutor for each of these topics so that the work would be distributed between workers.

Likely there will be a lot of rules in that scenario, so we’d hit the UV_THREADPOOL_SIZE=128 limit, so we’d need to finally wrap up the long-awaited changes to the node kafka driver to avoid that.

Event Timeline

Change 345411 had a related patch set uploaded (by Ppchelko):
[mediawiki/services/change-propagation/deploy@master] Config: Enable subscription to regex topics

https://gerrit.wikimedia.org/r/345411

Change 345411 merged by Mobrovac:
[mediawiki/services/change-propagation/deploy@master] Config: Enable subscription to regex topics

https://gerrit.wikimedia.org/r/345411

Pchelolo edited projects, added Services (done); removed Patch-For-Review, Services (later).

Depoyed, enabled, verified to be working.