Page MenuHomePhabricator

Support regexes in topic names in ChangeProp
Closed, ResolvedPublic

Description

In JobQueue we have one queue per job type, so we need a separate Kafka topic per job type. For better load distribution we not only should be able to connect a worker to all topics that match some regex (like job_queue.*), but also create a separate RuleExecutor for each of these topics so that the work would be distributed between workers.

Likely there will be a lot of rules in that scenario, so we’d hit the UV_THREADPOOL_SIZE=128 limit, so we’d need to finally wrap up the long-awaited changes to the node kafka driver to avoid that.

Event Timeline

Pchelolo created this task.Feb 3 2017, 12:27 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 3 2017, 12:27 AM

Change 345411 had a related patch set uploaded (by Ppchelko):
[mediawiki/services/change-propagation/deploy@master] Config: Enable subscription to regex topics

https://gerrit.wikimedia.org/r/345411

Change 345411 merged by Mobrovac:
[mediawiki/services/change-propagation/deploy@master] Config: Enable subscription to regex topics

https://gerrit.wikimedia.org/r/345411

Pchelolo closed this task as Resolved.Mar 29 2017, 8:25 PM
Pchelolo edited projects, added Services (done); removed Patch-For-Review, Services (later).

Depoyed, enabled, verified to be working.