Page MenuHomePhabricator

Datacenter aware configs for EventGate topic prefixes
Open, NormalPublic0 Story Points

Description

EventGate will replace EventBus. EventBus does handles DC failovers by prefixing topics in its datacenter with the datacenter name. E.g. in eqiad, mediawiki.revision-create becomes eqiad.mediawiki.revision-create. Topics are replicated between main Kafka clusters, eqiad -> codfw, and codfw -> eqiad.

Since EventBus is managed by puppet, we can easily configure it there with the datacenter name as the topic prefix. EventGate will be deployed via Helm & Kubernetes. We need a way to configure EventGate with current datacenter.

@fselles suggested labels and the k8s DownwardAPI, which would work if we can render the service-runner config.yaml template with values provided by it.

Event Timeline

Ottomata triaged this task as Normal priority.Jan 11 2019, 4:52 PM
Ottomata created this task.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 11 2019, 4:52 PM

can render the service-runner config.yaml template with values provided by it.

We can also include it as an env variable into the config.yaml if it's easier, there's support for that.

I think we will just have different values.yaml files in prod that specify --set topic_prefix=XXXX appropriately.