Page MenuHomePhabricator

Datacenter aware configs for EventGate topic prefixes
Closed, ResolvedPublic0 Estimated Story Points

Description

EventGate will replace EventBus. EventBus does handles DC failovers by prefixing topics in its datacenter with the datacenter name. E.g. in eqiad, mediawiki.revision-create becomes eqiad.mediawiki.revision-create. Topics are replicated between main Kafka clusters, eqiad -> codfw, and codfw -> eqiad.

Since EventBus is managed by puppet, we can easily configure it there with the datacenter name as the topic prefix. EventGate will be deployed via Helm & Kubernetes. We need a way to configure EventGate with current datacenter.

@fselles suggested labels and the k8s DownwardAPI, which would work if we can render the service-runner config.yaml template with values provided by it.

Event Timeline

Ottomata triaged this task as Medium priority.Jan 11 2019, 4:52 PM
Ottomata created this task.

can render the service-runner config.yaml template with values provided by it.

We can also include it as an env variable into the config.yaml if it's easier, there's support for that.

I think we will just have different values.yaml files in prod that specify --set topic_prefix=XXXX appropriately.

We have datacenter / k8s cluster specific values file overrides now, which accomplishes the goal here.