Page MenuHomePhabricator

Bulk elasticsearch update daemon in an omega/psi world
Closed, ResolvedPublic

Description

this daemon was built with multi-cluster in mind, but the choices we've made don't quite line up with the implementation.

Problem: We can't point daemon at search.svc.eqiad.wmnet:9200,search.svc.eqiad.wmnet:9400,search.svc.eqiad.wmnet:9600, because every elastic server "owns" that DNS, and only two of the three ports will be open on a given server.

Solution: Mjolnir needs to be provided multiple real host/port for each cluster. On loading it should de-duplicate them and assert it has the expected number of clusters. Some puppet magic will be needed to get the list of servers into the mjolnir systemd script. The daemon does not need to talk cross-datacenter, there is a kafka consumer group of daemons per cluster.

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 21 2018, 4:26 PM
TJones closed this task as Resolved.Jan 29 2019, 6:31 PM