We are currently running a Zookeeper cluster (a mini zoo) to elect the leader of the Kafka brokers but also Hive use Zookeeper to put a lock data in Hadoop I am sure in the future, we will have other distributed systems that will use Zookeeper as well. Having it's log files available for quick inspection and debugging is useful.
Version: wmf-deployment
Severity: enhancement
| Subject | Author | Repo | Branch | Lines +/- | |
|---|---|---|---|---|---|
| logstash: ship zookeeper logs to ELK | Herron | operations/puppet | production | +32 -19 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | fgiunchedi | T205849 Begin the implementation of Q1's Logging Infrastructure design (2018-19 Q2 Goal) | |||
| Resolved | fgiunchedi | T205852 Onboard at least 10 new non-sensitive log producers to the logging pipeline | |||
| Open | Feature | None | T63779 Add system logs to logstash (tracking) | ||
| Open | Feature | None | T63789 Add Zookeeper log files to logstash |
Change 476977 had a related patch set uploaded (by Herron; owner: Herron):
[operations/puppet@production] logstash: ship zookeeper logs to ELK
Change #476977 abandoned by Herron:
[operations/puppet@production] logstash: ship zookeeper logs to ELK
Reason:
spring cleaning -- stale patch
The reason we are interested in this is that we are working with releng on a new zuul system (https://phabricator.wikimedia.org/project/view/7592/).
The new zuul system requires zookeeper.
We are using the existing zookeeper puppet module.
But existing zookeeper has not used TLS before while zookeeper and zuul expect a mutual (m)TLS setup where client and server auth to each other.
For this we have already added initial TLS support to zookeeper (as a feature flag to avoid touching anything existing run by DPE).
But the absence of logging makes this near impossible to debug and blocks us.