Review znodes on Zookeeper cluster to possibly remove not-used data
Closed, ResolvedPublic3 Estimated Story Points
Actions

Assigned To

Authored By

	elukey
	Feb 25 2019, 7:15 AM

Description

The two Zookeeper clusters, main-eqiad (conf100[4-6]) and main-codfw (conf200[1-3]) are holding data for the following clusters:

Kafka main eqiad/codfw
Kafka Analytics eqiad
Kafka Jumbo eqiad
Kafka Logging
Kafka Burrow codfw/eqiad
Hadoop Test Analytics
Hadoop Analytics

The current znode allocation is the following:

main-eqiad -> ~50k
main-codfw -> ~3.7k

In order to preserve mental sanity when dealing with maintenance or (hopefully few) critical events, I think that it would be great to:

check what data is surely garbage that can be trashed (reducing the amount of znodes)
check if zookeeper is misused somehow, like an application storing data rather than a state. A huge amount of znodes under the same parent is, for example, fine but I am wondering if we could hit some limits sooner or later (or maybe subtle failures).

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		elukey	T216979 Review znodes on Zookeeper cluster to possibly remove not-used data
		Resolved		elukey	T216952 Hadoop Yarn stores a ton of znodes related to running/old applications

Event Timeline

elukey triaged this task as Medium priority.Feb 25 2019, 7:15 AM

elukey created this task.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptFeb 25 2019, 7:15 AM

elukey added a subtask: T216952: Hadoop Yarn stores a ton of znodes related to running/old applications.Feb 25 2019, 7:15 AM

elukey added a project: User-Elukey.

List of parent znodes in main-eqiad:

[zk: localhost:2181(CONNECTED) 0] ls /
[registry, brokers, zookeeper, yarn-leader-election, hadoop-ha, rmstore-analytics-test-hadoop, services, druid, etc, hive_zookeeper_namespace, kafka, rmstore, burrow, consumers]

registry (Testing znodes for Apache Slider, can be trashed)

[zk: localhost:2181(CONNECTED) 4] ls /registry/users/joal/services
[org-apache-slider]

brokers (probably old Kafka Analytics stuff)

[zk: localhost:2181(CONNECTED) 9] ls /brokers/topics
[]

zookeeper (seems internal usage for quotas)

[zk: localhost:2181(CONNECTED) 11] ls /zookeeper/quota
[]

yarn-leader-election (HA data for Hadoop Yarn)

[zk: localhost:2181(CONNECTED) 12] ls /yarn-leader-election
[analytics-hadoop, analytics-test-hadoop]

hadoop-ha (HA znodes for Hadoop HDFS)

[zk: localhost:2181(CONNECTED) 13] ls /hadoop-ha
[analytics-hadoop, analytics-test-hadoop]

services (Seems again Apache slider testing data)

[zk: localhost:2181(CONNECTED) 18] ls /services/slider/users/joal
[]

druid (old Druid zookeeper data, we currently have separate clusters for druid, can surely be cleaned up)

etc - This is old Burrow data, probably my mistake when I have set it up (see the burrow znode later on)

[zk: localhost:2181(CONNECTED) 19] ls /etc/burrow
[notifier, notifier-eqiad, notifier-main-eqiad, notifier-analytics, notifier-jumbo-eqiad]

rmstore-analytics-test-hadoop and rmstore, see T216952

hive_zookeeper_namespace - stil not sure what this is, seems old data though from a first look.

[zk: localhost:2181(CONNECTED) 20] ls /hive_zookeeper_namespace
[wmf, qchris, ironholds, Wmf, otto, wmf_Raw, yurik, ellery, wmf_raw]

kafka - Parent znode for all the Kafka clusters. It contains a lot of znodes but it seems all legit data

[zk: localhost:2181(CONNECTED) 38] ls /kafka
[main-codfw, jumbo-eqiad, logging-eqiad, eqiad, main-eqiad]

burrow - Parent znode for all the Burrow data.

[zk: localhost:2181(CONNECTED) 47] ls /burrow/notifier
[analytics, jumbo-eqiad, logging-eqiad, main-eqiad]

consumers - probably old Kafka znodes when consumers where handled via Zookeeper

[zk: localhost:2181(CONNECTED) 48] ls /consumers
[ebernhardson_test1, otto0, otto1, test_joal_flink, KafkaWordCount-otto-0, eventlogging-8c5a95e0-a8ef-11e5-b1da-782bcb0a0efc, eventlogging-group]

Main codfw is less crowded and probably doesn't need a clean up:

[zk: localhost:2181(CONNECTED) 0] ls /
[burrow, kafka, zookeeper]

[zk: localhost:2181(CONNECTED) 1] ls /burrow
[notifier-main-codfw, notifier]

[zk: localhost:2181(CONNECTED) 2] ls /kafka
[logging-codfw, main-codfw]

elukey added subscribers: Ottomata, fgiunchedi, JAllemandou.Feb 25 2019, 7:31 AM

• fdans moved this task from Incoming to Operational Excellence on the Analytics board.Feb 25 2019, 4:50 PM

Proposal for removal:

registry brokers services etc consumers

@Ottomata what do you think?

elukey added a project: Analytics-Kanban.Feb 28 2019, 3:55 PM

elukey moved this task from Next Up to In Code Review on the Analytics-Kanban board.

don't know about registry, services or etc, but /brokers and /consumers should be leftover from when we might have had an un-namespaced kafka cluster in zookeeper and should be safe to delete.

/etc is my fault when I've set up burrow the first time, and registry/services seems to be @joal's slider test (so safe to delete IIRC).

Got down to:

[zk: localhost:2181(CONNECTED) 37] ls /
[zookeeper, yarn-leader-election, hadoop-ha, hive_zookeeper_namespace, kafka, burrow]

That looks much nicer :)

elukey set the point value for this task to 3.Feb 28 2019, 4:39 PM

elukey moved this task from In Code Review to Done on the Analytics-Kanban board.

Mentioned in SAL (#wikimedia-operations) [2019-02-28T16:39:57Z] <elukey> clean up old/stale zookeeper znodes from conf100[4-6] - T216979

elukey moved this task from Backlog to Done on the User-Elukey board.Mar 4 2019, 9:34 AM

• Nuria closed this task as Resolved.Mar 11 2019, 6:13 PM

• Nuria closed subtask T216952: Hadoop Yarn stores a ton of znodes related to running/old applications as Resolved.Mar 18 2019, 4:13 PM

Review znodes on Zookeeper cluster to possibly remove not-used dataClosed, ResolvedPublic3 Estimated Story PointsActions

Description

Related ObjectsSearch...

Event Timeline

Review znodes on Zookeeper cluster to possibly remove not-used data
Closed, ResolvedPublic3 Estimated Story Points
Actions

Related Objects
Search...