There are some potentially helpful notes and links to reference patches here, from a ticket that I worked on some time ago: T255148: Refresh Druid nodes (druid100[1-3])

The nodes in this ticket are the druid 'public' cluster, as opposd to the druid 'analytics' cluster.
There are some differences, most notably:

• The public cluster has an LVS load-balancer connected to it, whereas the analytics cluster doesn't.
• The public cluster doesn't have a zookeeper cluster co-located with it.

From the previous tickets, the steps are roughly

• Create Keytabs
• Add the hosts to the role(druid::public::worker)
  • druid1009
  • druid1010
  • druid1011

Change 959147 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] Bring druid1009.equad.wmnet into service

https://gerrit.wikimedia.org/r/959147

Thanks @Stevemunene - The plan looks good, but we also need to take into account the zookeeper cluster which is co-located with this Druid cluster.
We are refreshing druid100[4-6] so those hosts are eventually going away (under T336043), but these three are currently the zookeeper cluster. See: https://github.com/wikimedia/operations-puppet/blob/production/hieradata/common.yaml#L684-L690

At the end of the operation we will still need a minimum of three hosts in the zookeeper cluster, so should we set these three new hosts (druid10[09-11]) as the new zookeeper cluster?
If we do this we could expand our zookeeper cluster up to six nodes, then gradually remove the three old nodes from the zookeeper cluster, so that we end up with three nodes.

We can make the changes in little steps to minimize the risk as well. There's no need to mix in the zookeeper changes at the same time as bringing a new druid worker into service.

I would advise checking the co-ordinator's web UI once you have added each server: https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Druid#Coordinators_Administration_UI
to make sure that it is balancing the data correctly and the load queues are behaving correctly as the new hosts arrive.

Also, be aware that the druid public server has an LVS endpoint of druid-public-broker.svc.eqiad.wmnet so the new servers will be able to pool and depool from this service.

btullis@lvs1019:~$ sudo ipvsadm -L -t druid-public-broker.svc.eqiad.wmnet:8082
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  druid-public-broker.svc.eqia wrr
  -> druid1004.eqiad.wmnet:8082   Route   10     119        7         
  -> druid1007.eqiad.wmnet:8082   Route   10     117        8         
  -> druid1005.eqiad.wmnet:8082   Route   10     108        8         
  -> druid1006.eqiad.wmnet:8082   Route   10     101        8         
  -> druid1008.eqiad.wmnet:8082   Route   10     112        10

I think puppet will set up the load-balancing automatically (by virtual of the profile::lvs::realserver being applioed) but it may be necessary to notify pybal of the change.
See: https://wikitech.wikimedia.org/wiki/LVS#Deploy_a_change_to_an_existing_service for more information.

I would reach out to the traffic team to verify whether this is required and if so, when would be a good time to do it.
For this reason alone, I would probably do each of the druid servers individually, as you have started to do in https://gerrit.wikimedia.org/r/959147

You might also want to do a PCC run against the O:lvs::balancer hosts as well, or lvs1019.eqiad.wmnet - I'm not sure whether puppet will make a change on them with a pcc run, but it would be useful to be sure.

I think puppet will set up the load-balancing automatically (by virtual of the profile::lvs::realserver being applioed) but it may be necessary to notify pybal of the change.
See: https://wikitech.wikimedia.org/wiki/LVS#Deploy_a_change_to_an_existing_service for more information.

I would reach out to the traffic team to verify whether this is required and if so, when would be a good time to do it.
For this reason alone, I would probably do each of the druid servers individually, as you have started to do in https://gerrit.wikimedia.org/r/959147

I think we need to add them here: https://github.com/wikimedia/operations-puppet/blob/production/conftool-data/node/eqiad.yaml#L394-L399 . Which from previous patches seems to be done separately.

In T336042#9182474, @Stevemunene wrote:

I think puppet will set up the load-balancing automatically (by virtual of the profile::lvs::realserver being applioed) but it may be necessary to notify pybal of the change.
See: https://wikitech.wikimedia.org/wiki/LVS#Deploy_a_change_to_an_existing_service for more information.

I would reach out to the traffic team to verify whether this is required and if so, when would be a good time to do it.
For this reason alone, I would probably do each of the druid servers individually, as you have started to do in https://gerrit.wikimedia.org/r/959147

I think we need to add them here: https://github.com/wikimedia/operations-puppet/blob/production/conftool-data/node/eqiad.yaml#L394-L399 . Which from previous patches seems to be done separately.

Great! I had forgotten about that.

Change 962248 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] druid: Bring druid1010.eqiad.wmnet into service

https://gerrit.wikimedia.org/r/962248

Change 962249 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] druid: Bring druid1011.eqiad.wmnet into service

https://gerrit.wikimedia.org/r/962249

Change 962250 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] druid: Add druid druid10[09-11] to druid_public_broker VIP

https://gerrit.wikimedia.org/r/962250

Change 959147 merged by Stevemunene:

[operations/puppet@production] druid: Bring druid1009.eqiad.wmnet into service

https://gerrit.wikimedia.org/r/959147

I think we might need to add each host to lvs first before it can be fully part of the druid cluster

What about running the sre.druid.roll-restart-workers cookbook on this cluster, so that it restarts the processes? I think that this is more likely to make sure that the new hosts joins the cluster than adding it to LVS.
In fact I'm pretty sure that adding it to LVS would be detrimental until druid1009 has got access to the same data as the rest of the cluster.

In T336042#9215049, @BTullis wrote:

What about running the sre.druid.roll-restart-workers cookbook on this cluster, so that it restarts the processes? I think that this is more likely to make sure that the new hosts joins the cluster than adding it to LVS.
In fact I'm pretty sure that adding it to LVS would be detrimental until druid1009 has got access to the same data as the rest of the cluster.

Thanks Ben, giving this a try

Mentioned in SAL (#wikimedia-analytics) [2023-10-02T13:40:20Z] <stevemunene> roll-restart druid public workers to pick up a new worker node. T336042

druid1009 fails the roll restart at the last stage pooling, since it is not part of the druid public broker VIP yet.

PASS |█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▊                                     |  83% (5/6) [1:01:05<12:13, 733.09s/hosts]
FAIL |████████████████████████████████████▋                                                                                                                                                                                       |  17% (1/6) [1:01:05<5:05:27, 3665.43s/hosts]
16.7% (1/6) of nodes failed to execute command 'pool': druid1009.eqiad.wmnet
83.3% (5/6) success ratio (< 100.0% threshold) of nodes successfully executed all commands. Aborting.: druid[1004-1008].eqiad.wmnet
Exception raised while executing cookbook sre.druid.roll-restart-workers:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 212, in run
    raw_ret = runner.run()
  File "/srv/deployment/spicerack/cookbooks/sre/druid/roll-restart-workers.py", line 90, in run
    self.druid_workers.run_async(
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 459, in run_async
    return self._execute(
  File "/usr/lib/python3/dist-packages/spicerack/remote.py", line 702, in _execute
    raise RemoteExecutionError(ret, "Cumin execution failed", worker.get_results())
spicerack.remote.RemoteExecutionError: Cumin execution failed (exit_code=2)
END (FAIL) - Cookbook sre.druid.roll-restart-workers (exit_code=99) for Druid public cluster: Roll restart of Druid jvm daemons.

the druid-historical.service errors are related to the fact that the server is not yet part of the zookeper cluster.

2023-10-02T14:50:49,454 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server druid1004.eqiad.wmnet/10.64.0.35:2181. Will not attempt to authenticate using SASL (unknown error)
2023-10-02T14:50:49,455 ERROR org.apache.druid.cli.CliHistorical: Error when starting up.  Failing.

The same applies to the druid-middlemanager.service

2023-10-02T14:55:14,178 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server druid1006.eqiad.wmnet/2620:0:861:107:10:64:48:171:2181. Will not attempt to authenticate using SASL (unknown error)
2023-10-02T14:55:14,186 ERROR org.apache.druid.cli.CliMiddleManager: Error when starting up.  Failing.

We expect to solve these once the servers are fully onboarded on zookeper, for now I have added some scheduled downtime for the two services to reduce the amount of alerts.

Icinga downtime and Alertmanager silence (ID=1bb0fee0-dd0c-4165-86e2-b81abeffa7d2) set by stevemunene@cumin1001 for 7 days, 0:00:00 on 1 host(s) and their services with reason: Downtime as we setup the host to join the druid and zookeper cluster

druid1009.eqiad.wmnet

Change 962248 merged by Stevemunene:

[operations/puppet@production] druid: Bring druid1010.eqiad.wmnet into service

https://gerrit.wikimedia.org/r/962248

Added druid1009 and druid1010 to the role(druid::public::worker) and to the druid_public_hosts firewall block and the two were able to join the druid cluster and are currently rebalancing without any issue.

Change 962249 merged by Stevemunene:

[operations/puppet@production] druid: Bring druid1011.eqiad.wmnet into service

https://gerrit.wikimedia.org/r/962249

Icinga downtime and Alertmanager silence (ID=1ef45f2c-6b73-432c-9d22-7d378d8653d7) set by stevemunene@cumin1001 for 1 day, 0:00:00 on 1 host(s) and their services with reason: Downtime as we setup the host to join the druid and zookeper cluster

druid1011.eqiad.wmnet

Change 965460 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[labs/private@master] Add dummy keytabs for new druid101[0-1]

https://gerrit.wikimedia.org/r/965460

The new druid workers have fully joined the cluster and we are ready to move on with the next steps.

• Disable puppet on druid100[4-6] and druid10[09-11] and an-launcher1002
• Disable the following four timers on an-launcher1002
  • eventlogging_to_druid_editattemptstep_hourly.timer
  • eventlogging_to_druid_navigationtiming_hourly.timer
  • eventlogging_to_druid_netflow_hourly.timer
  • eventlogging_to_druid_prefupdate_hourly.timer
• Disable the following three schedules in Hue
  • webrequest-druid-hourly-coord
  • pageview-druid-hourly-coord
  • edit-hourly-druid-coord
• Manually stop zookeeper on druid1004 (This being the host we plan to switch first to the newer druid1009)
• Manually disable zookeeper on druid1004
• Enable and run puppet with this change on druid1009 to start zookeeper
• Check zookeeper state with echo mntr | nc localhost 2181 to make sure that the ensemble of three nodes is up and running.
• Enable and run puppet on druid1005 to obtain the configuration change (the second host we plan to switch)
• Restart the zookeeper service on druid1005 to apply the configuration change
• Check zookeeper state with echo mntr | nc localhost 2181 to make sure that the ensemble of three nodes is up and running.
• Enable and run puppet on druid1006 to obtain the configuration change (the third host we plan to switch)
• Restart the zookeeper service on druid1006 to apply the configuration change
• Verify that the three zookeeper ensemble of three nodes is functioning correctly.
• Check for quorum and znodes presence etc with Grafana.
• Wait until any in-progress druid ingestions have finished (except wmf_netflow)
• Roll restart the druid cluster
• Repeat the procedure above twice in order to replace druid1005 and then druid1006.
• Once complete, re-enable systemd timers and hue jobs that were previously disabled.

In T336042#9245252, @Stevemunene wrote:

The new druid workers have fully joined the cluster and we are ready to move on with the next steps.

Great!

From the conversations on the last druid refresh, the servers were manually switched one by one as mentioned on the deployment plan discussion by @BTullis which can be combined and modified to

That's a good reference to start with, there are a few differences to be considered.

• The Druid cluster being operated on is different, so on that ticket (T255148) we were dealing with the analytics cluster, whereas this ticket is about the public cluster.
• This means that the jobs you mentioned (eventlogging, webrequest, pageview) etc. aren't relevant, because they don't get loaded to this cluster.
• The only dataset loaded to the public cluster is Mediawiki_history_reduced
• This dataset only gets generated and loaded to Druid once per month, so you're not going to need to disable any loading jobs for the duration of the move.
• However, when loading it currently uses druid1004, so this config in airflow-dags will need to be updated.
• All references to Oozie and Hue are out-of-date, since all of these druid loading jobs have been migrated to Airflow now.
• But that's OK anyway, as you don't need to stop them (see above).

So I've added some inline comments in to your plan below. Hope that's OK.

• Disable puppet on druid100[4-6] and druid10[09-11] and an-launcher1002

Yes, this is probably still a good idea to stop puppet and only enable it when you want to apply a new puppet configuration to a certain host.

• Disable the following four timers on an-launcher1002
  • eventlogging_to_druid_editattemptstep_hourly.timer
  • eventlogging_to_druid_navigationtiming_hourly.timer
  • eventlogging_to_druid_netflow_hourly.timer
  • eventlogging_to_druid_prefupdate_hourly.timer
• Disable the following three schedules in Hue
  • webrequest-druid-hourly-coord
  • pageview-druid-hourly-coord
  • edit-hourly-druid-coord

None of this is needed, since it's the druid-public cluster.

• Manually stop zookeeper on druid1004 (This being the host we plan to switch first to the newer druid1009)
• Manually disable zookeeper on druid1004
• Enable and run puppet with this change on druid1009 to start zookeeper

OK, have you got the puppet changes ready for the zookeeper modifications?
Will you be doing a chain of patches to add them, or one big change?

• Check zookeeper state with echo mntr | nc localhost 2181 to make sure that the ensemble of three nodes is up and running.
• Enable and run puppet on druid1005 to obtain the configuration change (the second host we plan to switch)
• Restart the zookeeper service on druid1005 to apply the configuration change
• Check zookeeper state with echo mntr | nc localhost 2181 to make sure that the ensemble of three nodes is up and running.
• Enable and run puppet on druid1006 to obtain the configuration change (the third host we plan to switch)
• Restart the zookeeper service on druid1006 to apply the configuration change
• Verify that the three zookeeper ensemble of three nodes is functioning correctly.
• Check for quorum and znodes presence etc with Grafana.

- Wait until any in-progress druid ingestions have finished (except wmf_netflow)

I commented out the line above because there shouldn't be any ingestions happening.

• Roll restart the druid cluster
• Repeat the procedure above twice in order to replace druid1005 and then druid1006.

- Once complete, re-enable systemd timers and hue jobs that were previously disabled.

Does this all make sense?

This does make sense, Thanks @BTullis. I shall be doing a string of patches for this

Change 965499 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] Switch druid1004 zookeeper node with druid1009

https://gerrit.wikimedia.org/r/965499

Change 965500 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] Switch druid1005 zookeeper node with druid1010

https://gerrit.wikimedia.org/r/965500

Change 965501 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] Switch druid1006 zookeeper node with druid1011

https://gerrit.wikimedia.org/r/965501

Updating the rollout plan as discussed

• Disable puppet on druid100[4-6] and druid10[09-11] and an-launcher1002
• Manually stop zookeeper on druid1004 (This being the host we plan to switch first to the newer druid1009)
• Manually disable zookeeper on druid1004
• Enable and run puppet with this change on druid1009 to start zookeeper
• Check zookeeper state with echo mntr | nc localhost 2181 to make sure that the ensemble of three nodes is up and running.
• Change the druid host required to index mediawiki_history_reduced to the druid-public cluster.
• Change the host and port configured on datahub injestion
• Change the druid host configured for profile::analytics::refinery::job::data_purge::public_druid_host: Checking with the DPE Engineers on the process involved for this.
• Enable and run puppet on druid1005 to obtain the configuration change (the second host we plan to switch)
• Restart the zookeeper service on druid1005 to apply the configuration change
• Check zookeeper state with echo mntr | nc localhost 2181 to make sure that the ensemble of three nodes is up and running.
• Enable and run puppet on druid1006 to obtain the configuration change (the third host we plan to switch)
• Restart the zookeeper service on druid1006 to apply the configuration change
• Verify that the three zookeeper ensemble of three nodes is functioning correctly.
• Check for quorum and znodes presence etc with Grafana.
• Roll restart the druid cluster
• Repeat the procedure above twice in order to replace druid1005 and then druid1006.

worth noting, we might need to start with druid1005 and druid1006 due the processes required to remove druid1004.

This should be the updated plan for our next course of action.

Updating the comment to show that we are starting with druid1006 switch with druid1011.
output from the zookeper leader druid1005 before start

stevemunene@druid1005:~$ echo mntr | nc localhost 2181
zk_version	3.4.13-2--1, built on Tue, 04 Jun 2019 21:22:04 -0700
zk_avg_latency	0
zk_max_latency	1657
zk_min_latency	0
zk_packets_received	6653499
zk_packets_sent	7200161
zk_num_alive_connections	19
zk_outstanding_requests	0
zk_server_state	leader
zk_znode_count	1668
zk_watch_count	10134
zk_ephemerals_count	1561
zk_approximate_data_size	3431542
zk_open_file_descriptor_count	64
zk_max_file_descriptor_count	524288
zk_fsync_threshold_exceed_count	4
zk_followers	2
zk_synced_followers	2
zk_pending_syncs	0
zk_last_proposal_size	203
zk_max_proposal_size	3487
zk_min_proposal_size	32

zk_followers 2 and zk_synced_followers 2 as expected. We expect to see this reduce to 1, the come back to 2 once druid1011 is up.

Mentioned in SAL (#wikimedia-analytics) [2023-10-26T09:18:54Z] <stevemunene> stop zookeper on druid1006 T336042

Disabled puppet on all the druid::public::worker hosts.
proceeding to disable zookeper on druid1006

stevemunene@druid1006:~$ sudo systemctl stop zookeeper 
stevemunene@druid1006:~$ sudo systemctl disable zookeeper
zookeeper.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable zookeeper

proceeding to merge the patch.

Change 965501 merged by Stevemunene:

[operations/puppet@production] Switch druid1006 zookeeper node with druid1011

https://gerrit.wikimedia.org/r/965501

Merged, running puppet on druid1011
status on zk leader

stevemunene@druid1005:~$ echo mntr | nc localhost 2181
zk_version	3.4.13-2--1, built on Tue, 04 Jun 2019 21:22:04 -0700
zk_avg_latency	0
zk_max_latency	1657
zk_min_latency	0
zk_packets_received	6667436
zk_packets_sent	7214160
zk_num_alive_connections	23
zk_outstanding_requests	0
zk_server_state	leader
zk_znode_count	1668
zk_watch_count	14733
zk_ephemerals_count	1561
zk_approximate_data_size	3431542
zk_open_file_descriptor_count	64
zk_max_file_descriptor_count	524288
zk_fsync_threshold_exceed_count	4
zk_followers	1
zk_synced_followers	1
zk_pending_syncs	0
zk_last_proposal_size	377
zk_max_proposal_size	3487
zk_min_proposal_size	32

druid1011 is having some trouble connecting to the current ensemble.

2023-10-26 09:23:56,186 - INFO  [main:QuorumPeer@668] - currentEpoch not found! Creating with a reasonable default of 0. This should only happen when you are upgrading your installation

And

2023-10-26 09:25:37,561 - WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperThread@60] - Exception occurred from thread NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181
java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;
	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:256)
	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:226)
	at java.lang.Thread.run(Thread.java:750)

Noting similar issues were had during the analytics zookeper swicth here which on following the link leads to the comment below that suggests restarting the leader allows us to form a quorum again just fine.

We found that it was actually due to https://issues.apache.org/jira/browse/ZOOKEEPER-2186. When the QuorumCnxManager thread dies, you can't get any more elections to happen. But, when you restart the leader, you can form a quorum again just fine. ZK servers which are already a part of the quorum continue to merrily participate.

Mentioned in SAL (#wikimedia-analytics) [2023-10-26T10:18:17Z] <stevemunene> restart zookeper leader to pick up new host druid1011 T336042

Restarted the ZK cluster leader after running puppet on the host to update the config.
druid1011 is yet to join the zk cluster due to constant timeouts.

2023-10-26 11:23:38,503 - INFO  [QuorumPeer[myid=1011]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@847] - Notification time out: 60000
2023-10-26 11:24:38,504 - INFO  [QuorumPeer[myid=1011]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@847] - Notification time out: 60000

and the only error on the leader being

2023-10-26 10:18:23,512 - INFO  [ProcessThread(sid:1005 cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0xec00001624bf000c type:create cxid:0x38412 zxid:0x3d00000035 txntype:-1 reqpath:n/a Error Path:/druid/public-eqiad/discovery/druid:broker/4f72099d-6cfe-475a-8c93-9cd9d0c00ac5 Error:KeeperErrorCode = NodeExists for /druid/public-eqiad/discovery/druid:broker/4f72099d-6cfe-475a-8c93-9cd9d0c00ac5

In T336042#9283701, @Stevemunene wrote:

Restarted the ZK cluster leader after running puppet on the host to update the config.
druid1011 is yet to join the zk cluster due to constant timeouts.

2023-10-26 11:23:38,503 - INFO  [QuorumPeer[myid=1011]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@847] - Notification time out: 60000
2023-10-26 11:24:38,504 - INFO  [QuorumPeer[myid=1011]/0:0:0:0:0:0:0:0:2181:FastLeaderElection@847] - Notification time out: 60000

and the only error on the leader being

2023-10-26 10:18:23,512 - INFO  [ProcessThread(sid:1005 cport:-1)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0xec00001624bf000c type:create cxid:0x38412 zxid:0x3d00000035 txntype:-1 reqpath:n/a Error Path:/druid/public-eqiad/discovery/druid:broker/4f72099d-6cfe-475a-8c93-9cd9d0c00ac5 Error:KeeperErrorCode = NodeExists for /druid/public-eqiad/discovery/druid:broker/4f72099d-6cfe-475a-8c93-9cd9d0c00ac5

The error was mainly caused by an unclean exit of the process during zookeper.service restarts, thus the service was not properly exited for a restart. This was fixed by manually killing the process via the PID and restarting the zookeper.service. Cluster back at full capacity

zk_followers	2
zk_synced_followers	2

Moving next to the druid1005 which is also the current leader of the cluster. A new leader should be elected in seconds after stopping the zookeper service on druid1005 then we can proceed after confirming all services are back up.

Mentioned in SAL (#wikimedia-analytics) [2023-10-26T15:29:25Z] <stevemunene> stop zookeper on druid1005 current leader for the druid-public-eqiad this will trigger the election of a new leader T336042

Change 965500 merged by Stevemunene:

[operations/puppet@production] Switch druid1005 zookeeper node with druid1010

https://gerrit.wikimedia.org/r/965500

Zookeper stopped on druid1005, druid1011 is now the new leader.

stevemunene@druid1011:~$ echo mntr | nc localhost 2181
zk_version	3.4.13-6--1, built on Sun, 07 Feb 2021 23:16:46 +0100
zk_avg_latency	0
zk_max_latency	0
zk_min_latency	0
zk_packets_received	1
zk_packets_sent	0
zk_num_alive_connections	1
zk_outstanding_requests	0
zk_server_state	leader
zk_znode_count	1668
zk_watch_count	0
zk_ephemerals_count	1561
zk_approximate_data_size	3431542
zk_open_file_descriptor_count	41
zk_max_file_descriptor_count	524288
zk_fsync_threshold_exceed_count	0
zk_followers	1
zk_synced_followers	1
zk_pending_syncs	0
zk_last_proposal_size	371
zk_max_proposal_size	377
zk_min_proposal_size	36

All seems okay to proceed with merging the druid1005 -> druid1010 change.

• • disabled puppet on all druid-public-eqiad hosts.
  • Merged the change from druid1005 ->druid1011
  • Enabled puppet on druid1010 to pick change
  • Enabled puppet on druid1011 and druid1004 to pick changes on zookeper.cfg
• Restarted zookeeper service on on druid1011 and druid1004 to pick changes
• Restarted zookeeper on druid1010 and got similar errors to previous change with the root cause being the same.

Oct 26 16:02:01 druid1010 systemd[1]: zookeeper.service: Found left-over process 2071045 (java) in control group while starting unit. Ignoring.
Oct 26 16:02:01 druid1010 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.

• stop the process sudo kill 2071045
• Restarted the zookeeper service on druid1010 and the host was able to join the cluster.

Output from ZK leader, all looks ok.

stevemunene@druid1011:~$ echo mntr | nc localhost 2181
zk_version	3.4.13-6--1, built on Sun, 07 Feb 2021 23:16:46 +0100
zk_avg_latency	0
zk_max_latency	0
zk_min_latency	0
zk_packets_received	3
zk_packets_sent	2
zk_num_alive_connections	1
zk_outstanding_requests	0
zk_server_state	leader
zk_znode_count	1668
zk_watch_count	0
zk_ephemerals_count	1561
zk_approximate_data_size	3431542
zk_open_file_descriptor_count	42
zk_max_file_descriptor_count	524288
zk_fsync_threshold_exceed_count	0
zk_followers	2
zk_synced_followers	2
zk_pending_syncs	0
zk_last_proposal_size	203
zk_max_proposal_size	2039
zk_min_proposal_size	36

Next is running the cookbook sre.druid.roll-restart-workers for the public cluster to pick up the new servers as well.

Mentioned in SAL (#wikimedia-analytics) [2023-10-26T16:18:25Z] <stevemunene> roll-restart druid public workers to pick up new zookeeper hosts. T336042

stevemunene opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/532

switch druid host to index to the druid-public cluster and datahub injestion.

stevemunene closed https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/532

switch druid host to index to the druid-public cluster and datahub injestion.

Change 970272 had a related patch set uploaded (by Stevemunene; author: Stevemunene):

[operations/puppet@production] switch druid host to run data_purge job

https://gerrit.wikimedia.org/r/970272

stevemunene opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/533

switch druid host to index to the druid-public cluster and datahub injestion.

Change 962250 merged by Ssingh:

[operations/puppet@production] druid: Add druid druid10[09-11] to druid_public_broker VIP

https://gerrit.wikimedia.org/r/962250

Mentioned in SAL (#wikimedia-analytics) [2023-11-06T13:25:43Z] <stevemunene> stop and disable zookeper on druid1004 T336042

Change 965499 merged by Stevemunene:

[operations/puppet@production] Switch druid1004 zookeeper node with druid1009

https://gerrit.wikimedia.org/r/965499

Mentioned in SAL (#wikimedia-analytics) [2023-11-06T13:32:36Z] <stevemunene> restart zookeper leader to pick up new host druid1009 T336042

In T336042#9284972, @Stevemunene wrote:

Zookeper stopped on druid1005, druid1011 is now the new leader.

stevemunene@druid1011:~$ echo mntr | nc localhost 2181
zk_version	3.4.13-6--1, built on Sun, 07 Feb 2021 23:16:46 +0100
zk_avg_latency	0
zk_max_latency	0
zk_min_latency	0
zk_packets_received	1
zk_packets_sent	0
zk_num_alive_connections	1
zk_outstanding_requests	0
zk_server_state	leader
zk_znode_count	1668
zk_watch_count	0
zk_ephemerals_count	1561
zk_approximate_data_size	3431542
zk_open_file_descriptor_count	41
zk_max_file_descriptor_count	524288
zk_fsync_threshold_exceed_count	0
zk_followers	1
zk_synced_followers	1
zk_pending_syncs	0
zk_last_proposal_size	371
zk_max_proposal_size	377
zk_min_proposal_size	36

All seems okay to proceed with merging the druid1005 -> druid1010 change.

• • disabled puppet on all druid-public-eqiad hosts.
  • Merged the change from druid1005 ->druid1011
  • Enabled puppet on druid1010 to pick change
  • Enabled puppet on druid1011 and druid1004 to pick changes on zookeper.cfg
• Restarted zookeeper service on on druid1011 and druid1004 to pick changes
• Restarted zookeeper on druid1010 and got similar errors to previous change with the root cause being the same.

Oct 26 16:02:01 druid1010 systemd[1]: zookeeper.service: Found left-over process 2071045 (java) in control group while starting unit. Ignoring.
Oct 26 16:02:01 druid1010 systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.

• stop the process sudo kill 2071045
• Restarted the zookeeper service on druid1010 and the host was able to join the cluster.

Output from ZK leader, all looks ok.

stevemunene@druid1011:~$ echo mntr | nc localhost 2181
zk_version	3.4.13-6--1, built on Sun, 07 Feb 2021 23:16:46 +0100
zk_avg_latency	0
zk_max_latency	0
zk_min_latency	0
zk_packets_received	3
zk_packets_sent	2
zk_num_alive_connections	1
zk_outstanding_requests	0
zk_server_state	leader
zk_znode_count	1668
zk_watch_count	0
zk_ephemerals_count	1561
zk_approximate_data_size	3431542
zk_open_file_descriptor_count	42
zk_max_file_descriptor_count	524288
zk_fsync_threshold_exceed_count	0
zk_followers	2
zk_synced_followers	2
zk_pending_syncs	0
zk_last_proposal_size	203
zk_max_proposal_size	2039
zk_min_proposal_size	36

Next is running the cookbook sre.druid.roll-restart-workers for the public cluster to pick up the new servers as well.

followed the same process to switch zookeper host druid1004 to druid1009.
This is the status on druid1009

stevemunene@druid1009:~$ echo mntr | nc localhost 2181
zk_version	3.4.13-6--1, built on Sat, 28 Oct 2023 23:16:44 +0200
zk_avg_latency	0
zk_max_latency	0
zk_min_latency	0
zk_packets_received	1
zk_packets_sent	0
zk_num_alive_connections	1
zk_outstanding_requests	0
zk_server_state	follower
zk_znode_count	5597
zk_watch_count	0
zk_ephemerals_count	5496
zk_approximate_data_size	6344423
zk_open_file_descriptor_count	41
zk_max_file_descriptor_count	524288
zk_fsync_threshold_exceed_count	0

The Zookeeper host is also back to having 2 followers as expected.

stevemunene@druid1011:~$ echo mntr | nc localhost 2181
zk_version	3.4.13-6--1, built on Sun, 07 Feb 2021 23:16:46 +0100
zk_avg_latency	0
zk_max_latency	15
zk_min_latency	0
zk_packets_received	6845
zk_packets_sent	6938
zk_num_alive_connections	9
zk_outstanding_requests	0
zk_server_state	leader
zk_znode_count	5828
zk_watch_count	11539
zk_ephemerals_count	5721
zk_approximate_data_size	6578923
zk_open_file_descriptor_count	51
zk_max_file_descriptor_count	524288
zk_fsync_threshold_exceed_count	0
zk_followers	2
zk_synced_followers	2
zk_pending_syncs	0
zk_last_proposal_size	204
zk_max_proposal_size	1830
zk_min_proposal_size	36

Mentioned in SAL (#wikimedia-analytics) [2023-11-06T13:57:43Z] <stevemunene> roll-restart druid public workers to pick up a new zookeeper node druid1009. T336042

xcollazo merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/533

switch druid host to index to the druid-public cluster and datahub injestion.

Change 970272 merged by Stevemunene:

[operations/puppet@production] switch druid host to run data_purge job

https://gerrit.wikimedia.org/r/970272

Change 965460 merged by Stevemunene:

[labs/private@master] Add dummy keytabs for new druid101[0-1]

https://gerrit.wikimedia.org/r/965460