These hosts are ready for use and should be added to the Kafka cluster.

checklist: replace kafka-logging1003 with kafka-logging1006

The insetup role is based on nftables, while the kafka role uses iptables. Switch to an iptables-based role, manually disable nftables, and reboot the host.

• Write/merge homer templates/cr/firewall.conf patch to include new broker(s) addresses This is no longer needed, as it is now handled by the Homer/Capirca/NetBox integration.
• Downtime kafka-logging1003 in icinga

cookbook sre.hosts.downtime -t T432444 --days 5 -r 'migrating to kafka-logging1006' 'kafka-logging1003*'

• Disable notifications for kafka-logging1006 (via hiera patch):

# add file hieradata/hosts/kafka-logging1006.yaml with content:
profile::monitoring::notifications_enabled: false

• Downtime alertmanager “under replicated partitions” checks on all members of the kafka cluster undergoing maintenance (e.g. all kafka-logging eqiad broker hosts) => 59e43fe9-1a46-41be-86f2-90a7baef5bac

# silence filters
alertname=KafkaUnderReplicatedPartitions isntance=kafka-logging100*

• kafka-logging100[36]: Disable puppet with message

sudo -i cumin 'P{I:facts.clientcert ~ "kafka-logging100(3|6)"}' 'disable-puppet T432444'

• kafka-logging1003: stop & mask service kafka

systemctl disable --now kafka.service
systemctlk mask kafka.service

• merge the patch that adds kafka-logging1006
• run puppet agent on kafka-logging1* nodes

sudo -i cumin 'P{I:facts.clientcert ~ "kafka-logging100(1|2|4|5)"}' 'disable-puppet T432444'

• run puppet agent on conf1* nodes

sudo -i cumin A:conf-eqiad run-puppet-agent

• run puppet agent on prometheus nodes

sudo -i cumin 'A:prometheus and A:eqiad' run-puppet-agent

• run puppet agent on grafana nodes

sudo -i cumin 'A:grafana' run-puppet-agent

• Setup kafka acls (on kafka-logging1001)

kafka acls --add --allow-principal User:CN=kafka-logging1006.eqiad.wmnet --operation All --cluster --topic '*' --group '*' --transactional-id '*' --delegation-token '*'

• Setting reserved blocks percentage to 0

tune2fs -m 0 /dev/mapper/vg0-srv

• kafka-logging1006: run puppet agent -> Ensure kafka, services come up cleanly (watch grafana dashboard for this kafka cluster, and watch consumer lag. it may be necessary to reload grafana in the browser to pick up new cluster hosts)
• Give Kafka cluster time to sync and settle down
• https://grafana.wikimedia.org/d/000000234/kafka-by-topic
• https://grafana.wikimedia.org/d/000000027/kafka
• if replica imbalance does not correct itself, issue a reelection with kafka leader-election --election-type preferred --all-topic-partitions
• Update broker list in eventgate-logging to include new broker(s) addresses This is no longer needed, as it is managed by Puppet, but it still needs to be deployed with Scap:

user@deploy:~$ sudo -i run-puppet-agent
user@deploy:~$ scap sync-world --k8s-only --k8s-confirm-diff -Dbuild_mw_container_image:False

• sync k8s external-services (check both sites)

deploy1003: kube-env admin eqiad; cd /srv/deployment-charts/helmfile.d/admin_ng; helmfile -e eqiad -lname=external-services diffl helmfile -e eqiad -lname=external-services sync

• Ensure kafka-logging1006 green in icinga and prometheus
• Enable notifications for kafka-logging1006

checklist: add kafka-logging1007 with kafka-logging1008

• Disable notifications for kafka-logging100[78] (via hiera patch):

# add file hieradata/hosts/kafka-logging100[78].yaml with content:
profile::monitoring::notifications_enabled: false

• Setting reserved blocks percentage to 0 on kafka-logging[78]

tune2fs -m 0 /dev/mapper/vg0-srv

• kafka-logging100[78]: disable puppet

sudo -i cumin 'P{I:facts.clientcert ~ "kafka-logging100(7|8)"}' 'disable-puppet T432444'

• Setup kafka acls (on kafka-logging1001)

kafka acls --add --allow-principal User:CN=kafka-logging1007.eqiad.wmnet --operation All --cluster --topic '*' --group '*' --transactional-id '*' --delegation-token '*'
kafka acls --add --allow-principal User:CN=kafka-logging1008.eqiad.wmnet --operation All --cluster --topic '*' --group '*' --transactional-id '*' --delegation-token '*'

• merge the patch that adds kafka-logging100[78]
• Update broker list in eventgate-logging to include new broker(s) addresses This is no longer needed, as it is managed by Puppet, but it still needs to be deployed with Scap: --> https://phabricator.wikimedia.org/P96506

user@deploy:~$ sudo -i run-puppet-agent
user@deploy:~$ scap sync-world --k8s-only --k8s-confirm-diff -Dbuild_mw_container_image:False

• sync k8s external-services (check both sites)

deploy1003: kube-env admin eqiad; cd /srv/deployment-charts/helmfile.d/admin_ng; helmfile -e eqiad -lname=external-services diff helmfile -e eqiad -lname=external-services sync

• run puppet agent on kafka-logging1* nodes

sudo -i cumin 'P{I:facts.clientcert ~ "kafka-logging100(1|2|4|5|6)"}' 'run-puppet-agent'

• run puppet agent on conf1* nodes

sudo -i cumin A:conf-eqiad run-puppet-agent

• run puppet agent on prometheus nodes

sudo -i cumin 'A:prometheus and A:eqiad' run-puppet-agent

• run puppet agent on grafana nodes

sudo -i cumin 'A:grafana' run-puppet-agent

• kafka-logging100[78]: run puppet agent -> Ensure kafka, services come up cleanly (watch grafana dashboard for this kafka cluster, and watch consumer lag. it may be necessary to reload grafana in the browser to pick up new cluster hosts)
• Give Kafka cluster time to sync and settle down
• https://grafana.wikimedia.org/d/000000234/kafka-by-topic
• https://grafana.wikimedia.org/d/000000027/kafka
• if replica imbalance does not correct itself, issue a reelection with kafka leader-election --election-type preferred --all-topic-partitions
• Ensure kafka-logging100[78] green in icinga and prometheus
• Enable notifications for kafka-logging100[78]

Rebalance/Rebuild

Only after kafka-logging1007 and kafka-logging1008 have joined the cluster is a rebalance needed to evict 1001 and 1002. There is no need to rebalance kafka-logging1003, since its partitions are already being taken over by kafka-logging1006, which has taken over node ID 1006, previously used by kafka-logging1003.

topicmappr rebuild --brokers "1004,1005,1006,1007,1008" --topics '.*' --out-file cluster-rebalance
sudo -E kafka reassign-partitions --reassignment-json-file ./cluster-rebalance.json --execute --throttle 50000000

After the rebuild, it will be possible to remove 1001 and 1002 from the cluster.

checklist: remove kafka-logging1001 with kafka-logging1002

• Downtime kafka-logging1001 in icinga

cookbook sre.hosts.downtime -t T432444 --days 5 -r 'removing kafka-logging1001' 'kafka-logging1001*'

• Downtime kafka-logging1002 in icinga

cookbook sre.hosts.downtime -t T432444 --days 5 -r 'removing kafka-logging1002' 'kafka-logging1002*'

• kafka-logging100[12]: Disable puppet with message

sudo -i cumin 'P{I:facts.clientcert ~ "kafka-logging100(1|2)"}' 'disable-puppet T432444'

• kafka-logging100[12]: stop & mask service kafka

systemctl disable --now kafka.service
systemctlk mask kafka.service

• merge the patch that removes kafka-logging100[12]
• run puppet agent on kafka-logging1* nodes

sudo -i cumin kafka-logging1* run-puppet-agent

• run puppet agent on conf1* nodes

sudo -i cumin -A:conf-eqiad run-puppet-agent

• run puppet agent on prometheus nodes

sudo -i cumin 'A:prometheus and A:eqiad' run-puppet-agent

• run puppet agent on grafana nodes

sudo -i cumin 'A:grafana' run-puppet-agent

• kafka-logging100[12]: run puppet agent
• Update broker list in eventgate-logging to include new broker(s) addresses This is no longer needed, as it is managed by Puppet, but it still needs to be deployed with Scap:

user@deploy:~$ sudo -i run-puppet-agent
user@deploy:~$ scap sync-world --k8s-only --k8s-confirm-diff -Dbuild_mw_container_image:False

• sync k8s external-services (check both sites)

deploy1003: kube-env admin eqiad; cd /srv/deployment-charts/helmfile.d/admin_ng; helmfile -e eqiad -lname=external-services diffl helmfile -e eqiad -lname=external-services sync

decom

• run decom cookbook for kafka-logging100[1-3]

checklist: add kafka-logging200[6-8]

• Disable notifications for kafka-logging200[6-8] (via hiera patch):

# add file hieradata/hosts/kafka-logging200[6-8].yaml with content:
profile::monitoring::notifications_enabled: false

• Setting reserved blocks percentage to 0 on kafka-logging[6-8]

tune2fs -m 0 /dev/mapper/vg0-srv

• kafka-logging200[6-8]: disable puppet

sudo -i cumin 'P{I:facts.clientcert ~ "kafka-logging200(6|7|8)"}' 'disable-puppet T432444'

• Setup kafka acls (on kafka-logging2001)

kafka acls --add --allow-principal User:CN=kafka-logging2006.codfw.wmnet --operation All --cluster --topic '*' --group '*' --transactional-id '*' --delegation-token '*'
kafka acls --add --allow-principal User:CN=kafka-logging2007.codfw.wmnet --operation All --cluster --topic '*' --group '*' --transactional-id '*' --delegation-token '*'
kafka acls --add --allow-principal User:CN=kafka-logging2008.codfw.wmnet --operation All --cluster --topic '*' --group '*' --transactional-id '*' --delegation-token '*'

• merge the patch that adds kafka-loggingì200[6-8]
• Update broker list in eventgate-logging to include new broker(s) addresses This is no longer needed, as it is managed by Puppet, but it still needs to be deployed with Scap:

user@deploy:~$ sudo -i run-puppet-agent
user@deploy:~$ scap sync-world --k8s-only --k8s-confirm-diff -Dbuild_mw_container_image:False

• sync k8s external-services (check both sites)

deploy1003: kube-env admin eqiad; cd /srv/deployment-charts/helmfile.d/admin_ng; helmfile -e eqiad -lname=external-services diff helmfile -e eqiad -lname=external-services sync
deploy1003: kube-env admin codfw; cd /srv/deployment-charts/helmfile.d/admin_ng; helmfile -e codfw -lname=external-services diff helmfile -e codfw -lname=external-services sync

• run puppet agent on kafka-logging200[1-5] nodes

sudo -i cumin 'P{I:facts.clientcert ~ "kafka-logging200(1|2|3|4|5)"}' 'run-puppet-agent'

• run puppet agent on conf1* nodes

sudo -i cumin A:conf-codfw run-puppet-agent

• run puppet agent on prometheus nodes

sudo -i cumin 'A:prometheus and A:codfw' run-puppet-agent

• run puppet agent on grafana nodes

sudo -i cumin 'A:grafana' run-puppet-agent

• kafka-logging200[6-8]: run puppet agent -> Ensure kafka, services come up cleanly (watch grafana dashboard for this kafka cluster, and watch consumer lag. it may be necessary to reload grafana in the browser to pick up new cluster hosts)
• Give Kafka cluster time to sync and settle down
• https://grafana.wikimedia.org/d/000000234/kafka-by-topic
• https://grafana.wikimedia.org/d/000000027/kafka
• if replica imbalance does not correct itself, issue a reelection with kafka leader-election --election-type preferred --all-topic-partitions
• Ensure kafka-logging200[6-8] green in icinga and prometheus
• Enable notifications for kafka-logging100[6-8]

Rebalance/Rebuild

• Only after kafka-logging200[6-8] have joined the cluster is a rebalance needed to evict 2001, 2002 and 2003.

topicmappr rebuild --brokers "2004,2005,2006,2007,2008" --topics '.*' --out-file cluster-rebalance
sudo -E kafka reassign-partitions --reassignment-json-file ./cluster-rebalance.json --execute --throttle 50000000

After the rebuild, it will be possible to remove 2001, 2002 and 2003 from the cluster.