Upgrade kafka-main nodes to buster
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	elukey
	Nov 29 2021, 3:20 PM

Description

The kafka main nodes are split between buster and stretch:

kafka-main[12]00[1-3] are on stretch
kafka-main[12]00[45] are on buster

The [45] nodes were added as part of T225005 and they got Buster straight away, the other ones are still todo. In theory we could do the following:

Create partman re-use recipes for kafka main, to keep the /srv partition while reimaging
Reimage one node at the time to Buster

From the past experience of Kafka Jumbo it seems fine to keep the list of brokers that clients use in their config intact, since they should be able to react to brokers not being reachable without issues. I'd avoid Bullseye for the moment since we have the puppet code already working for Buster, and it should be quick and easy to upgrade the remaining nodes.

I can take care of it if you like the idea :)

Details

Project	Branch	Lines +/-	Subject
operations/puppet	production	+2 -2	role::kafka::main: apply kafka fixed uid/gid to eqiad
operations/puppet	production	+1 -2	install_server: set the reuse recipe for all kafka-main hosts
operations/puppet	production	+2 -0	Move kafka test to fixed gid/uid for user kafka
operations/puppet	production	+16 -1	profile::kafka::broker: allow to specify kafka uid/gid
operations/puppet	production	+6 -0	admin: reserve kafka uid/gid
operations/puppet	production	+2 -1	install_server: set test reuse recipe for kafka-main2003
operations/puppet	production	+0 -0	install_server: rename reuse-kafka-main recipe
operations/puppet	production	+15 -0	install_server: add reuse recipe for kafka-main hosts

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	elukey	T296641 Upgrade kafka-main nodes to buster
Resolved	elukey	T296982 Move kafka clusters to fixed uid/gid
Resolved	elukey	T296990 Move kafka-jumbo to a fixed uid/gid
Resolved	herron	T298883 Move kafka logging eqiad/codfw to fixed uid/gid
Resolved	Papaul	T297422 Installation issues on PowerEdge R440 Kafka main codfw servers with buster / firmware update needed
Resolved	Cmjohnson	T298867 Installation issues on PowerEdge R440 Kafka main eqiad servers with buster / firmware update needed

Event Timeline

elukey created this task.Nov 29 2021, 3:20 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 29 2021, 3:20 PM

MoritzMuehlenhoff added a subscriber: MoritzMuehlenhoff.Nov 29 2021, 3:28 PM

Change 742926 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] install_server: add reuse recipe for kafka-main hosts

https://gerrit.wikimedia.org/r/742926

gerritbot added a project: Patch-For-Review.Dec 1 2021, 9:29 AM

Another interesting thing to consider is:

===== NODE GROUP =====                                                                                                                                                                                                                  
(4) kafka-main[2001-2003].codfw.wmnet,kafka-main1001.eqiad.wmnet                                                                                                                                                                        
----- OUTPUT of 'id kafka' -----                                                                                                                                                                                                        
uid=497(kafka) gid=498(kafka) groups=498(kafka)                                                                                                                                                                                         
===== NODE GROUP =====                                                                                                                                                                                                                  
(26) kafka-jumbo[1001-1009].eqiad.wmnet,kafka-logging[2001-2003].codfw.wmnet,kafka-logging[1001-1003].eqiad.wmnet,kafka-main[2004-2005].codfw.wmnet,kafka-main[1002-1005].eqiad.wmnet,kafka-test[1006-1010].eqiad.wmnet                 
----- OUTPUT of 'id kafka' -----                                                                                                                                                                                                        
uid=499(kafka) gid=499(kafka) groups=499(kafka)

So the reimage, even if we preserve /srv, will likely end up in kafka files owned by the wrong user (the old kafka id before the os reinstall). A chown -R easily solves the issue, but we could think about standardizing the kafka user in puppet and apply the common id across our clusters.

Change 742926 merged by Elukey:

[operations/puppet@production] install_server: add reuse recipe for kafka-main hosts

https://gerrit.wikimedia.org/r/742926

Maintenance_bot removed a project: Patch-For-Review.Dec 1 2021, 11:10 AM

Change 742964 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] install_server: rename reuse-kafka-main recipe

https://gerrit.wikimedia.org/r/742964

gerritbot added a project: Patch-For-Review.Dec 1 2021, 2:06 PM

Change 742964 merged by Elukey:

[operations/puppet@production] install_server: rename reuse-kafka-main recipe

https://gerrit.wikimedia.org/r/742964

The kafka-main[12]00[45] nodes running buster don't have, afaics, any specific bits related to buster, so there seems to be no need to do anything else than reimage the nodes.

Change 742969 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] install_server: set test reuse recipe for kafka-main2003

https://gerrit.wikimedia.org/r/742969

The layout of files under /srv is pretty simple, the kafka dir is the only relevant one and all files are owned by kafka:kafka. A quick chown -R kafka:kafka should be sufficient after the reimage.

An alternative could be to:

Select a fixed uid/gid in puppet's admin module (added commented)
Create a profile::kafka::user that is enabled by a hiera flag, to deploy a kafka user with fixed uid/gid.
Disable puppet on the target node to reimage, and enable the hiera flag for it.
Stop/mask kafka, modify kafka user/group (via usermod etc..), reimage

After the above the host should come up without any need of chown. When all brokers will transition to the new scheme the profile::kafka::user will not be needed.

Change 742969 merged by Elukey:

[operations/puppet@production] install_server: set test reuse recipe for kafka-main2003

https://gerrit.wikimedia.org/r/742969

Change 743130 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] admin: reserve kafka uid/gid

https://gerrit.wikimedia.org/r/743130

Change 743150 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Add profile::kafka::user to Kafka Brokers and Mirror Makers instances

https://gerrit.wikimedia.org/r/743150

Change 743163 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Move kafka test to fixed gid/uid for user kafka

https://gerrit.wikimedia.org/r/743163

@elukey would it facilitate the work if we disabled eventgate-main in eqiad during the work?

In T296641#7543719, @Joe wrote:

@elukey would it facilitate the work if we disabled eventgate-main in eqiad during the work?

I would keep eventgate pooled while doing the maintenance, on paper there shouldn't be any impact to it since a good kafka client should behave fine in these uses cases. If you are ok I'd keep ready-to-execute depool commands (if we start from main-codfw, to depool eventgate main codfw etc..) and observe eventgate's behavior during the first reimage. If it fails for any reason then it means something is really wrong and needs to fixed (sort of chaos engineering, better to see it failing while we are all watching rather than during a weekend). Lemme know your thoughts :)

Change 743130 merged by Elukey:

[operations/puppet@production] admin: reserve kafka uid/gid

https://gerrit.wikimedia.org/r/743130

elukey@kafka-main1001:~$ sudo find / \( -path /proc -o -path /mnt -o -path /sys -o -path /dev -o -path /media -o -path /srv/kafka/data \) -prune -false -o -user kafka
/etc/kafka/ssl
/etc/kafka/ssl/kafka_main-eqiad_broker.keystore.jks
/etc/kafka/ssl/truststore.jks
/etc/kafka/mirror
/etc/kafka/mirror/ssl
/etc/kafka/mirror/ssl/kafka_mirror_maker.keystore.jks
/etc/kafka/mirror/ssl/truststore.jks
/var/log/kafka
/var/log/kafka/controller.log.2
/var/log/kafka/controller.log.4
/var/log/kafka/controller.log.1
/var/log/kafka/server.log.3
/var/log/kafka/kafka-mirror-main-codfw_to_main-eqiad@0.log.1
/var/log/kafka/kafkaServer-gc.log.0
/var/log/kafka/controller.log.3
/var/log/kafka/server.log
/var/log/kafka/server.log.1
/var/log/kafka/kafkaServer-gc.log.3
/var/log/kafka/kafka-mirror-main-codfw_to_main-eqiad@0.log
/var/log/kafka/server.log.2
/var/log/kafka/kafka-request.log
/var/log/kafka/state-change.log
/var/log/kafka/kafka-authorizer.log
/var/log/kafka/kafkaServer-gc.log.2.current
/var/log/kafka/kafkaServer-gc.log.4
/var/log/kafka/kafkaServer-gc.log.1
/var/log/kafka/controller.log
/var/log/kafka/log-cleaner.log
/srv/kafka/data
/tmp/snappy-1.1.7-a8408a72-adcb-4d31-9219-e7b5addb32db-libsnappyjava.so
/tmp/hsperfdata_kafka
/tmp/hsperfdata_kafka/89989
/tmp/hsperfdata_kafka/88568
/tmp/snappy-1.1.7-1847af45-909a-4847-9230-5e99b73641d8-libsnappyjava.so

We can reuse something like T231067#6851675 to chown uid/gid automatically, node by node.

#!/bin/bash

set -x

change_uid() {
    # $1 new uid
    # $2 username
    if id "$2" &>/dev/null
    then
        OLD_UID=$(id -u $2)
        usermod -u $1 $2
        find / \( -path /proc -o -path /mnt -o -path /sys -o -path /dev -o -path /media \) -prune -false -o -user $OLD_UID -print0 | xargs -0 chown $1
    fi
}

change_gid() {
    # $1 new gid
    # $2 username
    if getent group $2 &>/dev/null
    then
        OLD_GID=$(getent group $2 | cut -d ":" -f 3)
        groupmod -g $1 $2
        find / \( -path /proc -o -path /mnt -o -path /sys -o -path /dev -o -path /media \) -prune -false -o -group $OLD_GID -print0  | xargs -0 chgrp $1
    fi
}

## hdfs


change_uid 916 kafka
change_gid 916 kafka

Change 743150 merged by Elukey:

[operations/puppet@production] profile::kafka::broker: allow to specify kafka uid/gid

https://gerrit.wikimedia.org/r/743150

Change 743163 merged by Elukey:

[operations/puppet@production] Move kafka test to fixed gid/uid for user kafka

https://gerrit.wikimedia.org/r/743163

The kafka-main codfw cluster is running with fixed gid/uid now, next steps:

sudo cookbook sre.hosts.reimage --os buster -t T296641 kafka-main2003
sudo cookbook sre.hosts.reimage --os buster -t T296641 kafka-main2002
sudo cookbook sre.hosts.reimage --os buster -t T296641 kafka-main2001

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kafka-main2003.codfw.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kafka-main2003.codfw.wmnet with OS buster executed with errors:

kafka-main2003 (FAIL)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- The reimage failed, see the cookbook logs for the details

This task is not actionable until the firmware of kafka-main2003 is updated, see task T297422

Papaul closed subtask T297422: Installation issues on PowerEdge R440 Kafka main codfw servers with buster / firmware update needed as Resolved.Dec 15 2021, 5:45 PM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kafka-main2003.codfw.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kafka-main2003.codfw.wmnet with OS buster completed:

kafka-main2003 (WARN)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202112161031_elukey_30224_kafka-main2003.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

elukey reopened subtask T297422: Installation issues on PowerEdge R440 Kafka main codfw servers with buster / firmware update needed as Open.Dec 16 2021, 11:16 AM

The BIOS+NIC upgrade for kafka-main2003 worked, it is now running buster. The partman reuse recipe and the fixed uid/gid worked as well.

Next steps:

Upgrade BIOS+NIC on kafka-main200[1,2] (see subtask)
Reimage the nodes to Buster
Rollout the fixed uid/gid change to Kafka Main eqiad
Upgrade BIOS+NIC on kafka-main100[1-3]
Reimage the nodes to Buster

Papaul closed subtask T297422: Installation issues on PowerEdge R440 Kafka main codfw servers with buster / firmware update needed as Resolved.Dec 16 2021, 5:11 PM

Thanks to Papaul, kafka-main200[12] are now ready to be reimaged :)

Change 747999 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] install_server: set the reuse recipe for all kafka-main hosts

https://gerrit.wikimedia.org/r/747999

Change 747999 merged by Elukey:

[operations/puppet@production] install_server: set the reuse recipe for all kafka-main hosts

https://gerrit.wikimedia.org/r/747999

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kafka-main2002.codfw.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kafka-main2002.codfw.wmnet with OS buster completed:

kafka-main2002 (PASS)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202112170923_elukey_29923_kafka-main2002.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kafka-main2001.codfw.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kafka-main2001.codfw.wmnet with OS buster completed:

kafka-main2001 (PASS)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202112171421_elukey_4434_kafka-main2001.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is optimal
- Icinga downtime removed
- Updated Netbox data from PuppetDB

Kafka main codfw on buster!

Next steps:

Rollout the fixed uid/gid change to Kafka main eqiad
Upgrade BIOS+NIC on kafka-main100[1-3]
Reimage the nodes to Buster

elukey added a subtask: T298867: Installation issues on PowerEdge R440 Kafka main eqiad servers with buster / firmware update needed.Jan 10 2022, 9:28 AM

Change 752613 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::kafka::main: apply kafka fixed uid/gid to eqiad

https://gerrit.wikimedia.org/r/752613

Mentioned in SAL (#wikimedia-operations) [2022-01-10T10:38:51Z] <elukey> stop/start kafka daemons on kafka-main1* nodes to move the kafka user to fixed uid/gid - T296641

Change 752613 merged by Elukey:

[operations/puppet@production] role::kafka::main: apply kafka fixed uid/gid to eqiad

https://gerrit.wikimedia.org/r/752613

Next steps:

Upgrade BIOS+NIC on kafka-main100[1-3] - T298867
Reimage the nodes to Buster

Cmjohnson closed subtask T298867: Installation issues on PowerEdge R440 Kafka main eqiad servers with buster / firmware update needed as Resolved.Jan 12 2022, 4:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kafka-main1003.eqiad.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kafka-main1003.eqiad.wmnet with OS buster completed:

kafka-main1003 (WARN)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201130808_elukey_10279_kafka-main1003.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kafka-main1002.eqiad.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kafka-main1002.eqiad.wmnet with OS buster completed:

kafka-main1002 (WARN)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201130926_elukey_22659_kafka-main1002.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host kafka-main1001.eqiad.wmnet with OS buster

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host kafka-main1001.eqiad.wmnet with OS buster completed:

kafka-main1001 (WARN)
- Downtimed on Icinga
- Disabled Puppet
- Removed from Puppet and PuppetDB if present
- Deleted any existing Puppet certificate
- Removed from Debmonitor if present
- Forced PXE for next reboot
- Host rebooted via IPMI
- Host up (Debian installer)
- Host up (new fresh buster OS)
- Generated Puppet certificate
- Signed new Puppet certificate
- Run Puppet in NOOP mode to populate exported resources in PuppetDB
- Found Nagios_host resource for this host in PuppetDB
- Downtimed the new host on Icinga
- First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201131029_elukey_21705_kafka-main1001.out
- Checked BIOS boot parameters are back to normal
- Rebooted
- Automatic Puppet run was successful
- Forced a re-check of all Icinga services for the host
- Icinga status is not optimal, downtime not removed
- Updated Netbox data from PuppetDB

All nodes on buster!

elukey closed subtask T296982: Move kafka clusters to fixed uid/gid as Resolved.Jan 13 2022, 4:37 PM

elukey reopened subtask T296982: Move kafka clusters to fixed uid/gid as Open.Jan 28 2022, 7:09 AM

elukey closed subtask T296982: Move kafka clusters to fixed uid/gid as Resolved.May 24 2022, 12:52 PM

elukey mentioned this in T332013: Migrate kafka-main to bullseye.Mar 17 2023, 2:52 PM

Maintenance_bot removed a project: Patch-For-Review.Mar 17 2023, 3:10 PM

Upgrade kafka-main nodes to busterClosed, ResolvedPublicActions

Description

Details

Related ObjectsSearch...

Event Timeline

Upgrade kafka-main nodes to buster
Closed, ResolvedPublic
Actions

Related Objects
Search...