| aborrero | |
| Jul 6 2023, 11:02 AM |
| F37132300: image.png | |
| Jul 7 2023, 3:58 PM |
| F37130483: image.png | |
| Jul 6 2023, 11:44 AM |
Netbox device: https://netbox.wikimedia.org/dcim/devices/3524/
Procedure: https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Rename_while_reimaging
| Subject | Repo | Branch | Lines +/- | |
|---|---|---|---|---|
| cloudlb: eqiad: bootstrap hiera data | operations/puppet | production | +18 -0 | |
| cloudlb1001/1002: add role | operations/puppet | production | +3 -4 |
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | aborrero | T296411 cloud: decide on general idea for having cloud-dedicated hardware provide service in the cloud realm & the internet | |||
| Resolved | aborrero | T297596 have cloud hardware servers in the cloud realm using a dedicated LB layer | |||
| Resolved | • taavi | T341060 openstack eqiad1: introduce cloud-private and cloudlb | |||
| Resolved | aborrero | T341061 eqiad1: repurpose 2 cloudswift servers as cloudlb | |||
| Resolved | Papaul | T341200 rename cloudswift1001 as cloudlb1001 |
cookbooks.sre.hosts.decommission executed by aborrero@cumin1001 for hosts: cloudswift1001.eqiad.wmnet
ERROR: some step on some host failed, check the bolded items above
Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudlb1001.eqiad.wmnet with OS bullseye
Change 936019 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudlb1001/1002: add role
Change 936019 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudlb1001/1002: add role
Change 936022 had a related patch set uploaded (by Arturo Borrero Gonzalez; author: Arturo Borrero Gonzalez):
[operations/puppet@production] cloudlb: eqiad: bootstrap hiera data
Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudlb1001.eqiad.wmnet with OS bullseye executed with errors:
T341223: Configure eqiad cloudsw devices to support cloud-private is no longer a blocker.
Now blocked by some kind of DHCP error preventing the boot into the installer, apparently. I need to research more.
Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudlb1001.eqiad.wmnet with OS bullseye
The host can only do DHCP boot while the reimage cookbook is running. This wasn't really any issue.
Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudlb1001.eqiad.wmnet with OS bullseye executed with errors:
hey @Papaul or @Jclark-ctr I'm requesting help with this host.
We are trying to reimage after renaming it from cloudswift1001 to cloudlb1001.
The debian installer shows up, but then it complained about a disks and root partition.
I checked on the iDrac GUI and found this warning:
I run the two commands in there, but now the reimage cookbook fails with:
aborrero@cumin1001:~ $ sudo cookbook sre.hosts.reimage --new --os bullseye --task-id T341200 cloudlb1001 ==> ATTENTION: destructive action for host: cloudlb1001 Are you sure to proceed? Type "go" to proceed or "abort" to interrupt the execution > go User input is: "go" Management Password: Running IPMI command: ipmitool -I lanplus -H cloudlb1001.mgmt.eqiad.wmnet -U root -E chassis power status Error: Unable to establish IPMI v2 / RMCP+ session Exception raised while initializing the Cookbook sre.hosts.reimage: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/spicerack/ipmi.py", line 85, in command output = run(command + command_parts, env=self.env.copy(), stdout=PIPE, check=True).stdout.decode() File "/usr/lib/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['ipmitool', '-I', 'lanplus', '-H', 'cloudlb1001.mgmt.eqiad.wmnet', '-U', 'root', '-E', 'chassis', 'power', 'status']' returned non-zero exit status 1.
Trying by hand:
aborrero@cumin1001:~$ ipmitool -I lanplus -H cloudlb1001.mgmt.eqiad.wmnet -U root -E chassis power status Unable to read password from environment Password: Error: Unable to establish IPMI v2 / RMCP+ session
Maybe this is something you know how to fix rather quick instead of me messing around. Help is appreciated.
Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host cloudlb1001.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host cloudlb1001.eqiad.wmnet with OS bullseye executed with errors:
@aborrero the issue was that IPMI was disable on the node I enable it and try to install the OS, the installation completed but failed for some other issue that i didn't take time to look into. So you can try to reimage again.
Change 936022 merged by Arturo Borrero Gonzalez:
[operations/puppet@production] cloudlb: eqiad: bootstrap hiera data
Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1001 for host cloudlb1001.eqiad.wmnet with OS bullseye
Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1001 for host cloudlb1001.eqiad.wmnet with OS bullseye completed:
