install db1207-db1225
Open, MediumPublic
Actions

Assigned To

Authored By

	RobH
	Jan 10 2023, 5:42 PM

Details

Project	Branch	Lines +/-	Subject
operations/cookbooks	master	+13 -8	sre.hosts.provision: fix NIC link detection
operations/cookbooks	master	+3 -2	sre.hosts.provision: handle the case of no NICs
operations/puppet	production	+1 -9	mariadb: Adjust new eqiad hosts

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
			Unknown Object (Task)
Open		Papaul	T326661 Q3:rack/setup/install db1207-db1225
Open		• Marostegui	T326669 Productionize db1206-db1225
Open		• Marostegui	T326683 Decommission db1106-db1225
Stalled	Request	• Marostegui	T327616 decommission db1106.eqiad.wmnet
Open		BTullis	T304492 Upgrade db1108 to Bullseye

Event Timeline

RobH created this task.Jan 10 2023, 5:42 PM

RobH added a parent task: Unknown Object (Task).

RobH moved this task from Backlog to Racking Tasks on the ops-eqiad board.

RobH removed a subscriber: RobH.

• Marostegui renamed this task from Q3:rack/setup/install db1207-db1225 to Q3:rack/setup/install db1207-db1229.Jan 10 2023, 5:48 PM

• Marostegui updated the task description. (Show Details)

RobH renamed this task from Q3:rack/setup/install db1207-db1229 to Q3:rack/setup/install db1207-db1225.Jan 10 2023, 6:14 PM

RobH updated the task description. (Show Details)

• Marostegui moved this task from Triage to Blocked on the DBA board.Jan 10 2023, 7:32 PM

Change 878182 had a related patch set uploaded (by Marostegui; author: Marostegui):

[operations/puppet@production] mariadb: Adjust new eqiad hosts

https://gerrit.wikimedia.org/r/878182

gerritbot added a project: Patch-For-Review.Jan 10 2023, 7:39 PM

Change 878182 merged by Marostegui:

[operations/puppet@production] mariadb: Adjust new eqiad hosts

https://gerrit.wikimedia.org/r/878182

Maintenance_bot removed a project: Patch-For-Review.Jan 10 2023, 8:29 PM

Jclark-ctr updated the task description. (Show Details)Feb 15 2023, 2:46 PM

Any ETA to get these (or some) racked and installed? Thanks!

I am in process of racking right now will have them finished being racked and cabled in the next day or so

Amazing, thank you!

db1207 a5 u22 Port 40 Cableid 2570
db1208 a5 u23 Port 41 Cableid 1880
db1209 a6 u24 Port 36 Cableid 1918
db1210 a6 u25 Port 41 Cableid 1949
db1211 b5 u27 Port 16 Cableid 3283
db1212 b5 u28 Port 17 Cableid 3282
db1213 b6 u37 Port 42 Cableid 23000001
db1214 b6 u38 Port 41 Cableid 23000012
db1215 b3 u38 Port 27 Cableid 1944
db1216 b3 u39 Port 14 Cableid 5236
db1217 c5 u39 Port 29 Cableid 4011
db1218 c5 u40 Port 28 Cableid 4010
db1219 c6 u29 Port 29 Cableid 1946
db1220 c6 u30 Port 26 Cableid 3248
db1221 d1 u34 Port 20 Cableid 3613
db1222 d3 u26 Port 26 Cableid 1961
db1223 d3 u27 Port 39 Cableid 5089
db1224 d6 u37 Port 37 Cableid 23000046
db1225 d6 u38 Port 38 Cableid 23000007

@Cmjohnson can you assist with next steps of these?

Jclark-ctr moved this task from Racking Tasks to Remote Work on the ops-eqiad board.Wed, Mar 15, 3:41 PM

Papaul mentioned this in T332523: Eqiad: Backlog HW failure-racking tasks - Decommision and remote work tasks .Sun, Mar 19, 8:38 PM

Papaul claimed this task.Tue, Mar 28, 5:19 PM

While running the provision cookbook on 2 of the db nodes (db1207 and db1208) and gerrit1003 i am getting the error .

Raised while handling: The `choices` argument is empty and no custom validator was provided.
Failed to run cookbooks.sre.hosts.provision.ProvisionRunner._config: The `choices` argument is empty and no custom validator was provided.

Papaul updated the task description. (Show Details)Tue, Mar 28, 10:53 PM

@Papaul please do not reimage db1206, that host is already in production. We bought it in advance to test the raid controller as it's a new one. So it's serving traffic.

@Marostegui it is db1207 and db1208 not db1206.

Great, as you mentioned db1206 earlier I got scared :)

@Volans

100.0% (1/1) success ratio (>= 100.0% threshold) for command: '/usr/local/sbin/...cludes -r commit'.
100.0% (1/1) success ratio (>= 100.0% threshold) of nodes successfully executed all commands.
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host db1207.mgmt.eqiad.wmnet with reboot policy FORCED
pt1979@cumin2002:~$ sudo cookbook sre.hosts.provision db1207 --no-dhcp --no-users
Management Password:
Testing Redfish API connection to cumin2002 (10.193.0.139)
==> Are you sure to proceed to apply BIOS/iDRAC settings for host db1207.mgmt.eqiad.wmnet with reboot policy FORCED?
Type "go" to proceed or "abort" to interrupt the execution
> go
User input is: "go"
START - Cookbook sre.hosts.provision for host db1207.mgmt.eqiad.wmnet with reboot policy FORCED
Testing Redfish API connection to db1207 (10.65.1.14)
==> Detected Hardware RAID. Please configure the RAID at this point (the password is still DELL default one). Once done select "modified" if the RAID was modified or "untouched" if it was not touched. If the RAID was modified the host will be rebooted to make sure the changes are applied.
> modified
User input is: "modified"
Rebooting the host with policy ChassisResetPolicy.FORCE_RESTART and waiting for 3 minutes
Resetting chassis power status for db1207 to ForceRestart
Testing Redfish API connection to db1207 (10.65.1.14)
[IDRAC.2.7.SYS057] Exporting Server Configuration Profile.
[1/30, retrying in 30.00s] Polling task: JID_800971819368 not completed yet: status=OK, state=Running, completed=10%
First attempt to load the new configuration failed, auto-retrying once
Testing Redfish API connection to db1207 (10.65.1.14)
[IDRAC.2.7.SYS057] Exporting Server Configuration Profile.
[1/30, retrying in 30.00s] Polling task: JID_800972141788 not completed yet: status=OK, state=Running, completed=10%
Raised while handling: The `choices` argument is empty and no custom validator was provided.
Failed to run cookbooks.sre.hosts.provision.ProvisionRunner._config: The `choices` argument is empty and no custom validator was provided.
==> What do you want to do? "retry" the last command, manually fix the issue and "skip" the last command to continue the execution or completely "abort" the execution.

Change 904201 had a related patch set uploaded (by Volans; author: Volans):

[operations/cookbooks@master] sre.hosts.provision: handle the case of no NICs

https://gerrit.wikimedia.org/r/904201

gerritbot added a project: Patch-For-Review.Wed, Mar 29, 3:43 PM

Change 904201 merged by jenkins-bot:

[operations/cookbooks@master] sre.hosts.provision: handle the case of no NICs

https://gerrit.wikimedia.org/r/904201

Maintenance_bot removed a project: Patch-For-Review.Wed, Mar 29, 4:10 PM

Change 904235 had a related patch set uploaded (by Volans; author: Volans):

[operations/cookbooks@master] sre.hosts.provision: fix NIC link detection

https://gerrit.wikimedia.org/r/904235

gerritbot added a project: Patch-For-Review.Wed, Mar 29, 4:59 PM

Change 904235 merged by jenkins-bot:

[operations/cookbooks@master] sre.hosts.provision: fix NIC link detection

https://gerrit.wikimedia.org/r/904235

Maintenance_bot removed a project: Patch-For-Review.Wed, Mar 29, 5:10 PM

After the switch configuration step I get the output below and

Testing Redfish API connection to db1209 (10.65.1.88)
Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1ba0fcf7f0>, 'Connection to 10.65.1.88 timed out. (connect timeout=10)')': /redfish
Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1ba0fcf130>, 'Connection to 10.65.1.88 timed out. (connect timeout=10)')': /redfish
Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x7f1ba0fcf070>, 'Connection to 10.65.1.88 timed out. (connect timeout=10)')': /redfish
Failed to run cookbooks.sre.hosts.provision.ProvisionRunner.run.<locals>.check_connection: Unable to connect to the Redfish API of db1209. Follow https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Dell_Documentation#Troubleshooting_2

Q3:rack/setup/install db1207-db1225
Open, MediumPublic
Actions

Description

Hostname / Racking / Installation Details

Per host setup checklist

db1207:

db1208:

db1209:

db1212:

db121:

db1213:

db1214:

db1215:

db1216:

db1217:

db1218:

db1219:

db1220:

db1221:

db1222:

db1223:

db1224:

db1225:

Details

Related Objects
Search...

Event Timeline

Q3:rack/setup/install db1207-db1225Open, MediumPublicActions

Description

Hostname / Racking / Installation Details

Per host setup checklist

db1207:

db1208:

db1209:

db1212:

db121:

db1213:

db1214:

db1215:

db1216:

db1217:

db1218:

db1219:

db1220:

db1221:

db1222:

db1223:

db1224:

db1225:

Details

Related ObjectsSearch...

Event Timeline

Q3:rack/setup/install db1207-db1225
Open, MediumPublic
Actions

Related Objects
Search...