b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	RobH
	Jul 8 2019, 10:44 PM

Description

This task will track the migration of the ps1 and ps2 to be replaced with new PDUs in rack B2-eqiad.

Each server & switch will need to have potential downtime scheduled, since this will be a live power change of the PDU towers.

These racks have a single tower for the old PDU (with and A and B side), with the new PDUs having independent A and B towers.

- schedule downtime for the entire list of switches and servers.
- Wire up one of the two towers, energize, and relocate power to it from existing/old pdu tower (now de-energized).
- confirm entire list of switches, routers, and servers have had their power restored from the new pdu tower
- Once new PDU tower is confirmed online, move on to next steps.
- Wire up remaining tower, energize, and relocate power to it from existing/old pdu tower (now de-energized).
- confirm entire list of switches, routers, and servers have had their power restored from the new pdu tower
- connect via serial / confirm serial connection works
- setup PDU following directions on https://wikitech.wikimedia.org/wiki/Platform-specific_documentation/ServerTech#Initial_Setup
- update PDU model in puppet per T233129.

List of routers, switches, and servers

device	role	SRE team coordination	notes
asw2-b2-eqiad	asw	@ayounsi
db1072	db	DBA	Host powered off, DO NOT POWER BACK ON pending onsite decommissioning steps : T228956
ms-be1020	ms-be	@fgiunchedi	poweroff / poweron
cloduvirt1002	cloudvirt	cloud-services-team	active VMs, please handle with care
cloudvirt1001	cloudvirt	cloud-services-team	active VMs, please handle with care
cloudvirt1024	cloudvirt	cloud-services-team	active VMs, but low risk
cloudvirt1018	cloudvirt	cloud-services-team	a lot of active VMs, please handle with care
cloudvirt1012	cloudvirt	cloud-services-team	active VMs, please handle with care
cloudvirt1009	cloudvirt	cloud-services-team	active VMs, please handle with care
cloudvirt1015	cloudvirt	cloud-services-team	Offline for maintenance
cloudvirt1008	cloudvirt	cloud-services-team	active VMs, please handle with care
db1099	db	DBA	@jcrespo to depool this host
ms-be1047	ms-be	@fgiunchedi	poweroff / poweron
cloudvirtan1002	cloudvirtan	@Ottomata	no longer in cloud services, not sure of the status
cloudvirtan1001	cloudvirtan	@Ottomata	no longer in cloud services, not sure of the status
an-worker1084		Analytics	fine to do any time
an-worker1083		Analytics	fine to do any time
cloudelastic1002	cloudelastic	Discovery-Search	@Gehel good to go
cp1080	cp	Traffic
cp1079	cp	Traffic
analytics1072		Analytics	fine to do any time

Details

	Subject	Repo	Branch	Lines +/-
	setting ps1-b2-eqiad model info	operations/puppet	production	+4 -3

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• Cmjohnson	T226778 Install new PDUs in rows A/B (Top level tracking task)
		Resolved		RobH	T227538 b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC)

Event Timeline

RobH created this task.Jul 8 2019, 10:44 PM

RobH mentioned this in T226778: Install new PDUs in rows A/B (Top level tracking task).Jul 8 2019, 10:50 PM

RobH updated the task description. (Show Details)Jul 10 2019, 10:49 PM

RobH added subscribers: ayounsi, fgiunchedi.

• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Jul 22 2019, 2:42 PM

cp1079 and cp1080 just need normal depooling process here.

From the DB side, this can be done after Thursday 25th as db1072 will no longer be a master

fgiunchedi updated the task description. (Show Details)Jul 23 2019, 9:33 AM

As of today, db1072 is no longer a master (T228243#5363931), so this rack is also good to go. db1072 will be decommissioned in a few days

Marostegui updated the task description. (Show Details)Jul 25 2019, 5:45 AM

RobH moved this task from High Priority Task to Blocked on the ops-eqiad board.Jul 26 2019, 1:37 PM

RobH removed RobH as the assignee of this task.Aug 14 2019, 4:53 PM

wiki_willy renamed this task from b2-eqiad pdu refresh to b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC).Aug 15 2019, 5:34 PM

CDanis triaged this task as Medium priority.Aug 16 2019, 1:01 PM

@jcrespo - I will be on holidays this day, hence I added you on the list as primary contact for db1099 :)

Gehel updated the task description. (Show Details)Aug 19 2019, 4:18 PM

Gehel subscribed.

elukey updated the task description. (Show Details)Sep 17 2019, 5:58 AM

elukey updated the task description. (Show Details)

Marostegui updated the task description. (Show Details)Sep 25 2019, 1:42 PM

RobH updated the task description. (Show Details)Oct 11 2019, 8:41 PM

Marostegui updated the task description. (Show Details)Oct 24 2019, 6:10 AM

wiki_willy assigned this task to • Cmjohnson.Oct 28 2019, 5:21 PM

• JHedden removed • Cmjohnson as the assignee of this task.Oct 28 2019, 5:24 PM

• JHedden updated the task description. (Show Details)

• JHedden added a subscriber: Ottomata.

wiki_willy assigned this task to • Cmjohnson.Oct 28 2019, 6:03 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-29T08:43:35Z] <jynus> shutting down db1099 T227538

db1099 is depooled and down, please ping me on IRC when done so I can put it up. dbs area all ready for maintenance.

Mentioned in SAL (#wikimedia-cloud) [2019-10-29T10:52:51Z] <arturo> icinga downtime cloudvirt1001/1002/1024/1018/1012/1009/1015/1008 for 1h T227538

starting pdu refresh

Finished with swapping of pdu`s

netbox updated/ serial connected

Mentioned in SAL (#wikimedia-operations) [2019-10-29T14:45:56Z] <robh> setting up ps1-b2-eqiad, librenms will output a couple reboots from it T227538

Change 546959 had a related patch set uploaded (by RobH; owner: RobH):
[operations/puppet@production] setting ps1-b2-eqiad model info

https://gerrit.wikimedia.org/r/546959

Change 546959 merged by RobH:
[operations/puppet@production] setting ps1-b2-eqiad model info

https://gerrit.wikimedia.org/r/546959

RobH removed a project: Patch-For-Review.Oct 29 2019, 3:03 PM

RobH updated the task description. (Show Details)

I will repool the db host, as I do not want to leave them off for a long time. Let me know if more operations on this row are planned at a later time.

I've completed all setup work from the software side for the PDU at this time.

all green in icinga, resolving task

b2-eqiad pdu refresh (Tuesday 10/29 @11am UTC)Closed, ResolvedPublicActions