b6-eqiad pdu refresh (Tuesday 9/10 @11am UTC)
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	RobH
	Jul 8 2019, 10:46 PM

Description

This task will track the migration of the ps1 and ps2 to be replaced with new PDUs in rack B6-eqiad.

Each server & switch will need to have potential downtime scheduled, since this will be a live power change of the PDU towers.

These racks have a single tower for the old PDU (with and A and B side), with the new PDUs having independent A and B towers.

- schedule downtime for the entire list of switches and servers.
- Wire up one of the two towers, energize, and relocate power to it from existing/old pdu tower (now de-energized).
- confirm entire list of switches, routers, and servers have had their power restored from the new pdu tower
- Once new PDU tower is confirmed online, move on to next steps.
- Wire up remaining tower, energize, and relocate power to it from existing/old pdu tower (now de-energized).
- confirm entire list of switches, routers, and servers have had their power restored from the new pdu tower

List of routers, switches, and servers

device	role	SRE team coordination
asw2-b6-eqiad
mc1027	mc host	serviceops ok to go with patch ready for easy removal
mc1026	mc host	serviceops ok to go with patch ready for easy removal
mc1025	mc host	serviceops ok to go with patch ready for easy removal
mc1024	mc host	serviceops ok to go with patch ready for easy removal
cloudvirt1029	hypervisor	@aborrero - good to go.
scb1002	scb cluster	serviceops - if poweroff, run `depool` as root first
wdqs1009	@Gehel good to go
aqs1008	Analytics	fine to do any time
kubernetes1002	kubernetes node	poweroff/poweron serviceops node must be put out of rotation
puppetmaster1001	puppetmaster frontend	https://wikitech.wikimedia.org/wiki/Puppet#Pool_/_depool_a_frontend @jbond @herron @colewhite
elastic1028	elastic	@Gehel good to go
elastic1047	elastic	@Gehel good to go
elastic1046	elastic	@Gehel good to go
mw1306	mw jobrunner	serviceops - good to go
mw1305	mw jobrunner	serviceops - good to go
mw1304	mw jobrunner	serviceops - good to go
mw1303	mw jobrunner	serviceops - good to go
mw1302	mw jobrunner	serviceops - good to go
mw1301	mw jobrunner	serviceops - good to go
mw1300	mw jobrunner	serviceops - good to go
mw1299	mw jobrunner	serviceops - good to go
mw1298	mw jobrunner	serviceops - good to go
mw1297	mw jobrunner	serviceops - good to go
mw1296	mw jobrunner	serviceops - good to go
mw1295	mw jobrunner	serviceops - good to go
mw1294	mw jobrunner	serviceops - good to go
mw1293	mw jobrunner	serviceops - good to go
thumbor1002	thumbor host	depool setting `pooled=inactive` serviceops
thumbor1001	thumbor host	depool setting `pooled=inactive` serviceops
mw1290	mw API host	serviceops - good to go
mw1289	mw API host	serviceops - good to go
mw1288	mw API host	serviceops - good to go
mw1287	mw API host	serviceops - good to go
mw1286	mw API host	serviceops - good to go
mw1285	mw API host	serviceops - good to go
mw1284	mw API host	serviceops - good to go

Details

Related Changes in Gerrit:

	Subject	Repo	Branch	Lines +/-
	facilities: ps1-b6-eqiad replaced with newer PDU	operations/puppet	production	+4 -3

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		• Cmjohnson	T226778 Install new PDUs in rows A/B (Top level tracking task)
		Resolved		• Cmjohnson	T227541 b6-eqiad pdu refresh (Tuesday 9/10 @11am UTC)

Event Timeline

RobH created this task.Jul 8 2019, 10:46 PM

RobH renamed this task from b5-eqiad pdu refresh to b6-eqiad pdu refresh.

RobH updated the task description. (Show Details)

RobH mentioned this in T226778: Install new PDUs in rows A/B (Top level tracking task).Jul 8 2019, 10:50 PM

• Cmjohnson moved this task from Backlog to High Priority Task on the ops-eqiad board.Jul 22 2019, 2:41 PM

• aborrero updated the task description. (Show Details)Jul 24 2019, 10:23 AM

• aborrero subscribed.

RobH moved this task from High Priority Task to Blocked on the ops-eqiad board.Jul 26 2019, 1:37 PM

akosiaris updated the task description. (Show Details)Aug 5 2019, 4:08 PM

RobH updated the task description. (Show Details)Aug 6 2019, 9:57 AM

• JHedden subscribed.Aug 6 2019, 4:08 PM

wiki_willy renamed this task from b6-eqiad pdu refresh to b6-eqiad pdu refresh (Tuesday 9/10 @11am UTC).Aug 15 2019, 5:37 PM

Gehel updated the task description. (Show Details)Aug 19 2019, 4:11 PM

Gehel subscribed.

colewhite updated the task description. (Show Details)Aug 26 2019, 4:38 PM

colewhite updated the task description. (Show Details)

colewhite added subscribers: jbond, herron.

colewhite subscribed.

• Phamhi subscribed.Aug 27 2019, 4:10 PM

Removing myself as assignee since this has all the servers populated in the task description.

mw and thumbor hosts @Joe stated he would add a followup comment in last weeks meeting.

Joe updated the task description. (Show Details)Sep 2 2019, 5:38 AM

Joe updated the task description. (Show Details)

jbond triaged this task as Medium priority.Sep 9 2019, 9:15 AM

wiki_willy assigned this task to • Cmjohnson.Sep 9 2019, 3:56 PM

Mentioned in SAL (#wikimedia-operations) [2019-09-10T11:20:54Z] <cmjohnson1> swapping the PDU in rack B6 eqiad T227541

• Cmjohnson updated the task description. (Show Details)Sep 10 2019, 12:42 PM

The PDU has been swapped and the new pdus are in netbox. @RobH can you help with the setup for serial console please.

Change 536140 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] facilities: ps1-b6-eqiad replaced with newer PDU

https://gerrit.wikimedia.org/r/536140

gerritbot added a project: Patch-For-Review.Sep 12 2019, 10:14 AM

Change 536140 merged by Filippo Giunchedi:
[operations/puppet@production] facilities: ps1-b6-eqiad replaced with newer PDU

https://gerrit.wikimedia.org/r/536140

Maintenance_bot removed a project: Patch-For-Review.Sep 12 2019, 11:10 AM

Trying to figure out why this is failing: https://icinga.wikimedia.org/cgi-bin/icinga/status.cgi?search_string=ps1-b6-eqiad
error is:

External command error: Error in packet
Reason: (noSuchName) There is no such variable name in this MIB.
Failed object: iso.3.6.1.4.1.1718.3.2.2.1.7.1.1

My first question is, is there a tower B ?

I don't see one on https://librenms.wikimedia.org/device/device=50/ while for example https://librenms.wikimedia.org/device/device=40/ does have one
Netbox says that there is a ps2.

or is it a bug/miss configuration?

Checked with @Cmjohnson , who says he'll follow up to check the connections.

elukey updated the task description. (Show Details)Sep 17 2019, 6:02 AM

15:15 <@RobH> : So, I can confirm in librenms it sees both towers
15:15 <@RobH> : so, this seems to me to be an icinga issue
15:15 <@RobH> : Does this seem reasonable? If so, we need to likely involve someone with some icinga knowledge.

Please note this is an issue that is happening on ALL the new PDUs. I'll update the parent task.

Please note that when I compare librenms output it seems like it sees both towers right now:

ps1-b6-eqiad:
https://librenms.wikimedia.org/device/device=50/
ps1-a4-eqiad:
https://librenms.wikimedia.org/device/device=40/

Both show voltages for AA phases and BB phases (that is the 3 phases per tower).

I think this is ok to resolve, but I'm not 100% that I am looking at the same thing as @ayounsi.

It was a PDU miss-configuration and a monitoring issue. Was solved in https://phabricator.wikimedia.org/T229328

Thanks for confirming @ayounsi Resolving task.

b6-eqiad pdu refresh (Tuesday 9/10 @11am UTC)Closed, ResolvedPublicActions