Since these should be moved while the systems are under load, there is an inherent risk involved.

Please do not move a system's power plugs without coordination in IRC with ops =]

ps1-a3-codfw
moving mw2215

ps1-c6-codfw
moving db2083

ps1-d6-codfw
moving db2063

In T163339#3194278, @Papaul wrote:

ps1-a3-codfw
moving mw2215

This is on bank XZ and move it to bank XY?

ps1-c6-codfw
moving db2083

This is on bank XY and we'll move onto YZ?

ps1-d6-codfw
moving db2063

This is on bank XZ and we'll move it to YZ?

mw2215 is on zx
db2083 is on zx
db2063 is on xy

In T163339#3194320, @Papaul wrote:

mw2215 is on zx

This one should work. Don't move until we confirm with opsen for that service group.

db2083 is on zx

We need to move a system off XY and moved onto YZ. Please pick a system on bank XY.

db2063 is on xy

We need to move a system off XZ and moved onto YZ. Please pick a system on bank XZ.

mw2215 is on zx (a3)
db2043 is on xy (c6)
db2061 is on xz (d6)

In T163339#3194384, @Papaul wrote:

mw2215 is on zx (a3)

I've just checked with @Joe, you can move the power plugs for mw2215 now. Just move them slowly, so one has time to come back online before you unplug the other, and the system can remain online.

The db systems we still need to clear with the DBA team.

Once mw2215 is moved, check and see if icinga clears, and we can both check the PDU load directly.

In T163339#3194384, @Papaul wrote:

db2043 is on xy (c6)
db2061 is on xz (d6)

Both of those db hosts are slaves in the s3 and s7 shards, not masters. So it should be ok to move them.

I've added the DBA tag to this task for DBA input on when we can move the power cables for those two systems. They shouldn't result in downtime, but since touching power cables is risky, better safe than sorry.

Don't move these until either @jcrespo or @Marostegui approve.

Ok, mw2215 has been moved, but a3 is still unhappy:

X 14.5 Y 8.3, Z 9.6

So now X is quite high, while Z is back to a more normal rate.

In T163339#3194453, @RobH wrote:

Ok, mw2215 has been moved, but a3 is still unhappy:

X 14.5 Y 8.3, Z 9.6

So now X is quite high, while Z is back to a more normal rate.

Lets try moving another system on a3-codfw from bank xz to bank yz.

Since the X is high, but the others are close, the lower power use systems are best. So ideally if there is a 1U system with fewer disks in xz, pick it for movement to yz and comment in here so I can clear the host for move.

I have no opening on yz

moving msw-a3-codfw from yz to xy

Ok, things shifted drastically on ps1-a3-codfw

X/Y/Z are at 9/9/14.

I've suggested to @Papaul we pick one mw system off xz, and one off yz, and move them both onto xy.

They should be mw appservers or api, not imagescaler or jobrunners.

In T163339#3194442, @RobH wrote:

In T163339#3194384, @Papaul wrote:

db2043 is on xy (c6)
db2061 is on xz (d6)

Both of those db hosts are slaves in the s3 and s7 shards, not masters. So it should be ok to move them.

I've added the DBA tag to this task for DBA input on when we can move the power cables for those two systems. They shouldn't result in downtime, but since touching power cables is risky, better safe than sorry.

Don't move these until either @jcrespo or @Marostegui approve.

Let's do this next week as things get more stable and we are sure about everything working fine load-wise on codfw.
@Papaul I will ping you here to decide a day when we can do it
From our side we only need to depool it and downtime it

@Papaul which day works for you to do those two servers? Monday? As @RobH said they shouldn't have downtime, but I would prefer to depool them just in case.

@Marostegui Anytime Monday at 9:30am works for me.

In T163339#3201510, @Papaul wrote:

@Marostegui Anytime Monday at 9:30am works for me.

Let's do it Monday at 9:30AM then.

Change 349882 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Depool db2043, db2061

https://gerrit.wikimedia.org/r/349882

Change 349882 merged by jenkins-bot:
[operations/mediawiki-config@master] db-codfw.php: Depool db2043, db2061

https://gerrit.wikimedia.org/r/349882

Mentioned in SAL (#wikimedia-operations) [2017-04-24T14:14:58Z] <marostegui@naos> Synchronized wmf-config/db-codfw.php: Depool db2043 and db2061 - T163339 (duration: 01m 08s)

Mentioned in SAL (#wikimedia-operations) [2017-04-24T14:17:02Z] <marostegui> Stop MySQL db2043 and db2061 for maintenance - https://phabricator.wikimedia.org/T163339

@Papaul you can do the maintenance on db2043 and db2061 now. They have been depooled. Please let me know when it is done, so I can start MySQL and repool them
Thanks!

Change 349970 had a related patch set uploaded (by Marostegui):
[operations/mediawiki-config@master] db-codfw.php: Repool db2043 and db2061

https://gerrit.wikimedia.org/r/349970

@Marostegui we are clear for db2061

Tower A Loads: X 11.16 Y 8.61 Z 10.46
Tower B Loads: X 11.03 Y 7.93 Z 10.66

no more warning ion Icinga

In T163339#3206426, @Papaul wrote:

@Marostegui we are clear for db2061

Tower A Loads: X 11.16 Y 8.61 Z 10.46
Tower B Loads: X 11.03 Y 7.93 Z 10.66

no more warring ion Icinga

Thanks, going to bring db2061 back up.
db2043 will remain stopped until you give me the good to go

Tower A Loads: X 10.27 Y8.14 Z 10.08
Tower B Loads: X10.97 Y 7.91 Z 10.81

no more warning ion Icinga

Change 349970 merged by Marostegui:
[operations/mediawiki-config@master] db-codfw.php: Repool db2043 and db2061

https://gerrit.wikimedia.org/r/349970

Mentioned in SAL (#wikimedia-operations) [2017-04-24T16:58:16Z] <marostegui@naos> Synchronized wmf-config/db-codfw.php: Repool db2043 and db2061 with less weight - T163339 (duration: 01m 16s)

Mentioned in SAL (#wikimedia-operations) [2017-04-24T17:28:36Z] <marostegui@naos> Synchronized wmf-config/db-codfw.php: Increase db2043 and db2061 weight - T163339 (duration: 00m 58s)

@Papaul is this done? I still see a SNMP WARNING - ps1-a3-codfw-infeed-load-tower-A-phase-Z *1233* alert right now.

@faidon a3 is not done yet only c6 and d6

mw2017 plug into ps1-a3 has jut one PSU and the reading on pss1-a3 is higher than ps2-a3 I will like to power that server down to move it from ps1-a3 to ps2-a3 since ps2-a3 has lower readings than ps1-a3

The reading on both PUD's shows

ps1-a3
X=9.73 Y= 9.65 Z=12.84
ps2-a3
X=1.96 Y=9.65 z= 1.97

ps1-a3 is pulling more power than ps2-a3 as @RobH suggested on T163362we we need to check the BIOS settings to each server to see because the readings are way too low on ps2-a3

@faidon

We should be able to balance it while the other tower 2 isnt being used, it will be more difficult but should be possible. I'm just not sure its worth the trouble during codfw being active, when we can fix the bios issue and rebalance more easily when it is fixed.

But if codfw were remaining active for multiple weeks, we should fix it while live. Since we aren't staying in codfw for that long, this could wait for the failover. (but either works!)

I moved the PSU's that are pulling lest power too ps1-a3 and the once pulling more power on ps2-a3 we should be good for now.

@Papaul: That will need to be swapped back once we fix the bios on the machines. Ideally all PSU1 are pulling from PDU1(tower1) and all PSU2 are pulling from PDU2.

So once we fail back to eqiad, and work to fix the bios, since it causes downtime for that, we can simply re-balance power properly.

@RobH let me know when you want to start working on this. Next week works for me.

is this still an issue?

@RobH @faidon if this is no longer an issue can we resolve it? it has been open for more then a year now.

Thanks.

@Papaul

I'll need you to review the phases on every PDU to know if it is resolved. You can do this by pulling up each one's web interface, or simply walking the rows and auditing the phases.

Can you audit the phases and just paste the output here?

Example:
A1-PS1:X at 5 amps, Y at 5 amps, Z at 4.5 amps
A1-PS2:X at 5 amps, Y at 5 amps, Z at 4.7 amps
A2-PS1:X at 7 amps, Y at 6.9 amps, Z at 7.5 amps
A2-PS2:X at 7.2 amps, Y at 6.8 amps, Z at 7.3 amps

To clarify, this is something that all on-sites should be doing regularly to ensure we don't have any imbalances. Please go ahead and just audit all 20 racks (40 PDUs.) This is something that should be done at least annually. We should all (all on-sites) regularly check/glance at all of these readouts. Anytime there is an imbalance of more than 10% overall between phases, its grossly out of balance. I try to get them as close as possible, depending on the contents of the rack. (Its harder to fully balance a rack of heavy power use 2U machines and disk shelves than it is to balance a rack full of 1U machines.)

@Papaul: Anywhere that the phases are more than 4% out of balance from the lowest phase, they need to be rebalanced. This is something that you, as the on-site engineer for codfw, will need to review and balance every time you are adding or removing anything from the rack, as well as ensuring they are balanced overall.

Since the majority of our devices (other than msws and mr1) are redundant power, this is something that should be able to be accomplished with no downtime. Lets take rack A2 as an example:

Those phases are quite out of balance, and should be within 4-5% of one another (and the lowest power draw.) In this case, it appears that phase Y is underutilized compared to the other phases. You need to move items off the XZ phase bank and split them evenly on the XY and YZ banks for both PS1 and PS2.

Does this make sense? Please let me know if not, and if it does, please go ahead and attempt to re-balance A2 to start! Once that one is fixed, we can review and you can move onto the other racks.

If the phases are this far out of balance, and we end up losing power or cycling power on an entire rack, the phase imbalance and lead to load spikes on the overloaded phases.

@RobH
Better to do this when I am back from vacation. Thanks.

In T163339#4430979, @Papaul wrote:

@RobH
Better to do this when I am back from vacation. Thanks.

Absolutely, no reason to go mucking about for a day before leaving! This is set to low priority already, as it is more 'general housekeeping' for onsite engineers, rather than an urgent task for immediate completion.

Once you are back, you can start balancing these. Let me know when and I'm happy to remotely assist as best I can (though most of this is the case of moving a single cable and waiting a few minutes to see power states normalize across phases.)