Due to the recent codfw migration, we've placed codfw under normal load conditions. The result is some of the PDU's are complaining about a phase imbalance.
With 3 phase power, please note power comes in on xyz phases, and is split into three banks: xy, yz, xz. The load (which is not the same as number of severs, but is closely related) must be spread evenly between these three power plug banks on each pdu tower.
SNMP WARNING - ps1-a3-codfw-infeed-load-tower-A-phase-Z *1588*
The tower A load breakdown breakdown is: X: 4.59 Amps Y: 3.74 Amps Z: 5.70 Amps
So we need to pick a single server on bank XZ and move it to bank XY. The loads are fairly close, so one server may do it. Once it's moved, we can check the balance again. Please list off the ideal server for the move, and we'll review before the actual power cable move.
SNMP WARNING - ps1-c6-codfw-infeed-load-tower-A-phase-X *1275*
SNMP WARNING - ps1-c6-codfw-infeed-load-tower-B-phase-X *1362*
Tower A Loads: X 12.71 Y 9.00 Z 8.23
Tower B Loads: X 13.58 Y 8.97 Z 8.40
So these have a very high load on X, and then a slightly higher on Y. It looks like a large server should be relocated off of XY and moved onto YZ for each tower (the same server). The imbalance is quite large, so check and see if perhaps we have new servers in this rack that aren't pulling under load? That would explain it some. Please list off the ideal server for the move, and we'll review before the actual power cable move.
SNMP WARNING - ps1-d6-codfw-infeed-load-tower-A-phase-X *1452
SNMP WARNING - ps1-d6-codfw-infeed-load-tower-B-phase-X *1461*
Tower A Loads: X 14.50 Y 8.41 Z 9.42
Tower B Loads: X 14 Y 7 Z 9
So X and Z are high, while Y is lower. I'd pick a single server on XZ and move it to YZ. The imbalance is quite large, so check and see if perhaps we have new servers in this rack that aren't pulling under load? That would explain it some. Please list off the ideal server for the move, and we'll review before the actual power cable move.
These are ideally fixed while they are still under load, as it will result in the most accurate power balance within the racks. However, moving systems under load is dangerous, and should be done carefully.
- Always ensure both power supply units are working in the system before moving power plugs.
- Move one power supply at a time, allowing time between plug moves for the power supply to recover and resume providing power.
- Make sure each power supply for a system plugs into the same bank on each tower. Example (not based in fact): mw2001 is plugged into the xz bank on tower A pdu, it should be plugged into the zx bank on tower B pdu.
- Do NOT move the server, just add longer power cables to route to the proper bank.
Please work within IRC to announce each system you are moving before you move it carefully, one power supply plug at a time, so it stays online during its power plug migration.