Page MenuHomePhabricator

Re-image sca1001, sca1002, sca2001, sca2002, as scb1003, scb1004 and scb2003, scb2004 respectively
Closed, ResolvedPublic

Description

Now that sca[12]00[34] have been added in T147409, we can reuse the hardware boxes sca1001, sca1002, sca2001, sca2002 as SCB boxes to increase that cluster's capacity. Reimage these boxes as scb1003, scb1004, scb2003, sc2004 respectively. Tasks

  • DNS changes
  • PXE/DHCP/install server changes
  • switchport descriptions
  • DC labels on the boxes themselves
  • Pooling the boxes into the various services

Details

Related Gerrit Patches:

Event Timeline

Restricted Application added a project: Operations. · View Herald TranscriptOct 17 2016, 9:01 AM
Restricted Application added subscribers: Southparkfan, Aklapper. · View Herald Transcript
mobrovac added a subscriber: mobrovac.

What is the ETA for this? If we are to install Node 4.6 on it right away then this is effectively blocked by T147849: ChangeProp failing on Node v4.6.0

I have the old 4.4.6 binaries still around, we can downgrade scb100[34] until T147849 is fixed. Or we could disable changeprop on those two new machines and stick with 4.6.

I have the old 4.4.6 binaries still around, we can downgrade scb100[34] until T147849 is fixed. Or we could disable changeprop on those two new machines and stick with 4.6.

We'd need to do the former, because we can't currently build repos for two different node versions at the same time.

What is the ETA for this? If we are to install Node 4.6 on it right away then this is effectively blocked by T147849: ChangeProp failing on Node v4.6.0

ETA is hopefully today

I have the old 4.4.6 binaries still around, we can downgrade scb100[34] until T147849 is fixed. Or we could disable changeprop on those two new machines and stick with 4.6.

We'd need to do the former, because we can't currently build repos for two different node versions at the same time.

OK, old packages it is. @MoritzMuehlenhoff should we downgrade the packages on apt.wikimedia.org while waiting for T147849 to be fixed ?

I'd rather keep 4.6.0 on apt.wikimedia.org, it works for all services except changeprop (parsoid is currently tested by Parsing team, will be upgraded once they're done) and fixes a number of security issues. Just ping me once these two systems are setup and I can downgrade them to 4.4.6.

Mentioned in SAL (#wikimedia-operations) [2016-10-17T09:51:47Z] <akosiaris> T148380 disable puppet on sca1001, sca1002, deactivate them on puppetmasters

Change 316308 had a related patch set uploaded (by Alexandros Kosiaris):
Rename sca1001, sca1002 to scb1003, scb1004

https://gerrit.wikimedia.org/r/316308

akosiaris renamed this task from Re-image sca1001, sca1002 as scb1003, scb1004 respectively to Re-image sca1001, sca1002, sca2001, sca2002, as scb1003, scb1004 and scb2003, scb2004 respectively.Oct 17 2016, 10:09 AM
akosiaris updated the task description. (Show Details)

Change 316308 merged by Alexandros Kosiaris:
Rename sca1001, sca1002, scb2001, scb2002

https://gerrit.wikimedia.org/r/316308

T148615: Update Node on SCB to v4.6.0 will be done soon, so let's finish that up before merging ex-SCA into SCB to make our lives easier.

akosiaris updated the task description. (Show Details)Oct 19 2016, 10:16 AM

Actually the boxes are up and running and ready to be pooled, sporting version 4.4.6 of nodejs

akosiaris removed akosiaris as the assignee of this task.Oct 19 2016, 2:17 PM
akosiaris updated the task description. (Show Details)
akosiaris added a subscriber: Cmjohnson.

That leaves only the DC labels stuff and switchport descriptions. @Cmjohnson could you update the DC labels ? Thanks!

elukey assigned this task to Cmjohnson.Oct 20 2016, 1:32 PM
elukey triaged this task as Medium priority.

update racktables and physical labels

Rename sca1001, sca1002 to scb1003, scb1004

Cmjohnson reassigned this task from Cmjohnson to Papaul.Oct 20 2016, 3:25 PM
Cmjohnson added a project: ops-codfw.
Cmjohnson added a subscriber: Papaul.

Eqiad: Labels have been changed, racktables updated, switch ports updated.

codfw: racktables partially updated, switch ports updated.

@Papaul: please update the physical labels on the servers and update that field in racktables please
sca2001 = scb2003
sca2002 = scb2004

Please resolve once completed. Thanks!

Papaul closed this task as Resolved.Oct 20 2016, 6:19 PM

Complete