Page MenuHomePhabricator
Create Task
Maniphest T165170

rack/setup/install ores2001-2009
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and installation of 9 systems: ores2001-2009. These were ordered on T161723, and originally requested by @akosiaris on T142578.

The racking locations have been assumed by @RobH, and must be verified by @akosiaris before this can be properly handled by @Papaul.

Alex: Please confirm if these should be racked with horizontal spread (across racks/rows) or if there is some limitation to this service that requires a different kind of setup. I've (@RobH) have assumed that we want to spread these out as much as possible. Also please review hostname proposal of oresXXXX. If this isn't going to work, please adjust this task's description with the acceptable hostname, and update naming conventions.

Racking Proposal: There are 9 systems, place them evenly spread between racks and rows. This means 2 per row, with one row having 3 instead of 2. Please place these in 1Gbit networking racks and otherwise place where you have the most power, network, and rackspace availability per row. None of these new hosts should be in the same rack as one another, to increase horizontal redundancy.

ores2001

  • - system received in from procurement task T161723.
  • - system racked according to racking proposal.
  • - bios/drac/serial setup/testing
  • - mgmt and production dns entries added (internal vlan)
  • - sub-task created in networking netops SRE project for network port setup, include all port info
  • - network port setup (description, enable, internal vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

ores2002

  • - system received in from procurement task T161723.
  • - system racked according to racking proposal.
  • - bios/drac/serial setup/testing
  • - mgmt and production dns entries added (internal vlan)
  • - sub-task created in networking netops SRE project for network port setup, include all port info
  • - network port setup (description, enable, internal vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

ores2003

  • - system received in from procurement task T161723.
  • - system racked according to racking proposal.
  • - bios/drac/serial setup/testing
  • - mgmt and production dns entries added (internal vlan)
  • - sub-task created in networking netops SRE project for network port setup, include all port info
  • - network port setup (description, enable, internal vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

ores2004

  • - system received in from procurement task T161723.
  • - system racked according to racking proposal.
  • - bios/drac/serial setup/testing
  • - mgmt and production dns entries added (internal vlan)
  • - sub-task created in networking netops SRE project for network port setup, include all port info
  • - network port setup (description, enable, internal vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

ores2005

  • - system received in from procurement task T161723.
  • - system racked according to racking proposal.
  • - bios/drac/serial setup/testing
  • - mgmt and production dns entries added (internal vlan)
  • - sub-task created in networking netops SRE project for network port setup, include all port info
  • - network port setup (description, enable, internal vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

ores2006

  • - system received in from procurement task T161723.
  • - system racked according to racking proposal.
  • - bios/drac/serial setup/testing
  • - mgmt and production dns entries added (internal vlan)
  • - sub-task created in networking netops SRE project for network port setup, include all port info
  • - network port setup (description, enable, internal vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

ores2007

  • - system received in from procurement task T161723.
  • - system racked according to racking proposal.
  • - bios/drac/serial setup/testing
  • - mgmt and production dns entries added (internal vlan)
  • - sub-task created in networking netops SRE project for network port setup, include all port info
  • - network port setup (description, enable, internal vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

ores2008

  • - system received in from procurement task T161723.
  • - system racked according to racking proposal.
  • - bios/drac/serial setup/testing
  • - mgmt and production dns entries added (internal vlan)
  • - sub-task created in networking netops SRE project for network port setup, include all port info
  • - network port setup (description, enable, internal vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

ores2009

  • - system received in from procurement task T161723.
  • - system racked according to racking proposal.
  • - bios/drac/serial setup/testing
  • - mgmt and production dns entries added (internal vlan)
  • - sub-task created in networking netops SRE project for network port setup, include all port info
  • - network port setup (description, enable, internal vlan)
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet/salt accept/initial run
  • - handoff for service implementation

Details

SubjectRepoBranchLines +/-
New ORES codfw cluster isn't provisioned yetmediawiki/services/ores/deploymaster+0 -9
ores: basic role for a worker-only nodeoperations/puppetproduction+9 -0
Reimage ores200* as stretchoperations/puppetproduction+18 -0
site: add ores200[19] as spare systemsoperations/puppetproduction+5 -0
DHCP/partman: Add DHCP and partman entries for ores200[1-9]operations/puppetproduction+46 -0
DNs: Add mgmt and production DNS entries for ores200[1-9]operations/dnsmaster+54 -4
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedakosiarisT162039 Prepare to service applications from kubernetes
 ResolvedakosiarisT162041 Expand the infrastructure to codfw
 ResolvedRobHT161700 CODFW: (4) hardware access request for kubernetes
 ResolvedRobHT142578 codfw/eqiad:(9+9) hardware access request for ORES
Unknown Object (Task)
 ResolvedakosiarisT165170 rack/setup/install ores2001-2009
 ResolvedRobHT166156 ores200[1-9] switch port configuration
ResolvedakosiarisT171851 Reimage ores* hosts with Debian Stretch
 ResolvedawightT169246 Stress/capacity test new ores* cluster
 ResolvedHalfakT174402 Review and fix file handle management in worker and celery processes
 ResolvedawightT177036 Clean up file handle and Redis connection management in ORES worker and celery processes
 ResolvedNoneT175736 Give ores admins read access to /srv/log/ores/main.log*
 ResolvedawightT179095 Wheels built on ores-misc-01 are incompatible with ores* and scb*
 ResolvedawightT181806 Problem with Redis server configuration on new ORES cluster
 ResolvedHalfakT182799 Make sure ORES is compatible with stretch
 ResolvedHalfakT184072 Rebuild ORES models on Stretch
 ResolvedHalfakT184073 Provision a Stretch box we can use to build ORES models
 ResolvedakosiarisT184074 Verify that all enchant/spelling dictionaries are available on Stretch. Port if needed.
 ResolvedHalfakT184135 Rebuild ORES wheels on Stretch
 ResolvedHalfakT184296 Convert CloudVPS instances to stretch.
ResolvedHalfakT184766 Convert ores-misc-01 to stretch
 ResolvedSumitT184765 Back up ores-misc-01 to ores-staging-01

Event Timeline

RobH created this task.May 12 2017, 4:13 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 12 2017, 4:13 PM
RobH updated the task description. (Show Details)May 18 2017, 4:28 PM
Papaul updated the task description. (Show Details)May 22 2017, 5:36 AM
Papaul claimed this task.May 23 2017, 3:22 PM
akosiaris added a comment.May 23 2017, 3:22 PM

Racking proposal sounds fine!

Papaul updated the task description. (Show Details)May 23 2017, 3:23 PM
Papaul created subtask T166156: ores200[1-9] switch port configuration .May 23 2017, 5:05 PM
gerritbot subscribed.May 23 2017, 7:11 PM

Change 355270 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNs: Add mgmt and production DNS entries for ores200[1-9]

https://gerrit.wikimedia.org/r/355270

gerritbot added a project: Patch-For-Review.May 23 2017, 7:11 PM
gerritbot added a comment.May 23 2017, 7:14 PM

Change 355270 merged by Dzahn:
[operations/dns@master] DNs: Add mgmt and production DNS entries for ores200[1-9]

https://gerrit.wikimedia.org/r/355270

Dzahn subscribed.May 23 2017, 7:16 PM
[bast1001:~] $ for orescodfw in $(seq 1 9); do host ores200${orescodfw}.codfw.wmnet; done
ores2001.codfw.wmnet has address 10.192.0.12
ores2002.codfw.wmnet has address 10.192.0.18
ores2003.codfw.wmnet has address 10.192.16.63
ores2004.codfw.wmnet has address 10.192.16.64
ores2005.codfw.wmnet has address 10.192.32.173
ores2006.codfw.wmnet has address 10.192.32.174
ores2007.codfw.wmnet has address 10.192.48.88
ores2008.codfw.wmnet has address 10.192.48.89
ores2009.codfw.wmnet has address 10.192.48.90
RobH closed subtask T166156: ores200[1-9] switch port configuration as Resolved.May 23 2017, 8:34 PM
RobH updated the task description. (Show Details)
Papaul added a comment.May 24 2017, 3:20 PM

@akosiaris can you please provide me with a partman recipe to use? Thanks

akosiaris added a comment.May 24 2017, 3:30 PM

IIRC these boxes don't have a RAID controller, so let's go for a RAID1 with LVM. Seems like raid1-lvm.cfg (https://github.com/wikimedia/puppet/blob/production/modules/install_server/files/autoinstall/partman/raid1-lvm.cfg) is fine.

RobH added a comment.EditedMay 24 2017, 4:15 PM

@akosiaris: I'd actually recommend we go with something that uses a /srv/ mount and ext4, with no swap like most servers. But that is just so more servers tend to follow the same partitioning scheme, not due to an actual problem with the raid1-lvm.cfg recipe.

raid1-lvm puts the following:

#   - /	:   ext3, RAID1, 50GB
#   - swap:       RAID1, 1GB, on LVM
#   - free space for the rest under RAID1/LVM

raid1-lvm-ext4-srv-noswap has:

# * two disks, sda & sdb
# * layout:
#   - /	:   ext4, RAID1, 50GB
#   - /srv: ext4, RAID1/LVM, up to 80% of the total space
#   - free space for the rest under RAID1/LVM

Overall, the elimination of swap is one of my secondary projects, referenced on T156955.

Since we order our systems with enough memory, swap space on the disks is not needed. I'd recommend using the raid1-lvm-ext4-srv-noswap unless Alex disagrees with me. There is discussion on that task about use of swap though, and it goes in both directions, but these systems are SATA, and seems they won't benefit from swap as much.

Mostly we should simply attempt to settle on more standard partitioning across the fleet.

akosiaris added a comment.May 24 2017, 7:37 PM

raid1-lvm-ext4-srv-noswap works too. Fine by me.

gerritbot added a comment.May 24 2017, 8:40 PM

Change 355501 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP/partman: Add DHCP and partman entries for ores200[1-9]

https://gerrit.wikimedia.org/r/355501

Papaul updated the task description. (Show Details)May 24 2017, 8:43 PM
gerritbot added a comment.May 24 2017, 8:43 PM

Change 355501 merged by Dzahn:
[operations/puppet@production] DHCP/partman: Add DHCP and partman entries for ores200[1-9]

https://gerrit.wikimedia.org/r/355501

Papaul reassigned this task from Papaul to akosiaris.May 25 2017, 11:54 PM
Papaul updated the task description. (Show Details)

@akosiaris This is complete at my end. It is all yours.

MoritzMuehlenhoff subscribed.May 31 2017, 2:21 PM

The salt key for ores2004 wasn't accepted (noticed that when rolling out the sudo update), this has been fixed.

Halfak added projects: Machine-Learning-Team (Active Tasks), ORES.Jun 16 2017, 7:01 PM
Halfak moved this task from Parked to Monitor (long term) on the Machine-Learning-Team (Active Tasks) board.
Halfak mentioned this in T168073: Switch ORES to dedicated cluster.Jun 16 2017, 7:03 PM
RobH removed projects: Patch-For-Review, ops-codfw.Jul 26 2017, 8:25 PM
akosiaris changed the task status from Open to Stalled.Aug 30 2017, 3:25 PM

Stalling a bit while T169246 is ongoing

Halfak edited projects, added Machine-Learning-Team; removed Machine-Learning-Team (Active Tasks).Sep 6 2017, 5:23 PM
Halfak moved this task from Unsorted to Backlog/Lift Wing on the Machine-Learning-Team board.Sep 6 2017, 5:26 PM
gerritbot added a comment.Nov 1 2017, 3:43 PM

Change 387811 had a related patch set uploaded (by Awight; owner: Awight):
[mediawiki/services/ores/deploy@master] New ORES codfw cluster isn't provisioned yet

https://gerrit.wikimedia.org/r/387811

gerritbot added a project: Patch-For-Review.Nov 1 2017, 3:43 PM
awight mentioned this in rORESDEPLOY02626a7dde13: New ORES codfw cluster isn't provisioned yet.Nov 1 2017, 3:44 PM
Dzahn added a comment.Dec 7 2017, 7:40 PM

Is this unstalled now? The reason was while T168246 is ongoing but that ticket is resolved. Is it really resolved though?

Dzahn mentioned this in T169246: Stress/capacity test new ores* cluster.Dec 7 2017, 7:41 PM
gerritbot added a comment.Dec 7 2017, 8:00 PM

Change 396101 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add ores200[19] as spare systems

https://gerrit.wikimedia.org/r/396101

gerritbot added a comment.Dec 7 2017, 8:14 PM

Change 396101 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] site: add ores200[19] as spare systems

https://gerrit.wikimedia.org/r/396101

gerritbot added a comment.Dec 7 2017, 8:15 PM

Change 396101 merged by Dzahn:
[operations/puppet@production] site: add ores200[19] as spare systems

https://gerrit.wikimedia.org/r/396101

akosiaris added a subtask: T171851: Reimage ores* hosts with Debian Stretch.Dec 8 2017, 1:10 PM
Dzahn added a comment.EditedDec 20 2017, 7:15 PM

So looks like we need a new role class for "regular ores server in production" (regular as opposed to ores::redis).

Because we have currently only ores::stresstest which is a temporary role for the stresstest (T169246) and ores::redis used on oresrdb*.

So the question is how does that role differ from the stresstest role, if at all?

The stresstest role currently includes profile::ores::worker and profile::ores::web, is that also what a regular (non-stresstest) role for an ores server would look like?

Would it be ores::worker which includes the worker profile but not the web profile?

gerritbot added a comment.Dec 20 2017, 7:20 PM

Change 399452 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ores: basic role for a worker-only node

https://gerrit.wikimedia.org/r/399452

akosiaris added a comment.Dec 21 2017, 9:46 AM
In T165170#3852233, @Dzahn wrote:

So looks like we need a new role class for "regular ores server in production" (regular as opposed to ores::redis).

Because we have currently only ores::stresstest which is a temporary role for the stresstest (T169246) and ores::redis used on oresrdb*.

ores::stresstest will be deleted. ores::redis clearly stays

So the question is how does that role differ from the stresstest role, if at all?

The stresstest role currently includes profile::ores::worker and profile::ores::web, is that also what a regular (non-stresstest) role for an ores server would look like?

Would it be ores::worker which includes the worker profile but not the web profile?

Yup pretty much the way you posted it

akosiaris changed the task status from Stalled to Open.Dec 21 2017, 9:48 AM

Unstalling. Note that we are in the year-end deployment freeze. The boxes should not be put in production until we are out of the freeze. Technically this currently means that the role should not be applied to them (we have no conftctl like switch for worker like loads). Same applies to changeprop.

gerritbot added a comment.Dec 21 2017, 9:50 AM

Change 399595 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Reimage ores200* as stretch

https://gerrit.wikimedia.org/r/399595

gerritbot added a comment.Dec 21 2017, 9:51 AM

Change 399595 merged by Alexandros Kosiaris:
[operations/puppet@production] Reimage ores200* as stretch

https://gerrit.wikimedia.org/r/399595

ops-monitoring-bot subscribed.Dec 21 2017, 4:36 PM

Script wmf-auto-reimage was launched by akosiaris on neodymium.eqiad.wmnet for hosts:

['ores2001.codfw.wmnet', 'ores2002.codfw.wmnet', 'ores2003.codfw.wmnet', 'ores2004.codfw.wmnet', 'ores2005.codfw.wmnet', 'ores2006.codfw.wmnet', 'ores2007.codfw.wmnet', 'ores2008.codfw.wmnet', 'ores2009.codfw.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201712211635_akosiaris_10155.log.

ops-monitoring-bot added a comment.Dec 21 2017, 5:22 PM

Completed auto-reimage of hosts:

['ores2008.codfw.wmnet', 'ores2002.codfw.wmnet', 'ores2003.codfw.wmnet', 'ores2005.codfw.wmnet', 'ores2007.codfw.wmnet', 'ores2009.codfw.wmnet', 'ores2006.codfw.wmnet', 'ores2004.codfw.wmnet', 'ores2001.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Dec 21 2017, 6:26 PM

Change 399452 merged by Dzahn:
[operations/puppet@production] ores: basic role for a worker-only node

https://gerrit.wikimedia.org/r/399452

akosiaris closed this task as Resolved.Feb 8 2018, 4:27 PM
akosiaris updated the task description. (Show Details)

This is finally done, resolving

akosiaris closed subtask T171851: Reimage ores* hosts with Debian Stretch as Resolved.Feb 14 2018, 8:31 AM
gerritbot added a comment.May 9 2018, 9:12 PM

Change 387811 abandoned by Awight:
New ORES codfw cluster isn't provisioned yet

https://gerrit.wikimedia.org/r/387811

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits