Page MenuHomePhabricator
Create Task
Maniphest T201346

rack/setup/install cumin1001.eqiad.wmnet (new cumin master)
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and installation of the new cumin master for eqiad, ordered to replace outdated hardware system neodymium.

Hostname Proposal: If we cannot come up with a decent cluster type name for this, we'll have to re-use old (no longer used) element names. As it is the role cluster::management, how about just clustermgmt1001?

Racking Proposal: Any 1G rack, this is replacing (not running alongside) neodymium, so its location in regards to it is irrelevant.

clustermgmt1001:

  • - receive in system on procurement task T195418
  • - rack system with proposed racking plan (see above) & update racktables (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run
  • - handoff for service implementation

Details

SubjectRepoBranchLines +/-
cumin: installation of cumin1001operations/puppetproduction+14 -3
Add cumin1001 IPs and PTRsoperations/dnsmaster+6 -2
Rename clustermgmt to cuminoperations/dnsmaster+2 -2
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
Unknown Object (Task)
 ResolvedVolansT201346 rack/setup/install cumin1001.eqiad.wmnet (new cumin master)

Event Timeline

RobH triaged this task as Medium priority.Aug 6 2018, 6:58 PM
RobH created this task.
Volans subscribed.Aug 7 2018, 6:41 AM

I think there was an agreement to install this a Stretch and perform this way the upgrade jessie->stretch of this cluster.
In order to do that we need to build some debian packages for stretch that right now are jessie only (cumin come to mind). It shouldn't be a problem but just mentioning to not forget.

About the naming given that it's a host used fairly often I would go with something quick and easy to type, unless everyone has some sort of autocompletion (for example using my script, that I need to put in a more official place). If we assume autocompletion cluster* looks good to me as we don't have any other host starting with clu and the autocompletion will be easy.
If we don't assume autocompletion than we might try to find a shorter name.

• Cmjohnson moved this task from Backlog to Racking Tasks on the ops-eqiad board.Aug 7 2018, 2:02 PM
• Cmjohnson updated the task description. (Show Details)Aug 8 2018, 3:03 PM
MoritzMuehlenhoff added a comment.Aug 9 2018, 7:19 AM

Let's simply call these cumin*, it's the primary service offered and the other bits on these hosts (like debdeploy) are also tied to it.

MoritzMuehlenhoff added a comment.Aug 9 2018, 7:20 AM
In T201346#4483500, @Volans wrote:

I think there was an agreement to install this a Stretch and perform this way the upgrade jessie->stretch of this cluster.

Ack, that was the plan.

• Cmjohnson updated the task description. (Show Details)Aug 16 2018, 3:22 PM
• Cmjohnson reassigned this task from • Cmjohnson to RobH.Aug 21 2018, 4:10 PM
• Cmjohnson moved this task from Racking Tasks to Blocked on the ops-eqiad board.

@RobH these are ready but I see there is another last minute name change. When you do production DNS can you update mgmt dns to cumin1001 please.

• Cmjohnson updated the task description. (Show Details)Aug 21 2018, 4:10 PM
• Marostegui subscribed.Aug 27 2018, 4:44 PM
Volans claimed this task.Sep 18 2018, 5:46 PM

I'm taking over this to use it for a live-session with the new hires. I'll take care of fixing the mgmt interface beforehand.

gerritbot subscribed.Sep 21 2018, 8:39 AM

Change 461919 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Rename clustermgmt to cumin

https://gerrit.wikimedia.org/r/461919

gerritbot added a project: Patch-For-Review.Sep 21 2018, 8:39 AM
gerritbot added a comment.Sep 21 2018, 10:50 AM

Change 461919 merged by Volans:
[operations/dns@master] Rename clustermgmt to cumin

https://gerrit.wikimedia.org/r/461919

gerritbot added a comment.Sep 23 2018, 9:33 AM

Change 462274 had a related patch set uploaded (by Volans; owner: Volans):
[operations/dns@master] Add cumin1001 IPs and PTRs

https://gerrit.wikimedia.org/r/462274

gerritbot added a comment.Sep 23 2018, 9:53 AM

Change 462278 had a related patch set uploaded (by Volans; owner: Volans):
[operations/puppet@production] cumin: installation of cumin1001

https://gerrit.wikimedia.org/r/462278

• GTirloni subscribed.Sep 25 2018, 5:18 PM
gerritbot added a comment.Sep 25 2018, 5:23 PM

Change 462274 merged by Volans:
[operations/dns@master] Add cumin1001 IPs and PTRs

https://gerrit.wikimedia.org/r/462274

gerritbot added a comment.Sep 25 2018, 5:26 PM

Change 462278 merged by Volans:
[operations/puppet@production] cumin: installation of cumin1001

https://gerrit.wikimedia.org/r/462278

ops-monitoring-bot subscribed.Sep 25 2018, 5:33 PM

Script wmf-auto-reimage was launched by volans on cumin2001.codfw.wmnet for hosts:

cumin1001.eqiad.wmnet

The log can be found in /var/log/wmf-auto-reimage/201809251732_volans_32407_cumin1001_eqiad_wmnet.log.

ops-monitoring-bot added a comment.Sep 25 2018, 6:08 PM

Completed auto-reimage of hosts:

['cumin1001.eqiad.wmnet']

and were ALL successful.

Volans updated the task description. (Show Details)Sep 25 2018, 7:42 PM
Volans added a comment.Sep 25 2018, 7:45 PM

Host installed, keyholder armed and a quick test with cumin worked as expected.

@MoritzMuehlenhoff we need to do some additional test to check everything is working but most likely it's all good

@Cmjohnson I think the physical label need to be updated from 'clustermgmt1001' to 'cumin1001', correct me if I'm wrong.

MoritzMuehlenhoff added a comment.Sep 26 2018, 7:43 AM

I've created T205513 for the router changes and made some tests with Cumin and debdeploy, looked all fine. Going to update my patch for the mysql grants to also cover cumin1001.

Volans closed this task as Resolved.Oct 19 2018, 11:35 AM
Volans removed a project: Patch-For-Review.
faidon renamed this task from rack/setup/install clustermgmt1001.eqiad.wmnet (new cumin master) to rack/setup/install cumin1001.eqiad.wmnet (new cumin master).Apr 19 2019, 11:52 AM
Aklapper edited projects, added Cumin; removed SRE-tools.May 10 2023, 11:15 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL