Page MenuHomePhabricator
Create Task
Maniphest T218985

rack/setup/install db1139|db1140.eqiad.wmnet (2 dump slaves)
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and installation of the 2 dedicated dump slaves

db1139:
Rack location: Anywhere on row B

  • - receive in system on procurement task T214066
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - RAID10 stripsize 256kb
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup
    • end on-site specific steps
  • - production dns entries
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch) - update netbox status to staged
  • - puppet accept/initial run
  • - handoff for service implementation - service implementation team must change status from staged to active

db1140:
Rack location: Anywhere but not on the following racks: C3,D8,A6, and not on the same rack of db1139

  • - receive in system on procurement task T214066
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location)
  • - bios/drac/serial setup/testing
  • - RAID10 stripsize 256kb
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup
    • end on-site specific steps
  • - production dns entries
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation (stretch) - update netbox status to staged
  • - puppet accept/initial run
  • - handoff for service implementation - service implementation team must change status from staged to active

Details

ProjectBranchLines +/-Subject
operations/puppetproduction+24 -4mariadb: Prepare db1139 and db1140 for reimage
operations/dnsmaster+4 -1db1139 and db1140 production dns
operations/dnsmaster+8 -0Adding mgmt dns for db1139/1140
operations/puppetproduction+2 -2site.pp: Add db1139 and db1140 as spares.
operations/puppetproduction+2 -2install_server: Add db1139,db1140
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Marostegui created this task.Mar 22 2019, 10:56 AM
Restricted Application added a project: SRE. · View Herald TranscriptMar 22 2019, 10:56 AM
• Marostegui changed the task status from Open to Stalled.Mar 22 2019, 10:56 AM
• Marostegui added a subtask: Unknown Object (Task).
• Marostegui moved this task from Triage to Blocked external/Not db team on the DBA board.
gerritbot added a comment.Mar 25 2019, 6:17 AM

Change 498768 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] install_server: Add db1139,db1140,dbprov200*

https://gerrit.wikimedia.org/r/498768

gerritbot added a project: Patch-For-Review.Mar 25 2019, 6:17 AM
Cmjohnson moved this task from Backlog to Racking Tasks on the ops-eqiad board.Mar 25 2019, 4:59 PM
gerritbot added a comment.Mar 26 2019, 5:55 AM

Change 498768 merged by Marostegui:
[operations/puppet@production] install_server: Add db1139,db1140

https://gerrit.wikimedia.org/r/498768

gerritbot added a comment.Mar 26 2019, 6:08 AM

Change 499048 had a related patch set uploaded (by Marostegui; owner: Marostegui):
[operations/puppet@production] site.pp: Add db1139 and db1140 as spares.

https://gerrit.wikimedia.org/r/499048

gerritbot added a comment.Mar 26 2019, 8:19 AM

Change 499048 merged by Marostegui:
[operations/puppet@production] site.pp: Add db1139 and db1140 as spares.

https://gerrit.wikimedia.org/r/499048

jcrespo added a parent task: T220002: Decommission dbstore2001.codfw.wmnet and dbstore2002.codfw.wmnet.Apr 3 2019, 3:22 PM
Cmjohnson closed subtask Unknown Object (Task) as Resolved.Apr 8 2019, 3:42 PM
• Marostegui added a comment.Apr 8 2019, 3:43 PM

Those two hosts were added to site.pp and have been added to the install recipe a few days ago.

• Marostegui mentioned this in Unknown Object (Task).Apr 8 2019, 3:46 PM
jcrespo added a parent task: T213406: Purchase and setup remaining hosts for database backups.Apr 8 2019, 5:01 PM
jcrespo mentioned this in T219399: rack/setup/deploy eqiad dedicated backup recovery/provisioning hosts.
• Marostegui changed the task status from Stalled to Open.Apr 9 2019, 5:34 AM

Opening as the hosts arrived

• Marostegui triaged this task as Medium priority.Apr 9 2019, 5:34 AM
• Marostegui added a parent task: T220572: Productionize eqiad and codfw source backup hosts & codfw backup test host.Apr 10 2019, 4:54 AM
• Marostegui mentioned this in T220572: Productionize eqiad and codfw source backup hosts & codfw backup test host.Apr 15 2019, 7:16 AM
gerritbot added a comment.Apr 16 2019, 7:25 PM

Change 504414 had a related patch set uploaded (by Cmjohnson; owner: Cmjohnson):
[operations/dns@master] Adding mgmt dns for db1139/1140

https://gerrit.wikimedia.org/r/504414

gerritbot added a comment.Apr 16 2019, 8:05 PM

Change 504414 merged by Cmjohnson:
[operations/dns@master] Adding mgmt dns for db1139/1140

https://gerrit.wikimedia.org/r/504414

Cmjohnson reassigned this task from Cmjohnson to RobH.Apr 16 2019, 8:06 PM
Cmjohnson updated the task description. (Show Details)
Cmjohnson added a subscriber: Cmjohnson.

@Marostegui @RobH these are racked and all on-site work is completed.

• Marostegui added a subscriber: jcrespo.Apr 16 2019, 8:19 PM

So from DC Ops side only missing the production DNS entries?
Thanks Chris!

gerritbot added a comment.Apr 18 2019, 8:49 PM

Change 504982 had a related patch set uploaded (by RobH; owner: RobH):
[operations/dns@master] db1139 and db1140 production dns

https://gerrit.wikimedia.org/r/504982

RobH updated the task description. (Show Details)Apr 18 2019, 8:49 PM
RobH updated the task description. (Show Details)
gerritbot added a comment.Apr 18 2019, 8:50 PM

Change 504982 merged by RobH:
[operations/dns@master] db1139 and db1140 production dns

https://gerrit.wikimedia.org/r/504982

• Marostegui moved this task from Blocked external/Not db team to Pending comment on the DBA board.Apr 25 2019, 6:46 AM
RobH removed RobH as the assignee of this task.Apr 29 2019, 4:56 PM

IRC Update: This is ready for installation by the DBA team, one of them will steal this task later this week.

jcrespo claimed this task.Apr 29 2019, 4:57 PM
gerritbot added a comment.May 2 2019, 10:15 AM

Change 507764 had a related patch set uploaded (by Jcrespo; owner: Jcrespo):
[operations/puppet@production] mariadb: Prepare db1139 and db1140 for reimage

https://gerrit.wikimedia.org/r/507764

gerritbot added a comment.May 2 2019, 11:20 AM

Change 507764 merged by Jcrespo:
[operations/puppet@production] mariadb: Prepare db1139 and db1140 for reimage

https://gerrit.wikimedia.org/r/507764

jcrespo added a comment.May 2 2019, 11:25 AM

Either dns, remote ipmi or password may not be configured properly:

Error: Unable to establish IPMI v2 / RMCP+ session
11:23:36 | Unable to run wmf-auto-reimage: Remote IPMI failed for mgmt 'db1139.mgmt.eqiad.wmnet': Command '['ipmitool', '-I', 'lanplus', '-H', 'db1139.mgmt.eqiad.wmnet', '-U', 'root', '-E', 'chassis', 'power', 'status']' returned non-zero exit status 1

trying to debug following workbook.

ops-monitoring-bot added a comment.May 2 2019, 12:14 PM

Script wmf-auto-reimage was launched by jynus on cumin1001.eqiad.wmnet for hosts:

['db1139.eqiad.wmnet', 'db1140.eqiad.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201905021213_jynus_190723.log.

jcrespo added a comment.May 2 2019, 12:22 PM

@Cmjohnson In case this is useful for you, I have documented how to enable ipmi on ilo5 from the web interface here: https://wikitech.wikimedia.org/w/index.php?title=Management_Interfaces&diff=1824940&oldid=1823217

jcrespo updated the task description. (Show Details)May 2 2019, 12:23 PM
ops-monitoring-bot added a comment.May 2 2019, 12:33 PM

Completed auto-reimage of hosts:

['db1140.eqiad.wmnet', 'db1139.eqiad.wmnet']

and were ALL successful.

jcrespo closed this task as Resolved.May 2 2019, 12:37 PM
jcrespo reassigned this task from jcrespo to Cmjohnson.
jcrespo updated the task description. (Show Details)

installed, implementation (provisioning) will be handled at T220572.

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL