Page MenuHomePhabricator
Create Task
Maniphest T135253

Rack and set up 16 db's db1079-1094
Closed, ResolvedPublic
Actions

Description

All 16 servers arrived and have been racked. I evenly distributed them across 4 rows.
4 in A2
4 in B1 (@jcrespo I need you to decom 1 database in row B1....i need the switch port, it's only a 24P switch)
2 in C2
2 in C3
4 in D2

  • - receive in normally
  • - rack
  • - update racktables
  • - label and asset tag servers
  • - cabling
  • - setup bios/idrac/ilom
  • - Raid Setup
  • - add mgmt dns entries for both asset tag and hostname
  • - add production dns entries
  • - setup network ports (description, enable, vlan)
  • - update install_server module
  • - install OS
  • puppet/salt keys signed/accepted

Details

Related Changes in Gerrit:
SubjectAuthorRepoBranchLines +/-
Remove db1027 from internal dns entriesJcrespooperations/dnsmaster+0 -3
remove db1027 remnants (reverse lookup)Dzahnoperations/dnsmaster+0 -2
s/wment/wmnet/ for db1080Jcrespooperations/dnsmaster+1 -1
Update autoinstall to include db1079-94Jcrespooperations/puppetproduction+1 -1
Remove (almost) all references to db1027 on production puppetJcrespooperations/puppetproduction+1 -6
Remove all mentions to db1027, db2008 and db2009 from mediawikiJcrespooperations/mediawiki-configmaster+0 -6
Take db1027 out of production needed for unrackingJcrespooperations/mediawiki-configmaster+3 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Cmjohnson created this task.May 13 2016, 6:32 PM
Restricted Application added subscribers: Zppix, Southparkfan, Aklapper. · View Herald TranscriptMay 13 2016, 6:32 PM
gerritbot subscribed.May 16 2016, 6:54 AM

Change 288882 had a related patch set uploaded (by Jcrespo):
Take db1027 out of production needed for unracking

https://gerrit.wikimedia.org/r/288882

gerritbot added a project: Patch-For-Review.May 16 2016, 6:54 AM

Change 288882 merged by Jcrespo:
Take db1027 out of production needed for unracking

https://gerrit.wikimedia.org/r/288882

gerritbot added a comment.May 16 2016, 2:07 PM

Change 288943 had a related patch set uploaded (by Jcrespo):
Remove (almost) all references to db1027 on production puppet

https://gerrit.wikimedia.org/r/288943

gerritbot added a comment.May 16 2016, 2:13 PM

Change 288945 had a related patch set uploaded (by Jcrespo):
Remove all mentions to db1027, db2008 and db2009 from mediawiki

https://gerrit.wikimedia.org/r/288945

gerritbot added a comment.May 16 2016, 2:52 PM

Change 288945 merged by Jcrespo:
Remove all mentions to db1027, db2008 and db2009 from mediawiki

https://gerrit.wikimedia.org/r/288945

gerritbot added a comment.May 17 2016, 10:58 AM

Change 288943 merged by Jcrespo:
Remove (almost) all references to db1027 on production puppet

https://gerrit.wikimedia.org/r/288943

• jcrespo mentioned this in rOPUP9d7f0eb3be40: Remove (almost) all references to db1027 on production puppet.May 17 2016, 10:59 AM
gerritbot added a comment.May 17 2016, 11:16 AM

Change 289168 had a related patch set uploaded (by Jcrespo):
Remove db1027 from internal dns entries

https://gerrit.wikimedia.org/r/289168

• jcrespo added a comment.May 17 2016, 11:19 AM

@Cmjohnson db1027 has been removed from puppet (sites, dhcp -not netboot, as usual, a range is used-, salt, puppet cert). DNS is above pending, as usual, waiting for you ok.

db1027 is currently down and ready to be disconnected/unracked.

• Cmjohnson added a comment.May 17 2016, 3:47 PM

@jcrespo Do you know the raid configuration you want for each of these servers? I am assuming it's H/W raid. I will update during the ILO setup

• jcrespo added a comment.May 17 2016, 3:49 PM

HW RAID 10 with 256K stripe or larger. Documented on: https://wikitech.wikimedia.org/wiki/Raid_and_MegaCli#Raid_setup_at_Wikimedia

• jcrespo added a parent task: T133398: Install, configure and provision recently arrived db core machines.May 19 2016, 9:33 AM
Volans mentioned this in T111992: Physical location SPOF because of database server distribution on a single rack (D1).May 19 2016, 9:35 AM
Volans added a parent task: T111992: Physical location SPOF because of database server distribution on a single rack (D1).
• jcrespo removed a parent task: T111992: Physical location SPOF because of database server distribution on a single rack (D1).May 19 2016, 9:37 AM
• Cmjohnson added a comment.May 19 2016, 4:21 PM

@Volans @jcrespo

Slight change in racking for row B
db1085 and db1086 will be in B3 now.

• Cmjohnson updated the task description. (Show Details)May 19 2016, 4:26 PM
Ricordisamoa subscribed.May 20 2016, 1:35 AM
• Cmjohnson updated the task description. (Show Details)May 25 2016, 2:10 PM
• Cmjohnson updated the task description. (Show Details)
• Cmjohnson updated the task description. (Show Details)May 27 2016, 5:51 PM
gerritbot added a comment.May 31 2016, 4:44 PM

Change 291948 had a related patch set uploaded (by Jcrespo):
Update autoinstall to include db1079-94

https://gerrit.wikimedia.org/r/291948

gerritbot added a comment.May 31 2016, 4:48 PM

Change 291948 merged by Jcrespo:
Update autoinstall to include db1079-94

https://gerrit.wikimedia.org/r/291948

• jcrespo mentioned this in rOPUP7c9c06d7d8c0: Update autoinstall to include db1079-94.May 31 2016, 4:49 PM
• jcrespo claimed this task.May 31 2016, 4:51 PM
• jcrespo updated the task description. (Show Details)
• jcrespo changed the task status from Open to Stalled.May 31 2016, 6:04 PM

Stalling until T133398 is completed as I've been warned there could be some network issues.

• jcrespo changed the task status from Stalled to Open.Jun 1 2016, 11:34 AM
• jcrespo reassigned this task from • jcrespo to • Cmjohnson.
• jcrespo mentioned this in T133398: Install, configure and provision recently arrived db core machines.

These are the issues I found:

DHCP/network issues
db1082
db1085
db1086

cannot connect to mgmt
db1089
db1094

gerritbot added a comment.Jun 1 2016, 3:28 PM

Change 292159 had a related patch set uploaded (by Jcrespo):
s/wment/wmnet/ for db1080

https://gerrit.wikimedia.org/r/292159

gerritbot added a comment.Jun 1 2016, 3:55 PM

Change 292159 merged by Jcrespo:
s/wment/wmnet/ for db1080

https://gerrit.wikimedia.org/r/292159

• Cmjohnson added a comment.Jun 2 2016, 2:59 PM

Mgmt issues on db1089 and db1094 are fixed and they're installing now.

• Cmjohnson added a comment.Jun 2 2016, 4:19 PM

db1082 cable was disconnected. Re-attached and no issues with installations.
db1085/86 are having issues with vlans. There may be a switch bug.

Having issues removing from vlans...I delete the interfaces from the range and commit and do show vlans and the interfaces remain. They are double listed in labs-instances and private1-b. Cannot remove from either.

• Cmjohnson added a comment.Jun 2 2016, 4:44 PM

with a little help from faidon the vlans were fixed but failed during install.

Chris

• jcrespo closed this task as Resolved.Jun 3 2016, 11:08 AM

All installed, work now and boot by default to disk.

• jcrespo mentioned this in rOPUP189dc8b1904c: Remove (almost) all references to db1027 on production puppet.Jun 17 2016, 6:09 PM
gerritbot added a comment.Aug 31 2016, 7:04 AM

Change 289168 merged by Jcrespo:
Remove db1027 from internal dns entries

https://gerrit.wikimedia.org/r/289168

gerritbot added a comment.Nov 2 2016, 11:32 PM

Change 319495 had a related patch set uploaded (by Dzahn):
remove db1027 remnants (reverse lookup)

https://gerrit.wikimedia.org/r/319495

gerritbot added a comment.Nov 3 2016, 1:21 AM

Change 319495 merged by Dzahn:
remove db1027 remnants (reverse lookup)

https://gerrit.wikimedia.org/r/319495

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits