Page MenuHomePhabricator
Create Task
Maniphest T265419

(Need By: TBD) rack/setup/install ms-be20[58-61]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of <enter the FQDN/hostname of the hosts being setup here>

These will replace hosts ms-be[2016-2027] and expand ms-be service ability.

Hostname / Racking / Installation Details

Hostnames: ms-be20[58-61]
Racking Proposal: One host per row. Better not to share racks with existing ms-be hosts (that we are not refreshing), but if we can't avoid it that's fine too.
Networking/Subnet/VLAN/IP: 10G private vlan
Partitioning/Raid: existing partman recipe as other ms-be hsots
OS Distro: Stretch

Per host setup checklist

Each host should have its own setup checklist copied and pasted into the list below.

ms-be2058: Row C rack C4 U1/U2 xe-4/0/0

  • - receive in system on procurement task T264139 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

ms-be2059: row D rack D7 U3/U4 xe-7/0/1

  • - receive in system on procurement task T264139 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

ms-be2060: Row A rack A4 U16/U17 xe-4/0/25

  • - receive in system on procurement task T264139 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

ms-be2061: Row D rack D2 U2/U3 xe-2/0/1

  • - receive in system on procurement task T264139 & in coupa
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
site: assign roles for all ms-be / ms-fe hostsoperations/puppetproduction+4 -4
Add new ms-be nodes to site.ppoperations/puppetproduction+1 -1
DHCP Add MAC address for ms-be205[8-9] and ms-be206[0-1]operations/puppetproduction+24 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedfgiunchediT266016 Refresh and expand Swift hardware capacity
Unknown Object (Task)
 ResolvedPapaulT265419 (Need By: TBD) rack/setup/install ms-be20[58-61]

Event Timeline

RobH created this task.Oct 13 2020, 10:05 PM
Restricted Application added a project: SRE. · View Herald TranscriptOct 13 2020, 10:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
RobH added a parent task: Unknown Object (Task).Oct 13 2020, 10:05 PM
RobH unsubscribed.
fgiunchedi added a parent task: T266016: Refresh and expand Swift hardware capacity.Oct 20 2020, 1:34 PM
Papaul moved this task from Backlog to Racking Tasks on the ops-codfw board.Oct 22 2020, 3:35 PM
Papaul updated the task description. (Show Details)Oct 28 2020, 3:23 PM
Papaul updated the task description. (Show Details)Oct 29 2020, 5:34 PM
Papaul updated the task description. (Show Details)Nov 18 2020, 7:22 PM
Papaul updated the task description. (Show Details)Nov 18 2020, 11:31 PM
Papaul updated the task description. (Show Details)Nov 23 2020, 5:07 PM
Papaul updated the task description. (Show Details)Nov 23 2020, 5:13 PM
Papaul updated the task description. (Show Details)Nov 23 2020, 6:10 PM
Papaul updated the task description. (Show Details)Nov 24 2020, 3:56 PM
gerritbot added a comment.Nov 24 2020, 5:53 PM

Change 643320 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP Add MAC address for ms-be205[8-9] and ms-be206[0-1]

https://gerrit.wikimedia.org/r/643320

gerritbot added a project: Patch-For-Review.Nov 24 2020, 5:53 PM
gerritbot added a comment.Nov 24 2020, 5:55 PM

Change 643320 merged by Papaul:
[operations/puppet@production] DHCP Add MAC address for ms-be205[8-9] and ms-be206[0-1]

https://gerrit.wikimedia.org/r/643320

Maintenance_bot removed a project: Patch-For-Review.Nov 24 2020, 6:10 PM
ops-monitoring-bot added a comment.Nov 24 2020, 7:56 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2059.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202011241956_pt1979_27659_ms-be2059_codfw_wmnet.log.

ops-monitoring-bot added a comment.Nov 24 2020, 7:57 PM

Completed auto-reimage of hosts:

['ms-be2059.codfw.wmnet']

Of which those FAILED:

['ms-be2059.codfw.wmnet']
ops-monitoring-bot added a comment.Nov 24 2020, 8:18 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2059.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202011242017_pt1979_32176_ms-be2059_codfw_wmnet.log.

Papaul updated the task description. (Show Details)Nov 24 2020, 8:29 PM
ops-monitoring-bot added a comment.Nov 24 2020, 9:12 PM

Completed auto-reimage of hosts:

['ms-be2059.codfw.wmnet']

and were ALL successful.

Papaul updated the task description. (Show Details)Nov 25 2020, 1:23 AM
ops-monitoring-bot added a comment.Nov 30 2020, 8:02 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2059.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202011302001_pt1979_1719_ms-be2059_codfw_wmnet.log.

ops-monitoring-bot added a comment.Nov 30 2020, 8:18 PM

Completed auto-reimage of hosts:

['ms-be2059.codfw.wmnet']

Of which those FAILED:

['ms-be2059.codfw.wmnet']
ops-monitoring-bot added a comment.Nov 30 2020, 8:27 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2059.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202011302027_pt1979_6775_ms-be2059_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 1 2020, 12:32 AM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2058.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012010032_pt1979_18452_ms-be2058_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 1 2020, 12:33 AM

Completed auto-reimage of hosts:

['ms-be2058.codfw.wmnet']

Of which those FAILED:

['ms-be2058.codfw.wmnet']
ops-monitoring-bot added a comment.Dec 1 2020, 12:33 AM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2058.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012010033_pt1979_18501_ms-be2058_codfw_wmnet.log.

Papaul updated the task description. (Show Details)Dec 1 2020, 1:50 AM
Papaul added a subscriber: fgiunchedi.Dec 1 2020, 2:00 AM

@fgiunchedi I re-imaged ms-be2059 and ms-be2058, puppet is not happy

WARNING: Puppet has 1 failures. Last run 42 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[mkfs-/dev/sdc1]

Can you please take a look and see why puppet is not happy so I can resume the install on the other 2 hosts?

Thanks

ops-monitoring-bot added a comment.Dec 1 2020, 2:00 AM

Completed auto-reimage of hosts:

['ms-be2058.codfw.wmnet']

Of which those FAILED:

['ms-be2058.codfw.wmnet']
fgiunchedi added a comment.Dec 1 2020, 8:46 AM
In T265419#6657950, @Papaul wrote:

@fgiunchedi I re-imaged ms-be2059 and ms-be2058, puppet is not happy

WARNING: Puppet has 1 failures. Last run 42 seconds ago with 1 failures. Failed resources (up to 3 shown): Exec[mkfs-/dev/sdc1]

Can you please take a look and see why puppet is not happy so I can resume the install on the other 2 hosts?

Thanks

So two different failure modes AFAICS:

  • ms-be2058 for some reason has got its sdc and sdb swapped post debian-installer:
ms-be2058:~$ cat /proc/partitions 
major minor  #blocks  name

   8        0  468320256 sda
   8        1   58592256 sda1
   8        2     976896 sda2
   8        3   97655808 sda3
   8        4  311093248 sda4
   8       32  468320256 sdc
   8       33   58592256 sdc1
   8       34     976896 sdc2
   8       35   97655808 sdc3
   8       36  311093248 sdc4
...
   8       16 7813464064 sdb

So the installer completed correctly with sda/sdb being the ssd, but then on reboot the disks were renumbered and puppet is confused by this fact. I tested a reboot of ms-be2058 to see if it'd restore the disk order, but the host isn't coming back.

  • ms-be2059 it looks like there was an existing vfat filesystem on sdc1 and that's what confused puppet. I wiped the filesystem and now puppet's happy again:
ms-be2059:~$ sudo wipefs -a /dev/sdc1
/dev/sdc1: 8 bytes were erased at offset 0x00000052 (vfat): 46 41 54 33 32 20 20 20
/dev/sdc1: 1 byte was erased at offset 0x00000000 (vfat): eb
/dev/sdc1: 2 bytes were erased at offset 0x000001fe (vfat): 55 aa
ms-be2059:~$ sudo puppet agent --test --verbose
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for ms-be2059.codfw.wmnet
Info: Applying configuration version '(0254fe8eb7) Elukey - hadoop: Migrate hiera() to lookup() and setting datatype in spark2'
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Exec[mkfs-/dev/sdc1]/returns: executed successfully
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Swift::Mount_filesystem[/dev/sdc1]/Exec[mountpoint-root-/srv/swift-storage/sdc1]/returns: executed successfully
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Swift::Mount_filesystem[/dev/sdc1]/Mount[/srv/swift-storage/sdc1]/ensure: defined 'ensure' as 'defined'
Info: Computing checksum on file /etc/fstab
Info: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Swift::Mount_filesystem[/dev/sdc1]/Mount[/srv/swift-storage/sdc1]: Scheduling refresh of Mount[/srv/swift-storage/sdc1]
Notice: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Swift::Mount_filesystem[/dev/sdc1]/Mount[/srv/swift-storage/sdc1]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Profile::Swift::Storage/Swift::Init_device[/dev/sdc]/Swift::Mount_filesystem[/dev/sdc1]/Mount[/srv/swift-storage/sdc1]: Scheduling refresh of Mount[/srv/swift-storage/sdc1]
Notice: Applied catalog in 19.97 seconds

Hope that helps!

Papaul added a comment.Dec 1 2020, 2:21 PM

@fgiunchedi thanks . ms-be2058 has memory error the same DIMM we were having problem with on msbe2057 was used in ms-be2058 so I will go ahead and ask for replacement . But the server is back up.

ops-monitoring-bot added a comment.Dec 1 2020, 2:22 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2060.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012011422_pt1979_9982_ms-be2060_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 1 2020, 2:31 PM

Completed auto-reimage of hosts:

['ms-be2060.codfw.wmnet']

Of which those FAILED:

['ms-be2060.codfw.wmnet']
ops-monitoring-bot added a comment.Dec 1 2020, 2:33 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2060.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012011433_pt1979_11336_ms-be2060_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 1 2020, 2:59 PM

Completed auto-reimage of hosts:

['ms-be2060.codfw.wmnet']

Of which those FAILED:

['ms-be2060.codfw.wmnet']
ops-monitoring-bot added a comment.Dec 1 2020, 3:06 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2060.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012011506_pt1979_16777_ms-be2060_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 1 2020, 3:20 PM

Completed auto-reimage of hosts:

['ms-be2060.codfw.wmnet']

Of which those FAILED:

['ms-be2060.codfw.wmnet']
ops-monitoring-bot added a comment.Dec 1 2020, 6:19 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2060.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012011819_pt1979_22321_ms-be2060_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 1 2020, 7:12 PM

Completed auto-reimage of hosts:

['ms-be2060.codfw.wmnet']

Of which those FAILED:

['ms-be2060.codfw.wmnet']
ops-monitoring-bot added a comment.Dec 1 2020, 7:15 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2060.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012011915_pt1979_32688_ms-be2060_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 1 2020, 7:30 PM

Completed auto-reimage of hosts:

['ms-be2060.codfw.wmnet']

Of which those FAILED:

['ms-be2060.codfw.wmnet']
ops-monitoring-bot added a comment.Dec 1 2020, 7:30 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2060.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012011930_pt1979_2357_ms-be2060_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 1 2020, 7:30 PM

Completed auto-reimage of hosts:

['ms-be2060.codfw.wmnet']

Of which those FAILED:

['ms-be2060.codfw.wmnet']
ops-monitoring-bot added a comment.Dec 1 2020, 9:21 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2061.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012012121_pt1979_23205_ms-be2061_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 1 2020, 9:35 PM

Completed auto-reimage of hosts:

['ms-be2061.codfw.wmnet']

Of which those FAILED:

['ms-be2061.codfw.wmnet']
gerritbot added a comment.Dec 2 2020, 1:52 PM

Change 644803 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] Add new ms-be nodes to site.pp

https://gerrit.wikimedia.org/r/644803

gerritbot added a project: Patch-For-Review.Dec 2 2020, 1:52 PM
gerritbot added a comment.Dec 2 2020, 1:55 PM

Change 644803 merged by Papaul:
[operations/puppet@production] Add new ms-be nodes to site.pp

https://gerrit.wikimedia.org/r/644803

ops-monitoring-bot added a comment.Dec 2 2020, 2:01 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2060.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012021401_pt1979_12323_ms-be2060_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2020, 2:01 PM

Completed auto-reimage of hosts:

['ms-be2060.codfw.wmnet']

Of which those FAILED:

['ms-be2060.codfw.wmnet']
ops-monitoring-bot added a comment.Dec 2 2020, 2:02 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2060.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012021401_pt1979_12379_ms-be2060_codfw_wmnet.log.

Maintenance_bot removed a project: Patch-For-Review.Dec 2 2020, 2:10 PM
ops-monitoring-bot added a comment.Dec 2 2020, 2:45 PM

Completed auto-reimage of hosts:

['ms-be2060.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Dec 2 2020, 2:58 PM

Change 644847 had a related patch set uploaded (by Filippo Giunchedi; owner: Filippo Giunchedi):
[operations/puppet@production] site: assign roles for all ms-be / ms-fe hosts

https://gerrit.wikimedia.org/r/644847

gerritbot added a project: Patch-For-Review.Dec 2 2020, 2:58 PM
ops-monitoring-bot added a comment.Dec 2 2020, 3:40 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2061.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012021539_pt1979_4680_ms-be2061_codfw_wmnet.log.

Papaul updated the task description. (Show Details)Dec 2 2020, 3:40 PM
ops-monitoring-bot added a comment.Dec 2 2020, 4:04 PM

Completed auto-reimage of hosts:

['ms-be2061.codfw.wmnet']

Of which those FAILED:

['ms-be2061.codfw.wmnet']
ops-monitoring-bot added a comment.Dec 2 2020, 4:06 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

ms-be2061.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202012021606_pt1979_12228_ms-be2061_codfw_wmnet.log.

ops-monitoring-bot added a comment.Dec 2 2020, 5:34 PM

Completed auto-reimage of hosts:

['ms-be2061.codfw.wmnet']

and were ALL successful.

Papaul updated the task description. (Show Details)Dec 2 2020, 5:37 PM
Papaul closed this task as Resolved.Dec 2 2020, 5:40 PM

@fgiunchedi complete

gerritbot added a comment.Dec 3 2020, 8:36 AM

Change 644847 merged by Filippo Giunchedi:
[operations/puppet@production] site: assign roles for all ms-be / ms-fe hosts

https://gerrit.wikimedia.org/r/644847

Maintenance_bot removed a project: Patch-For-Review.Dec 3 2020, 9:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits