Page MenuHomePhabricator
Create Task
Maniphest T247340

(Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2]
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of cp2027 to cp2042

Hostname / Racking / Installation Details

Hostnames:
cp2027-cp2042
Racking Proposal:
Preferably rack these with maximum redundancy spread like we did in eqiad (4 per row, 2 per rack, in two different 10G racks of each row).
Networking/Subnet/VLAN/IP: 10G, single connection
Partitioning/Raid: match esams cp systems

  • receive in system on procurement task T242044 missing rails
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare)
  • - host state in netbox set to staged

Details

SubjectRepoBranchLines +/-
partman: clean up cacheproxy selectorsoperations/puppetproduction+2 -3
site: Reimage cp2028 as cache::uploadoperations/puppetproduction+4 -2
acme_chief: expand cp nodes regexoperations/puppetproduction+10 -10
site: Reimage cp2027 as cache::textoperations/puppetproduction+4 -2
Add IPv6 for cp2027-42operations/dnsmaster+37 -0
cp2027-42: add IPs to cache_hosts dataoperations/puppetproduction+32 -0
cp2027-42: define ATS storage as nvmeoperations/puppetproduction+64 -0
Add new cp nodes cp2027 to cp2042 to site.ppoperations/puppetproduction+5 -0
DHCP Partman: Add MAC address and partman for cp2027 to cp2042operations/puppetproduction+81 -1
DNS: Add mgmt and production DNS for cp2027 to cp2042operations/dnsmaster+97 -1
Customize query in gerrit

Related Objects

Event Timeline

Papaul created this task.Mar 10 2020, 6:39 PM
Restricted Application added a project: SRE. · View Herald TranscriptMar 10 2020, 6:39 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Papaul triaged this task as Medium priority.Mar 10 2020, 6:40 PM
• ema moved this task from Backlog to Hardware on the Traffic board.Mar 11 2020, 1:53 PM
Papaul moved this task from Backlog to Racking Tasks on the ops-codfw board.Mar 12 2020, 8:51 PM
Papaul renamed this task from (Need by: TBD) rack/setup/install cp202[7-0], cp203[0-9], cp204[0-2] to (Need by: TBD) rack/setup/install cp202[7-9], cp203[0-9], cp204[0-2].Mar 13 2020, 5:54 PM
Papaul updated the task description. (Show Details)Mar 24 2020, 7:10 PM
gerritbot added a comment.Mar 24 2020, 8:52 PM

Change 583144 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/dns@master] DNS: Add mgmt and production DNS for cp2027 to cp2042

https://gerrit.wikimedia.org/r/583144

gerritbot added a project: Patch-For-Review.Mar 24 2020, 8:52 PM
gerritbot added a comment.Mar 25 2020, 3:38 PM

Change 583144 merged by Papaul:
[operations/dns@master] DNS: Add mgmt and production DNS for cp2027 to cp2042

https://gerrit.wikimedia.org/r/583144

Maintenance_bot removed a project: Patch-For-Review.Mar 25 2020, 4:10 PM
Papaul updated the task description. (Show Details)Mar 25 2020, 4:13 PM
gerritbot added a comment.Mar 25 2020, 4:14 PM

Change 583381 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] DHCP Partman: Add MAC address and partman for cp2027 to cp2042

https://gerrit.wikimedia.org/r/583381

gerritbot added a project: Patch-For-Review.Mar 25 2020, 4:14 PM
gerritbot added a comment.Mar 25 2020, 10:30 PM

Change 583469 had a related patch set uploaded (by Papaul; owner: Papaul):
[operations/puppet@production] Add new cp nodes cp2027 to cp2042 to site.pp

https://gerrit.wikimedia.org/r/583469

gerritbot added a comment.Mar 26 2020, 12:41 PM

Change 583381 merged by Dzahn:
[operations/puppet@production] DHCP Partman: Add MAC address and partman for cp2027 to cp2042

https://gerrit.wikimedia.org/r/583381

gerritbot added a comment.Mar 26 2020, 12:47 PM

Change 583469 merged by Dzahn:
[operations/puppet@production] Add new cp nodes cp2027 to cp2042 to site.pp

https://gerrit.wikimedia.org/r/583469

Maintenance_bot removed a project: Patch-For-Review.Mar 26 2020, 1:10 PM
Papaul updated the task description. (Show Details)Mar 26 2020, 1:53 PM
ops-monitoring-bot added a comment.Mar 26 2020, 1:58 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2027.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261358_pt1979_4873_cp2027_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 26 2020, 2:30 PM

Completed auto-reimage of hosts:

['cp2027.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 2:31 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2028.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261431_pt1979_9983_cp2028_codfw_wmnet.log.

gerritbot added a comment.Mar 26 2020, 2:50 PM

Change 583608 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] Add IPv6 for cp2027-42

https://gerrit.wikimedia.org/r/583608

gerritbot added a project: Patch-For-Review.Mar 26 2020, 2:50 PM
ops-monitoring-bot added a comment.Mar 26 2020, 2:55 PM

Completed auto-reimage of hosts:

['cp2028.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 2:56 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2029.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261456_pt1979_14283_cp2029_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 26 2020, 3:01 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2030.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261501_pt1979_14691_cp2030_codfw_wmnet.log.

gerritbot added a comment.Mar 26 2020, 3:08 PM

Change 583613 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] partman: clean up cacheproxy selectors

https://gerrit.wikimedia.org/r/583613

gerritbot added a comment.Mar 26 2020, 3:16 PM

Change 583623 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] cp2027-42: define ATS storage as nvme

https://gerrit.wikimedia.org/r/583623

ops-monitoring-bot added a comment.Mar 26 2020, 3:19 PM

Completed auto-reimage of hosts:

['cp2029.codfw.wmnet']

and were ALL successful.

gerritbot added a comment.Mar 26 2020, 3:23 PM

Change 583623 merged by BBlack:
[operations/puppet@production] cp2027-42: define ATS storage as nvme

https://gerrit.wikimedia.org/r/583623

ops-monitoring-bot added a comment.Mar 26 2020, 3:25 PM

Completed auto-reimage of hosts:

['cp2030.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 3:30 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2031.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261530_pt1979_21215_cp2031_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 26 2020, 3:32 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2032.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261532_pt1979_21383_cp2032_codfw_wmnet.log.

gerritbot added a comment.Mar 26 2020, 3:37 PM

Change 583634 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] cp2027-42: add IPs to cache_hosts data

https://gerrit.wikimedia.org/r/583634

gerritbot added a comment.Mar 26 2020, 3:37 PM

Change 583635 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] acme_chief: expand cp nodes regex

https://gerrit.wikimedia.org/r/583635

ops-monitoring-bot added a comment.Mar 26 2020, 3:53 PM

Completed auto-reimage of hosts:

['cp2031.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 3:53 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2033.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261553_pt1979_26852_cp2033_codfw_wmnet.log.

gerritbot added a comment.Mar 26 2020, 3:57 PM

Change 583634 merged by BBlack:
[operations/puppet@production] cp2027-42: add IPs to cache_hosts data

https://gerrit.wikimedia.org/r/583634

ops-monitoring-bot added a comment.Mar 26 2020, 3:58 PM

Completed auto-reimage of hosts:

['cp2032.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 3:58 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2034.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261558_pt1979_27384_cp2034_codfw_wmnet.log.

gerritbot added a comment.Mar 26 2020, 4:12 PM

Change 583608 merged by BBlack:
[operations/dns@master] Add IPv6 for cp2027-42

https://gerrit.wikimedia.org/r/583608

ops-monitoring-bot added a comment.Mar 26 2020, 4:17 PM

Completed auto-reimage of hosts:

['cp2033.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 4:24 PM

Completed auto-reimage of hosts:

['cp2034.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 4:25 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2035.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261625_pt1979_1523_cp2035_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 26 2020, 4:34 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2036.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261634_pt1979_2441_cp2036_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 26 2020, 4:49 PM

Completed auto-reimage of hosts:

['cp2035.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 4:58 PM

Completed auto-reimage of hosts:

['cp2036.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 6:47 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2037.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261847_pt1979_22587_cp2037_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 26 2020, 6:57 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2038.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261857_pt1979_23424_cp2038_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 26 2020, 7:10 PM

Completed auto-reimage of hosts:

['cp2037.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 7:20 PM

Completed auto-reimage of hosts:

['cp2038.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 7:20 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2039.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261920_pt1979_29089_cp2039_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 26 2020, 7:20 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2040.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003261920_pt1979_29122_cp2040_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 26 2020, 7:44 PM

Completed auto-reimage of hosts:

['cp2039.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 7:45 PM

Completed auto-reimage of hosts:

['cp2040.codfw.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Mar 26 2020, 9:11 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2041.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003262110_pt1979_10526_cp2041_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 26 2020, 9:15 PM

Script wmf-auto-reimage was launched by pt1979 on cumin2001.codfw.wmnet for hosts:

cp2042.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003262114_pt1979_12244_cp2042_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 26 2020, 9:37 PM

Completed auto-reimage of hosts:

['cp2041.codfw.wmnet']

and were ALL successful.

Papaul updated the task description. (Show Details)Mar 26 2020, 9:38 PM
ops-monitoring-bot added a comment.Mar 26 2020, 9:41 PM

Completed auto-reimage of hosts:

['cp2042.codfw.wmnet']

and were ALL successful.

Papaul closed this task as Resolved.Mar 26 2020, 9:47 PM

@BBlack servers are ready for service.

gerritbot added a comment.Mar 27 2020, 3:06 PM

Change 583978 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] site: Reimage cp2027 as cache::text

https://gerrit.wikimedia.org/r/583978

gerritbot added a comment.Mar 30 2020, 4:46 AM

Change 583978 merged by Vgutierrez:
[operations/puppet@production] site: Reimage cp2027 as cache::text

https://gerrit.wikimedia.org/r/583978

gerritbot added a comment.Mar 30 2020, 4:47 AM

Change 583635 merged by Vgutierrez:
[operations/puppet@production] acme_chief: expand cp nodes regex

https://gerrit.wikimedia.org/r/583635

ops-monitoring-bot added a comment.Mar 30 2020, 4:49 AM

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2027.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003300449_vgutierrez_131472_cp2027_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 30 2020, 5:15 AM

Completed auto-reimage of hosts:

['cp2027.codfw.wmnet']

Of which those FAILED:

['cp2027.codfw.wmnet']
Stashbot added a comment.Mar 30 2020, 5:40 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-30T05:40:24Z] <vgutierrez> pool cp2027 - T247340

gerritbot added a comment.Mar 30 2020, 5:50 AM

Change 584290 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] site: Reimage cp2028 as cache::upload

https://gerrit.wikimedia.org/r/584290

gerritbot added a comment.Mar 30 2020, 6:03 AM

Change 584290 merged by Vgutierrez:
[operations/puppet@production] site: Reimage cp2028 as cache::upload

https://gerrit.wikimedia.org/r/584290

ops-monitoring-bot added a comment.Mar 30 2020, 6:06 AM

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

cp2028.codfw.wmnet

The log can be found in /var/log/wmf-auto-reimage/202003300605_vgutierrez_142681_cp2028_codfw_wmnet.log.

ops-monitoring-bot added a comment.Mar 30 2020, 6:31 AM

Completed auto-reimage of hosts:

['cp2028.codfw.wmnet']

Of which those FAILED:

['cp2028.codfw.wmnet']
Stashbot added a comment.Mar 30 2020, 6:52 AM

Mentioned in SAL (#wikimedia-operations) [2020-03-30T06:52:27Z] <vgutierrez> pool cp2028 - T247340

Vgutierrez mentioned this in T248816: Replace cp20[01-26] with cp20[27-42].Mar 30 2020, 7:08 AM
gerritbot added a comment.Apr 6 2020, 7:00 AM

Change 583613 merged by Vgutierrez:
[operations/puppet@production] partman: clean up cacheproxy selectors

https://gerrit.wikimedia.org/r/583613

Maintenance_bot removed a project: Patch-For-Review.Apr 6 2020, 7:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits