Page MenuHomePhabricator
Create Task
Maniphest T233242

rack/setup/install cp30[50-65].esams.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking/setup/installation of 16 new cp nodes in esams, ordered via T230619.

Common Specifications

Hostnames: cp3050+
Racking Proposal: These will be racked according to the layout detailed on this google sheet.
Networking/Subnet/VLAN/IP: 10G single connection, internal subnet.
Partitioning/Raid: These will have dual SSD for OS, and a single NVMe SSD for caching. This will likely require a new partman recipie be written to support this.
Initial Puppet Role: Traffic will need to advise if we use role::spare or a specific role for these for initial installation.

Individual Server Checklists

cp3050

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3051

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3052

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3053

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3054

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3055

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3056

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3057

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3058

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3059

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3061

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3062

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3060

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3063

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname

[x]x - network port setup (description, enable, vlan)

    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3064

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

cp3065

  • - receive in system on procurement task T230619
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - disable embedded NIC
  • - mgmt dns entries added for both asset tag and hostname
  • - network port setup (description, enable, vlan)
    • end on-site specific steps
  • - production dns entries added
  • - operations/puppet update (install_server at minimum, other files if possible)
  • - OS installation
  • - puppet accept/initial run (with role:spare or perhaps with actual role, up to traffic to comment above)
  • - host state in netbox set to staged
  • - handoff for service implementation
  • - service implementer changes from 'staged' status to 'active' status in netbox'

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 Resolvedwiki_willyT235805 ESAMS Refresh/Rebuild (October 2019)
Unknown Object (Task)
 ResolvedPapaulT233242 rack/setup/install cp30[50-65].esams.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Papaul added a comment.Oct 23 2019, 4:35 PM

@BBlack here is the information for the CP servers in rack 15

cp3055 : xe-5/0/15
cp3056: xe-5/0/16
cp3057: xe-5/0/17
cp3058: xe-5/0/18
cp3059: xe-5/0/19
cp3060: xe-5/0/20

Maintenance_bot removed a project: Patch-For-Review.Oct 23 2019, 5:11 PM
RobH removed a subscriber: RobH.Oct 23 2019, 6:02 PM
Papaul updated the task description. (Show Details)Oct 23 2019, 7:32 PM
gerritbot added a comment.Oct 23 2019, 8:42 PM

Change 545658 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Basic install for new esams hosts

https://gerrit.wikimedia.org/r/545658

gerritbot added a project: Patch-For-Review.Oct 23 2019, 8:42 PM
gerritbot added a comment.Oct 23 2019, 9:05 PM

Change 545662 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] basic DNS entries for new esams hosts

https://gerrit.wikimedia.org/r/545662

gerritbot added a comment.Oct 24 2019, 3:11 AM

Change 545662 merged by BBlack:
[operations/dns@master] basic DNS entries for new esams hosts

https://gerrit.wikimedia.org/r/545662

gerritbot added a comment.Oct 24 2019, 3:42 AM

Change 545691 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] cp30[56][0-9]: add hiera/conftool data

https://gerrit.wikimedia.org/r/545691

gerritbot added a comment.Oct 24 2019, 3:50 AM

Change 545693 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] acme_chief: Grant new esams cp hosts access to the unified certificate

https://gerrit.wikimedia.org/r/545693

gerritbot added a comment.Oct 24 2019, 3:51 AM

Change 545693 merged by Vgutierrez:
[operations/puppet@production] acme_chief: Grant new esams cp hosts access to the unified certificate

https://gerrit.wikimedia.org/r/545693

gerritbot added a comment.Oct 24 2019, 4:01 AM

Change 545658 merged by BBlack:
[operations/puppet@production] Basic install for new esams hosts

https://gerrit.wikimedia.org/r/545658

ops-monitoring-bot added a comment.Oct 24 2019, 4:19 AM

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

['cp3055.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910240419_vgutierrez_126191.log.

ops-monitoring-bot added a comment.Oct 24 2019, 4:45 AM

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

['cp3056.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910240445_vgutierrez_131290.log.

Vgutierrez updated the task description. (Show Details)Oct 24 2019, 5:31 AM
Vgutierrez updated the task description. (Show Details)Oct 24 2019, 5:39 AM
Vgutierrez updated the task description. (Show Details)Oct 24 2019, 5:45 AM
Vgutierrez updated the task description. (Show Details)Oct 24 2019, 5:51 AM
gerritbot added a comment.Oct 24 2019, 6:11 AM

Change 545701 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] install_server: Fix MAC addresses for new esams boxes

https://gerrit.wikimedia.org/r/545701

gerritbot added a comment.Oct 24 2019, 6:13 AM

Change 545701 merged by Vgutierrez:
[operations/puppet@production] install_server: Fix MAC addresses for new esams boxes

https://gerrit.wikimedia.org/r/545701

ops-monitoring-bot added a comment.Oct 24 2019, 7:05 AM

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

['cp3055.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910240704_vgutierrez_158045.log.

ops-monitoring-bot added a comment.Oct 24 2019, 7:28 AM

Completed auto-reimage of hosts:

['cp3055.esams.wmnet']

Of which those FAILED:

['cp3055.esams.wmnet']
gerritbot added a comment.Oct 24 2019, 7:38 AM

Change 545706 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Provide storage configuration for ats-backend on cp3055

https://gerrit.wikimedia.org/r/545706

gerritbot added a comment.Oct 24 2019, 7:46 AM

Change 545706 merged by Vgutierrez:
[operations/puppet@production] hiera: Provide storage configuration for ats-backend on cp3055

https://gerrit.wikimedia.org/r/545706

gerritbot added a comment.Oct 24 2019, 7:57 AM

Change 545711 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Provide ats storage config for new esams upload hosts

https://gerrit.wikimedia.org/r/545711

gerritbot added a comment.Oct 24 2019, 8:00 AM

Change 545712 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Provide varnish storage config for new cp text hosts

https://gerrit.wikimedia.org/r/545712

gerritbot added a comment.Oct 24 2019, 8:14 AM

Change 545711 merged by Vgutierrez:
[operations/puppet@production] hiera: Provide ats storage config for new esams upload hosts

https://gerrit.wikimedia.org/r/545711

gerritbot added a comment.Oct 24 2019, 8:14 AM

Change 545712 merged by Vgutierrez:
[operations/puppet@production] hiera: Provide varnish storage config for new cp text hosts

https://gerrit.wikimedia.org/r/545712

gerritbot added a comment.Oct 24 2019, 8:29 AM

Change 545691 merged by Vgutierrez:
[operations/puppet@production] cp30[56][0-9]: add hiera/conftool data

https://gerrit.wikimedia.org/r/545691

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2019, 9:10 AM
ops-monitoring-bot added a comment.Oct 24 2019, 9:12 AM

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

['cp3055.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910240912_vgutierrez_187246.log.

ops-monitoring-bot added a comment.Oct 24 2019, 9:58 AM

Completed auto-reimage of hosts:

['cp3055.esams.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Oct 24 2019, 10:36 AM

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3060.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910241036_ema_206514.log.

ops-monitoring-bot added a comment.Oct 24 2019, 10:41 AM

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

['cp3057.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910241041_vgutierrez_207264.log.

ops-monitoring-bot added a comment.Oct 24 2019, 10:52 AM

Completed auto-reimage of hosts:

['cp3060.esams.wmnet']

Of which those FAILED:

['cp3060.esams.wmnet']
gerritbot added a comment.Oct 24 2019, 11:07 AM

Change 545814 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp3060: add to cache::nodes

https://gerrit.wikimedia.org/r/545814

gerritbot added a project: Patch-For-Review.Oct 24 2019, 11:07 AM

Change 545814 merged by Ema:
[operations/puppet@production] cp3060: add to cache::nodes

https://gerrit.wikimedia.org/r/545814

ops-monitoring-bot added a comment.Oct 24 2019, 11:09 AM

Script wmf-auto-reimage was launched by vgutierrez on cumin1001.eqiad.wmnet for hosts:

['cp3059.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910241109_vgutierrez_214246.log.

ops-monitoring-bot added a comment.Oct 24 2019, 11:09 AM

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3060.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910241109_ema_214315.log.

ops-monitoring-bot added a comment.Oct 24 2019, 11:10 AM

Completed auto-reimage of hosts:

['cp3060.esams.wmnet']

Of which those FAILED:

['cp3060.esams.wmnet']
ops-monitoring-bot added a comment.Oct 24 2019, 11:10 AM

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3060.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910241110_ema_214404.log.

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2019, 11:10 AM
ops-monitoring-bot added a comment.Oct 24 2019, 11:16 AM

Completed auto-reimage of hosts:

['cp3057.esams.wmnet']

and were ALL successful.

gerritbot added a comment.Oct 24 2019, 11:34 AM

Change 545819 had a related patch set uploaded (by Vgutierrez; owner: Vgutierrez):
[operations/puppet@production] hiera: Add cp305[5,7,9] to cache::nodes

https://gerrit.wikimedia.org/r/545819

gerritbot added a project: Patch-For-Review.Oct 24 2019, 11:34 AM
Vgutierrez updated the task description. (Show Details)Oct 24 2019, 11:35 AM
ops-monitoring-bot added a comment.Oct 24 2019, 11:45 AM

Completed auto-reimage of hosts:

['cp3059.esams.wmnet']

and were ALL successful.

gerritbot added a comment.Oct 24 2019, 11:47 AM

Change 545820 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] esams cp nodes: size storage correctly

https://gerrit.wikimedia.org/r/545820

gerritbot added a comment.Oct 24 2019, 11:49 AM

Change 545819 merged by Vgutierrez:
[operations/puppet@production] hiera: Add cp305[5,7,9] to cache::nodes

https://gerrit.wikimedia.org/r/545819

gerritbot added a comment.Oct 24 2019, 12:10 PM

Change 545820 merged by BBlack:
[operations/puppet@production] esams cp nodes: size storage correctly

https://gerrit.wikimedia.org/r/545820

Stashbot added a comment.Oct 24 2019, 12:25 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T12:25:20Z] <ema> cp3060: powercycle -- NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [charon:1226] T233242

ops-monitoring-bot added a comment.Oct 24 2019, 12:31 PM

Completed auto-reimage of hosts:

['cp3060.esams.wmnet']

and were ALL successful.

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2019, 1:10 PM
Stashbot added a comment.Oct 24 2019, 1:17 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T13:17:50Z] <ema> set ats-be weights on new esams upload nodes T233242

gerritbot added a comment.Oct 24 2019, 1:49 PM

Change 545852 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] cp3056: add to cache::nodes

https://gerrit.wikimedia.org/r/545852

gerritbot added a project: Patch-For-Review.Oct 24 2019, 1:49 PM
ops-monitoring-bot added a comment.Oct 24 2019, 1:50 PM

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3056.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910241350_ema_246247.log.

gerritbot added a comment.Oct 24 2019, 1:50 PM

Change 545852 merged by Ema:
[operations/puppet@production] cp3056: add to cache::nodes

https://gerrit.wikimedia.org/r/545852

Stashbot added a comment.Oct 24 2019, 1:59 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T13:59:43Z] <ema> pool cp3060 T233242

gerritbot added a comment.Oct 24 2019, 2:08 PM

Change 545857 had a related patch set uploaded (by Ema; owner: Ema):
[operations/puppet@production] Add new esams cp hosts to cache::nodes

https://gerrit.wikimedia.org/r/545857

gerritbot added a comment.Oct 24 2019, 2:14 PM

Change 545857 merged by Ema:
[operations/puppet@production] Add new esams cp hosts to cache::nodes

https://gerrit.wikimedia.org/r/545857

Stashbot added a comment.Oct 24 2019, 2:16 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T14:16:31Z] <ema> power-cycle cp3056, stuck rebooting into d-i T233242

ops-monitoring-bot added a comment.Oct 24 2019, 2:19 PM

Script wmf-auto-reimage was launched by ema on cumin1001.eqiad.wmnet for hosts:

['cp3058.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910241418_ema_251786.log.

Papaul added a comment.Oct 24 2019, 2:24 PM

@BBlack here is the information for the CP servers in rack 16

cp3061 : xe-6/0/15
cp3062: xe-6/0/16
cp3063: xe-6/0/17
cp3064: xe-6/0/18
cp3065: xe-6/0/19

ops-monitoring-bot added a comment.Oct 24 2019, 2:54 PM

Completed auto-reimage of hosts:

['cp3058.esams.wmnet']

and were ALL successful.

Stashbot added a comment.Oct 24 2019, 3:09 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T15:09:37Z] <ema> pool cp3055 (cache_upload) T233242

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2019, 3:10 PM
Stashbot added a comment.Oct 24 2019, 3:19 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T15:19:12Z] <ema> pool cp3058 (cache_text) T233242

gerritbot added a comment.Oct 24 2019, 3:35 PM

Change 545880 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] esams: mgmt dns for rack 16

https://gerrit.wikimedia.org/r/545880

gerritbot added a project: Patch-For-Review.Oct 24 2019, 3:35 PM

Change 545880 merged by BBlack:
[operations/dns@master] esams: mgmt dns for rack 16

https://gerrit.wikimedia.org/r/545880

Stashbot added a comment.Oct 24 2019, 3:40 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T15:40:46Z] <ema> depool cp3030 (cache_text) T233242

Stashbot added a comment.Oct 24 2019, 3:45 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T15:45:42Z] <ema> depool cp3034 (cache_upload) T233242

Stashbot added a comment.Oct 24 2019, 3:51 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T15:51:20Z] <ema> depool cp3032 (cache_text) T233242

Stashbot added a comment.Oct 24 2019, 4:07 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T16:07:05Z] <ema> pool cp3057 (cache_upload) T233242

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2019, 4:10 PM
Stashbot added a comment.Oct 24 2019, 4:28 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T16:28:13Z] <ema> depool cp3035 (cache_upload) T233242

gerritbot added a comment.Oct 24 2019, 4:45 PM

Change 545893 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] Add dhcp macaddrs for esams rack 16 hosts

https://gerrit.wikimedia.org/r/545893

gerritbot added a project: Patch-For-Review.Oct 24 2019, 4:45 PM
gerritbot added a comment.Oct 24 2019, 4:48 PM

Change 545893 merged by BBlack:
[operations/puppet@production] Add dhcp macaddrs for esams rack 16 hosts

https://gerrit.wikimedia.org/r/545893

Stashbot added a comment.Oct 24 2019, 4:54 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T16:54:14Z] <ema> depool cp3036 (cache_upload) T233242

gerritbot added a comment.Oct 24 2019, 5:05 PM

Change 545895 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/dns@master] esams mgmt dns for rack 14

https://gerrit.wikimedia.org/r/545895

gerritbot added a comment.Oct 24 2019, 5:09 PM

Change 545895 merged by BBlack:
[operations/dns@master] esams mgmt dns for rack 14

https://gerrit.wikimedia.org/r/545895

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2019, 5:10 PM
Stashbot added a comment.Oct 24 2019, 5:38 PM

Mentioned in SAL (#wikimedia-operations) [2019-10-24T17:38:38Z] <ema> pool cp3059 (cache_upload) T233242

gerritbot added a comment.Oct 24 2019, 6:03 PM

Change 545908 had a related patch set uploaded (by BBlack; owner: BBlack):
[operations/puppet@production] esams: macaddrs for all new rack 14 hosts

https://gerrit.wikimedia.org/r/545908

gerritbot added a project: Patch-For-Review.Oct 24 2019, 6:03 PM

Change 545908 merged by BBlack:
[operations/puppet@production] esams: macaddrs for all new rack 14 hosts

https://gerrit.wikimedia.org/r/545908

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2019, 6:11 PM
ops-monitoring-bot added a comment.Oct 24 2019, 6:35 PM

Script wmf-auto-reimage was launched by bblack on cumin1001.eqiad.wmnet for hosts:

['cp3061.esams.wmnet', 'cp3062.esams.wmnet', 'cp3063.esams.wmnet', 'cp3064.esams.wmnet', 'cp3065.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910241835_bblack_44406.log.

ops-monitoring-bot added a comment.Oct 24 2019, 6:35 PM

Script wmf-auto-reimage was launched by bblack on cumin1001.eqiad.wmnet for hosts:

['cp3050.esams.wmnet', 'cp3051.esams.wmnet', 'cp3052.esams.wmnet', 'cp3053.esams.wmnet', 'cp3054.esams.wmnet']

The log can be found in /var/log/wmf-auto-reimage/201910241835_bblack_44412.log.

ops-monitoring-bot added a comment.Oct 24 2019, 7:36 PM

Completed auto-reimage of hosts:

['cp3062.esams.wmnet', 'cp3063.esams.wmnet', 'cp3061.esams.wmnet', 'cp3064.esams.wmnet', 'cp3065.esams.wmnet']

and were ALL successful.

ops-monitoring-bot added a comment.Oct 24 2019, 8:03 PM

Completed auto-reimage of hosts:

['cp3053.esams.wmnet', 'cp3050.esams.wmnet', 'cp3051.esams.wmnet', 'cp3052.esams.wmnet', 'cp3054.esams.wmnet']

and were ALL successful.

Papaul updated the task description. (Show Details)Oct 24 2019, 8:21 PM
BBlack closed this task as Resolved.Oct 29 2019, 1:28 PM
BBlack updated the task description. (Show Details)

As a batch these servers are complete in general. Note cp3056 had an early hardware issue that prevented progress, but this is tracked separately in: T236497

Content licensed under Creative Commons Attribution-ShareAlike 3.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL