Page MenuHomePhabricator
Create Task
Maniphest T290694

Q1:(Need By: TBD) rack/setup/install cp403[3-6].ulsfo.wmnet
Closed, ResolvedPublic
Actions

Description

This task will track the racking, setup, and OS installation of cp403[3-6].ulsfo.wmnet

Hostname / Racking / Installation Details

Hostnames: cp403[3-6].ulsfo.wmnet
Racking Proposal: Odd number hostnames in rack 22, even numbered in rack 23
Networking/Subnet/VLAN/IP: single 10g connection to private1-vlan
Partitioning/Raid: sw standard 2dev (the nmve card can remain unused by installer)
OS Distro: Buster (default unless otherwise specified)

Per host setup checklist

cp4033:

  • - receive in system on procurement task T286590 & in coupa - systems arrived but rails haven't yet.
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/735461
  • - OS installation & initital puppet run via cookbook

cp4034:

  • - receive in system on procurement task T286590 & in coupa - systems arrived but rails haven't yet.
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/735461
  • - OS installation & initital puppet run via cookbook

cp4035:

  • - receive in system on procurement task T286590 & in coupa - systems arrived but rails haven't yet.
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/735461
  • - OS installation & initital puppet run via cookbook

cp4036:

  • - receive in system on procurement task T286590 & in coupa - systems arrived but rails haven't yet.
  • - rack system with proposed racking plan (see above) & update netbox (include all system info plus location, state of planned)
  • - bios/drac/serial setup/testing
  • - add mgmt dns (asset tag and hostname) and production dns entries in netbox, run cookbook sre.dns.netbox.
  • - network port setup via netbox, run homer to commit
  • - firmware update (idrac, bios, network, raid controller)
  • - operations/puppet update - https://gerrit.wikimedia.org/r/735461
  • - OS installation & initital puppet run via cookbook

Once the system(s) above have had all checkbox steps completed, this task can be resolved.

Details

SubjectRepoBranchLines +/-
install_server: Update instance hardware categoryoperations/puppetproduction+14 -2
site: Add new cache instances in ulsfo DCoperations/puppetproduction+18 -7
insetup_nofermoperations/puppetproduction+1 -1
ferm not firmoperations/puppetproduction+1 -1
new cp40[3-6]operations/puppetproduction+5 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
Unknown Object (Task)
 ResolvedRobHT290694 Q1:(Need By: TBD) rack/setup/install cp403[3-6].ulsfo.wmnet

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 9 2021, 9:16 PM
RobH added a parent task: Unknown Object (Task).Sep 9 2021, 9:16 PM
RobH mentioned this in Unknown Object (Task).
Maintenance_bot added a project: SRE.Sep 9 2021, 9:45 PM
RobH moved this task from Backlog to Racking Tasks on the ops-ulsfo board.Sep 30 2021, 6:12 PM
RobH updated the task description. (Show Details)Oct 4 2021, 3:06 PM
BBlack moved this task from Backlog to Caching on the Traffic board.Oct 8 2021, 7:49 PM
BBlack moved this task from Caching to Traffic team actively servicing on the Traffic board.Oct 8 2021, 8:15 PM
RobH claimed this task.Oct 28 2021, 8:05 PM
RobH updated the task description. (Show Details)
RobH updated the task description. (Show Details)Oct 28 2021, 8:24 PM
RobH updated the task description. (Show Details)Oct 28 2021, 11:33 PM
gerritbot added a comment.Oct 28 2021, 11:36 PM

Change 735461 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] new cp40[3-6]

https://gerrit.wikimedia.org/r/735461

gerritbot added a project: Patch-For-Review.Oct 28 2021, 11:36 PM
gerritbot added a comment.Oct 28 2021, 11:38 PM

Change 735461 merged by RobH:

[operations/puppet@production] new cp40[3-6]

https://gerrit.wikimedia.org/r/735461

ops-monitoring-bot added a comment.Oct 28 2021, 11:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host cp4033.ulsfo.wmnet with OS buster

RobH removed a project: Patch-For-Review.Oct 28 2021, 11:46 PM
RobH updated the task description. (Show Details)
ops-monitoring-bot added a comment.Oct 28 2021, 11:46 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host cp4034.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 28 2021, 11:50 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host cp4034.ulsfo.wmnet with OS buster executed with errors:

  • cp4034 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Oct 28 2021, 11:50 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host cp4033.ulsfo.wmnet with OS buster executed with errors:

  • cp4033 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
RobH added a comment.Oct 28 2021, 11:51 PM

Not sure why these are failing, but I'm out of mental bandwidth for them today.

They are remotely accessible via idrac and will accept script commands. Something isn't right that is having them hang on install and needs investigation.

RobH added a comment.Oct 29 2021, 12:32 AM

Please note the librenms alerts didnt clear for these, until they were powered down. I need to check the settings for their power redundancy to ensure they are pulling evenly instead of one then other for failover.

Volans added a subscriber: Volans.Oct 29 2021, 8:02 AM
In T290694#7467061, @RobH wrote:

Not sure why these are failing, but I'm out of mental bandwidth for them today.

They are remotely accessible via idrac and will accept script commands. Something isn't right that is having them hang on install and needs investigation.

Both runs of the cookbook were interrupted by the operator:

2021-10-28 23:50:05,984 robh 31550 [ERROR] Ctrl+c pressed
...
2021-10-28 23:50:18,273 robh 31327 [ERROR] Ctrl+c pressed

In both cases the cookbook was polling the host to check when it was rebooted into the Debian installer environment.
For cp4034 it was interrupted after just 90s of polling, for cp4033 it was interrupted after 180s.
Depending on the hardware it can take a quite different amount of time for a host to be rebooted, those doesn't seem alarming polling times yet, it's possible that that they would have just continued the reimage without any problem.
In case of doubts you can always follow the reimage along connecting to the remote console.

RobH added a comment.Oct 29 2021, 6:26 PM
In T290694#7467448, @Volans wrote:
In T290694#7467061, @RobH wrote:

Not sure why these are failing, but I'm out of mental bandwidth for them today.

They are remotely accessible via idrac and will accept script commands. Something isn't right that is having them hang on install and needs investigation.

Both runs of the cookbook were interrupted by the operator:

2021-10-28 23:50:05,984 robh 31550 [ERROR] Ctrl+c pressed
...
2021-10-28 23:50:18,273 robh 31327 [ERROR] Ctrl+c pressed

In both cases the cookbook was polling the host to check when it was rebooted into the Debian installer environment.
For cp4034 it was interrupted after just 90s of polling, for cp4033 it was interrupted after 180s.
Depending on the hardware it can take a quite different amount of time for a host to be rebooted, those doesn't seem alarming polling times yet, it's possible that that they would have just continued the reimage without any problem.
In case of doubts you can always follow the reimage along connecting to the remote console.

Oh, I was watchign ON the hardware and it gave an error screen so I knew it was a bios setting. Sorry if it seemed like I was blaming the script, i was not =]

ops-monitoring-bot added a comment.Oct 29 2021, 7:15 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host cp4033.ulsfo.wmnet with OS buster

RobH changed the task status from Open to In Progress.Oct 29 2021, 7:41 PM

UEFI boot mode was enabled, which is why it was failing rather than attempting to actually hit our PXE server. Changed to bios mode and it immediately worked. I've corrected the bios setttings on cp4033 and once its fully done with a successful install, the rest will follow suit.

ops-monitoring-bot added a comment.Oct 29 2021, 7:49 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host cp4033.ulsfo.wmnet with OS buster executed with errors:

  • cp4033 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • The reimage failed, see the cookbook logs for the details
RobH added a comment.Oct 29 2021, 7:50 PM

hung on loading ramdisk post install reboot... not sure why

gerritbot added a comment.Oct 29 2021, 7:52 PM

Change 735687 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] ferm not firm

https://gerrit.wikimedia.org/r/735687

gerritbot added a project: Patch-For-Review.Oct 29 2021, 7:52 PM

Change 735687 merged by RobH:

[operations/puppet@production] ferm not firm

https://gerrit.wikimedia.org/r/735687

ops-monitoring-bot added a comment.Oct 29 2021, 7:56 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host cp4033.ulsfo.wmnet with OS buster

Maintenance_bot removed a project: Patch-For-Review.Oct 29 2021, 8:10 PM
gerritbot added a comment.Oct 29 2021, 8:22 PM

Change 735692 had a related patch set uploaded (by RobH; author: RobH):

[operations/puppet@production] insetup_noferm

https://gerrit.wikimedia.org/r/735692

gerritbot added a project: Patch-For-Review.Oct 29 2021, 8:22 PM

Change 735692 merged by RobH:

[operations/puppet@production] insetup_noferm

https://gerrit.wikimedia.org/r/735692

ops-monitoring-bot added a comment.Oct 29 2021, 8:39 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host cp4033.ulsfo.wmnet with OS buster completed:

  • cp4033 (WARN)
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110291954_robh_12618_cp4033.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Oct 29 2021, 8:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host cp4034.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 29 2021, 8:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host cp4035.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 29 2021, 8:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host cp4036.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Oct 29 2021, 9:06 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host cp4036.ulsfo.wmnet with OS buster executed with errors:

  • cp4036 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • The reimage failed, see the cookbook logs for the details
Maintenance_bot removed a project: Patch-For-Review.Oct 29 2021, 9:10 PM
ops-monitoring-bot added a comment.Oct 29 2021, 9:15 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host cp4035.ulsfo.wmnet with OS buster completed:

  • cp4035 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110292043_robh_22622_cp4035.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Oct 29 2021, 9:17 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host cp4034.ulsfo.wmnet with OS buster completed:

  • cp4034 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110292042_robh_22567_cp4034.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
ops-monitoring-bot added a comment.Oct 29 2021, 9:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by robh@cumin1001 for host cp4036.ulsfo.wmnet with OS buster

RobH updated the task description. (Show Details)Oct 29 2021, 9:33 PM
ops-monitoring-bot added a comment.Oct 29 2021, 10:02 PM

Cookbook cookbooks.sre.hosts.reimage started by robh@cumin1001 for host cp4036.ulsfo.wmnet with OS buster completed:

  • cp4036 (WARN)
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202110292132_robh_31974_cp4036.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Updated Netbox status planned -> staged
RobH closed this task as Resolved.Oct 29 2021, 10:03 PM
RobH updated the task description. (Show Details)
gerritbot added a comment.Nov 1 2021, 5:21 PM

Change 735994 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] site: Add new cache instances in ulsfo DC

https://gerrit.wikimedia.org/r/735994

gerritbot added a project: Patch-For-Review.Nov 1 2021, 5:21 PM
gerritbot added a comment.Nov 1 2021, 6:02 PM

Change 735994 merged by MMandere:

[operations/puppet@production] site: Add new cache instances in ulsfo DC

https://gerrit.wikimedia.org/r/735994

Maintenance_bot removed a project: Patch-For-Review.Nov 1 2021, 6:10 PM
Dzahn added a subscriber: Dzahn.Nov 1 2021, 7:27 PM

Icinga alerts that were added by puppet started firing and raised some questions but confirmed it was just about these new hosts and they just switched from PENDING to active in Icinga. also discussed on IRC

19:26 <+icinga-wm> ACKNOWLEDGEMENT - Check systemd state on cp4033 is CRITICAL: CRITICAL - degraded: The following units failed: prometheus-varnish-exporter.service,varnishncsa.service 
                   daniel_zahn https://phabricator.wikimedia.org/T290694 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
19:26 <+icinga-wm> ACKNOWLEDGEMENT - Check systemd state on cp4034 is CRITICAL: CRITICAL - degraded: The following units failed: prometheus-varnish-exporter.service,varnishncsa.service 
                   daniel_zahn https://phabricator.wikimedia.org/T290694 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
19:26 <+icinga-wm> ACKNOWLEDGEMENT - Check systemd state on cp4035 is CRITICAL: CRITICAL - degraded: The following units failed: prometheus-varnish-exporter.service,varnishncsa.service 
                   daniel_zahn https://phabricator.wikimedia.org/T290694 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
19:26 <+icinga-wm> ACKNOWLEDGEMENT - traffic-pool service on cp4035 is CRITICAL: CRITICAL - Expecting active but unit traffic-pool is inactive daniel_zahn 
                   https://phabricator.wikimedia.org/T290694 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
19:26 <+icinga-wm> ACKNOWLEDGEMENT - Check systemd state on cp4036 is CRITICAL: CRITICAL - degraded: The following units failed: prometheus-varnish-exporter.service,varnishncsa.service 
                   daniel_zahn https://phabricator.wikimedia.org/T290694 https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
19:26 <+icinga-wm> ACKNOWLEDGEMENT - traffic-pool service on cp4036 is CRITICAL: CRITICAL - Expecting active but unit traffic-pool is inactive daniel_zahn 
                   https://phabricator.wikimedia.org/T290694 https://wikitech.wikimedia.org/wiki/Monitoring/systemd_unit_state
BBlack added a subscriber: BBlack.Nov 1 2021, 7:34 PM

Yeah sorry for the noise - we weren't anticipating the hosts re-puppeting themselves into the productions roles (incorrectly!) and should've just puppet-disabled them until we were ready to reimage them. Eitherway, definitely not an actual production concern.

ops-monitoring-bot added a comment.Nov 2 2021, 12:02 PM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp4033.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Nov 2 2021, 12:51 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp4033.ulsfo.wmnet with OS buster completed:

  • cp4033 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp4033.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_upload,service=varnish-fe"}
{"cp4033.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_upload,service=ats-tls"}
{"cp4033.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_upload,service=ats-be"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111021119_mmandere_3810_cp4033.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 2 2021, 1:32 PM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp4035.ulsfo.wmnet with OS buster

Stashbot added a comment.Nov 2 2021, 1:45 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-02T13:45:21Z] <mmandere> pool cp4033.ulsfo.wmnet - T290694

ops-monitoring-bot added a comment.Nov 2 2021, 2:24 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp4035.ulsfo.wmnet with OS buster completed:

  • cp4035 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp4035.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_text,service=ats-be"}
{"cp4035.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_text,service=ats-tls"}
{"cp4035.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_text,service=varnish-fe"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111021331_mmandere_15322_cp4035.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 2 2021, 2:31 PM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp4034.ulsfo.wmnet with OS buster

Stashbot added a comment.Nov 2 2021, 2:34 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-02T14:34:44Z] <mmandere> pool cp4035.ulsfo.wmnet - T290694

ops-monitoring-bot added a comment.Nov 2 2021, 3:32 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp4034.ulsfo.wmnet with OS buster completed:

  • cp4034 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp4034.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_upload,service=varnish-fe"}
{"cp4034.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_upload,service=ats-be"}
{"cp4034.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_upload,service=ats-tls"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111021430_mmandere_25896_cp4034.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
  • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Nov 2 2021, 3:38 PM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp4036.ulsfo.wmnet with OS buster

Stashbot added a comment.Nov 2 2021, 3:41 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-02T15:41:46Z] <mmandere> pool cp4034.ulsfo.wmnet - T290694

ops-monitoring-bot added a comment.Nov 2 2021, 4:30 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp4036.ulsfo.wmnet with OS buster completed:

  • cp4036 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp4036.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_text,service=ats-be"}
{"cp4036.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_text,service=ats-tls"}
{"cp4036.ulsfo.wmnet": {"weight": 0, "pooled": "inactive"}, "tags": "dc=ulsfo,cluster=cache_text,service=varnish-fe"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111021538_mmandere_6478_cp4036.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:
  • Updated Netbox data from PuppetDB
Stashbot added a comment.Nov 2 2021, 4:38 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-02T16:38:43Z] <mmandere> pool cp4036.ulsfo.wmnet - T290694

gerritbot added a comment.Nov 3 2021, 2:16 PM

Change 736475 had a related patch set uploaded (by MMandere; author: MMandere):

[operations/puppet@production] install_server: Update instance hardware category

https://gerrit.wikimedia.org/r/736475

gerritbot added a project: Patch-For-Review.Nov 3 2021, 2:16 PM
gerritbot added a comment.Nov 3 2021, 2:45 PM

Change 736475 merged by MMandere:

[operations/puppet@production] install_server: Update instance hardware category

https://gerrit.wikimedia.org/r/736475

Maintenance_bot removed a project: Patch-For-Review.Nov 3 2021, 3:10 PM
Stashbot added a comment.Nov 3 2021, 3:58 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-03T15:58:40Z] <mmandere> depool cp4033.ulsfo.wmnet - T290694

ops-monitoring-bot added a comment.Nov 3 2021, 4:04 PM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp4033.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Nov 3 2021, 4:52 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp4033.ulsfo.wmnet with OS buster completed:

  • cp4033 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp4033.ulsfo.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_upload,service=ats-tls"}
{"cp4033.ulsfo.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_upload,service=varnish-fe"}
{"cp4033.ulsfo.wmnet": {"weight": 100, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_upload,service=ats-be"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111031602_mmandere_22907_cp4033.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=ulsfo,cluster=cache_upload,service=ats-tls' set/pooled=no
sudo confctl select 'dc=ulsfo,cluster=cache_upload,service=varnish-fe' set/pooled=no
sudo confctl select 'dc=ulsfo,cluster=cache_upload,service=ats-be' set/pooled=no

  • Updated Netbox data from PuppetDB
Stashbot added a comment.Nov 3 2021, 5:06 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-03T17:06:15Z] <mmandere> pool cp4033.ulsfo.wmnet - T290694

Stashbot added a comment.Nov 3 2021, 5:48 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-03T17:48:18Z] <mmandere> depool cp4035.ulsfo.wmnet - T290694

ops-monitoring-bot added a comment.Nov 3 2021, 5:50 PM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp4035.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Nov 3 2021, 6:51 PM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp4035.ulsfo.wmnet with OS buster completed:

  • cp4035 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp4035.ulsfo.wmnet": {"weight": 100, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_text,service=ats-be"}
{"cp4035.ulsfo.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_text,service=varnish-fe"}
{"cp4035.ulsfo.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_text,service=ats-tls"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111031750_mmandere_25895_cp4035.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=ulsfo,cluster=cache_text,service=ats-be' set/pooled=no
sudo confctl select 'dc=ulsfo,cluster=cache_text,service=varnish-fe' set/pooled=no
sudo confctl select 'dc=ulsfo,cluster=cache_text,service=ats-tls' set/pooled=no

  • Updated Netbox data from PuppetDB
Stashbot added a comment.Nov 3 2021, 7:26 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-03T19:26:14Z] <mmandere> pool cp4035.ulsfo.wmnet - T290694

Stashbot added a comment.Nov 4 2021, 9:09 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-04T09:09:28Z] <mmandere> depool cp4034.ulsfo.wmnet - T290694

ops-monitoring-bot added a comment.Nov 4 2021, 9:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp4034.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Nov 4 2021, 10:01 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp4034.ulsfo.wmnet with OS buster completed:

  • cp4034 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp4034.ulsfo.wmnet": {"weight": 100, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_upload,service=ats-be"}
{"cp4034.ulsfo.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_upload,service=ats-tls"}
{"cp4034.ulsfo.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_upload,service=varnish-fe"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111040911_mmandere_26533_cp4034.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=ulsfo,cluster=cache_upload,service=ats-be' set/pooled=no
sudo confctl select 'dc=ulsfo,cluster=cache_upload,service=ats-tls' set/pooled=no
sudo confctl select 'dc=ulsfo,cluster=cache_upload,service=varnish-fe' set/pooled=no

  • Updated Netbox data from PuppetDB
Stashbot added a comment.Nov 4 2021, 10:21 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-04T10:21:16Z] <mmandere> pool cp4034.ulsfo.wmnet - T290694

Stashbot added a comment.Nov 4 2021, 10:27 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-04T10:27:10Z] <mmandere> depool cp4036.ulsfo.wmnet - T290694

ops-monitoring-bot added a comment.Nov 4 2021, 10:28 AM

Cookbook cookbooks.sre.hosts.reimage was started by mmandere@cumin1001 for host cp4036.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Nov 4 2021, 11:28 AM

Cookbook cookbooks.sre.hosts.reimage started by mmandere@cumin1001 for host cp4036.ulsfo.wmnet with OS buster completed:

  • cp4036 (WARN)
    • Downtimed on Icinga
    • Set pooled=inactive for the following services on confctl:

{"cp4036.ulsfo.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_text,service=ats-tls"}
{"cp4036.ulsfo.wmnet": {"weight": 100, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_text,service=ats-be"}
{"cp4036.ulsfo.wmnet": {"weight": 1, "pooled": "no"}, "tags": "dc=ulsfo,cluster=cache_text,service=varnish-fe"}

  • Disabled Puppet
  • Removed from Puppet and PuppetDB if present
  • Deleted any existing Puppet certificate
  • Removed from Debmonitor if present
  • Forced PXE for next reboot
  • Host rebooted via IPMI
  • Host up (Debian installer)
  • Host up (new fresh buster OS)
  • Generated Puppet certificate
  • Signed new Puppet certificate
  • Run Puppet in NOOP mode to populate exported resources in PuppetDB
  • Found Nagios_host resource for this host in PuppetDB
  • Downtimed the new host on Icinga
  • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202111041028_mmandere_6637_cp4036.out
  • Checked BIOS boot parameters are back to normal
  • Rebooted
  • Automatic Puppet run was successful
  • Forced a re-check of all Icinga services for the host
  • Icinga status is not optimal, downtime not removed
  • Services in confctl are not automatically pooled, to restore the previous state you have to run the following commands:

sudo confctl select 'dc=ulsfo,cluster=cache_text,service=ats-tls' set/pooled=no
sudo confctl select 'dc=ulsfo,cluster=cache_text,service=ats-be' set/pooled=no
sudo confctl select 'dc=ulsfo,cluster=cache_text,service=varnish-fe' set/pooled=no

  • Updated Netbox data from PuppetDB
Stashbot added a comment.Nov 4 2021, 11:53 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-04T11:53:50Z] <mmandere> pool cp4036.ulsfo.wmnet - T290694

• ema mentioned this in T295120: Varnish packages installed from the wrong component on host reimage.Nov 5 2021, 9:47 AM
Volans mentioned this in T320955: Netbox accounting report: exclude removed hosts.Apr 4 2023, 5:43 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL