Page MenuHomePhabricator
Create Task
Maniphest T330261

Upgrade DSE to k8s 1.23
Closed, ResolvedPublic
Actions

Description

The procedure should be the following:

  1. Add new Intermediate PKI CAs for DSE (requirement for k8s 1.23)
  2. Prepare puppet and deployment-charts changes. Examples:

Don't merge them yet :)

  1. Run the sre.k8s.upgrade cookbook on a cumin node (without the dry run):
sudo cookbook --dry-run sre.k8s.upgrade-cluster --reason "Upgrade to k8s 1.23" --k8s-cluster "dse-eqiad" --os bullseye --etcd-wipe-only

As part of the cookbook you'll need to merge the patches prepared in 1), the cookbook will ask for them at the right moment.

Details

SubjectRepoBranchLines +/-
admin_ng: refactor DSE 1.23 config and disable istio sidecars in nsoperations/deployment-chartsmaster+32 -54
admin_ng: add istio network policy config for DSEoperations/deployment-chartsmaster+9 -0
role::dse_k8s::worker: update istio-cni versionoperations/puppetproduction+1 -0
admin_ng: upgrade the DSE cluster to k8s 1.23operations/deployment-chartsmaster+32 -60
role::dse_k8s::{master,worker}: update settings to k8s 1.23operations/puppetproduction+31 -1
Add K8s DSE intermediate PKI configs and public certsoperations/puppetproduction+60 -0
Add fake intermediate PKI key for DSE k8slabs/privatemaster+0 -0
profile::pki::root_ca: add new pkis for the DSE k8s clusteroperations/puppetproduction+2 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gmodena subscribed.Feb 22 2023, 12:49 PM
gmodena added a comment.Feb 22 2023, 12:52 PM

Is there a known timeline / ETA for this upgrade?

elukey added a comment.Feb 22 2023, 2:18 PM

@gmodena are you doing any experiments on it? If not I can try to do it tomorrow or Friday, I have to wipe everything so this is why I am asking :)

gerritbot added a comment.Feb 22 2023, 2:33 PM

Change 891280 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::dse_k8s::{master,worker}: update settings to k8s 1.23

https://gerrit.wikimedia.org/r/891280

gerritbot added a project: Patch-For-Review.Feb 22 2023, 2:33 PM
gerritbot added a comment.Feb 22 2023, 2:55 PM

Change 891284 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: upgrade the DSE cluster to k8s 1.23

https://gerrit.wikimedia.org/r/891284

gmodena added a subscriber: Ottomata.Feb 22 2023, 3:52 PM

@elukey we have some Flink ops tasks this sprint that will require re-deploying our PoC app on DSE. It's unlikely that we'll deploy this week, and if you give us a maintenance window (or some heads up) we'll work around it.
tl;dr: let me know when we can use DSE, so that we don't get in your way :).

cc / @Ottomata

Ottomata added a comment.Feb 22 2023, 3:57 PM

@gmodena I think elukey can just do this anytime, no? We don't mind if our stuff is deleted, we can redeploy, right?

gmodena added a comment.Feb 22 2023, 3:59 PM

@Ottomata absolutely. Just wanted to sync so we avoid attempting deployments / experiments during a maintenance window.

gerritbot added a comment.Feb 22 2023, 4:14 PM

Change 891321 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] profile::pki::root_ca: add new pkis for the DSE k8s cluster

https://gerrit.wikimedia.org/r/891321

elukey updated the task description. (Show Details)Feb 22 2023, 4:16 PM

@gmodena we'll probably do it on Friday, there is some prep-work to be done and it will be done tomorrow :)

Ottomata added a comment.Feb 22 2023, 4:18 PM

Ah got it, sounds good!

gerritbot added a comment.Feb 22 2023, 5:36 PM

Change 891321 merged by Elukey:

[operations/puppet@production] profile::pki::root_ca: add new pkis for the DSE k8s cluster

https://gerrit.wikimedia.org/r/891321

gerritbot added a comment.Feb 22 2023, 5:48 PM

Change 891344 had a related patch set uploaded (by Elukey; author: Elukey):

[labs/private@master] Add fake intermediate PKI key for DSE k8s

https://gerrit.wikimedia.org/r/891344

gerritbot added a comment.Feb 22 2023, 5:48 PM

Change 891344 merged by Elukey:

[labs/private@master] Add fake intermediate PKI key for DSE k8s

https://gerrit.wikimedia.org/r/891344

elukey mentioned this in rLPRI6584ca8a4ec1: Add fake intermediate PKI key for DSE k8s.Feb 22 2023, 5:49 PM
gerritbot added a comment.Feb 22 2023, 5:52 PM

Change 891346 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] Add K8s DSE intermediate PKI configs and public certs

https://gerrit.wikimedia.org/r/891346

gerritbot added a comment.Feb 23 2023, 10:23 AM

Change 891346 merged by Elukey:

[operations/puppet@production] Add K8s DSE intermediate PKI configs and public certs

https://gerrit.wikimedia.org/r/891346

elukey added a comment.Feb 23 2023, 10:32 AM

New PKI intermediates added, puppet/deployment-charts changes are out for review as well :)

ops-monitoring-bot added a comment.Feb 24 2023, 8:00 AM

Icinga downtime and Alertmanager silence (ID=74cbf082-10f9-46b9-9315-de46465fbfba) set by elukey@cumin1001 for 8:00:00 on 8 host(s) and their services with reason: Downtime DSE workers for cluster upgrade

dse-k8s-worker[1001-1008].eqiad.wmnet
gerritbot added a comment.Feb 24 2023, 8:09 AM

Change 891280 merged by Elukey:

[operations/puppet@production] role::dse_k8s::{master,worker}: update settings to k8s 1.23

https://gerrit.wikimedia.org/r/891280

ops-monitoring-bot added a comment.Feb 24 2023, 9:10 AM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1002.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 9:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1003.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 9:11 AM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1004.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 9:12 AM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1005.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 9:13 AM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 9:13 AM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1007.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 9:13 AM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1008.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 9:27 AM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1008.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1008 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 24 2023, 10:07 AM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1003.eqiad.wmnet with OS bullseye completed:

  • dse-k8s-worker1003 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302240911_elukey_3529676_dse-k8s-worker1003.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 24 2023, 10:10 AM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1002.eqiad.wmnet with OS bullseye completed:

  • dse-k8s-worker1002 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302240910_elukey_3529569_dse-k8s-worker1002.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 24 2023, 10:12 AM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1004.eqiad.wmnet with OS bullseye completed:

  • dse-k8s-worker1004 (PASS)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302240911_elukey_3529799_dse-k8s-worker1004.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
ops-monitoring-bot added a comment.Feb 24 2023, 10:29 AM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1008.eqiad.wmnet with OS bullseye

gerritbot added a comment.Feb 24 2023, 10:32 AM

Change 891284 merged by Elukey:

[operations/deployment-charts@master] admin_ng: upgrade the DSE cluster to k8s 1.23

https://gerrit.wikimedia.org/r/891284

ops-monitoring-bot added a comment.Feb 24 2023, 10:52 AM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1005.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1005 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 24 2023, 10:59 AM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1006 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302240912_elukey_3530060_dse-k8s-worker1006.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 24 2023, 10:59 AM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1008.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1008 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 24 2023, 11:02 AM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1007.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1007 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302240913_elukey_3530187_dse-k8s-worker1007.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • The reimage failed, see the cookbook logs for the details
Maintenance_bot removed a project: Patch-For-Review.Feb 24 2023, 11:10 AM
ops-monitoring-bot added a comment.Feb 24 2023, 2:10 PM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1005.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 2:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 2:22 PM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1007.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 2:23 PM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1008.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 24 2023, 2:23 PM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1005.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1005 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 24 2023, 2:31 PM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1006 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • The reimage failed, see the cookbook logs for the details
elukey added a comment.Feb 24 2023, 2:33 PM

Full exception:

Exception raised while executing cookbook sre.hosts.reimage:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/spicerack/_menu.py", line 234, in run
    raw_ret = runner.run()
  File "/srv/deployment/spicerack/cookbooks/sre/hosts/reimage.py", line 499, in run
    fingerprint = self.puppet_installer.regenerate_certificate()[self.fqdn]
  File "/usr/lib/python3/dist-packages/spicerack/puppet.py", line 276, in regenerate_certificate
    raise PuppetHostsError(
spicerack.puppet.PuppetHostsError: Unable to find CSR fingerprints for all hosts, detected errors are:
dse-k8s-worker1006.eqiad.wmnet: Error: request https://puppet:8140//puppet-ca/v1/certificate/ca failed: Failed to open TCP connection to puppet:8140 (getaddrinfo: Temporary failure in name resolution)
dse-k8s-worker1006.eqiad.wmnet: Error: Could not request certificate: Failed to open TCP connection to puppet:8140 (getaddrinfo: Temporary failure in name resolution)
**The reimage failed, see the cookbook logs for the details**
Reimage executed with errors:
- dse-k8s-worker1006 (**FAIL**)
  - Downtimed on Icinga/Alertmanager
  - //Unable to disable Puppet, the host may have been unreachable//
  - Removed from Puppet and PuppetDB if present
  - Deleted any existing Puppet certificate
  - Removed from Debmonitor if present
  - Forced PXE for next reboot
  - Host rebooted via IPMI
  - Host up (Debian installer)
  - Host up (new fresh bullseye OS)
  - **The reimage failed, see the cookbook logs for the details**
ops-monitoring-bot added a comment.Feb 24 2023, 2:36 PM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1008.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1008 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • The reimage failed, see the cookbook logs for the details
elukey changed the task status from Open to Stalled.Feb 24 2023, 2:49 PM

There is a problem/bug triggered while reimaging nodes in row E/F in eqiad, tracked in T306421. Until it is fixed we cannot really complete the reimages of the dse1005->1008 workers.

ops-monitoring-bot added a comment.Feb 24 2023, 2:50 PM

Icinga downtime and Alertmanager silence (ID=b1fe3d5f-2fa2-4c9e-92d5-c7b84f294e1e) set by elukey@cumin1001 for 7 days, 0:00:00 on 1 host(s) and their services with reason: Cluster half broken, in the middle of upgrading

dse-k8s-worker1007.eqiad.wmnet
ops-monitoring-bot added a comment.Feb 24 2023, 2:50 PM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1007.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1007 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed, asking the operator what to do
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 25 2023, 11:03 AM

Icinga downtime and Alertmanager silence (ID=42315681-7e02-4c17-bd5c-eef6680a2aa9) set by elukey@cumin1001 for 4 days, 0:00:00 on 5 host(s) and their services with reason: Downtime DSE workers for cluster upgrade

dse-k8s-worker[1001-1004,1007].eqiad.wmnet
ops-monitoring-bot added a comment.Feb 27 2023, 9:19 AM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 27 2023, 9:32 AM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 27 2023, 10:26 AM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 27 2023, 11:05 AM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 27 2023, 11:07 AM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 27 2023, 11:20 AM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 27 2023, 11:51 AM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 27 2023, 12:04 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 27 2023, 12:21 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 27 2023, 12:34 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1006 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 27 2023, 2:14 PM

Cookbook cookbooks.sre.hosts.reimage was started by cmooney@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 27 2023, 2:49 PM

Cookbook cookbooks.sre.hosts.reimage started by cmooney@cumin1001 for host dse-k8s-worker1006.eqiad.wmnet with OS bullseye completed:

  • dse-k8s-worker1006 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302271414_cmooney_239320_dse-k8s-worker1006.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Feb 27 2023, 2:54 PM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1007.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 27 2023, 3:35 PM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1007.eqiad.wmnet with OS bullseye completed:

  • dse-k8s-worker1007 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302271454_elukey_249516_dse-k8s-worker1007.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Feb 27 2023, 3:35 PM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1007.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1007 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Unable to disable Puppet, the host may have been unreachable
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • Removed previous downtime on Alertmanager (old OS)
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302271454_elukey_249516_dse-k8s-worker1007.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
    • The reimage failed, see the cookbook logs for the details
ops-monitoring-bot added a comment.Feb 27 2023, 3:43 PM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1005.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 27 2023, 3:44 PM

Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host dse-k8s-worker1008.eqiad.wmnet with OS bullseye

ops-monitoring-bot added a comment.Feb 27 2023, 4:47 PM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1005.eqiad.wmnet with OS bullseye completed:

  • dse-k8s-worker1005 (PASS)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302271543_elukey_263784_dse-k8s-worker1005.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Feb 27 2023, 5:38 PM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1008.eqiad.wmnet with OS bullseye completed:

  • dse-k8s-worker1008 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302271544_elukey_263871_dse-k8s-worker1008.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
ops-monitoring-bot added a comment.Feb 27 2023, 5:38 PM

Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host dse-k8s-worker1008.eqiad.wmnet with OS bullseye executed with errors:

  • dse-k8s-worker1008 (FAIL)
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • First Puppet run failed, asking the operator what to do
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202302271544_elukey_263871_dse-k8s-worker1008.out
    • Checked BIOS boot parameters are back to normal
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (row E/F)
    • The reimage failed, see the cookbook logs for the details
gerritbot added a comment.Feb 27 2023, 5:50 PM

Change 892519 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: add istio network policy config for DSE

https://gerrit.wikimedia.org/r/892519

gerritbot added a project: Patch-For-Review.Feb 27 2023, 5:50 PM
gerritbot added a comment.Feb 27 2023, 5:55 PM

Change 892522 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/puppet@production] role::dse_k8s::worker: update istio-cni version

https://gerrit.wikimedia.org/r/892522

gerritbot added a comment.Feb 27 2023, 5:56 PM

Change 892522 merged by Elukey:

[operations/puppet@production] role::dse_k8s::worker: update istio-cni version

https://gerrit.wikimedia.org/r/892522

gerritbot added a comment.Feb 27 2023, 6:00 PM

Change 892519 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: add istio network policy config for DSE

https://gerrit.wikimedia.org/r/892519

elukey added a comment.Feb 27 2023, 6:05 PM

The DSE cluster is on k8s 1.23! I deployed everything up to istio/cfssl, we'll do more as soon as we need. There seems to be an issue with hosts in row E/F (see T306421) but I managed to bootstrap all of them anyway.

elukey changed the task status from Stalled to Open.Feb 27 2023, 6:05 PM
elukey triaged this task as Medium priority.
Maintenance_bot removed a project: Patch-For-Review.Feb 27 2023, 6:10 PM
cmooney awarded a token.Feb 27 2023, 11:05 PM
gerritbot added a comment.Feb 28 2023, 7:29 AM

Change 892870 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: refactor DSE 1.23 config and disable istio sidecars in ns

https://gerrit.wikimedia.org/r/892870

gerritbot added a project: Patch-For-Review.Feb 28 2023, 7:29 AM
gerritbot added a comment.Feb 28 2023, 7:37 AM

Change 892870 merged by Elukey:

[operations/deployment-charts@master] admin_ng: refactor DSE 1.23 config and disable istio sidecars in ns

https://gerrit.wikimedia.org/r/892870

Maintenance_bot removed a project: Patch-For-Review.Feb 28 2023, 8:10 AM
elukey claimed this task.Feb 28 2023, 1:35 PM
elukey moved this task from Unsorted to Complete Q3 2022/23 on the Machine-Learning-Team board.

Cluster upgraded!

elukey closed this task as Resolved.Mar 2 2023, 9:49 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits