Page MenuHomePhabricator
Create Task
Maniphest T363307

Co-locate kube-apiserver and etcd on new staging control plane nodes
Closed, ResolvedPublic
Actions

Description

In order to test migrating the wikikube control planes to hardware nodes and colocate with etcd we should migrate the wikikube clusters from the current setup with 2 control planes + 3 etcd VMs to 3 VMs that co-locate kube-apiserver and etcd.

This will be useful for testing new puppet roles as well as the procedure and will ensure that staging and prod clusters are set up the same way as prod (which is required so we can catch errors early during future k8s upgrades etc.).

  • Add two additional new stacked masters to staging-codfw
  • Remove the old control-planes and etcd nodes (delete VMs etc)
  • Repeat the process for staging-eqiad

Details

SubjectRepoBranchLines +/-
Add wikikube-worker configoperations/puppetproduction+6 -4
Remove role etcd::v3::kubernetes::stagingoperations/puppetproduction+1 -37
Remove remaining occurrences of kubernetes::staging::master roleoperations/puppetproduction+2 -20
Decom kubestagetcd100[123]operations/puppetproduction+0 -5
Remove kubestagetcd100[123] from etcd SRV recordsoperations/dnsmaster+0 -6
Decom kubestagemaster100[12]operations/puppetproduction+0 -15
Add kubestagemaster1005 to the etcd-server SRV recordoperations/dnsmaster+1 -0
Add kubestagemaster1004 to the etcd-server SRV recordoperations/dnsmaster+1 -0
Remove kubestagemaster100[45] from server SRV recordoperations/dnsmaster+0 -2
Add kubestagemaster100[345] as master_stackedoperations/puppetproduction+11 -6
Add kubestagemaster100[345] to the etcd-server SRV recordoperations/dnsmaster+3 -0
Decom kubestagetcd200[123]operations/puppetproduction+0 -5
prometheus/ops: Refactor etcd scraping, fix hostnamesoperations/puppetproduction+1 -0
prometheus/ops: Refactor etcd scrapingoperations/puppetproduction+31 -46
Remove kubernetesMasters definition from staging-codfwoperations/deployment-chartsmaster+0 -7
kubernetes::master: Make etcd_urls optionaloperations/puppetproduction+14 -13
Remove kubestagetcd200[123] from etcd SRV recordsoperations/dnsmaster+0 -8
Decom kubestagemaster200[12]operations/puppetproduction+7 -12
kubernetes::master: Retry kube-publish-sa-certs 5 timesoperations/puppetproduction+6 -0
Add kubestagemaster2005 as master_stackedoperations/puppetproduction+4 -5
Add kubestagemaster2005 to the etcd-server SRV recordoperations/dnsmaster+2 -0
Add kubestagemaster2004 as master_stackedoperations/puppetproduction+5 -2
Add kubestagemaster2004 to the etcd-server SRV recordoperations/dnsmaster+2 -0
Fix all-etcd, wikikube-master and wikikube-etcd aliasesoperations/puppetproduction+5 -5
Add kubestagemaster200[45] as insetup::serviceopsoperations/puppetproduction+4 -0
cfssl::cert: Add before_services parameteroperations/puppetproduction+20 -6
etcd: Notify etcd on PKI cert generation and renevaloperations/puppetproduction+2 -1
Fix copy/paste error for kubestagemaster2003operations/puppetproduction+1 -1
Add kubestage2003 to staging-codfw and conftooloperations/puppetproduction+3 -0
Add kubestagemaster2003 to the etcd-server SRV recordoperations/dnsmaster+1 -0
kubernetes::master: Add stacked control plane optionoperations/puppetproduction+81 -4
kubernetes::master: Add stacked control plane optionoperations/puppetproduction+30 -7
kubestagemaster2003: Add as insetup::serviceopsoperations/puppetproduction+4 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
JMeybohm updated the task description. (Show Details)Apr 24 2024, 12:31 PM
gerritbot added a comment.Apr 26 2024, 7:21 AM

Change #1024543 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubestagemaster2003: Add as insetup::serviceops

https://gerrit.wikimedia.org/r/1024543

gerritbot added a project: Patch-For-Review.Apr 26 2024, 7:21 AM
gerritbot added a comment.Apr 26 2024, 7:24 AM

Change #1024543 merged by JMeybohm:

[operations/puppet@production] kubestagemaster2003: Add as insetup::serviceops

https://gerrit.wikimedia.org/r/1024543

Maintenance_bot removed a project: Patch-For-Review.Apr 26 2024, 4:48 PM
gerritbot added a comment.Apr 29 2024, 9:03 AM

Change #1025278 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes::master: Add stacked control plane option

https://gerrit.wikimedia.org/r/1025278

gerritbot added a project: Patch-For-Review.Apr 29 2024, 9:03 AM
JMeybohm closed subtask T363310: Site: codfw 1 VM request for staging-codfw kube-apiserver as Resolved.Apr 29 2024, 9:15 AM
gerritbot added a comment.Apr 29 2024, 10:39 AM

Change #1025295 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes::master: Add stacked control plane option

https://gerrit.wikimedia.org/r/1025295

gerritbot added a comment.Apr 29 2024, 11:32 AM

Change #1025278 abandoned by JMeybohm:

[operations/puppet@production] kubernetes::master: Add stacked control plane option

Reason:

in favor of Ie59a97fd1934e219c5d1f20fe1186b8135f7118d

https://gerrit.wikimedia.org/r/1025278

JMeybohm triaged this task as High priority.Apr 29 2024, 11:38 AM
JMeybohm updated the task description. (Show Details)
gerritbot added a comment.Apr 29 2024, 1:09 PM

Change #1025278 restored by JMeybohm:

[operations/puppet@production] kubernetes::master: Add stacked control plane option

https://gerrit.wikimedia.org/r/1025278

gerritbot added a comment.Apr 29 2024, 1:36 PM

Change #1025295 abandoned by JMeybohm:

[operations/puppet@production] kubernetes::master: Add stacked control plane option

Reason:

In favor of I758b43e5e523b7f258a59a3bbbeb92a19c2850f0

https://gerrit.wikimedia.org/r/1025295

gerritbot added a comment.Apr 29 2024, 3:26 PM

Change #1025278 merged by JMeybohm:

[operations/puppet@production] kubernetes::master: Add stacked control plane option

https://gerrit.wikimedia.org/r/1025278

gerritbot added a comment.Apr 29 2024, 3:37 PM

Change #1025397 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Add kubestage2003 to staging-codfw and conftool

https://gerrit.wikimedia.org/r/1025397

gerritbot added a comment.Apr 29 2024, 3:49 PM

Change #1025399 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/dns@master] Add kubestagemaster2003 to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1025399

gerritbot added a comment.Apr 29 2024, 3:52 PM

Change #1025399 merged by JMeybohm:

[operations/dns@master] Add kubestagemaster2003 to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1025399

gerritbot added a comment.Apr 29 2024, 3:56 PM

Change #1025397 merged by JMeybohm:

[operations/puppet@production] Add kubestage2003 to staging-codfw and conftool

https://gerrit.wikimedia.org/r/1025397

gerritbot added a comment.Apr 29 2024, 3:59 PM

Change #1025404 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Fix copy/paste error for kubestagemaster2003

https://gerrit.wikimedia.org/r/1025404

gerritbot added a comment.Apr 29 2024, 4:01 PM

Change #1025404 merged by JMeybohm:

[operations/puppet@production] Fix copy/paste error for kubestagemaster2003

https://gerrit.wikimedia.org/r/1025404

JMeybohm added a comment.Apr 29 2024, 4:56 PM

I've added the new, stacked control-plan with some manual intervention as etcd did not come up initially which makes kube-apiserver-safe-restart wait forever for to acquire as lock.

Seems like etcd got started before the certificates where available (and not restarted after):

Apr 29 15:29:08 kubestagemaster2003 etcd[33145]: open /var/lib/etcd/ssl/etcd___etcd-server-ssl__tcp_k8s3-staging_codfw_wmnet.chained.pem: no such file or directory  
Apr 29 15:29:08 kubestagemaster2003 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE  
Apr 29 15:29:08 kubestagemaster2003 systemd[1]: etcd.service: Failed with result 'exit-code'.  
Apr 29 15:29:08 kubestagemaster2003 systemd[1]: Failed to start etcd - highly-available key value store.

After manual restart I figured that I should have added the new control-plane to the etcd server SRV record before running puppet:

Apr 29 15:44:46 kubestagemaster2003 etcd[45706]: error setting up initial cluster: cannot find local etcd member "kubestagemaster2003" in SRV records  
Apr 29 15:44:46 kubestagemaster2003 systemd[1]: etcd.service: Main process exited, code=exited, status=1/FAILURE  
Apr 29 15:44:46 kubestagemaster2003 systemd[1]: etcd.service: Failed with result 'exit-code'.  
Apr 29 15:44:46 kubestagemaster2003 systemd[1]: Failed to start etcd - highly-available key value store.

After doing that, the waiting puppet process finished successfully.

Now I'm blocked by T345823: Wikikube staging clusters are out of IPv4 Pod IP's as there is no free IPv4 pool for the new control-plane to claim, so I need to remove one of the existing ones.

gerritbot added a comment.Apr 29 2024, 5:18 PM

Change #1025422 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] etcd: Notify etcd on PKI cert generation and reneval

https://gerrit.wikimedia.org/r/1025422

gerritbot added a comment.Apr 30 2024, 8:49 AM

Change #1025422 abandoned by JMeybohm:

[operations/puppet@production] etcd: Notify etcd on PKI cert generation and reneval

Reason:

AIUI etcd does reload the Cert from disk with every client connection, so there is no need to restart (https://github.com/etcd-io/etcd/commit/4e21f87e3d014d606bb3ba2a89731a7d24806611). But we still need a way to describe the relation between the pki cert and the service.

https://gerrit.wikimedia.org/r/1025422

gerritbot added a comment.Apr 30 2024, 8:59 AM

Change #1025690 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] cfssl::cert: Add before_services parameter

https://gerrit.wikimedia.org/r/1025690

gerritbot added a comment.May 2 2024, 7:54 AM

Change #1025690 merged by JMeybohm:

[operations/puppet@production] cfssl::cert: Add before_services parameter

https://gerrit.wikimedia.org/r/1025690

gerritbot added a comment.May 13 2024, 1:21 PM

Change #1030955 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Add kubestagemaster200[45] as insetup::serviceops

https://gerrit.wikimedia.org/r/1030955

gerritbot added a comment.May 13 2024, 1:29 PM

Change #1030957 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/dns@master] Add kubestagemaster2004 to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1030957

gerritbot added a comment.May 13 2024, 1:31 PM

Change #1030958 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Add kubestagemaster2004 to staging-codfw conftool

https://gerrit.wikimedia.org/r/1030958

gerritbot added a comment.May 13 2024, 2:11 PM

Change #1030955 merged by JMeybohm:

[operations/puppet@production] Add kubestagemaster200[45] as insetup::serviceops

https://gerrit.wikimedia.org/r/1030955

gerritbot added a comment.May 13 2024, 2:46 PM

Change #1030995 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Add stacked kubernetes masters to appropriate aliases

https://gerrit.wikimedia.org/r/1030995

JMeybohm closed subtask T364740: Site: codfw 2 VM request for staging-codfw kube-apiserver as Resolved.May 14 2024, 8:31 AM
gerritbot added a comment.May 14 2024, 8:34 AM

Change #1030995 merged by JMeybohm:

[operations/puppet@production] Fix all-etcd, wikikube-master and wikikube-etcd aliases

https://gerrit.wikimedia.org/r/1030995

gerritbot added a comment.May 14 2024, 9:35 AM

Change #1030957 merged by JMeybohm:

[operations/dns@master] Add kubestagemaster2004 to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1030957

gerritbot added a comment.May 14 2024, 9:40 AM

Change #1030958 merged by JMeybohm:

[operations/puppet@production] Add kubestagemaster2004 as master_stacked

https://gerrit.wikimedia.org/r/1030958

JMeybohm closed subtask T364746: Site: eqiad 3 VM request for staging-eqiad kube-apiserver as Resolved.May 14 2024, 9:55 AM
Clement_Goubert mentioned this in T362323: Move 100% of external traffic to Kubernetes.May 14 2024, 12:04 PM
gerritbot added a comment.May 14 2024, 2:05 PM

Change #1031457 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/dns@master] Add kubestagemaster2005 to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1031457

gerritbot added a comment.May 14 2024, 2:06 PM

Change #1031460 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Add kubestagemaster2005 as master_stacked

https://gerrit.wikimedia.org/r/1031460

gerritbot added a comment.May 14 2024, 2:08 PM

Change #1031457 merged by JMeybohm:

[operations/dns@master] Add kubestagemaster2005 to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1031457

gerritbot added a comment.May 14 2024, 2:11 PM

Change #1031460 merged by JMeybohm:

[operations/puppet@production] Add kubestagemaster2005 as master_stacked

https://gerrit.wikimedia.org/r/1031460

gerritbot added a comment.May 14 2024, 4:24 PM

Change #1031507 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes::master: Retry kube-publish-sa-certs 5 times

https://gerrit.wikimedia.org/r/1031507

gerritbot added a comment.May 15 2024, 9:02 AM

Change #1031816 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/dns@master] Remove kubestagetcd200[123] from etcd SRV records

https://gerrit.wikimedia.org/r/1031816

gerritbot added a comment.May 15 2024, 9:04 AM

Change #1031507 merged by JMeybohm:

[operations/puppet@production] kubernetes::master: Retry kube-publish-sa-certs 5 times

https://gerrit.wikimedia.org/r/1031507

gerritbot added a comment.May 15 2024, 9:50 AM

Change #1031825 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Decom kubestagemaster200[12]

https://gerrit.wikimedia.org/r/1031825

ops-monitoring-bot added a comment.May 15 2024, 9:53 AM

cookbooks.sre.hosts.decommission executed by jayme@cumin1002 for hosts: kubestagemaster[2001-2002].codfw.wmnet

  • kubestagemaster2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
  • kubestagemaster2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
gerritbot added a comment.May 15 2024, 9:53 AM

Change #1031825 merged by JMeybohm:

[operations/puppet@production] Decom kubestagemaster200[12]

https://gerrit.wikimedia.org/r/1031825

gerritbot added a comment.May 15 2024, 12:20 PM

Change #1031882 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] kubernetes::master: Make etcd_urls optional

https://gerrit.wikimedia.org/r/1031882

gerritbot added a comment.May 15 2024, 12:21 PM

Change #1031816 merged by JMeybohm:

[operations/dns@master] Remove kubestagetcd200[123] from etcd SRV records

https://gerrit.wikimedia.org/r/1031816

ops-monitoring-bot added a comment.May 15 2024, 12:23 PM

Icinga downtime and Alertmanager silence (ID=5c048aeb-57ce-4f8d-8159-53dcf8b5fb78) set by jayme@cumin1002 for 2 days, 0:00:00 on 3 host(s) and their services with reason: decom

kubestagetcd[2001-2003].codfw.wmnet
gerritbot added a comment.May 15 2024, 12:54 PM

Change #1031882 merged by JMeybohm:

[operations/puppet@production] kubernetes::master: Make etcd_urls optional

https://gerrit.wikimedia.org/r/1031882

ops-monitoring-bot added a comment.May 15 2024, 12:57 PM

cookbooks.sre.hosts.decommission executed by jayme@cumin1002 for hosts: kubestagetcd[2001-2003].codfw.wmnet

  • kubestagetcd2001.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
  • kubestagetcd2002.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
  • kubestagetcd2003.codfw.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster codfw to Netbox
JMeybohm updated the task description. (Show Details)May 15 2024, 1:05 PM
gerritbot added a comment.May 15 2024, 1:19 PM

Change #1031901 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Remove kubernetesMasters definition from staging-codfw

https://gerrit.wikimedia.org/r/1031901

gerritbot added a comment.May 15 2024, 1:23 PM

Change #1031901 merged by JMeybohm:

[operations/deployment-charts@master] Remove kubernetesMasters definition from staging-codfw

https://gerrit.wikimedia.org/r/1031901

JMeybohm updated the task description. (Show Details)May 15 2024, 1:50 PM
gerritbot added a comment.May 16 2024, 8:41 AM

Change #1032394 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] prometheus/ops: Refactor etcd scraping

https://gerrit.wikimedia.org/r/1032394

gerritbot added a comment.May 16 2024, 11:53 AM

Change #1032394 merged by JMeybohm:

[operations/puppet@production] prometheus/ops: Refactor etcd scraping

https://gerrit.wikimedia.org/r/1032394

gerritbot added a comment.May 16 2024, 11:59 AM

Change #1032453 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] prometheus/ops: Refactor etcd scraping, fix hostnames

https://gerrit.wikimedia.org/r/1032453

gerritbot added a comment.May 16 2024, 12:05 PM

Change #1032453 merged by JMeybohm:

[operations/puppet@production] prometheus/ops: Refactor etcd scraping, fix hostnames

https://gerrit.wikimedia.org/r/1032453

JMeybohm updated the task description. (Show Details)May 16 2024, 12:19 PM
gerritbot added a comment.May 17 2024, 7:40 AM

Change #1032633 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/dns@master] Add kubestagemaster100[345] to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1032633

gerritbot added a comment.May 17 2024, 7:43 AM

Change #1032634 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Add kubestagemaster100[345] as master_stacked

https://gerrit.wikimedia.org/r/1032634

gerritbot added a comment.May 17 2024, 7:55 AM

Change #1032706 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Decom kubestagemaster100[12]

https://gerrit.wikimedia.org/r/1032706

gerritbot added a comment.May 17 2024, 8:04 AM

Change #1032707 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Decom kubestagetcd200[123]

https://gerrit.wikimedia.org/r/1032707

gerritbot added a comment.May 17 2024, 8:06 AM

Change #1032708 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/dns@master] Remove kubestagetcd100[123] from etcd SRV records

https://gerrit.wikimedia.org/r/1032708

gerritbot added a comment.May 17 2024, 8:07 AM

Change #1032709 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Decom kubestagetcd100[123]

https://gerrit.wikimedia.org/r/1032709

gerritbot added a comment.May 17 2024, 8:08 AM

Change #1032707 merged by JMeybohm:

[operations/puppet@production] Decom kubestagetcd200[123]

https://gerrit.wikimedia.org/r/1032707

gerritbot added a comment.May 17 2024, 9:27 AM

Change #1032633 merged by JMeybohm:

[operations/dns@master] Add kubestagemaster100[345] to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1032633

gerritbot added a comment.May 17 2024, 9:32 AM

Change #1032634 merged by JMeybohm:

[operations/puppet@production] Add kubestagemaster100[345] as master_stacked

https://gerrit.wikimedia.org/r/1032634

gerritbot added a comment.May 17 2024, 9:49 AM

Change #1032746 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/dns@master] Remove kubestagemaster100[45] from server SRV record

https://gerrit.wikimedia.org/r/1032746

gerritbot added a comment.May 17 2024, 9:51 AM

Change #1032746 merged by JMeybohm:

[operations/dns@master] Remove kubestagemaster100[45] from server SRV record

https://gerrit.wikimedia.org/r/1032746

gerritbot added a comment.May 17 2024, 10:11 AM

Change #1032748 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/dns@master] Add kubestagemaster1004 to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1032748

gerritbot added a comment.May 17 2024, 10:11 AM

Change #1032749 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/dns@master] Add kubestagemaster1005 to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1032749

gerritbot added a comment.May 17 2024, 10:24 AM

Change #1032748 merged by JMeybohm:

[operations/dns@master] Add kubestagemaster1004 to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1032748

gerritbot added a comment.May 17 2024, 10:37 AM

Change #1032749 merged by JMeybohm:

[operations/dns@master] Add kubestagemaster1005 to the etcd-server SRV record

https://gerrit.wikimedia.org/r/1032749

ops-monitoring-bot added a comment.May 17 2024, 11:51 AM

Icinga downtime and Alertmanager silence (ID=d858a874-17ca-4ab5-8c9c-7fea35f1c823) set by jayme@cumin1002 for 2 days, 0:00:00 on 2 host(s) and their services with reason: decom

kubestagemaster[1001-1002].eqiad.wmnet
gerritbot added a comment.May 17 2024, 12:09 PM

Change #1032706 merged by JMeybohm:

[operations/puppet@production] Decom kubestagemaster100[12]

https://gerrit.wikimedia.org/r/1032706

ops-monitoring-bot added a comment.May 17 2024, 12:12 PM

cookbooks.sre.hosts.decommission executed by jayme@cumin1002 for hosts: kubestagemaster[1001-1002].eqiad.wmnet

  • kubestagemaster1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
  • kubestagemaster1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
gerritbot added a comment.May 17 2024, 12:18 PM

Change #1032763 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] cumin/aliases: Remove role kubernetes::staging::master

https://gerrit.wikimedia.org/r/1032763

ops-monitoring-bot added a comment.May 17 2024, 12:24 PM

Icinga downtime and Alertmanager silence (ID=dd087345-70da-428c-8704-76433fe47872) set by jayme@cumin1002 for 2 days, 0:00:00 on 3 host(s) and their services with reason: decom

kubestagetcd[1004-1006].eqiad.wmnet
gerritbot added a comment.May 17 2024, 12:33 PM

Change #1032708 merged by JMeybohm:

[operations/dns@master] Remove kubestagetcd100[123] from etcd SRV records

https://gerrit.wikimedia.org/r/1032708

ops-monitoring-bot added a comment.May 17 2024, 12:56 PM

cookbooks.sre.hosts.decommission executed by jayme@cumin1002 for hosts: kubestagetcd[1004-1006].eqiad.wmnet

  • kubestagetcd1004.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
  • kubestagetcd1005.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
  • kubestagetcd1006.eqiad.wmnet (PASS)
    • Downtimed host on Icinga/Alertmanager
    • Found Ganeti VM
    • VM shutdown
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
    • VM removed
    • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
gerritbot added a comment.May 17 2024, 12:58 PM

Change #1032709 merged by JMeybohm:

[operations/puppet@production] Decom kubestagetcd100[123]

https://gerrit.wikimedia.org/r/1032709

JMeybohm updated the task description. (Show Details)May 17 2024, 1:53 PM
gerritbot added a comment.May 17 2024, 2:05 PM

Change #1032763 merged by JMeybohm:

[operations/puppet@production] Remove remaining occurrences of kubernetes::staging::master role

https://gerrit.wikimedia.org/r/1032763

JMeybohm closed this task as Resolved.May 17 2024, 2:20 PM

Both staging clusters have been migrated to stacked control-planes

gerritbot added a comment.May 20 2024, 11:18 PM

Change #1034193 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] cumin: Remove etcd::v3::kubernetes::staging from A:wikikube-staging-etcd

https://gerrit.wikimedia.org/r/1034193

gerritbot added a comment.May 21 2024, 8:42 AM

Change #1034193 merged by JMeybohm:

[operations/puppet@production] Remove role etcd::v3::kubernetes::staging

https://gerrit.wikimedia.org/r/1034193

gerritbot added a comment.May 22 2024, 3:08 PM

Change #1034956 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] Add wikikube-worker config

https://gerrit.wikimedia.org/r/1034956

gerritbot added a comment.May 23 2024, 7:26 AM

Change #1034956 merged by JMeybohm:

[operations/puppet@production] Add wikikube-worker config

https://gerrit.wikimedia.org/r/1034956

JMeybohm mentioned this in T366465: Extend puppet ipresolve() to support SRV records.Jun 3 2024, 11:36 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits