Page MenuHomePhabricator
Create Task
Maniphest T293729

setup/install kubestage100[34]
Closed, ResolvedPublic
Actions

Description

New nodes kubestage100[34] have been handed over by DC-Ops and need to be setup/added do the cluster.

These are replacements for kubestage100[12], so those need to be decommissioned afterwards.

I don't think we have a proper documentation on how to do that (in Kubernetes context). That should be an outcome of this as well.
We actually had a bit of documentation from the "Create a new Cluster" perspective. I moved that out and extended it a bit (but I might not have caught every step/aspect): https://wikitech.wikimedia.org/wiki/Kubernetes/Clusters/Add_or_remove_nodes

After this is done, we should remote the workaround for mediawiki images in CI: T284628

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
kubernetes: remove kubestage1001 & kubestage1002operations/homer/publicmaster+0 -2
kubernetes: remove kubestage1001 & kubestage1002operations/puppetproduction+1 -9
kubernetes: point to new kubestage nodeoperations/dnsmaster+1 -1
changeprop: increase memory limit for stagingoperations/deployment-chartsmaster+1 -1
sites: add new kubestage nodesoperations/homer/publicmaster+4 -2
hieradata: add kubestage bgp peersoperations/puppetproduction+2 -0
site: include new k8s hosts on kubestage groupoperations/puppetproduction+7 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

JMeybohm created this task.Oct 19 2021, 7:41 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 19 2021, 7:41 AM
JMeybohm updated the task description. (Show Details)Oct 19 2021, 11:14 AM
Dzahn subscribed.Oct 21 2021, 11:00 PM

Want help with this? I / we could put OS and role on it, see if any puppet issues, then meet with you to go through the actual adding-to-the-cluster part in a shared session. We'd and take notes for docs.

JMeybohm updated the task description. (Show Details)Nov 18 2021, 11:27 AM
gerritbot added a comment.Nov 18 2021, 4:09 PM

Change 739857 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] site: include new k8s hosts on kubestage group

https://gerrit.wikimedia.org/r/739857

gerritbot added a project: Patch-For-Review.Nov 18 2021, 4:09 PM
gerritbot added a comment.Nov 18 2021, 5:18 PM

Change 739879 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/homer/public@master] sites: add new kubestage nodes

https://gerrit.wikimedia.org/r/739879

gerritbot added a comment.Nov 18 2021, 7:54 PM

Change 739857 merged by AOkoth:

[operations/puppet@production] site: include new k8s hosts on kubestage group

https://gerrit.wikimedia.org/r/739857

gerritbot added a comment.Nov 20 2021, 8:59 PM

Change 740314 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] hieradata: add kubestage bgp peers

https://gerrit.wikimedia.org/r/740314

gerritbot added a comment.Nov 20 2021, 9:00 PM

Change 740314 merged by AOkoth:

[operations/puppet@production] hieradata: add kubestage bgp peers

https://gerrit.wikimedia.org/r/740314

gerritbot added a comment.Nov 22 2021, 3:20 PM

Change 739879 merged by jenkins-bot:

[operations/homer/public@master] sites: add new kubestage nodes

https://gerrit.wikimedia.org/r/739879

jenkins-bot mentioned this in rOHPU6321fe1d58e2: sites: add new kubestage nodes.Nov 22 2021, 3:20 PM
Maintenance_bot removed a project: Patch-For-Review.Nov 22 2021, 4:10 PM
Stashbot added a comment.Nov 23 2021, 9:57 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-23T09:57:34Z] <jayme> cordoned kubestage1001.eqiad.wmnet kubestage1002.eqiad.wmnet - T293729

Stashbot added a comment.Nov 23 2021, 11:05 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-23T11:05:42Z] <jayme> uncordoned kubestage1001.eqiad.wmnet kubestage1002.eqiad.wmnet (we have issues with POD IP prefix allocation) - T293729

Stashbot added a comment.Nov 23 2021, 11:05 AM

Mentioned in SAL (#wikimedia-operations) [2021-11-23T11:05:55Z] <jayme> cordoned kubestage1003.eqiad.wmnet kubestage1004.eqiad.wmnet (we have issues with POD IP prefix allocation) - T293729

JMeybohm mentioned this in T296303: New Kubernetes nodes may end up with no Pod IPv4 block assigned.Nov 23 2021, 2:41 PM
Stashbot added a comment.Nov 25 2021, 2:25 PM

Mentioned in SAL (#wikimedia-operations) [2021-11-25T14:25:08Z] <jayme> uncordoned kubestage1003.eqiad.wmnet kubestage1004.eqiad.wmnet - T293729

JMeybohm assigned this task to Arnoldokoth.Nov 25 2021, 2:30 PM

@Arnoldokoth the new nodes now have a ipam block assigned (I moved some test workload there to verify). From my POV you can continue with this when you have time (decom kubestage100[12]).

gerritbot added a comment.Dec 20 2021, 6:28 PM

Change 748781 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/deployment-charts@master] changeprop: increase memory limit for staging

https://gerrit.wikimedia.org/r/748781

gerritbot added a project: Patch-For-Review.Dec 20 2021, 6:28 PM
gerritbot added a comment.Jan 4 2022, 2:52 PM

Change 748781 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop: increase memory limit for staging

https://gerrit.wikimedia.org/r/748781

Maintenance_bot removed a project: Patch-For-Review.Jan 4 2022, 3:10 PM
ops-monitoring-bot added a comment.Jan 5 2022, 9:48 AM

cookbooks.sre.hosts.decommission executed by aokoth@cumin1001 for hosts: kubestage1001.eqiad.wmnet

  • kubestage1001.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)

ERROR: some step on some host failed, check the bolded items above

ops-monitoring-bot added a comment.Jan 5 2022, 11:34 AM

cookbooks.sre.hosts.decommission executed by aokoth@cumin1001 for hosts: kubestage1002.eqiad.wmnet

  • kubestage1002.eqiad.wmnet (PASS)
    • Downtimed host on Icinga
    • Found physical host
    • Downtimed management interface on Icinga
    • Wiped all swraid, partition-table and filesystem signatures
    • Powered off
    • Set Netbox status to Decommissioning and deleted all non-mgmt interfaces and related IPs
    • Removed from DebMonitor
    • Removed from Puppet master and PuppetDB
  • COMMON_STEPS (FAIL)
    • Failed to run the sre.dns.netbox cookbook: Cumin execution failed (exit_code=2)

ERROR: some step on some host failed, check the bolded items above

gerritbot added a comment.Jan 5 2022, 3:05 PM

Change 751752 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] kubernetes: remove kubestage1001 & kubestage1002

https://gerrit.wikimedia.org/r/751752

gerritbot added a project: Patch-For-Review.Jan 5 2022, 3:05 PM
gerritbot added a comment.Jan 5 2022, 3:20 PM

Change 751754 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/homer/public@master] kubernetes: remove kubestage1001 & kubestage1002

https://gerrit.wikimedia.org/r/751754

gerritbot added a comment.Jan 6 2022, 6:30 PM

Change 751976 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/dns@master] kubernetes: point to new kubestage node

https://gerrit.wikimedia.org/r/751976

gerritbot added a comment.Jan 7 2022, 3:26 PM

Change 751976 merged by AOkoth:

[operations/dns@master] kubernetes: point to new kubestage node

https://gerrit.wikimedia.org/r/751976

gerritbot added a comment.Jan 7 2022, 4:02 PM

Change 751752 merged by AOkoth:

[operations/puppet@production] kubernetes: remove kubestage1001 & kubestage1002

https://gerrit.wikimedia.org/r/751752

gerritbot added a comment.Jan 7 2022, 4:04 PM

Change 751754 merged by AOkoth:

[operations/homer/public@master] kubernetes: remove kubestage1001 & kubestage1002

https://gerrit.wikimedia.org/r/751754

Arnoldokoth added a subtask: T299142: decommission kubestage100[12]-eqiad.Jan 13 2022, 3:13 PM
• Cmjohnson closed subtask T299142: decommission kubestage100[12]-eqiad as Resolved.Jan 20 2022, 5:05 PM
Arnoldokoth moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.Jan 27 2022, 5:31 PM
akosiaris closed this task as Resolved.Mar 21 2022, 3:52 PM
akosiaris subscribed.

This has been done, resolving!

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits