Page MenuHomePhabricator
Create Task
Maniphest T384450

Update wikikube-staging-codfw to kubernetes 1.31
Closed, ResolvedPublic
Actions

Description

Things are going to be a bit different this time (compared to T326340: Update staging-codfw to k8s 1.23) given we need to be able to update clusters without the need to reimage all workers.

The very generic plan is:

  • Downtime the cluster (ctrl, worker, etcd)
  • Disable puppet on ctrl and worker nodes
  • Stop k8s components on ctrl and worker nodes
  • Delete etcd data
  • Merge updated version and calico_version in hieradata/common/kubernetes.yaml (1110813)
  • Enable and run puppet on ctrl
  • Enable and run puppet on workers
  • Deploy admin_ng
  • Deploy services
  • Deploy some version of mediawiki to validate PSP to VAP migration (T273507)

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
staging-codfw: Unset image.tag for coredns to apply the default versionoperations/deployment-chartsmaster+5 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

JMeybohm created this task.Jan 22 2025, 11:12 AM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJan 22 2025, 11:12 AM
Gehel edited projects, added Data-Platform-SRE (2025.01.11 - 2025.01.31); removed Data-Platform-SRE.Jan 24 2025, 9:37 AM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.01.11 - 2025.01.31) board.
Gehel subscribed.

Moving to our current work board to keep visibility on the progress for DPE SRE.

JMeybohm changed the task status from Open to In Progress.Jan 30 2025, 5:19 PM
JMeybohm claimed this task.
Stashbot added a comment.Jan 30 2025, 5:20 PM

Mentioned in SAL (#wikimedia-operations) [2025-01-30T17:20:12Z] <jayme> staging-codfw k8s cluster is currently being updated to k8s 1.31 and in an unusable state - T384450

JMeybohm added a comment.Feb 3 2025, 10:28 AM

FTR: A CertManagerCertNotReady critical alert fired for staging-codw which should not happen for staging-codfw at all.

jijiki triaged this task as Medium priority.Feb 3 2025, 1:33 PM
jijiki moved this task from Incoming 🐫 to Doing 😎 on the serviceops-deprecated board.
Gehel edited projects, added Data-Platform-SRE (2025.02.10 - 2025.02.28); removed Data-Platform-SRE (2025.01.11 - 2025.01.31).Feb 11 2025, 10:03 AM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.02.10 - 2025.02.28) board.
JMeybohm raised the priority of this task from Medium to High.Feb 28 2025, 1:05 PM
Gehel edited projects, added Data-Platform-SRE (2025.03.01 - 2025.03.21); removed Data-Platform-SRE (2025.02.10 - 2025.02.28).Feb 28 2025, 1:26 PM
Gehel moved this task from Backlog - project to Blocked/Waiting on the Data-Platform-SRE (2025.03.01 - 2025.03.21) board.
LSobanski moved this task from Incoming to K8s on the collaboration-services board.Mar 4 2025, 9:36 AM
Gehel removed a project: Data-Platform-SRE (2025.03.01 - 2025.03.21).Mar 4 2025, 4:38 PM
gerritbot added a comment.Mar 5 2025, 4:01 PM

Change #1124831 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] staging-codfw: Unset image.tag for coredns to apply the default version

https://gerrit.wikimedia.org/r/1124831

gerritbot added a project: Patch-For-Review.Mar 5 2025, 4:02 PM
gerritbot added a comment.Mar 7 2025, 3:13 PM

Change #1124831 merged by jenkins-bot:

[operations/deployment-charts@master] staging-codfw: Unset image.tag for coredns to apply the default version

https://gerrit.wikimedia.org/r/1124831

JMeybohm merged a task: T388398: Alert in need of triage: HelmfileAdminNGPendingChanges (instance deploy1003:9100).Mar 10 2025, 12:31 PM
JMeybohm added a subscriber: LSobanski.
Jelto added a subscriber: Raine.Mar 11 2025, 3:49 PM

@JMeybohm @kamila and I did a upgrade of staging-codfw to 1.31. We used the sre.k8s.wipe-cluster cookbook from https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1115380.

After fixing a few dependency issues in admin-ng we were able to apply admin-ng properly to the fresh cluster. However the next step, deploying istio, failed. The istio pods which were running on the control plane nodes had DNS issues and did not became ready. This was related to the pod ip range change from 10.192.75.0/24 to 10.192.64.0/21 which we combined with the cluster reinstall.

After running homer on the core routers the istio pods became ready and inter-pod traffic worked fine. Thanks @JMeybohm for troubleshooting this.

Deploying a service (ipoid) was successful as well.

JMeybohm mentioned this in T386232: Add pod ip address blocks to staging.Mar 11 2025, 5:31 PM
JMeybohm closed this task as Resolved.Mar 17 2025, 2:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits