Page MenuHomePhabricator
Create Task
Maniphest T427401

Update istio to 1.29
Closed, ResolvedPublic
Actions

Description

In preparation of the next k8s upgrade(s) we need to update istio to 1.29 which is the last version supporting our currently running kubernetes version (1.31)

Update process

  • cp deployment-charts/custom_deploy.d/istio/<CLUSTER GROUP>/config.yaml deployment-charts/custom_deploy.d/istio/<CLUSTER GROUP>/config_1.29.yaml
  • Update the image tag to 1.29:
--- config.yaml 2026-07-21 16:44:17.712393143 +0000
+++ config_1.29.yaml    2026-07-21 16:44:17.712393143 +0000
@@ -2,7 +2,7 @@
 kind: IstioOperator
 spec:
   hub: docker-registry.discovery.wmnet/istio
-  tag: 1.24.2-1
+  tag: 1.29.4-1
   meshConfig:
     # This enables access logging in istio-proxy, which means all sidecars
     # as well as all ingress and egress gateways.
  • Run kube-env admin <CLUSTER>; istioctl-1.29.4 x precheck to validate the state of the current installation
  • Run kube-env admin <CLUSTER>; istioctl-1.29.4 install --dry-run --verify -f config_1.29.yaml
  • Run kube-env admin <CLUSTER>; istioctl-1.29.4 install --verify -f config_1.29.yaml
  • When you're done with all clusters of the cluster group, please git mv config_1.29.yaml config.yaml

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenJMeybohmT427069 Update Kubernetes clusters to 1.34
 ResolvedJMeybohmT427401 Update istio to 1.29
Restricted Task
 OpenNoneT434198 Update istio-cni to 1.29

Event Timeline

JMeybohm created this task.May 27 2026, 2:37 PM
JMeybohm removed JMeybohm as the assignee of this task.
jijiki triaged this task as Medium priority.May 28 2026, 8:28 AM
jijiki moved this task from Inbox to Scheduled (this Q) on the ServiceOps board.
MLechvien-WMF subscribed.Jun 8 2026, 12:24 PM

Will this patch {T423685} too?

JMeybohm added a comment.Jun 8 2026, 12:33 PM
In T427401#11993792, @MLechvien-WMF wrote:

Will this patch {T423685} too?

yes

JMeybohm added a subtask: Restricted Task.Jun 8 2026, 12:34 PM
gerritbot added a comment.Jun 10 2026, 12:35 PM

Change #1300115 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/puppet@production] aptrepo: Add components for calico, istio and kubernetes

https://gerrit.wikimedia.org/r/1300115

gerritbot added a project: Patch-For-Review.Jun 10 2026, 12:35 PM
gerritbot added a comment.Jun 10 2026, 1:17 PM

Change #1300138 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/debs/istioctl@master] Add 1.29.4, drop 1.15.7

https://gerrit.wikimedia.org/r/1300138

gerritbot added a comment.Jun 10 2026, 1:50 PM

Change #1300115 merged by JMeybohm:

[operations/puppet@production] aptrepo: Add components for calico, istio and kubernetes

https://gerrit.wikimedia.org/r/1300115

gerritbot added a comment.Jun 16 2026, 9:13 AM

Change #1300138 merged by JMeybohm:

[operations/debs/istioctl@master] Add 1.29.4, drop 1.15.7

https://gerrit.wikimedia.org/r/1300138

JMeybohm claimed this task.Jun 16 2026, 9:15 AM
JMeybohm moved this task from Scheduled (this Q) to In Progress on the ServiceOps board.
Stashbot added a comment.Jun 16 2026, 9:31 AM

Mentioned in SAL (#wikimedia-operations) [2026-06-16T09:31:21Z] <jayme> imported istioctl 1.29.4-1 to bookworm-/trixie-wikimedia - T427401

Maintenance_bot removed a project: Patch-For-Review.Jun 16 2026, 9:31 AM
JMeybohm mentioned this in rODIS78a9dfe48826: Add 1.29.4, drop 1.15.7.Jun 16 2026, 9:49 AM
gerritbot added a comment.Jun 17 2026, 3:42 PM

Change #1303475 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/docker-images/production-images@master] Update istio to 1.29.4

https://gerrit.wikimedia.org/r/1303475

gerritbot added a project: Patch-For-Review.Jun 17 2026, 3:42 PM
gerritbot added a comment.Jun 22 2026, 10:27 AM

Change #1303475 merged by JMeybohm:

[operations/docker-images/production-images@master] Update istio to 1.29.4

https://gerrit.wikimedia.org/r/1303475

Maintenance_bot removed a project: Patch-For-Review.Jun 22 2026, 10:31 AM
gerritbot added a comment.Jun 29 2026, 1:44 PM

Change #1306279 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Copy wikikube istio config to config_1.29.yaml

https://gerrit.wikimedia.org/r/1306279

gerritbot added a project: Patch-For-Review.Jun 29 2026, 1:44 PM

Change #1306280 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] istio/main: Bump to istio 1.29.4-1

https://gerrit.wikimedia.org/r/1306280

gerritbot added a comment.Jun 29 2026, 2:04 PM

Change #1306292 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/debs/istioctl@master] Readd 1.15.7

https://gerrit.wikimedia.org/r/1306292

gerritbot added a comment.Jun 29 2026, 2:17 PM

Change #1306292 merged by JMeybohm:

[operations/debs/istioctl@master] Readd 1.15.7

https://gerrit.wikimedia.org/r/1306292

gerritbot added a comment.Jun 29 2026, 2:24 PM

Change #1306319 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Rakefile: Support multiple istio versions

https://gerrit.wikimedia.org/r/1306319

gerritbot added a comment.Jun 29 2026, 2:31 PM

Change #1306321 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[integration/config@master] helm-linter: Update istioctl package to 1.29.4-2

https://gerrit.wikimedia.org/r/1306321

gerritbot added a comment.Jun 29 2026, 2:31 PM

Change #1306322 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[integration/config@master] helm-linter: Use version 0.8.2

https://gerrit.wikimedia.org/r/1306322

gerritbot added a comment.Jun 29 2026, 2:40 PM

Change #1306321 merged by jenkins-bot:

[integration/config@master] helm-linter: Update istioctl package to 1.29.4-2

https://gerrit.wikimedia.org/r/1306321

gerritbot added a comment.Jun 29 2026, 2:45 PM

Change #1306327 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Clean up istio configs

https://gerrit.wikimedia.org/r/1306327

gerritbot added a comment.Jun 29 2026, 2:52 PM

Change #1306322 merged by jenkins-bot:

[integration/config@master] jjb: [helm-lint] Use version 0.8.2

https://gerrit.wikimedia.org/r/1306322

JMeybohm mentioned this in rODISadb1c4cb765a: Readd 1.15.7.Jun 29 2026, 3:12 PM
gerritbot added a comment.Jun 29 2026, 3:29 PM

Change #1306319 merged by jenkins-bot:

[operations/deployment-charts@master] Rakefile: Support multiple istio versions

https://gerrit.wikimedia.org/r/1306319

gerritbot added a comment.Jun 29 2026, 4:24 PM

Change #1306279 merged by JMeybohm:

[operations/deployment-charts@master] Copy wikikube istio config to config_1.29.yaml

https://gerrit.wikimedia.org/r/1306279

gerritbot added a comment.Jun 29 2026, 4:24 PM

Change #1306280 merged by JMeybohm:

[operations/deployment-charts@master] istio/main: Bump to istio 1.29.4-1

https://gerrit.wikimedia.org/r/1306280

gerritbot added a comment.Jul 6 2026, 8:26 AM

Change #1306327 merged by jenkins-bot:

[operations/deployment-charts@master] Clean up istio configs

https://gerrit.wikimedia.org/r/1306327

Maintenance_bot removed a project: Patch-For-Review.Jul 6 2026, 8:31 AM
Stashbot added a comment.Jul 7 2026, 1:15 PM

Mentioned in SAL (#wikimedia-operations) [2026-07-07T13:15:36Z] <jayme> Istio is being upgraded from 1.24.2 to 1.29.4 on wikikube staging eqiad and codfw - T427401

gerritbot added a comment.Jul 9 2026, 3:50 PM

Change #1309242 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] kind.sh: Use istio 1.29.4

https://gerrit.wikimedia.org/r/1309242

gerritbot added a project: Patch-For-Review.Jul 9 2026, 3:50 PM
gerritbot added a comment.Jul 9 2026, 3:55 PM

Change #1309242 merged by jenkins-bot:

[operations/deployment-charts@master] kind.sh: Use istio 1.29.4

https://gerrit.wikimedia.org/r/1309242

Maintenance_bot removed a project: Patch-For-Review.Jul 9 2026, 4:31 PM
• Scott_French added subscribers: ssingh, • Scott_French.EditedJul 15 2026, 7:22 PM

@ssingh pointed out earlier today that PyBal on the low-traffic and secondary LVS hosts in codfw is alerting with:

PYBAL CRITICAL - CRITICAL - k8s-ingress-staging_30443: Servers kubestage2003.codfw.wmnet, kubestage2004.codfw.wmnet are marked down but pooled

It looks like none of the 4 codfw staging workers are responsive on 30443 (the alert will only mention two, since the depool threshold is 50%).

In any case, I wanted to connect these dots here in case it's an artifact of the testing work ahead of the Istio upgrade. This is of course "fine" since it's staging-codfw (which we generally use for disruptive testing like this), but if it is related, might indicate something isn't quite right there.

Edit: Alright, interesting - I just dumped staging-codfw's ingressgateway envoy config, and it contains no 8443 listener (i.e., the upstream port to which the istio-ingressgateway service forwards the 30443 NodePort).

In fact, it contains no dynamic listeners at all, and in contrast to staging-eqiad, where the config looks like what I'd expect.

Anyway, that's about the depth of my Istio knowledge, but it does indeed look like something is missing in the resources that drive that part of the config.

JMeybohm added a comment.Jul 27 2026, 1:39 PM

Sorry...this is due to the fact that the listener is only configured by istio when there is at least one virtual service (e.g. one thing using ingress) in the cluster. I would assume that's not the case for staging-codfw currently. While that is fine technically, I will find something appropriate to deploy there so that istio is set up "properly" from pybal perspective

Stashbot added a comment.Aug 6 2026, 7:37 AM

Mentioned in SAL (#wikimedia-operations) [2026-08-06T07:37:41Z] <jayme> updated istio to 1.29.4 on wikikube codfw - T427401

JMeybohm updated the task description. (Show Details)Aug 6 2026, 7:47 AM
JMeybohm updated the task description. (Show Details)
gerritbot added a comment.Aug 6 2026, 1:18 PM

Change #1321949 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/debs/istio@master] Update to 1.29.4

https://gerrit.wikimedia.org/r/1321949

gerritbot added a project: Patch-For-Review.Aug 6 2026, 1:18 PM
JMeybohm created subtask T434198: Update istio-cni to 1.29.Aug 6 2026, 1:23 PM
Stashbot added a comment.Aug 7 2026, 6:35 AM

Mentioned in SAL (#wikimedia-operations) [2026-08-07T06:35:57Z] <jayme> updated istio to 1.29.4 on wikikube eqiad - T427401

JMeybohm closed subtask Restricted Task as Resolved.Aug 7 2026, 6:40 AM
gerritbot added a comment.Aug 7 2026, 6:42 AM

Change #1322553 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] istio/main: Make istio 1.29 the default for wikikube

https://gerrit.wikimedia.org/r/1322553

JMeybohm updated the task description. (Show Details)Aug 19 2026, 10:43 AM
JMeybohm closed this task as Resolved.Sep 9 2026, 9:22 AM

Updated on all wikikube-clusters

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits