Page MenuHomePhabricator

Upgrade Envoy to v1.35.7
Closed, ResolvedPublic

Description

As of this writing we're still concluding T405808 for the 1.29 -> 1.32 bump, but in parallel we can start planning the next step, 1.32 -> 1.35.

This will conclude the multi-stage push in T380211, and it will be the last upgrade for the immediate future (until 1.38 is released in the spring of 2026, or we have an operational need for a feature or bug fix sooner).

Release notes of potential interest (1.33, 1.34, 1.35):

Config changes post-upgrade

  • (1.33.0) cluster: DNS-related fields in Cluster are deprecated when using strict and logical DNS clusters. Instead, use the cluster_type extension point with typed_config of type DnsCluster.

Probably no effect

  • (1.33.0) http: RFC1918 addresses are no longer considered to be internal addresses by default. This addresses a security issue for Envoy’s in multi-tenant mesh environments. Please explicit set internal_address_config to retain the prior behavior. This change can be temporarily reverted by setting runtime guard envoy.reloadable_features.explicit_internal_address_config to false.
    • In previous upgrades we started explicitly setting internal_address_config, so this should be a no-op. If we have any Envoy installation without an explicit config, it's already logging warnings at startup as of 1.32.
  • (1.34.0) http2: Sets runtime guard envoy.reloadable_features.http2_use_oghttp2 to true by default.
    • This finally enables oghttp2 by default, after some back and forth in previous versions. Our Envoys receive no untrusted traffic so we likely won't notice any difference, but documenting in case of edge-case behavior changes.

Bare-metal hosts still upgradable:

  • an-tool[1007-1008]
  • aphlict[1002,2001]
  • apus-fe[1003-5,2003-5] to v1.35.9
  • chartmuseum[1001,2001]
  • cloudweb[1003-1004]
  • config-master[1001,2001]
  • debmonitor[1003,2003]
  • doc[1004,2003]
  • idm-test1001
  • idp[1005,2005]
  • lists1004
  • logstash[1023-1025,1030-1032,2023-2025,2030-2032]
  • matomo1003
  • moss-fe[1001-1002,2001-2002] (decommissioned)
  • ms-fe[1009-1024,2009-2024] to v1.35.9
  • people[1005,2004]
  • phab1004
  • planet[1003,2003]
  • prometheus[1005-10082,005-2008,3004,4002,5002,6002,7002]
  • puppetboard[1003,2003]
  • puppetserver[1001-1003,2001-2002,2004]
  • restbase[1031-1045,2024-2038]
  • schema[1003-1004,2003-2004]
  • thanos-fe[1004-1007,2004-2007] to v1.35.9
  • titan[1001-1002,2001-2002]
  • vrts1003
  • wcqs[1001-1003,2001-2003]
  • wdqs[1011-1027,2007-2027]

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

Change #1210776 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/debs/envoyproxy@v1.35] Update to v1.35.6

https://gerrit.wikimedia.org/r/1210776

Change #1210776 merged by RLazarus:

[operations/debs/envoyproxy@v1.35] Update to v1.35.6

https://gerrit.wikimedia.org/r/1210776

Mentioned in SAL (#wikimedia-operations) [2025-11-25T02:36:05Z] <rzl> rzl@apt1002:~$ sudo -i reprepro -C component/envoy-future include bullseye-wikimedia /home/rzl/envoyproxy_1.35.6-1_amd64.changes # T410975

Change #1210806 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/docker-images/production-images@master] envoy-future: Update to v1.35.6

https://gerrit.wikimedia.org/r/1210806

Change #1210806 merged by RLazarus:

[operations/docker-images/production-images@master] envoy-future: Update to v1.35.6

https://gerrit.wikimedia.org/r/1210806

Change #1211765 had a related patch set uploaded (by RLazarus; author: RLazarus):

[integration/config@master] helm-linter: Bump for Envoy 1.35.6

https://gerrit.wikimedia.org/r/1211765

Change #1211766 had a related patch set uploaded (by RLazarus; author: RLazarus):

[integration/config@master] jjb: Update to helm-linter:0.7.6 to pick up envoy-future 1.35.6

https://gerrit.wikimedia.org/r/1211766

Change #1211765 merged by jenkins-bot:

[integration/config@master] helm-linter: Bump for Envoy 1.35.6

https://gerrit.wikimedia.org/r/1211765

Change #1211766 merged by jenkins-bot:

[integration/config@master] jjb: Update to helm-linter:0.7.6 to pick up envoy-future 1.35.6

https://gerrit.wikimedia.org/r/1211766

Envoy 1.35.7 is about to come out, with security fixes: https://groups.google.com/g/envoy-announce/c/zr2OzwmJFqY

None of these issues affect us urgently, but since we're early in the 1.35 process anyway, I'll bump to 1.35.7 instead of 1.35.6.

RLazarus renamed this task from Upgrade Envoy to v1.35.6 to Upgrade Envoy to v1.35.7.Dec 3 2025, 11:44 PM

Change #1215349 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/debs/envoyproxy@v1.35] Update to v1.35.7

https://gerrit.wikimedia.org/r/1215349

Change #1215349 merged by RLazarus:

[operations/debs/envoyproxy@v1.35] Update to v1.35.7

https://gerrit.wikimedia.org/r/1215349

Change #1215363 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/docker-images/production-images@master] envoy-future: Update to v1.35.7

https://gerrit.wikimedia.org/r/1215363

Change #1215363 merged by RLazarus:

[operations/docker-images/production-images@master] envoy-future: Update to v1.35.7

https://gerrit.wikimedia.org/r/1215363

Change #1215366 had a related patch set uploaded (by RLazarus; author: RLazarus):

[integration/config@master] helm-linter: Bump for Envoy 1.35.6

https://gerrit.wikimedia.org/r/1215366

Change #1215367 had a related patch set uploaded (by RLazarus; author: RLazarus):

[integration/config@master] jjb: Update to helm-linter:0.7.7 to pick up envoy-future 1.35.7

https://gerrit.wikimedia.org/r/1215367

Change #1215366 merged by jenkins-bot:

[integration/config@master] Docker: [helm-linter] Bump for Envoy 1.35.7

https://gerrit.wikimedia.org/r/1215366

Mentioned in SAL (#wikimedia-releng) [2025-12-05T13:39:40Z] <James_F> Docker: [helm-linter] Bump for Envoy 1.35.7, for T410975

Change #1215367 merged by jenkins-bot:

[integration/config@master] jjb: [helm-lint] Update to helm-linter:0.7.7 to pick up envoy-future 1.35.7

https://gerrit.wikimedia.org/r/1215367

Change #1216701 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] mathoid: Upgrade to envoy-future:1.35.7 for validation

https://gerrit.wikimedia.org/r/1216701

Change #1216702 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] {api,rest}-gateway: Update staging to Envoy 1.35.7 for validation

https://gerrit.wikimedia.org/r/1216702

Change #1216701 merged by jenkins-bot:

[operations/deployment-charts@master] mathoid: Upgrade to envoy-future:1.35.7 for validation

https://gerrit.wikimedia.org/r/1216701

Mentioned in SAL (#wikimedia-operations) [2025-12-10T23:36:07Z] <rzl> rzl@deploy2002:/srv/deployment-charts/helmfile.d/services/mw-debug$ helmfile -e codfw -i apply -l name=pinkunicorn --set mesh.image_name=envoy-future --set mesh.image_version=1.35.7-1 --context=5 # T410975

Mentioned in SAL (#wikimedia-operations) [2025-12-10T23:40:59Z] <rzl> rzl@deploy2002:/srv/deployment-charts/helmfile.d/services/mw-debug$ helmfile -e codfw -i apply -l name=pinkunicorn --context=5 # T410975

Change #1216702 merged by jenkins-bot:

[operations/deployment-charts@master] {api,rest}-gateway: Update staging to Envoy 1.35.7 for validation

https://gerrit.wikimedia.org/r/1216702

Mentioned in SAL (#wikimedia-operations) [2025-12-11T00:16:11Z] <rzl> rzl@apt1002:~$ sudo -i reprepro -C main includedeb bullseye-wikimedia /srv/wikimedia/pool/component/envoy-future/e/envoyproxy/envoyproxy_1.35.7-1_amd64.deb # T410975

Mentioned in SAL (#wikimedia-operations) [2025-12-11T00:16:43Z] <rzl> rzl@apt1002:~$ sudo -i reprepro copy bookworm-wikimedia bullseye-wikimedia envoyproxy # T410975

Mentioned in SAL (#wikimedia-operations) [2025-12-11T00:16:49Z] <rzl> rzl@apt1002:~$ sudo -i reprepro copy trixie-wikimedia bullseye-wikimedia envoyproxy # T410975

Change #1217344 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/docker-images/production-images@master] envoy: Update to v1.35.7

https://gerrit.wikimedia.org/r/1217344

Change #1217347 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] mw-*: Upgrade to Envoy 1.35.7 in the MW canary releases and mw-debug

https://gerrit.wikimedia.org/r/1217347

Change #1217344 merged by RLazarus:

[operations/docker-images/production-images@master] envoy: Update to v1.35.7

https://gerrit.wikimedia.org/r/1217344

Change #1217347 merged by jenkins-bot:

[operations/deployment-charts@master] mw-*: Upgrade to Envoy 1.35.7 in the MW canary releases and mw-debug

https://gerrit.wikimedia.org/r/1217347

Mentioned in SAL (#wikimedia-operations) [2025-12-11T18:48:30Z] <rzl@deploy2002> rzl: https://gerrit.wikimedia.org/r/1217347 T410975 synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there.

Mentioned in SAL (#wikimedia-operations) [2025-12-11T18:51:48Z] <rzl@deploy2002> Finished scap sync-world: https://gerrit.wikimedia.org/r/1217347 T410975 (duration: 04m 54s)

Change #1217609 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] mw-videoscaler: Update to Envoy 1.35.7

https://gerrit.wikimedia.org/r/1217609

Change #1217610 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] mw-*: Update to Envoy 1.35.7

https://gerrit.wikimedia.org/r/1217610

Change #1217611 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/deployment-charts@master] {api,rest}-gateway: Update to Envoy 1.35.7 in production

https://gerrit.wikimedia.org/r/1217611

Change #1217610 merged by jenkins-bot:

[operations/deployment-charts@master] mw-*: Update to Envoy 1.35.7

https://gerrit.wikimedia.org/r/1217610

Mentioned in SAL (#wikimedia-operations) [2025-12-15T18:13:20Z] <rzl@deploy2002> Finished scap sync-world: https://gerrit.wikimedia.org/r/1217610 T410975 (duration: 05m 20s)

Change #1217611 merged by RLazarus:

[operations/deployment-charts@master] {api,rest}-gateway: Update to Envoy 1.35.7 in production

https://gerrit.wikimedia.org/r/1217611

Change #1218799 had a related patch set uploaded (by RLazarus; author: RLazarus):

[operations/puppet@production] kubernetes: Set default Envoy version to 1.35.7

https://gerrit.wikimedia.org/r/1218799

Change #1218799 merged by RLazarus:

[operations/puppet@production] kubernetes: Set default Envoy version to 1.35.7

https://gerrit.wikimedia.org/r/1218799

Mentioned in SAL (#wikimedia-operations) [2025-12-17T13:44:40Z] <moritzm> upgtrade Envoy on grafana* T410975

Mentioned in SAL (#wikimedia-operations) [2025-12-17T15:28:35Z] <moritzm> upgrade Envoy on etherpad* T410975

Change #1217609 merged by jenkins-bot:

[operations/deployment-charts@master] mw-videoscaler: Update to Envoy 1.35.7

https://gerrit.wikimedia.org/r/1217609

For the record, 121761 updates the rest-gateway to 1.35.7

RLazarus changed the task status from Open to In Progress.Jan 13 2026, 2:12 AM
RLazarus triaged this task as Medium priority.
RLazarus moved this task from Inbox to In Progress on the ServiceOps new board.

@RLazarus can we close this? If not can you update the description to what is exactly needed here?

We're very close. There are some bare-metal hosts still left to upgrade, I've updated the description (see also Debmonitor).

@RLazarus can we close this in Q3? if not, how much effort should we factor in Q4 plan?

Resolving; the remaining hosts will go straight to 1.35.9 in T419637 instead.

@RLazarus can we close this in Q3? if not, how much effort should we factor in Q4 plan?

Sorry to miss this. Thinking about 1.35.9 instead, we'll either close it in Q3 or the remaining work in Q4 will be negligible from a quarterly planning perspective.

Mentioned in SAL (#wikimedia-operations) [2026-04-01T08:07:00Z] <moritzm> upgrading Envoy on the Puppet servers to 1.35.9 T419637 T410975

Mentioned in SAL (#wikimedia-operations) [2026-04-01T11:48:24Z] <moritzm> upgrading Envoy on the idp-test servers to 1.35.9 T419637 T410975

Mentioned in SAL (#wikimedia-operations) [2026-04-02T09:19:48Z] <moritzm> upgrading Envoy on the config-master servers to 1.35.9 T419637 T410975

Mentioned in SAL (#wikimedia-operations) [2026-04-15T15:26:36Z] <Emperor> update & restart envoy on apus frontends T410975 T419637

Mentioned in SAL (#wikimedia-operations) [2026-04-15T15:30:11Z] <Emperor> update & restart envoy on thanos frontends T410975 T419637

Mentioned in SAL (#wikimedia-operations) [2026-04-15T15:30:43Z] <Emperor> update & restart envoy on ms swift frontends T410975 T419637

Mentioned in SAL (#wikimedia-operations) [2026-04-16T16:27:57Z] <urandom> upgrade envoyproxy, restbase[1031,2024] (canary) — T419637 & T410975

Mentioned in SAL (#wikimedia-operations) [2026-04-20T14:02:01Z] <urandom> upgrade envoyproxy, restbase — T419637 & T410975

Mentioned in SAL (#wikimedia-operations) [2026-04-29T10:29:59Z] <moritzm> installing Envoy upgrades on chartmuseum* T410975 T419637

Mentioned in SAL (#wikimedia-operations) [2026-04-29T10:32:38Z] <moritzm> installing Envoy upgrades on webperf* T410975 T419637