Page MenuHomePhabricator

Upgrade Envoy to v1.39.0
Closed, ResolvedPublic

Description

Release notes of potential interest (1.36.0, 1.37.0, 1.38.*):

  • maximum concurrent http2 streams per connection default changed
  • http default reset code changed
  • Minimum acceptable TLS version is 1.0 and maximum is 1.3 as of 1.39.0

Once the upgrade is complete, we will be able to configure one or both of:

Either one would have circumvented what we saw in T421343: Some traffic still flowing to mw-api-int after the switchover

Related Objects

Event Timeline

Blake triaged this task as Medium priority.Jun 17 2026, 9:01 AM
Blake added a subtask: Restricted Task.
MLechvien-WMF updated Other Assignee, added: Blake.
MLechvien-WMF moved this task from Inbox to Scheduled (this Q) on the ServiceOps board.
RLazarus renamed this task from Upgrade Envoy to >=1.37 to Upgrade Envoy to v1.38.3.Aug 4 2026, 9:34 PM
RLazarus updated the task description. (Show Details)
RLazarus updated Other Assignee, added: RLazarus; removed: Blake.
RLazarus subscribed.

As discussed offline, over to you!

1.38.3 (the newest patch version on 1.38) is the natural destination due to the config deprecation window we talked about, so retitled the task accordingly.

(1.39.0 came out a couple of weeks ago, and you could consider going straight there if you think there are features we'd benefit from, and the config version skew is acceptable -- for example, no config fields are fully removed in 1.39.0 whose replacement wasn't even available yet in 1.35.9, the version we're running now. I don't know of any compelling reason to jump ahead, but just highlighting that it's your call to make.)

After a review of the changelogs, I'm not seeing anything that jumps out as being particularly likely to break. It also seems as though we'd be able to proceed to 1.39, and I'm inclined to get us on the latest available version, given that there are no additional concerning changes. I'll proceed to attempt to build a new version.

Change #1321526 had a related patch set uploaded (by Blake; author: Blake):

[operations/debs/envoyproxy@v1.39] Update to v1.39

https://gerrit.wikimedia.org/r/1321526

Change #1321526 merged by Blake:

[operations/debs/envoyproxy@v1.39] Update to v1.39.0

https://gerrit.wikimedia.org/r/1321526

We are now at the step where we build the envoy-future Docker image.

Change #1324312 had a related patch set uploaded (by Blake; author: Blake):

[operations/docker-images/production-images@master] envoy-future: Update envoy-future to 1.39.0.

https://gerrit.wikimedia.org/r/1324312

Change #1324312 merged by Blake:

[operations/docker-images/production-images@master] envoy-future: Update envoy-future to 1.39.0.

https://gerrit.wikimedia.org/r/1324312

Blake renamed this task from Upgrade Envoy to v1.38.3 to Upgrade Envoy to v1.39.0.Aug 12 2026, 12:03 PM

Change #1324699 had a related patch set uploaded (by Blake; author: Blake):

[integration/config@master] helm-linter: Update envoy-future to 1.39.0

https://gerrit.wikimedia.org/r/1324699

Change #1324730 had a related patch set uploaded (by Blake; author: Blake):

[integration/config@master] jjb: update job to releng/helm-linter:0.8.4

https://gerrit.wikimedia.org/r/1324730

Change #1324699 merged by jenkins-bot:

[integration/config@master] helm-linter: Update envoy-future to 1.39.0

https://gerrit.wikimedia.org/r/1324699

Change #1324730 merged by jenkins-bot:

[integration/config@master] jjb: update job to releng/helm-linter:0.8.4

https://gerrit.wikimedia.org/r/1324730

Working through https://wikitech.wikimedia.org/wiki/Envoy#Validate_the_new_version, and I've successfully deployed, tested, and subsequently rolled back envoy-future on mw-debug. @Scott_French, do you happen to have a suggestion for a "low traffic non-Mediawiki" service I might try to upgrade the Envoy of, to continue validation? Thanks!

Change #1326221 had a related patch set uploaded (by Blake; author: Blake):

[operations/deployment-charts@master] rest-gateway: Use envoy-future in staging.

https://gerrit.wikimedia.org/r/1326221

I tested this with the proton service (chromium-render) in codfw, and envoy started up with no errors, and no deprecation warnings. I've sent out a patch to switch over the staging deployment of the rest gateway.

Change #1326221 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Use envoy-future in staging.

https://gerrit.wikimedia.org/r/1326221

Change #1326818 had a related patch set uploaded (by Blake; author: Blake):

[operations/docker-images/production-images@master] envoy: New upstream version 1.39.0.

https://gerrit.wikimedia.org/r/1326818

Change #1326818 merged by Blake:

[operations/docker-images/production-images@master] envoy: New upstream version 1.39.0.

https://gerrit.wikimedia.org/r/1326818

Testing this on restbase-canary:

blake@restbase1033:~$ curl -s localhost:9631/server_info | jq .version
"75e220883447543d35571aecae826d7b1a2646b9/1.35.9/Clean/RELEASE/BoringSSL"
blake@restbase1033:~$ sudo pkill -HUP -f envoyproxy-hot-restarter
blake@restbase1033:~$ curl -s localhost:9631/server_info | jq .version
"8eea3285d6bdb89f8ea34632cfe7ce1608a8f374/1.39.0/Clean/RELEASE/BoringSSL"

Logs look good, no deprecation notices or other warnings. We'll proceed to roll out the new version in Kubernetes next week.

Change #1328532 had a related patch set uploaded (by Blake; author: Blake):

[operations/deployment-charts@master] mw-*: Upgrade to envoy 1.39.0 in the MW canary releases and mw-debug.

https://gerrit.wikimedia.org/r/1328532

Change #1328532 merged by jenkins-bot:

[operations/deployment-charts@master] mw-*: Upgrade to envoy 1.39.0 in the MW canary releases and mw-debug.

https://gerrit.wikimedia.org/r/1328532

Mentioned in SAL (#wikimedia-operations) [2026-08-24T10:05:55Z] <blake@deploy1003> Started scap sync-world: upgrade envoy in debug and canary for T421418

Mentioned in SAL (#wikimedia-operations) [2026-08-24T10:11:16Z] <blake@deploy1003> Finished scap sync-world: upgrade envoy in debug and canary for T421418 (duration: 06m 46s)

Change #1328549 had a related patch set uploaded (by Blake; author: Blake):

[operations/deployment-charts@master] rest-gateway: Upgrade to envoy 1.39.0-1 in production.

https://gerrit.wikimedia.org/r/1328549

Change #1328549 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Upgrade to envoy 1.39.0-1 in production.

https://gerrit.wikimedia.org/r/1328549

Change #1329325 had a related patch set uploaded (by Blake; author: Blake):

[operations/deployment-charts@master] mw-*: switch to envoy 1.39.0-1.

https://gerrit.wikimedia.org/r/1329325

Change #1329325 merged by jenkins-bot:

[operations/deployment-charts@master] mw-*: switch to envoy 1.39.0-1.

https://gerrit.wikimedia.org/r/1329325

mw-* have been updated.

The remaining pieces are mw-videoscaler, the non-mw k8s services, and the rest of the baremetal servers.

Change #1329565 had a related patch set uploaded (by Blake; author: Blake):

[operations/deployment-charts@master] mw-videoscaler: bump envoy to 1.39.0-1.

https://gerrit.wikimedia.org/r/1329565

Change #1329565 merged by jenkins-bot:

[operations/deployment-charts@master] mw-videoscaler: bump envoy to 1.39.0-1.

https://gerrit.wikimedia.org/r/1329565

Bare metal hosts have been updated, as well as mw-videoscaler. The only thing left to do is set this version as the default for all services, which I'm going to chat with Reuven about later today.

Well, and maybe also start the security release for 1.39.1. Perhaps that ought to be a different task, we'll see.

Change #1330515 had a related patch set uploaded (by Blake; author: Blake):

[operations/puppet@production] kubernetes: switch the default envoy version to 1.39.0

https://gerrit.wikimedia.org/r/1330515

Change #1330515 merged by Blake:

[operations/puppet@production] kubernetes: switch the default envoy version to 1.39.0

https://gerrit.wikimedia.org/r/1330515

Ran charlie over everything today, here are the services which didn't cooperate difflessly or otherwise:

staging

media-analytics timeout
miscweb diff
mcrouter diff
flink diff
mw-script diff
ratelimit diff
rdf-streaming-updater diff
tegola-vector-tiles diff
thumbor diff
toolhub diff

codfw

miscweb diff
mw-experimental diff
mw-page-content-change-enrich diff
rdf-streaming-updater diff
tegola-vector-tiles diff
toolhub diff
wikifunctions diff

eqiad

kartotherian timeout
miscweb diff
mw-page-content-change-enrich diff
rdf-streaming-updater diff
tegola-vector-tiles diff
thumbor diff
toolhub diff
wikifunctions diff

Here are the services which will now have an additional diff, the next time helmfile apply is run in the relevant environment, organized by the team that owns them. It should be safe to apply the Envoy upgrade for all of these services, as we've rolled it out everywhere else with no problems.

Collaboration Services:

miscweb (diffs in staging, codfw, eqiad)

DPE SRE:

flink (diffs in staging)

Serviceops:

mcrouter (diffs in staging)
ratelimit (diffs in staging)
mw-experimental (diffs in codfw)

Search Platform:

rdf-streaming-updater (diffs in staging, codfw, eqiad)
mw-page-content-change-enrich (diffs in codfw, eqiad)

Mediawiki Content Platform:

tegola-vector-tiles (diffs in staging, codfw, eqiad)
kartotherian (timed out in eqiad, likely diff present)

Tools Platform:

toolhub (diffs in staging, codfw, eqiad)

Mediawiki Media Platform:

thumbor (diffs in staging, eqiad)

Abstract Wikipedia:

wikifunctions (diffs in staging, codfw)

Closing this out, as there's nothing remaining from serviceops. Thanks!

MLechvien-WMF closed subtask Restricted Task as Resolved.Fri, Sep 4, 2:20 PM