Page MenuHomePhabricator

Support ECH on Wikimedia servers
Open, MediumPublicFeature

Description

ECH (Encrypted Client Hello) encrypts SNI, which is the only field in cleartext in a HTTPS connection. It requires use of an encrypted DNS server (Edit: no longer mandatory). Enabling this will further protect user privacy by shadowing what WMF sites they are connecting to.

It could also make WMF sites more censorship-resistant. Note that packets with ECH has a distinct signature, so a censor may choose to block it at all.

See also

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
operations/puppetproduction+3 -15
operations/dnsmaster+0 -1
operations/dnsmaster+109 -33
operations/dnsmaster+28 -0
operations/puppetproduction+15 -6
operations/debs/nginx-echmaster+9 K -0
operations/dnsmaster+1 -0
operations/puppetproduction+1 -2
operations/puppetproduction+3 -3
operations/puppetproduction+78 -4
operations/puppetproduction+72 -4
labs/privatemaster+7 -0
integration/configmaster+4 -0
integration/configmaster+4 -0
integration/configmaster+1 -0
operations/puppetproduction+4 -0
operations/puppetproduction+1 -0
operations/puppetproduction+7 -0
operations/puppetproduction+16 -0
operations/dnsmaster+4 -0
operations/dnsmaster+161 -1
operations/puppetproduction+1 -0
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
BBlack subscribed.

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all tickets that aren't are neither part of our current planned work nor clearly a recent, higher-priority emergent issue. This is simply one step in a larger task cleanup effort. Further triage of these tickets (and especially, organizing future potential project ideas from them into a new medium) will occur afterwards! For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

Diskdance renamed this task from Enable ESNI support on Wikimedia servers to Support ECH on Wikimedia servers.Dec 2 2021, 12:14 PM
Diskdance updated the task description. (Show Details)
Diskdance subscribed.

ESNI has been superseded by the ECH, hence updating the task.

unsubing, as I think I was added to this ticket by mistake. This is traffic/traffic security expertise, and they already triaged and aware of the task.

Mozilla have now launched ECH in Firefox. Cloudflare have also launched server side support globally. Chrome will be shipping ECH imminently.

Is it worth re-triaging this feature?

@DennisJJackson Hi and welcome to Phabricator! What in this ticket led you to asking for "retriage" (and what does that mean)?

@Aklapper - It looks like this issue was originally raised several years ago and put in the icebox. I'm flagging that the situation around standardization and deployment of ECH has changed rather dramatically since then. This work would also be closely aligned with Wikimedia's recent work on hosting secure DNS.

I'm just flagging that its worth taking a look at and seeing whether its still icebox priority for Wikimedia or whether its worth actively working on.

Hi @DennisJJackson: Thanks for the question. We do plan to work on ECH and enable it for our sites and have had some discussions internally. There is no timeline yet as such, for a variety of reasons, the limited browser support being one, though that has clearly changed over the past few weeks. There are some other considerations here as well such as the lack of server-side options for turning it on but we are hoping the DEfO project will provide the much needed support there for HAProxy, which is what we use for TLS termination.

Once we have defined some timeline, we will re-triage this task and share updates here. Thanks for checking!

Change #1122155 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] wikimedia-ech: add ncredir-parking

https://gerrit.wikimedia.org/r/1122155

Change #1122155 merged by Ssingh:

[operations/dns@master] wikimedia-ech: add ncredir-parking

https://gerrit.wikimedia.org/r/1122155

Change #1132669 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] P:durum: add conditional to enable ECH

https://gerrit.wikimedia.org/r/1132669

Change #1132730 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] aptrepo: add component for ECH-enabled nginx

https://gerrit.wikimedia.org/r/1132730

Change #1132730 merged by Ssingh:

[operations/puppet@production] aptrepo: add component for ECH-enabled nginx

https://gerrit.wikimedia.org/r/1132730

Change #1133135 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/puppet@production] Add pbuilder hook for ECH builds

https://gerrit.wikimedia.org/r/1133135

Change #1133190 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: acme_chief: add wikimedia-ech.org

https://gerrit.wikimedia.org/r/1133190

Change #1133917 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] utils: add a script to generate HTTPS TYPE65 records for ECH

https://gerrit.wikimedia.org/r/1133917

Change #1133917 merged by Ssingh:

[operations/dns@master] utils: add a script to generate HTTPS TYPE65 records for ECH

https://gerrit.wikimedia.org/r/1133917

Change #1133135 merged by Muehlenhoff:

[operations/puppet@production] Add pbuilder hook for ECH builds

https://gerrit.wikimedia.org/r/1133135

Change #1133190 merged by Ssingh:

[operations/puppet@production] hiera: acme_chief: add wikimedia-ech.org

https://gerrit.wikimedia.org/r/1133190

Change #1134726 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] package_builder: add bc package for OpenSSL build

https://gerrit.wikimedia.org/r/1134726

Change #1134726 merged by Ssingh:

[operations/puppet@production] package_builder: add bc package for OpenSSL build

https://gerrit.wikimedia.org/r/1134726

Mentioned in SAL (#wikimedia-operations) [2025-04-09T15:38:40Z] <sukhe> reprepro -C component/nginx-ech include bookworm-wikimedia openssl_3.4.1-1+ech1_amd64.changes: T205378

Change #1135728 had a related patch set uploaded (by Ssingh; author: Ssingh):

[integration/config@master] Zuul: [operations/debs/nginx-ech] Add debian-glue CI

https://gerrit.wikimedia.org/r/1135728

Change #1135731 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] package_builder: add packages for nginx build

https://gerrit.wikimedia.org/r/1135731

Change #1135733 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/debs/nginx-ech@master] Release 1.22.1-9+deb12u1+ech1

https://gerrit.wikimedia.org/r/1135733

Change #1135731 abandoned by Ssingh:

[operations/puppet@production] package_builder: add packages for nginx build

Reason:

sticking to gitlab for now

https://gerrit.wikimedia.org/r/1135731

Mentioned in SAL (#wikimedia-operations) [2025-04-11T13:33:46Z] <sukhe> reprepro -C component/nginx-ech include bookworm-wikimedia openssl_3.4.1-1+ech2_amd64.changes: T205378

Change #1135728 merged by jenkins-bot:

[integration/config@master] Zuul: [operations/debs/nginx-ech] Add debian-glue CI

https://gerrit.wikimedia.org/r/1135728

Mentioned in SAL (#wikimedia-operations) [2025-04-11T15:23:28Z] <sukhe> reprepro -C component/nginx-ech include bookworm-wikimedia openssl_3.4.1-1+ech3_amd64.changes: T205378

Mentioned in SAL (#wikimedia-operations) [2025-04-11T15:37:24Z] <sukhe> reprepro -C component/nginx-ech include bookworm-wikimedia nginx_1.22.1-9+deb12u1+ech1_amd64.changes: T205378

Change #1136001 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] Zuul: [operations/debs/nginx-ech] use component/nginx-ech

https://gerrit.wikimedia.org/r/1136001

Change #1136001 merged by jenkins-bot:

[integration/config@master] Zuul: [operations/debs/nginx-ech] use component/nginx-ech

https://gerrit.wikimedia.org/r/1136001

Change #1136005 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: debian-glue should export COMPONENT

https://gerrit.wikimedia.org/r/1136005

Change #1136005 merged by jenkins-bot:

[integration/config@master] jjb: debian-glue should export COMPONENT

https://gerrit.wikimedia.org/r/1136005

Change #1136376 had a related patch set uploaded (by Ssingh; author: Ssingh):

[labs/private@master] hiera: durum: add dummy ECH private key

https://gerrit.wikimedia.org/r/1136376

Change #1136376 merged by Ssingh:

[labs/private@master] hiera: durum: add dummy ECH private key

https://gerrit.wikimedia.org/r/1136376

Mentioned in SAL (#wikimedia-operations) [2025-04-14T13:38:05Z] <sukhe> reprepro -C component/nginx-ech include bookworm-wikimedia nginx_1.22.1-9+deb12u1+ech2_amd64.changes: T205378

Change #1132669 merged by Ssingh:

[operations/puppet@production] P:durum: add conditional to enable ECH (durum2002)

https://gerrit.wikimedia.org/r/1132669

Mentioned in SAL (#wikimedia-operations) [2025-04-16T14:22:06Z] <sukhe> reprepro -C component/nginx-ech include bookworm-wikimedia nginx_1.22.1-9+deb12u1+ech3_amd64.changes: T205378

Change #1137021 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] wikimedia-dns.org: add TYPE65 records for check.wikimedia-dns.org

https://gerrit.wikimedia.org/r/1137021

Change #1138823 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] P:durum: add conditional to enable ECH (esams)

https://gerrit.wikimedia.org/r/1138823

Change #1138823 merged by Ssingh:

[operations/puppet@production] P:durum: add conditional to enable ECH (esams)

https://gerrit.wikimedia.org/r/1138823

Mentioned in SAL (#wikimedia-operations) [2025-04-28T14:47:56Z] <sukhe> reprepro -C component/nginx-ech include bookworm-wikimedia nginx_1.22.1-9+deb12u1+ech4_amd64.changes: T205378

Change #1139525 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] P:durum: only log ECH status for ECH-enabled clients

https://gerrit.wikimedia.org/r/1139525

Change #1139525 merged by Ssingh:

[operations/puppet@production] P:durum: only log ECH status for ECH-enabled clients

https://gerrit.wikimedia.org/r/1139525

Change #1139542 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: durum: set do_ech true for all durum hosts

https://gerrit.wikimedia.org/r/1139542

Change #1139542 merged by Ssingh:

[operations/puppet@production] hiera: durum: set do_ech true for all durum hosts

https://gerrit.wikimedia.org/r/1139542

Change #1137021 merged by Ssingh:

[operations/dns@master] wikimedia-dns.org: add TYPE65 records for check.wikimedia-dns.org

https://gerrit.wikimedia.org/r/1137021

Change #1139911 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] P:durum: hiera: log only ech_status

https://gerrit.wikimedia.org/r/1139911

Change #1135733 abandoned by Ssingh:

[operations/debs/nginx-ech@master] Release 1.22.1-9+deb12u1+ech1

Reason:

for reason mentioned above

https://gerrit.wikimedia.org/r/1135733

Change #1139911 merged by Ssingh:

[operations/puppet@production] P:durum: hiera: log only ech_status

https://gerrit.wikimedia.org/r/1139911

Change #1140189 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] wikimedia-ech.org: update zone file and add A/AAAA records

https://gerrit.wikimedia.org/r/1140189

Change #1140189 merged by Ssingh:

[operations/dns@master] wikimedia-ech.org: update zone file and add A/AAAA records

https://gerrit.wikimedia.org/r/1140189

Change #1142631 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] type65.py: add support for generation of additional HTTPS SvcParams

https://gerrit.wikimedia.org/r/1142631

Change #1142631 merged by Ssingh:

[operations/dns@master] type65.py: add support for generation of additional HTTPS SvcParams

https://gerrit.wikimedia.org/r/1142631

Change #1156355 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/puppet@production] hiera: durum: revert ECH experiment

https://gerrit.wikimedia.org/r/1156355

Change #1156373 had a related patch set uploaded (by Ssingh; author: Ssingh):

[operations/dns@master] wikimedia-dns.org: remove TYPE65 record

https://gerrit.wikimedia.org/r/1156373

Change #1156373 merged by Ssingh:

[operations/dns@master] wikimedia-dns.org: remove TYPE65 record for check

https://gerrit.wikimedia.org/r/1156373

Change #1156355 merged by Ssingh:

[operations/puppet@production] hiera: durum: revert ECH experiment

https://gerrit.wikimedia.org/r/1156355

The ECH experiment has been reverted as of today.