Page MenuHomePhabricator
Create Task
Maniphest T271421

Test envoyproxy as a WMF's CDN TLS terminator with real traffic
Closed, ResolvedPublic
Actions

Description

Envoy is one of the candidates to replace ats-tls as the TLS terminator used in the WMF caching infrastructure. To fully validate its performance and stability a real traffic test is needed.
To be able to perform this test several requirements need to be fulfilled:

  • Adapt the current envoy puppetization to be able to meet Traffic requirements -> currently blocked by T265880
  • Test envoyproxy in Traffic WCMS environment
  • Test envoyproxy in ulsfo
    • Currently running on:
      • cp4025 (upload)
      • cp4031 (text)
  • Test envoyproxy in eqsin
    • Currently running on:
      • cp5005 (upload)
      • cp5011 (text)
  • Test envoyproxy in esams
    • Currently running on:
      • cp3063 (upload)
      • cp3062 (text)
  • Test envoyproxy in codfw
    • Currently running on:
      • cp2040 (upload)
      • cp2039 (text)
  • Test envoyproxy in eqiad
    • Currently running on:
      • cp1088 (upload)
      • cp1087 (text)

Details

SubjectRepoBranchLines +/-
site: Reimage cp3062 as cache::text_haproxyoperations/puppetproduction+10 -8
site: Reimage cp2039 as cache::text_haproxyoperations/puppetproduction+11 -9
site: Reimage cp5011 as cache::text_haproxyoperations/puppetproduction+10 -8
site: Reimage cp1088 as cache::upload_haproxyoperations/puppetproduction+10 -8
site: Reimage cp3063 as cache::upload_haproxyoperations/puppetproduction+10 -8
site: Reimage cp5005 as cache::haproxy_uploadoperations/puppetproduction+10 -8
site: Reimage cp2040 as cache::upload_haproxyoperations/puppetproduction+10 -8
site: Reimage cp4025 as cache::upload_haproxyoperations/puppetproduction+10 -13
cache::envoy: Bound envoy to the same NUMA node as the main NICoperations/puppetproduction+20 -0
cumin: Add cache::text_envoy to cp-text aliasoperations/puppetproduction+1 -1
site: Reimage cp1087 as cache::text_envoyoperations/puppetproduction+9 -1
cache::envoy: Reduce downstream_idle_timeoutoperations/puppetproduction+1 -1
site: Reimage cp2039 as cache::text_envoyoperations/puppetproduction+9 -1
site: Reimage cp3062 as cache::text_envoyoperations/puppetproduction+9 -1
site: Reimage cp5011 as cache::text_envoyoperations/puppetproduction+9 -1
prometheus::ops: Gather varnish mtail metrics on text_envoyoperations/puppetproduction+15 -0
site: Reimage cp4031 as cache::text_envoyoperations/puppetproduction+9 -1
cache: Provide a text_envoy roleoperations/puppetproduction+292 -0
site: Reimage cp1088 as cache::upload_envoyoperations/puppetproduction+9 -1
site: Reimage cp2040 as cache::upload_envoyoperations/puppetproduction+9 -1
site: Reimage cp3063 as cache::upload_envoyoperations/puppetproduction+9 -1
cache::envoy: Set the delayed_close_timeout to 20soperations/puppetproduction+3 -0
envoy: Allow configuring delayed_closed_timeoutoperations/puppetproduction+18 -0
cumin: Add cache::upload_envoy to cp aliasesoperations/puppetproduction+1 -1
cache::envoy: Decrease upstream idle_timeoutoperations/puppetproduction+1 -1
cache::envoy: Decrease upstream idle_timeout to 30soperations/puppetproduction+1 -1
cache::envoy: Set upstream idle timeoutoperations/puppetproduction+3 -0
cache::envoy: Disable circuit breakersoperations/puppetproduction+1 -0
envoy: Allow disabling circuit breakersoperations/puppetproduction+19 -0
cache::envoy: Limit number of requests per varnish connoperations/puppetproduction+1 -0
cache::envoy: Set stream_idle/request/request_headers timeoutoperations/puppetproduction+10 -1
envoyproxy: Add stream_idle/request/request_headers timeoutoperations/puppetproduction+54 -0
cache::envoy: Bump request timeout to 300soperations/puppetproduction+1 -1
cache::envoy: Increase upstream response timeoutoperations/puppetproduction+1 -1
cache::envoy: Add sslcert::ocsp::hookoperations/puppetproduction+13 -0
sslcert::ocsp: Allow configuring group ownership on /var/cache/ocspoperations/puppetproduction+29 -6
cache::envoy: Allow envoy to read sslcert managed TLS materialoperations/puppetproduction+1 -0
cache::envoy: Allow envoyproxy unit to write on /var/cache/ocspoperations/puppetproduction+1 -1
site: Reimage cp5005 as cache::upload_envoyoperations/puppetproduction+9 -1
cache::envoy: Increase LimitNOFILEoperations/puppetproduction+13 -0
prometheus::ops: Add varnish/ATS metrics for cache::upload_envoy roleoperations/puppetproduction+15 -0
prometheus::ops: Gather full metrics for cache::envoyoperations/puppetproduction+17 -6
cache::envoy: Disable x-request-id generationoperations/puppetproduction+1 -0
envoyproxy: Allow disabling x-request-id generationoperations/puppetproduction+16 -0
cache::envoy: Strip [] from X-Client-IPoperations/puppetproduction+1 -1
cache::envoy: Fix ocsp systemd config file contentoperations/puppetproduction+1 -1
role::cache: Add missing upload_envoy.yamloperations/puppetproduction+95 -0
site: Reimage cp4025 as cache::upload_envoyoperations/puppetproduction+9 -1
prometheus::ops: Collect envoy metrics for profile::cache::envoyoperations/puppetproduction+5 -0
cache: Provide a Envoy upload roleoperations/puppetproduction+245 -0
envoyproxy: Allow setting http2 protocol optionsoperations/puppetproduction+32 -0
envoyproxy: Add downstream idle_timeout config optionoperations/puppetproduction+19 -0
envoyproxy: Allow setting per_connection_buffer_limit_bytesoperations/puppetproduction+10 -0
envoyproxy: Allow configuring TLS handshake timeoutoperations/puppetproduction+9 -0
envoyproxy: Support PreserveCase HeaderKeyFormatoperations/puppetproduction+35 -18
cache: Use envoy lua API to provide TLS infooperations/puppetproduction+73 -0
envoyproxy: Allow setting a global lua scriptoperations/puppetproduction+25 -2
envoyproxy: Support TLS min/max version configoperations/puppetproduction+28 -2
envoyproxy: Support alpn_protocols configurationoperations/puppetproduction+26 -7
varnish: add tests for unknown XCPS session reuseoperations/puppetproduction+30 -0
varnish: Allow SSR=2 on XCPSoperations/puppetproduction+5 -3
envoyproxy: Provide support for UDS upstreamsoperations/puppetproduction+74 -26
cache: Provide an envoy STEK manager scriptoperations/puppetproduction+111 -0
envoyproxy: Add STEK configuration supportoperations/puppetproduction+31 -0
envoyproxy: Support ECDH curves configurationoperations/puppetproduction+39 -13
envoyproxy: Support ciphersuite configurationoperations/puppetproduction+21 -0
envoyproxy: Add dual stack cert supportoperations/puppetproduction+81 -57
envoyproxy: Add upstream PROXY protocol supportoperations/puppetproduction+41 -1
envoyproxy: Add prefetched OCSP staple supportoperations/puppetproduction+8 -0
envoyproxy: Allow configuring the admin addressoperations/puppetproduction+8 -4
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jan 24 2022, 10:59 AM

Change 756541 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp1088 as cache::upload_envoy

https://gerrit.wikimedia.org/r/756541

ops-monitoring-bot added a comment.Jan 24 2022, 11:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp1088.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 24 2022, 11:49 AM

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp1088.eqiad.wmnet with OS buster completed:

  • cp1088 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201241103_vgutierrez_20056_cp1088.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 24 2022, 11:50 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-24T11:50:46Z] <vgutierrez> pool cp1088 using envoy as TLS termination layer - T271421

Vgutierrez updated the task description. (Show Details)Jan 24 2022, 11:51 AM
gerritbot added a comment.Jan 26 2022, 11:26 AM

Change 757415 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache: Provide a text_envoy role

https://gerrit.wikimedia.org/r/757415

gerritbot added a comment.Jan 27 2022, 10:32 AM

Change 757415 merged by Vgutierrez:

[operations/puppet@production] cache: Provide a text_envoy role

https://gerrit.wikimedia.org/r/757415

gerritbot added a comment.Jan 27 2022, 10:43 AM

Change 757627 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp4031 as cache::text_envoy

https://gerrit.wikimedia.org/r/757627

Stashbot added a comment.Jan 27 2022, 11:12 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-27T11:12:00Z] <vgutierrez> depool cp4031 to be reimaged as cache::text_envoy - T271421

gerritbot added a comment.Jan 27 2022, 11:25 AM

Change 757627 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp4031 as cache::text_envoy

https://gerrit.wikimedia.org/r/757627

ops-monitoring-bot added a comment.Jan 27 2022, 11:29 AM

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp4031.ulsfo.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 27 2022, 12:09 PM

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp4031.ulsfo.wmnet with OS buster completed:

  • cp4031 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201271129_vgutierrez_30656_cp4031.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
RLazarus mentioned this in T300324: Upgrade Envoy to supported version.Jan 28 2022, 1:10 AM
Stashbot added a comment.Jan 28 2022, 3:41 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-28T15:41:55Z] <vgutierrez> pool cp4031 using envoy as TLS termination layer - T271421

Vgutierrez updated the task description. (Show Details)Jan 28 2022, 3:44 PM
gerritbot added a comment.Jan 28 2022, 4:11 PM

Change 757917 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] prometheus::ops: Gather varnish mtail metrics on text_envoy

https://gerrit.wikimedia.org/r/757917

gerritbot added a comment.Jan 28 2022, 4:27 PM

Change 757917 merged by Vgutierrez:

[operations/puppet@production] prometheus::ops: Gather varnish mtail metrics on text_envoy

https://gerrit.wikimedia.org/r/757917

gerritbot added a comment.Jan 31 2022, 9:46 AM

Change 758430 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp5011 as cache::text_envoy

https://gerrit.wikimedia.org/r/758430

Stashbot added a comment.Jan 31 2022, 9:53 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-31T09:53:24Z] <vgutierrez> depool cp5011 to be reimaged as cache::text_envoy - T271421

gerritbot added a comment.Jan 31 2022, 9:54 AM

Change 758430 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp5011 as cache::text_envoy

https://gerrit.wikimedia.org/r/758430

ops-monitoring-bot added a comment.Jan 31 2022, 9:55 AM

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp5011.eqsin.wmnet with OS buster

ops-monitoring-bot added a comment.Jan 31 2022, 10:57 AM

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp5011.eqsin.wmnet with OS buster completed:

  • cp5011 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202201310955_vgutierrez_8847_cp5011.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Jan 31 2022, 10:58 AM

Mentioned in SAL (#wikimedia-operations) [2022-01-31T10:58:13Z] <vgutierrez> pool cp5011 running envoy as TLS terminator - T271421

Vgutierrez updated the task description. (Show Details)Jan 31 2022, 10:59 AM
gerritbot added a comment.Jan 31 2022, 4:16 PM

Change 758512 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp3062 as cache::text_envoy

https://gerrit.wikimedia.org/r/758512

Stashbot added a comment.Feb 1 2022, 9:01 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-01T09:01:31Z] <vgutierrez> depool cp3062 to be reimaged as cache::text_envoy - T271421

gerritbot added a comment.Feb 1 2022, 9:03 AM

Change 758512 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp3062 as cache::text_envoy

https://gerrit.wikimedia.org/r/758512

ops-monitoring-bot added a comment.Feb 1 2022, 9:03 AM

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp3062.esams.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 1 2022, 10:01 AM

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp3062.esams.wmnet with OS buster completed:

  • cp3062 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202010903_vgutierrez_28355_cp3062.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 1 2022, 10:14 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-01T10:14:19Z] <vgutierrez> pool cp3062 running envoy as TLS terminator - T271421

Vgutierrez updated the task description. (Show Details)Feb 1 2022, 10:16 AM
gerritbot added a comment.Feb 1 2022, 4:04 PM

Change 758879 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp2039 as cache::text_envoy

https://gerrit.wikimedia.org/r/758879

Stashbot added a comment.Feb 1 2022, 4:10 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-01T16:10:43Z] <vgutierrez> depool cp2039 to be reimaged as cache::text_envoy - T271421

gerritbot added a comment.Feb 1 2022, 4:11 PM

Change 758879 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp2039 as cache::text_envoy

https://gerrit.wikimedia.org/r/758879

ops-monitoring-bot added a comment.Feb 1 2022, 4:12 PM

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp2039.codfw.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 1 2022, 4:58 PM

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp2039.codfw.wmnet with OS buster completed:

  • cp2039 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202011612_vgutierrez_14398_cp2039.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 1 2022, 5:21 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-01T17:21:48Z] <vgutierrez> pool cp2039 running envoy as TLS terminator - T271421

Vgutierrez updated the task description. (Show Details)Feb 1 2022, 5:22 PM
gerritbot added a comment.Feb 2 2022, 11:08 AM

Change 759224 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::envoy: Reduce downstream_idle_timeout

https://gerrit.wikimedia.org/r/759224

gerritbot added a comment.Feb 2 2022, 11:12 AM

Change 759224 merged by Vgutierrez:

[operations/puppet@production] cache::envoy: Reduce downstream_idle_timeout

https://gerrit.wikimedia.org/r/759224

gerritbot added a comment.Feb 2 2022, 11:47 AM

Change 759231 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp1087 as cache::text_envoy

https://gerrit.wikimedia.org/r/759231

Stashbot added a comment.Feb 2 2022, 11:48 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-02T11:48:40Z] <vgutierrez> depool cp1087 to be reimaged as cache::text_envoy - T271421

gerritbot added a comment.Feb 2 2022, 11:49 AM

Change 759231 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp1087 as cache::text_envoy

https://gerrit.wikimedia.org/r/759231

ops-monitoring-bot added a comment.Feb 2 2022, 11:50 AM

Cookbook cookbooks.sre.hosts.reimage was started by vgutierrez@cumin1001 for host cp1087.eqiad.wmnet with OS buster

ops-monitoring-bot added a comment.Feb 2 2022, 12:43 PM

Cookbook cookbooks.sre.hosts.reimage started by vgutierrez@cumin1001 for host cp1087.eqiad.wmnet with OS buster completed:

  • cp1087 (WARN)
    • Downtimed on Icinga
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present
    • Deleted any existing Puppet certificate
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Host up (new fresh buster OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202202021150_vgutierrez_28641_cp1087.out
    • Checked BIOS boot parameters are back to normal
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is not optimal, downtime not removed
    • Updated Netbox data from PuppetDB
Stashbot added a comment.Feb 2 2022, 2:13 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-02T14:13:17Z] <vgutierrez> pool cp1087 running envoy as TLS terminator - T271421

Vgutierrez updated the task description. (Show Details)Feb 2 2022, 2:13 PM
taavi mentioned this in T300366: Problem loading thumbnail images due to Envoy (HTTP/1.0 clients getting '426 Upgrade Required').Feb 2 2022, 8:03 PM
Dzahn subscribed.EditedFeb 2 2022, 8:04 PM

There are some user reports / IRC chatter and this ticket T300366 that seem like they are related to this. ('426 issues")

BBlack subscribed.Feb 2 2022, 8:26 PM

summarizing about the link above: apparently we do have HTTP/1.0 clients, and it does work with our other terminators, but not envoy.

Envoy does have some config for this (cf https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#config-core-v3-http1protocoloptions ), but it's not on by default, was added later and not part of the original design, and it has some scary language about being "not fully standards compliant". It also requires a default hostname for when the Host header is lacking. This probably need some further investigation to determine the path forward here. I can imagine three loosely defined buckets of where we could end up:

  1. Maybe we can turn this on safely, with a reasonable (perhaps invalid) default hostname.
  2. Maybe we don't actually need to support HTTP/1.0 and can justify turning it off (seems unlikely, but possible)
  3. Maybe neither of the above is true, which gives us a useful data point in making decisions about using envoy :)
Ahecht subscribed.Feb 2 2022, 10:21 PM
This comment was removed by Ahecht.
Dzahn added a subtask: T300366: Problem loading thumbnail images due to Envoy (HTTP/1.0 clients getting '426 Upgrade Required').Feb 2 2022, 11:14 PM
TheDJ subscribed.EditedFeb 2 2022, 11:36 PM
In T271421#7672538, @BBlack wrote:

Envoy does have some config for this (cf https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#config-core-v3-http1protocoloptions ), but it's not on by default, was added later and not part of the original design, and it has some scary language about being "not fully standards compliant". It also requires a default hostname for when the Host header is lacking. This probably need some further investigation to determine the path forward here. I can imagine three loosely defined buckets of where we could end up:

This has the original discussion on this feature I think: https://github.com/envoyproxy/envoy/issues/170
Here is the paper on differences between 1.0 and 1.1 which might help with risk assessment: https://web.archive.org/web/20180302204914/http://www.research.att.com/people/Krishnamurthy_Balachander/papers/h0vh1.html

Support was written by experienced envoy developer and google dev. Maybe we can ask Google for their experiences ?

AntiCompositeNumber subscribed.Feb 3 2022, 1:24 AM

Another report of user-facing impact in #mediawiki from someone using the w3m browser: https://wm-bot.wmcloud.org/logs/%23mediawiki/20220203.txt

Vgutierrez changed the status of subtask T300366: Problem loading thumbnail images due to Envoy (HTTP/1.0 clients getting '426 Upgrade Required') from Open to In Progress.Feb 3 2022, 9:16 AM
Vgutierrez closed subtask T300366: Problem loading thumbnail images due to Envoy (HTTP/1.0 clients getting '426 Upgrade Required') as Resolved.Feb 3 2022, 9:48 AM
gerritbot added a comment.Feb 3 2022, 11:34 AM

Change 759474 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cumin: Add cache::text_envoy to cp-text alias

https://gerrit.wikimedia.org/r/759474

gerritbot added a comment.Feb 3 2022, 11:36 AM

Change 759474 merged by Vgutierrez:

[operations/puppet@production] cumin: Add cache::text_envoy to cp-text alias

https://gerrit.wikimedia.org/r/759474

Esanders reopened subtask T300366: Problem loading thumbnail images due to Envoy (HTTP/1.0 clients getting '426 Upgrade Required') as Open.Feb 3 2022, 11:05 PM
gerritbot added a comment.Feb 15 2022, 12:04 PM

Change 762802 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] cache::envoy: Bound envoy to the same NUMA node as the main NIC

https://gerrit.wikimedia.org/r/762802

gerritbot added a comment.Feb 15 2022, 1:29 PM

Change 762802 merged by Vgutierrez:

[operations/puppet@production] cache::envoy: Bound envoy to the same NUMA node as the main NIC

https://gerrit.wikimedia.org/r/762802

gerritbot added a comment.Feb 25 2022, 10:48 AM

Change 766073 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp4025 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/766073

gerritbot added a comment.Feb 25 2022, 10:53 AM

Change 766073 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp4025 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/766073

Stashbot mentioned this in T290005: Test haproxy as a WMF's CDN TLS terminator with real traffic.Feb 25 2022, 11:40 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-25T11:40:19Z] <vgutierrez> pool cp4025 running HAProxy as TLS termination layer - T290005 T271421

Vgutierrez closed this task as Resolved.Feb 25 2022, 11:41 AM

envoy instances are currently being reimaged as HAProxy ones. We're cleaning up and pausing the envoyproxy experiment

gerritbot added a comment.Feb 25 2022, 11:46 AM

Change 766078 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp2040 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/766078

gerritbot added a comment.Feb 25 2022, 11:52 AM

Change 766078 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp2040 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/766078

Stashbot added a comment.Feb 25 2022, 12:32 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-25T12:32:45Z] <vgutierrez> pool cp2040 running HAProxy as TLS termination layer - T290005 T271421

gerritbot added a comment.Feb 25 2022, 1:58 PM

Change 766102 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp5005 as cache::haproxy_upload

https://gerrit.wikimedia.org/r/766102

gerritbot added a comment.Feb 25 2022, 2:03 PM

Change 766102 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp5005 as cache::haproxy_upload

https://gerrit.wikimedia.org/r/766102

Stashbot added a comment.Feb 25 2022, 3:25 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-25T15:25:19Z] <vgutierrez> pool cp5005 running HAProxy as TLS termination layer - T290005 T271421

gerritbot added a comment.Feb 25 2022, 3:34 PM

Change 766119 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp3063 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/766119

gerritbot added a comment.Feb 25 2022, 3:37 PM

Change 766119 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp3063 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/766119

Stashbot added a comment.Feb 25 2022, 4:35 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-25T16:35:06Z] <vgutierrez> pool cp3063 running HAProxy as TLS termination layer - T290005 T271421

gerritbot added a comment.Feb 28 2022, 10:17 AM

Change 766586 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp1088 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/766586

gerritbot added a comment.Feb 28 2022, 10:27 AM

Change 766586 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp1088 as cache::upload_haproxy

https://gerrit.wikimedia.org/r/766586

Stashbot added a comment.Feb 28 2022, 11:09 AM

Mentioned in SAL (#wikimedia-operations) [2022-02-28T11:09:40Z] <vgutierrez> pool cp1088 running HAProxy as TLS termination layer - T290005 T271421

gerritbot added a comment.Feb 28 2022, 11:21 AM

Change 766597 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp5011 as cache::text_haproxy

https://gerrit.wikimedia.org/r/766597

gerritbot added a comment.Feb 28 2022, 11:28 AM

Change 766597 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp5011 as cache::text_haproxy

https://gerrit.wikimedia.org/r/766597

Stashbot added a comment.Feb 28 2022, 12:24 PM

Mentioned in SAL (#wikimedia-operations) [2022-02-28T12:24:02Z] <vgutierrez> pool cp5011 running HAProxy as TLS termination layer - T290005 T271421

gerritbot added a comment.Mar 1 2022, 8:51 AM

Change 767051 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp2039 as cache::text_haproxy

https://gerrit.wikimedia.org/r/767051

gerritbot added a comment.Mar 1 2022, 8:53 AM

Change 767051 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp2039 as cache::text_haproxy

https://gerrit.wikimedia.org/r/767051

Stashbot added a comment.Mar 1 2022, 10:05 AM

Mentioned in SAL (#wikimedia-operations) [2022-03-01T10:05:44Z] <vgutierrez> pool cp2039 running HAProxy as TLS termination layer - T290005 T271421

gerritbot added a comment.Mar 1 2022, 10:36 AM

Change 767065 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site: Reimage cp3062 as cache::text_haproxy

https://gerrit.wikimedia.org/r/767065

gerritbot added a comment.Mar 1 2022, 10:39 AM

Change 767065 merged by Vgutierrez:

[operations/puppet@production] site: Reimage cp3062 as cache::text_haproxy

https://gerrit.wikimedia.org/r/767065

Stashbot added a comment.Mar 1 2022, 12:49 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-01T12:49:58Z] <vgutierrez> pool cp3062 running HAProxy as TLS termination layer - T290005 T271421

Stashbot added a comment.Mar 1 2022, 2:36 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-01T14:36:30Z] <vgutierrez> pool cp1087 running HAProxy as TLS termination layer - T290005 T271421

Vgutierrez closed subtask T300366: Problem loading thumbnail images due to Envoy (HTTP/1.0 clients getting '426 Upgrade Required') as Resolved.Mar 1 2022, 3:00 PM
Stashbot added a comment.Mar 2 2022, 3:47 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-02T15:47:06Z] <vgutierrez> pool cp5014 running HAProxy as TLS termination layer - T290005 T271421

Stashbot added a comment.Mar 2 2022, 4:51 PM

Mentioned in SAL (#wikimedia-operations) [2022-03-02T16:51:42Z] <vgutierrez> pool cp3061 running HAProxy as TLS termination layer - T290005 T271421

BCornwall closed subtask T271407: Upgrade envoyproxy to 1.16.2 as Resolved.Feb 17 2023, 4:25 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits