Page MenuHomePhabricator
Create Task
Maniphest T238593

Phabricator downtime due to aphlict and websockets (aphlict current disabled)
Closed, ResolvedPublic
Actions

Description

On Friday November 15th we had a short Phabricator downtime.

It was related to the aphlict service (realtime notification server) which uses websockets (T112765) behind cache (formerly cache_misc T134870).

All php-fpm child processes were used by it.

Disabling the aphlict service (and puppet) was the immediate fix and brought Phabricator back normally.

For now aphlict is disabled. Users will still get normal notifications but no realtime pop-up notifications.

There is a theory this is related to recent changes on the caching (ATS) side because we did not have this problem before and it's not a new setup.

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
aphlict: add second envoy TLS terminator for admin portoperations/puppetproduction+12 -2
aphlict: listen on IPv6 instead IPv4 for client and admin portsoperations/puppetproduction+2 -2
enable envoy websockets support for role::aphlictoperations/puppetproduction+1 -0
ATS: set caching to 'websockets' for Phabricatoroperations/puppetproduction+1 -1
ATS: add new backend for phabricator aphlictoperations/puppetproduction+6 -8
ssl: update aphlict TLS cert, add phabricator to SANsoperations/puppetproduction+21 -20
ATS/phabricator: directly talk wss:// to aphlictoperations/puppetproduction+6 -8
phabricator: add envoy TLS terminator for aphlict (DO NOT MERGE)operations/puppetproduction+46 -6
add aphlict.discovery.wmnet with CNAME aphlict1001operations/dnsmaster+2 -0
aphlict: let the service listen on %{::ipaddress}operations/puppetproduction+1 -0
aphlict: make port and IP for the admin interface configurableoperations/puppetproduction+15 -2
phabricator: move phabricator_name key to common.yaml in Hieraoperations/puppetproduction+12 -7
aphlict: add parameter/ferm rule to let phab server talk to admin portoperations/puppetproduction+13 -4
aphlict: set envoy-proxy upstream port to 22280, no SNIoperations/puppetproduction+4 -0
phabricator: set aphlict to disabled in eqiadoperations/puppetproduction+1 -1
aphlict: add envoy for TLS terminationoperations/puppetproduction+2 -1
add fake key for aphlict.discovery.wmnetlabs/privatemaster+3 -0
ssl: add certificate for aphlict.discovery.wmnetoperations/puppetproduction+23 -0
aphlict: require php-cli packageoperations/puppetproduction+2 -0
phabricator: fix the basedir/base_dir parameter name for aphlictoperations/puppetproduction+1 -1
access: let existing phabricator-root-admins get on aphlict machineoperations/puppetproduction+2 -0
phabricator/aphlict: set base_dir parameter when using aphlictoperations/puppetproduction+1 -0
aphlict: remove requirement of the phab_deploy_finalize scriptoperations/puppetproduction+0 -1
aphlict: add scap deploy target and missing parametersoperations/puppetproduction+28 -8
phabricator: enable TLS for aphlictoperations/puppetproduction+11 -4
phabricator: add envoy TLS terminator for aphlictoperations/puppetproduction+44 -6
phabricator: add non-global cert/key path for aphlict envoy terminatoroperations/puppetproduction+2 -0
ATS/phabricator: enable aphlict certificate in hiera.operations/puppetproduction+12 -12
ATS/phabricator: configure aphlict certificateoperations/puppetproduction+129 -39
phabricator: re-enable aphlict serviceoperations/puppetproduction+1 -1
phabricator: remove wstunnel for aphlict from webserver configoperations/puppetproduction+0 -21
varnish: remove config for disabled phab_aphlictoperations/puppetproduction+0 -15
ATS: Disable websocket remap rules for phabricatoroperations/puppetproduction+8 -6
phabricator: if aphlict is disabled also unload wstunnel httpd moduleoperations/puppetproduction+4 -0
phabricator: stop aphlict service if disabled in Hieraoperations/puppetproduction+6 -2
prometheus,ATS: track current active http/http2/websocket connectionsoperations/puppetproduction+11 -1
phabricator: disable aphlictoperations/puppetproduction+1 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
gerritbot added a comment.Jan 31 2020, 12:14 AM

Change 569104 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: directly talk wss:// to aphlict

https://gerrit.wikimedia.org/r/569104

gerritbot added a project: Patch-For-Review.Jan 31 2020, 12:14 AM
Sebastian_Berlin-WMSE subscribed.Jan 31 2020, 2:04 PM
matmarex subscribed.Feb 26 2020, 7:26 PM
Aklapper removed a subscriber: GerritBot.Mar 22 2020, 9:32 PM
Jdforrester-WMF edited projects, added Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)); removed Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)).Apr 1 2020, 3:04 PM
Jdforrester-WMF moved this task from INBOX to Maintenance on the Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)) board.
gerritbot added a comment.Apr 6 2020, 11:37 AM

Change 586338 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: re-enable aphlict service

https://gerrit.wikimedia.org/r/586338

gerritbot added a comment.Apr 6 2020, 11:46 AM

Change 586338 merged by Dzahn:
[operations/puppet@production] phabricator: re-enable aphlict service

https://gerrit.wikimedia.org/r/586338

gerritbot added a comment.Apr 6 2020, 10:44 PM

Change 586461 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] ATS/phabricator: configure aphlict certificate

https://gerrit.wikimedia.org/r/586461

Dzahn added a comment.Apr 7 2020, 9:25 AM

The aphlict service has been re-enabled on phab1001.

The plan is to have ATS (caching layer) talk directly to the nodejs (aphlict) service using wss:// (TLS) on port 22280 and avoiding to go via Apache first.

https://gerrit.wikimedia.org/r/c/operations/puppet/+/569104

For this we can use the existing certificate created for envoy. Or, as an alternative, we can go via envoy for TLS termination and from there to nodejs, also avoiding Apache (httpd) in the mix. Since the Apache module caused the issue that took down all of Phabricator.

The existing certificate has these SANs on it:

alt_names: ["phabricator.discovery.wmnet","phabricator.svc.eqiad.wmnet","phabricator.svc.codfw.wmnet","phabricator.wikimedia.org","phab.wmfusercontent.org","bugs.wikimedia.org","bugzilla.wikimedia.org","git.wikimedia.org"]

and exists in this location:

certificate_chain: { filename: "/etc/ssl/localcerts/phabricator.discovery.wmnet.crt" }
private_key: { filename: "/etc/ssl/private/phabricator.discovery.wmnet.key" }

If we can tell aphlict to find it in this path we should be good to go without further action.

gerritbot added a comment.Apr 7 2020, 9:50 AM

Change 586461 merged by Dzahn:
[operations/puppet@production] ATS/phabricator: configure aphlict certificate

https://gerrit.wikimedia.org/r/586461

gerritbot added a comment.Apr 7 2020, 10:20 AM

Change 587224 had a related patch set uploaded (by 20after4; owner: 20after4):
[operations/puppet@production] ATS/phabricator: enable aphlict certificate in hiera.

https://gerrit.wikimedia.org/r/587224

• mmodell added a comment.EditedApr 7 2020, 10:28 AM

So we have just one last remaining issue to deal with:

Unable to open file ("/etc/ssl/private/phabricator.discovery.wmnet.key"). Check that permissions are set correctly.

Error: EACCES: permission denied, open '/etc/ssl/private/phabricator.discovery.wmnet.key'

That file is owned by root.envoy:

$ ls -la /etc/ssl/private/phabricator.discovery.wmnet.key
-r--r----- 1 root envoy 227 Dec  5 02:58 /etc/ssl/private/phabricator.discovery.wmnet.key

But aphlict runs as aphlict.aphlict

gerritbot added a comment.Apr 7 2020, 10:31 AM

Change 587225 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: enable TLS for aphlict

https://gerrit.wikimedia.org/r/587225

gerritbot added a comment.Apr 7 2020, 10:45 AM

Change 587224 abandoned by 20after4:
ATS/phabricator: enable aphlict certificate in hiera.

https://gerrit.wikimedia.org/r/587224

gerritbot added a comment.Apr 7 2020, 11:42 AM

Change 587233 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: add envoy TLS terminator for aphlict

https://gerrit.wikimedia.org/r/587233

Dzahn added a comment.Apr 7 2020, 12:06 PM

The new plan is to do TLS termination in envoy rather than in nodejs itself. Hence the new patch above to achieve that and add a second TLS listener in addition to the existing one for main Phabricator. That way we still avoid using the buggy apache module for websockets.

• mmodell changed the task status from Open to Stalled.May 14 2020, 3:09 PM
• mmodell removed • mmodell as the assignee of this task.

I am currently unable to drive this forward as all the changes are pending merge in gerrit (operations/puppet)

@Dzahn: Do you think you will have any time to work on this in the foreseeable future? I can make my self available at any time that works best for your schedule, even at midnight CST or later.

gerritbot added a comment.Jun 9 2020, 7:26 AM

Change 587233 merged by Dzahn:
[operations/puppet@production] phabricator: add envoy TLS terminator for aphlict

https://gerrit.wikimedia.org/r/587233

gerritbot added a comment.Jun 9 2020, 8:01 AM

Change 603883 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: add non-global cert/key path for aphlict envoy terminator

https://gerrit.wikimedia.org/r/603883

gerritbot added a comment.Jun 9 2020, 8:09 AM

Change 603883 merged by Dzahn:
[operations/puppet@production] phabricator: add non-global cert/key path for aphlict envoy terminator

https://gerrit.wikimedia.org/r/603883

gerritbot added a comment.Jun 9 2020, 8:33 AM

Change 603895 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: add envoy TLS terminator for aphlict (DO NOT MERGE)

https://gerrit.wikimedia.org/r/603895

gerritbot added a comment.Jun 9 2020, 8:59 AM

Change 587225 abandoned by Dzahn:
phabricator: enable TLS for aphlict

https://gerrit.wikimedia.org/r/587225

thcipriani edited projects, added Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1)); removed Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)).Jul 8 2020, 5:34 PM
gerritbot added a comment.Jul 23 2020, 9:53 PM

Change 615797 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: add backend for aphlict

https://gerrit.wikimedia.org/r/615797

gerritbot added a comment.Jul 23 2020, 9:53 PM

Change 615796 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: set aphlict to disabled in eqiad

https://gerrit.wikimedia.org/r/615796

gerritbot added a comment.Jul 23 2020, 9:55 PM

Change 615842 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aphlict: add scap deploy target and missing parameters

https://gerrit.wikimedia.org/r/615842

gerritbot added a comment.Jul 23 2020, 11:07 PM

Change 615842 merged by Dzahn:
[operations/puppet@production] aphlict: add scap deploy target and missing parameters

https://gerrit.wikimedia.org/r/615842

gerritbot added a comment.Jul 23 2020, 11:19 PM

Change 615879 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aphlict: add phab_deploy_finalize and rollback scripts

https://gerrit.wikimedia.org/r/615879

gerritbot added a comment.Jul 24 2020, 7:18 PM

Change 615879 merged by Dzahn:
[operations/puppet@production] aphlict: remove requirement of the phab_deploy_finalize script

https://gerrit.wikimedia.org/r/615879

gerritbot added a comment.Jul 24 2020, 7:52 PM

Change 616154 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator/aphlict: set base_dir parameter when using aphlict

https://gerrit.wikimedia.org/r/616154

gerritbot added a comment.Jul 24 2020, 7:55 PM

Change 616154 merged by Dzahn:
[operations/puppet@production] phabricator/aphlict: set base_dir parameter when using aphlict

https://gerrit.wikimedia.org/r/616154

gerritbot added a comment.Jul 24 2020, 8:09 PM

Change 616159 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] access: let existing phabricator admins get on aphlict machine

https://gerrit.wikimedia.org/r/616159

gerritbot added a comment.Jul 24 2020, 8:11 PM

Change 616160 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: fix the basedir/base_dir parameter name for aphlict

https://gerrit.wikimedia.org/r/616160

gerritbot added a comment.Jul 24 2020, 8:20 PM

Change 616159 merged by Dzahn:
[operations/puppet@production] access: let existing phabricator-root-admins get on aphlict machine

https://gerrit.wikimedia.org/r/616159

gerritbot added a comment.Jul 24 2020, 8:22 PM

Change 616160 merged by Dzahn:
[operations/puppet@production] phabricator: fix the basedir/base_dir parameter name for aphlict

https://gerrit.wikimedia.org/r/616160

gerritbot added a comment.Jul 24 2020, 8:46 PM

Change 616165 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aphlict: require php-cli package

https://gerrit.wikimedia.org/r/616165

gerritbot added a comment.Jul 24 2020, 8:50 PM

Change 616165 merged by Dzahn:
[operations/puppet@production] aphlict: require php-cli package

https://gerrit.wikimedia.org/r/616165

gerritbot added a comment.Jul 24 2020, 10:04 PM

Change 616171 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ssl: add certificate for aphlict.discovery.wmnet

https://gerrit.wikimedia.org/r/616171

gerritbot added a comment.Jul 24 2020, 10:10 PM

Change 616171 merged by Dzahn:
[operations/puppet@production] ssl: add certificate for aphlict.discovery.wmnet

https://gerrit.wikimedia.org/r/616171

gerritbot added a comment.Jul 24 2020, 10:13 PM

Change 616173 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[labs/private@master] add fake key for aphlict.discovery.wmnet

https://gerrit.wikimedia.org/r/616173

gerritbot added a comment.Jul 24 2020, 10:47 PM

Change 616173 merged by Dzahn:
[labs/private@master] add fake key for aphlict.discovery.wmnet

https://gerrit.wikimedia.org/r/616173

Dzahn mentioned this in rLPRI1731a4f14b58: add fake key for aphlict.discovery.wmnet.Jul 24 2020, 10:48 PM
gerritbot added a comment.Jul 24 2020, 11:57 PM

Change 616184 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aphlict: add envoy for TLS termination

https://gerrit.wikimedia.org/r/616184

gerritbot added a comment.Jul 25 2020, 12:06 AM

Change 616184 merged by Dzahn:
[operations/puppet@production] aphlict: add envoy for TLS termination

https://gerrit.wikimedia.org/r/616184

gerritbot added a comment.Jul 27 2020, 11:08 PM

Change 615796 merged by Dzahn:
[operations/puppet@production] phabricator: set aphlict to disabled in eqiad

https://gerrit.wikimedia.org/r/615796

gerritbot added a comment.Jul 28 2020, 12:07 AM

Change 616630 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aphlict: set envoy-proxy upstream port to 22280, no SNI

https://gerrit.wikimedia.org/r/616630

gerritbot added a comment.Jul 28 2020, 12:52 AM

Change 616630 merged by Dzahn:
[operations/puppet@production] aphlict: set envoy-proxy upstream port to 22280, no SNI

https://gerrit.wikimedia.org/r/616630

• mmodell changed the task status from Stalled to Open.Jul 28 2020, 5:41 PM
gerritbot added a comment.Jul 28 2020, 6:52 PM

Change 616890 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aphlict: add parameter/ferm rule to let phab server talk to admin port

https://gerrit.wikimedia.org/r/616890

gerritbot added a comment.Jul 28 2020, 7:00 PM

Change 616890 merged by Dzahn:
[operations/puppet@production] aphlict: add parameter/ferm rule to let phab server talk to admin port

https://gerrit.wikimedia.org/r/616890

gerritbot added a comment.Jul 28 2020, 7:16 PM

Change 616894 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] phabricator: move phabricator_name key to common.yaml in Hiera

https://gerrit.wikimedia.org/r/616894

gerritbot added a comment.Jul 28 2020, 7:23 PM

Change 616894 merged by Dzahn:
[operations/puppet@production] phabricator: move phabricator_name key to common.yaml in Hiera

https://gerrit.wikimedia.org/r/616894

gerritbot added a comment.Jul 28 2020, 7:39 PM

Change 616896 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aphlict: make port and IP for the admin interface configurable

https://gerrit.wikimedia.org/r/616896

gerritbot added a comment.Jul 28 2020, 8:09 PM

Change 616896 merged by Dzahn:
[operations/puppet@production] aphlict: make port and IP for the admin interface configurable

https://gerrit.wikimedia.org/r/616896

gerritbot added a comment.Jul 28 2020, 8:19 PM

Change 616906 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aphlict: let the service listen on %{::ipaddress}

https://gerrit.wikimedia.org/r/616906

gerritbot added a comment.Jul 28 2020, 8:22 PM

Change 616906 merged by Dzahn:
[operations/puppet@production] aphlict: let the service listen on %{::ipaddress}

https://gerrit.wikimedia.org/r/616906

gerritbot added a comment.Jul 28 2020, 8:30 PM

Change 616909 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/dns@master] add aphlict.discovery.wmnet with CNAME aphlict1001

https://gerrit.wikimedia.org/r/616909

gerritbot added a comment.Jul 28 2020, 8:51 PM

Change 616909 merged by Dzahn:
[operations/dns@master] add aphlict.discovery.wmnet with CNAME aphlict1001

https://gerrit.wikimedia.org/r/616909

Dzahn added a subscriber: 20after4.EditedJul 28 2020, 9:13 PM

@20after4

Current status is now

  • aphlict1001.eqiad.wmnet up and running
  • phabricator-roots admin group has access
  • aphlict.service Active: active (running)
  • admin IP and port now configurable in Hiera with defaults 127.0.0.1 and 22281
  • nodejs listening on port 22280 for clients and port 22281 for the connection from phabricator (aka admin port)
  • admin port listening on actual IP on the first interface and not just 127.0.0.1 on the standalone server
tcp        0      0 0.0.0.0:22280           0.0.0.0:*               LISTEN      499        6634586    3091/nodejs         
tcp        0      0 10.64.48.39:22281       0.0.0.0:*               LISTEN      499        6634587    3091/nodejs
  • name of the active phabricator server moved to "common.yaml" in Hiera so aphlict class can use it as well
  • ferm / iptables opens admin port only for phabricator server but client port for all
[aphlict1001:~] $ sudo iptables -L | grep 222
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:22280
ACCEPT     tcp  --  phab1001.eqiad.wmnet  anywhere             tcp dpt:22281
  • aphlict.discovery.wmnet created in DNS with CNAME to aphlict1001
  • aphlict.discovery.wmnet TLS cert for envoy created, added in private and public repos
  • envoy-proxy added to aphlict1001 for TLS termination and is up and running
  • 1 envoy TLS terminator configured that does "port 443 -> port 22280" for the client port and uses the cert
gerritbot added a comment.Jul 28 2020, 9:49 PM

Change 616917 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aphlict: add second envoy TLS terminator for client port

https://gerrit.wikimedia.org/r/616917

• mmodell added a comment.Jul 29 2020, 5:17 PM

So I think that this is all that remains:

  • cache layer proxy wss://phabricator.wikimedia.org to aphlict1001
  • enable notifications in phabricator
  • test
gerritbot added a comment.Aug 4 2020, 8:26 PM

Change 603895 abandoned by Dzahn:
[operations/puppet@production] phabricator: add envoy TLS terminator for aphlict (DO NOT MERGE)

Reason:
done on a dedicated VM now with separate role aphlict

https://gerrit.wikimedia.org/r/603895

gerritbot added a comment.Aug 4 2020, 8:27 PM

Change 569104 abandoned by Dzahn:
[operations/puppet@production] ATS/phabricator: directly talk wss:// to aphlict

Reason:
basically duplicate of https://gerrit.wikimedia.org/r/c/operations/puppet/ /615797 but no aphlict is on a separate VM

https://gerrit.wikimedia.org/r/569104

gerritbot added a comment.Aug 4 2020, 8:36 PM

Change 618392 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ssl: update aphlict TLS cert, add phabricator to SANs

https://gerrit.wikimedia.org/r/618392

gerritbot added a comment.Aug 4 2020, 8:53 PM

Change 618392 merged by Dzahn:
[operations/puppet@production] ssl: update aphlict TLS cert, add phabricator to SANs

https://gerrit.wikimedia.org/r/618392

gerritbot added a comment.Aug 7 2020, 11:23 AM

Change 615797 merged by Ema:
[operations/puppet@production] ATS: add new backend for phabricator aphlict

https://gerrit.wikimedia.org/r/615797

hashar merged a task: T259895: Phabricator reports and admin issue: "Unable to Connect to Notification Server".Aug 7 2020, 3:05 PM
hashar mentioned this in T259895: Phabricator reports and admin issue: "Unable to Connect to Notification Server".
hashar subscribed.

I filed a dupe of this task. The admin interface states the notification server is not reachable. At https://phabricator.wikimedia.org/config/issue/aphlict.connect/ it says it can not connect to the service on aphlict1001.eqiad.wmnet due to some curl error (CURLE_COULDNT_CONNECT)

gerritbot added a comment.Aug 7 2020, 8:44 PM

Change 619036 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] ATS: set caching to 'websockets' for Phabricator

https://gerrit.wikimedia.org/r/619036

gerritbot added a comment.Aug 10 2020, 11:40 AM

Change 619036 merged by Ema:
[operations/puppet@production] ATS: set caching to 'websockets' for Phabricator

https://gerrit.wikimedia.org/r/619036

hashar unsubscribed.Aug 10 2020, 1:45 PM

Went to https://phabricator.wikimedia.org/config/issue/aphlict.connect/ again and it states:

Issue Resolved
This setup issue has been resolved. Return to Open Issue List

Seems like setting caching to websockets solved the connection issue.

• mmodell added a subscriber: hashar.Aug 10 2020, 5:08 PM
In T238593#6372927, @hashar wrote:

Seems like setting caching to websockets solved the connection issue.

Unfortunately it didn't actually fix it. That setup issue was related to the connection from phabricator to aphlict. The connection from users to aphlict still isn't working.

Concretely, when I attempt to connect from outside to https://phabricator.wikimedia.org:443 with appropriate websocket upgrade headers, I get the phabricator homepage.

Command:

curl --include \
     --no-buffer \
     --header "Connection: keep-alive, Upgrade" \
     --header "Upgrade: websocket" \
     --header "Cache-Control: no-cache" \
     --header "Accept: */*" \
     --header "Host: phabricator.wikimedia.org" \
     --header "Origin: https://phabricator.wikimedia.org" \
     --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIZ==" \
     --header "Sec-WebSocket-Version: 13" \
     https://phabricator.wikimedia.org:443/

Response

HTTP/2 200 
date: Mon, 10 Aug 2020 16:45:21 GMT
server: Apache
x-frame-options: Deny
content-security-policy: default-src https://phab.wmfusercontent.org; img-src https://phab.wmfusercontent.org data:; style-src https://phab.wmfusercontent.org 'unsafe-inline'; script-src https://phab.wmfusercontent.org; connect-src 'self'; frame-src 'self' https://commons.wikimedia.org; frame-ancestors 'none'; object-src 'none'; form-action 'self'  https://www.mediawiki.org https://m.mediawiki.org; base-uri 'none'
referrer-policy: no-referrer
cache-control: private, max-age=0, s-maxage=0
expires: Sat, 01 Jan 2000 00:00:00 GMT
x-content-type-options: nosniff
set-cookie: phsid=A%2Fkjiwjmnitjzy47cgqr3m745k2ushb73d4keee2r7; expires=Sat, 09-Aug-2025 16:45:21 GMT; Max-Age=157680000; path=/; domain=phabricator.wikimedia.org; secure; HttpOnly
set-cookie: next_uri=1597077921%2C%2Fws%2F; path=/; domain=phabricator.wikimedia.org; secure; HttpOnly
set-cookie: phcid=rpk5istax2wuzdfa; path=/; domain=phabricator.wikimedia.org; secure; HttpOnly
vary: Accept-Encoding
content-type: text/html; charset=UTF-8
age: 0
x-cache: cp2037 miss, cp2031 pass
x-cache-status: pass
server-timing: cache;desc="pass"
strict-transport-security: max-age=106384710; includeSubDomains; preload
set-cookie: WMF-Last-Access=10-Aug-2020;Path=/;HttpOnly;secure;Expires=Fri, 11 Sep 2020 12:00:00 GMT
x-client-ip: ****
accept-ranges: bytes

<!DOCTYPE html><html><head><meta charset="UTF-8" /><title>Login</title><meta name="viewport" content="width=device-width, initial-scale=1,
...

Interestingly, if I tell curl to use HTTP 1.1 with the --http1.1 argument, then the front-end proxy gets me to envoy which returns 403:

Command:

curl --include \
     --no-buffer \
     --http1.1 \
     --header "Connection: keep-alive, Upgrade" \
     --header "Upgrade: websocket" \
     --header "Cache-Control: no-cache" \
     --header "Accept: */*" \
     --header "Host: phabricator.wikimedia.org" \
     --header "Origin: https://phabricator.wikimedia.org" \
     --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkID==" \
     --header "Sec-WebSocket-Version: 13" \
     https://phabricator.wikimedia.org:443/

Response:

HTTP/1.1 403 Forbidden
date: Mon, 10 Aug 2020 17:00:01 GMT
server: envoy
content-length: 0
Age: 0
X-Cache-Int: cp2033 pass
Connection: close

So apparently there is something related to HTTP 2 which breaks the websocket forwarding?

Vgutierrez subscribed.Aug 11 2020, 1:06 PM

hmm that's interesting, please note that this is not the first time we use websockets. etherpad.wm.o is already using websockets successfully (even when HTTP/2 is used to perform the upgrade request)

gerritbot added a comment.Aug 11 2020, 1:21 PM

Change 619465 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/puppet@production] enable envoy websockets support for role::aphlict

https://gerrit.wikimedia.org/r/619465

gerritbot added a comment.Aug 11 2020, 1:24 PM

Change 619465 merged by CDanis:
[operations/puppet@production] enable envoy websockets support for role::aphlict

https://gerrit.wikimedia.org/r/619465

CDanis subscribed.Aug 11 2020, 1:48 PM

The Envoy TLS terminator is now configured to allow websocket upgrades -- however, it's improperly configured. It's trying to connect via IPv6 address to the nodejs process, but the nodejs process is not listening there -- it listens only on the IPv4 address.

I'm not familiar with this setup so I'll leave it to @Dzahn to fix.

gerritbot added a comment.Aug 11 2020, 8:09 PM

Change 619560 had a related patch set uploaded (by Dzahn; owner: Dzahn):
[operations/puppet@production] aphlict: listen on IPv6 instead IPv4 for client and admin ports

https://gerrit.wikimedia.org/r/619560

Dzahn added a comment.Aug 11 2020, 8:10 PM

test comment

gerritbot added a comment.Aug 11 2020, 8:12 PM

Change 619560 merged by Dzahn:
[operations/puppet@production] aphlict: listen on IPv6 instead IPv4 for client and admin ports

https://gerrit.wikimedia.org/r/619560

• mmodell added a comment.Aug 11 2020, 8:16 PM

Will it blend?

• mmodell added a comment.Aug 11 2020, 8:19 PM

it appears to blend.

Dzahn added a comment.EditedAug 11 2020, 8:36 PM
In T238593#6376059, @CDanis wrote:

The Envoy TLS terminator is now configured to allow websocket upgrades

Thank you for adding that line. Was aware of it but somehow lost it when splitting aphlict away from phab.

  • however, it's improperly configured. It's trying to connect via IPv6 address to the nodejs process, but the nodejs process is not listening there -- it listens only on the IPv4 address.

This is fixed now. Now it is listening on IPv6-only for both the client and the "admin" port (where phabricator connects)

tcp6 0 0 2620:0:861:107:10:22280 :::* LISTEN 499 16341385 20751/nodejs
tcp6 0 0 2620:0:861:107:10:22281 :::* LISTEN 499 16341386 20751/nodejs

I'm not familiar with this setup so I'll leave it to @Dzahn to fix.

Done:) Confirmed with Mukunda this works now. We both see pop-up notifications and realtime changes when things move on workboards. Yay!

Dzahn closed this task as Resolved.Aug 11 2020, 8:43 PM
Dzahn claimed this task.

We are seeing realtime notifications again and aphlict is now separated from Phabricator on a dedicated VM.

We are also not using mod_ws any longer which caused the instability that made us create this ticket.

The traffic flow is now:

client->ATS->Varnish->ATS->envoy-proxy->aphlict

and

phabricator->aphlict

with envoy-proxy doing TLS termination on the aphlict1001 server

• mmodell awarded a token.Aug 13 2020, 11:18 PM
gerritbot added a comment.Jun 22 2023, 9:16 PM

Change 616917 abandoned by Dzahn:

[operations/puppet@production] aphlict: add second envoy TLS terminator for admin port

Reason:

to be considered in the future

https://gerrit.wikimedia.org/r/616917

Maintenance_bot removed a project: Patch-For-Review.Jun 22 2023, 9:30 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits