Page MenuHomePhabricator

Phab Aphlict Notification Servers config shows Connection error
Open, MediumPublic

Description

https://phabricator.wikimedia.org/config/cluster/notifications/ lists a Connection Error for phabricator.wikimedia.org:443 with a message Got HTTP 200, but expected HTTP 501 (WebSocket Upgrade)!

(I'm very clueless about Aphlict stuff, maybe this issue is quite ignorable.)

That output is thrown in https://phabricator.wikimedia.org/source/phabricator/browse/wmf%252Fstable/src/applications/notification/client/PhabricatorNotificationServerRef.php$181-203

Event Timeline

I think it's ignorable because we are not using phabricator.wikimedia.org:443 as an aphlict host anymore since aphlict machines were outsourced to dedicated VMs.

In that screen you link you can see the working connection to aphlict.discovery.wmnet right below it.

But let's still clean this up and get the config right if that is still lingering in there.

You are not saying this is new, or are you?

Generally, if desktop notifications still work then aphlict is fine.

I do not know at all if this is new, sorry. I'm quite sure though it is not new, given how long ago T257617, T322369, T333452 got resolved.
I'm wondering if the Phorge upstream code is "too simple" here. And if there's any way that this check could be made work as expected.

Shall we close this very ticket as declined?

@Aklapper I checked the actual live config in the filesystem and I see this:

notification_servers:
- type: client
  host: phabricator.wikimedia.org
  port: 443
  protocol: https
- type: admin
  host: aphlict.discovery.wmnet
  port: 22281
  protocol: http

Here are some facts around that:

  • we know there are 2 ports for aphlict, a client port and an admin port
  • but at least my expectation was that both are configured to aphlict.discovery.wmnet and connect there
  • we also configured both ports to be allowed on the aphlict side, where we have opened 22280 (client) and 22281 (admin)
  • the nodejs process that is aphlict is running on aphlict1002. but should I see one on phab1004?
  • realtime notifications apparently work just fine all this time regardless

Please keep the task open for now with low priority.

Oh, of course.. so.. the configuration in the CDN has this:

- type: map
  target: ws://phabricator.wikimedia.org
  replacement: wss://aphlict.discovery.wmnet

This means that if the connection is to phabricator.wikimedia.org on port 443, like the client port is configured, AND it's a websocket connection rather than https.. then it ends up on the aphlict server, not the main phorge server.

edit: No, wait, it says "ws://" in the target and only "wss://" in the replacement. So that means port 443 from external would not work. (?!)

LSobanski triaged this task as Medium priority.Aug 28 2025, 1:25 PM