Analysis written by Claude (Fable 5) and reviewed by @ABran-WMF. Some questions are still open, especially about the mesh module.

Related: T257617: eqiad: 1 VM request for aphlict, T322369: create aphlict2001 (Phabricator realtime notifications codfw) (current aphlict hosts), T416125: Migrate turnilo.wikimedia.org to the kubernetes deployment (turnilo migration, the puppet-side precedent).

Current state

Aphlict is Phabricator's real-time notification relay: a Node.js process shipped inside the phabricator/deployment scap repo, running on two Ganeti VMs.

The client leg carries websockets from browsers:

browser wss://phabricator.wikimedia.org
  -> varnish (cache/text.yaml, caching: 'websockets' -> vcl_pipe)
  -> ATS map ws://phabricator.wikimedia.org -> wss://aphlict.discovery.wmnet
  -> envoy:443 on aphlict1002 (TLS termination, cfssl discovery2026)
  -> plaintext ws to [aphlict1002 IPv6]:22280

(varnish policy: hieradata/role/common/cache/text.yaml:161; ATS map: hieradata/common/profile/trafficserver/backend.yaml:225-226)

The admin leg pushes notifications from server to server:

phab1005 -> http://aphlict.discovery.wmnet:22281  (plaintext, firewalled srange=[phab1005])

Both ports listen on the host's routable IPv6, not on loopback (client_listen and admin_listen in hieradata/role/common/aphlict.yaml; the 127.0.0.1 / 0.0.0.0 values are module defaults the role overrides). The srange is hiera-driven from $phabricator_active_server, not a literal. The client port has no srange at all, so plaintext 22280 is reachable from the whole internal network today. A k8s NetworkPolicy would tighten that, which should be weighed against the admin-leg regression in Open questions.

Other details:

• Hosts: aphlict1002 (eqiad, active), aphlict2001 (codfw, ensure: absent). Ganeti VMs (preseed.yaml:71).
• Failover is a manual DNS comment swap: operations/dns templates/wmnet:1106-1107.
• There is no LVS service, conftool entry or service::catalog entry. aphlict.discovery.wmnet is a plain CNAME, not a real discovery record.
• Nothing monitors aphlict. modules/profile/manifests/phabricator/monitoring.pp covers HTTP/SMTP/PHD on the phab hosts only, and nothing watches ports 22280/22281.
• Code path: scap deploys phabricator/deployment, and bin/aphlict, a PHP wrapper, daemonises node aphlict_server.js. node_modules is a symlink into the deploy tree, so the VM has both php-cli and nodejs installed for a pure Node service. php-cli comes from profile::phabricator::aphlict, not from the module, and its comment there ("needed by deployment scripts only") is wrong: ExecStart invokes the PHP wrapper, so PHP is on the runtime path. The wrapper also refuses to start if the pidfile exists and refuses to run as root; the container must not inherit either behaviour.

Why move it

1. Retires two Ganeti VMs and a bespoke puppet module (modules/phabricator/manifests/aphlict.pp plus three templates) whose only job is to daemonise one Node process.
2. Removes the manual-DNS-edit failover.
3. Gains monitoring that does not exist today.
4. Removes php-cli from a service that does not need PHP.

Risks below lists the costs, the largest being version skew between the web tier and aphlict.

Target architecture

Namespace aphlict on aux-k8s-eqiad and aux-k8s-codfw, reached through the shared k8s-ingress-aux ingress (port 30443), following the jaeger/zarcillo/kafka-ui pattern. It needs no new LVS service IP, conftool node or PKI intermediate.

browser wss://phabricator.wikimedia.org
  -> varnish (unchanged)
  -> ATS map -> wss://aphlict.discovery.wmnet:30443
  -> istio-ingressgateway on aux-k8s (namespace cert, cfssl discovery2026 profile k8s_aux)
  -> envoy mesh sidecar (websocket upgrade, see Phase 0)
  -> aphlict client port in the pod

phab1005 -> https://aphlict-admin.discovery.wmnet:30443
  -> same gateway, authority-matched route
  -> mesh restricted_listener -> aphlict admin port in the pod

Decisions taken but still open for discussion:

Phase 0: websockets through the WMF mesh (blocking)

The change is service-agnostic: it lands in the shared mesh module, where any future websocket service on the cluster can use it.

Traffic reaches the pod through the envoy mesh sidecar, which terminates TLS for the gateway as envoy does on the VM today. No modules/mesh/configuration_*.tpl emits upgrade_configs, and Envoy's HTTP connection manager rejects Upgrade: websocket without it. The bare-metal equivalent already exists, and aphlict uses it today: profile::tlsproxy::envoy::websockets: true renders upgrade_type: "websocket" at listener.yaml.erb:36. Phase 0 ports that capability to the chart module.

This blocks the migration because aphlict has no fallback transport, such as long-polling, to degrade to. Its client server is a plain ws WebSocket server that answers every non-upgraded request with HTTP/501 Use Websockets (see AphlictClientServer.js:73 on the deployed wmf/stable branch, and new WebSocket.Server at line 103).

Only one setting is missing. The two timeouts that look relevant are already exposed as values and need no module change:

_local_listener serves both mesh.public_port and every mesh.restricted_listeners entry (lines 198-207), so a single upgrade_configs stanza covers both the websocket client port and the admin port.

The version target is 1.16.1, a patch bump, not 1.17.0. configuration_1.16.0 landed on 2026-09-04 for Host-based listener splits (6709d4be5) and is unrelated to this.

Both hops on the client leg were tested.

For the sidecar, the test rendered the mesh sidecar Envoy config twice with one difference, upgrade_configs on the inbound HCM, and left timeouts at production defaults. Without the field, Envoy answers the upgrade itself with a 403 and never selects an upstream, so the application sees nothing:

"GET / HTTP/1.1" 403 - 0 0 0 - "-" "-" "..." "ws-harness.svc.cluster.local" "-"

With it, the response is 101 Switching Protocols, the Upgrade header reaches the application, and the connection survives a 90s idle hold. The config Envoy loaded is byte-identical to what configuration_1.17.0 renders with mesh.websockets.enabled, so the protobuf loader accepts the stanza and the feature works at runtime. That is a stronger check than the envoy --mode validate run in CI.

For the ingressgateway, the test used a single-node kind cluster with Istio installed from custom_deploy.d/istio/main/config_1.29.yaml and no override, the three objects modules/ingress/istio_1.5.0.tpl renders written out by hand, and a websocket echo behind an nginx playing the sidecar's TLS-terminator role. A raw HTTP/1.1 upgrade with the production SNI returns 101 Switching Protocols with sec-websocket-accept correctly computed, and the frame is echoed back. The gateway's own access log shows response_code 101, upstream_cluster outbound|8443||wsapp.wstest.svc.cluster.local, bytes_sent 46 and bytes_received 8, so an upstream was selected and bytes crossed the tunnel in both directions after the handshake. This fits with WMF adding no timeout or upgrade override to Istio: the gateway runs Istio defaults, and Istio relays websockets by default.

Not covered: the DestinationRule's subjectAltNames verification did not program in the test cluster. The test shows that Istio relays the upgrade to a TLS upstream, but not with caCertificates and subjectAltNames as production renders them. That verification is independent of connection upgrades, and every aux-k8s service on this template already goes through it.

The stanza itself is small; the cost is the version cascade. Per modules/mesh/module.json, mesh.name:1.2 depends on mesh.configuration:1.16 and mesh.certificate:1.2 depends on mesh.name:1.2, so bumping configuration to 1.17 pulls name 1.3 and certificate 1.3 behind it, plus ingress.istio 1.5 and a re-vendor in every chart that wants the feature. A patch bump of 1.16 would satisfy mesh.name:1.2 and mesh.certificate:1.2 as they stand and need none of those copies. modules/README.md leans minor, but configuration_1.15.3, service 1.2.1 and networkpolicy 1.2.2 all shipped this shape at patch level, and a flag that is off by default meets both hard constraints for a patch release: no incompatibility and no dependency change. Resolved: JMeybohm confirmed a patch bump is fine as long as it stays backward compatible (see the thread on 1338751, comment, and {T436657#12353881}).

The patches: 1338959, the version bump, then 1338751, the websocket feature, stacked in that order. 1338960, a values.yaml documentation fix, was split out of this chain at JMeybohm's suggestion and stands alone. 1338752, the scaffold re-pin, is abandoned: the move from a minor to a patch bump made it obsolete.

Phase 1: build an aphlict image

A new GitLab repo, repos/phabricator/aphlict, has a Blubber .pipeline/blubber.yaml and vendors support/aphlict/server/ from the Phabricator source. The dependency surface is small: package.json declares one dependency, ws, which has no required transitive dependencies.

support/aphlict/server/ is not self-contained, and an image built from it alone will not boot. lib/javelin.js resolves ../../../../webroot/rsrc/externals/javelin/, and JX.require reads files off disk at runtime and evals them in a vm sandbox, so webroot/rsrc/externals/javelin/core/{init_node,util,install,Event,Stratcom}.js must be present at that relative path. The full image contents are nine JS files under support/aphlict/server/, those five javelin files, ws, and a config JSON. Nothing else in the tree is touched, and no PHP is needed at runtime.

Both repos live on GitLab (repos/phabricator/...); the Gerrit copies are frozen 2022 mirrors, which is what the phabricator/phabricator links in this task point at.

• The entrypoint runs node aphlict_server.js in the foreground. Drop bin/aphlict (the PHP wrapper) and the pidfile; the container has no reason to carry PHP.
• Config comes from a ConfigMap instead of modules/phabricator/templates/aphlict-config.json.erb, with the same shape: a servers list with a client and an admin entry.
• Logs go to stdout, not /var/log/aphlict/aphlict.log, so logrotate_aphlict goes away.
• No TLS in the config: the mesh sidecar terminates it, as envoy does on the VM today. Aphlict can terminate TLS itself (ssl.key / ssl.cert / ssl.chain per server entry, validated in PhabricatorAphlictManagementWorkflow and used in aphlict_server.js to build an https listener), but puppet already made the other choice (modules/phabricator/manifests/aphlict.pp:19, enable_ssl: false in every role). Keep ssl.* null and let the sidecar do it.

The most important part is the version pin. The deployed Phabricator revision is the phabricator submodule gitlink in phabricator/deployment@wmf/stable (git ls-tree HEAD phabricator): scap deploys the branch tip with git_submodules: True, and no tag scheme is in use (scripts/tag-release.sh still exists, but the GitLab repo carries no tags). Tag the image with that SHA and rebuild when the gitlink moves, so the pin is correct by construction and needs no documented manual procedure. Something still has to notice that the gitlink moved. Without a trigger, this degrades to the same manual bump etherpad uses (version: 2026-07-21-122453, edited by hand in the helmfile values), with a less readable tag. Do not build on every wmf/stable push of phabricator/phabricator: the gitlink currently lags the branch tip by about a week, so that would ship aphlict ahead of the web tier and create the skew described in Risks.

Phase 2: k8s plumbing

deployment-charts

• helmfile.d/admin_ng/values/aux-k8s.yaml, new namespace:

aphlict:
  tlsHostnames:
    - aphlict
    - aphlict-admin
  tlsExtraSANs:
    - phabricator.wikimedia.org

The phabricator.wikimedia.org SAN is required; Open questions explains why.

• New charts/aphlict scaffolded with create_new_service.sh, using the Phase 0 mesh module version.
• mesh.public_port for the client/websocket port, mesh.restricted_listeners for the admin port (configuration_1.16.0.tpl:201-208). service 1.2.1 adds the TLS service ports for restricted listeners and networkpolicy 1.2.2 adds their ingress ports, so the admin-leg design needs nothing hand-rolled.
• ingress.httproutes: an authority match on aphlict-admin.* routed to the admin port, default route to the websocket port.
• resources.replicas: 1 and strategy: {type: Recreate}.
• Aphlict makes no outbound calls, so it needs no egress. Ingress comes from the ingressgateway only.
• Copy the aux-k8s deployment skeleton from etherpad and zarcillo rather than deriving it: the helmfile.yaml boilerplate, mesh.enabled: true with a public_port distinct from app.port, docker.registry: docker-registry.discovery.wmnet (the chart default points at the public registry), and config as a ConfigMap mounted as a file rather than through config.public / config.private, since app.generic only turns config.public into environment variables and aphlict needs a JSON file on disk.

dns

Add the aphlict and aphlict-admin CNAMEs to the svc.eqiad.wmnet and svc.codfw.wmnet blocks in this phase, not in Phase 4. They are additive, they point at a gateway that already exists, and Phase 3 cannot run without them. Only the aphlict.discovery.wmnet repoint belongs in Phase 4, because it moves a live name off the VM. etherpad used the same order: its svc and -next discovery records merged a week before its chart.

puppet

• hieradata/common/profile/kubernetes/deployment_server.yaml:1007, under the existing aux-k8s: block:

aphlict:
  usernames:
    - name: aphlict
    - name: aphlict-deploy

Phase 3: parallel run and validation

Deploy to aux-k8s-eqiad only, with no DNS or ATS change, so nothing in production points at it yet.

1262151 looked like a precedent for pointing a test instance at this deployment, but it does something unrelated: it enables aphlict locally on a Cloud VPS devtools instance, not against k8s. No patch exists yet that points any Phabricator instance's admin leg at the k8s deployment; check 5 below needs one.

Acceptance checks:

1. Websocket handshake completes through the gateway (wscat or equivalent against aphlict.svc.eqiad.wmnet:30443 with SNI).
2. A connection survives more than 10 minutes idle. This is the check that tests Phase 0. If it fails, measure the same scenario against the current production path before treating it as a blocker: the bar is parity with today's behaviour, and the aphlict client auto-reconnects either way.
3. An admin POST to aphlict-admin.svc.eqiad.wmnet:30443 is delivered to a connected client.
4. After kubectl delete pod, clients reconnect on their own.
5. The admin /status/ endpoint answers a direct HTTP request. This proves reachability only, since aphlict has no version check (see Risks). Confirming it from the Phabricator config page needs an instance whose notification_servers admin entry already points here, which is Phase 4 work; if that confirmation matters before cutover, point a non-production instance at it first.
6. A push to the admin endpoint from a source other than the Phabricator servers is rejected. This is the only check that tests the AuthorizationPolicy's security boundary.

Phase 4: cutover

Follow the turnilo ordering from T416125: Migrate turnilo.wikimedia.org to the kubernetes deployment: plumbing first, one-line traffic flip, deletions last.

1. puppet, hieradata/common/service.yaml: aphlict and aphlict-admin catalog entries reusing the ingress anchors (ip: *k8s-ingress-aux_ips at line 125, port: *k8s-ingress-aux_port at line 160), encryption: true, probes: [{type: tcp}], state: service_setup. No lvs: key, same as jaeger and codesearch.
2. dns, templates/wmnet:1106: replace with aphlict 300 IN CNAME k8s-ingress-aux-rw.discovery.wmnet. (the -rw record is active/passive, matching today's single-active-DC behaviour) and drop the commented line 1107. The svc.{eqiad,codfw}.wmnet CNAMEs land in Phase 2; this step only moves the live name. The failover semantics change: today aphlict fails over through its own DNS edit, and afterwards it follows the shared k8s-ingress-aux-rw discovery flip together with every other -rw consumer on aux (wmf-navigator today). That is the intended cluster-level behaviour.
3. puppet, hieradata/common/profile/trafficserver/backend.yaml:226: append the port, replacement: wss://aphlict.discovery.wmnet:30443. This step is the traffic flip. aphlict is the only ws:// remap in that file without an explicit port; etherpad (:7443) and gitlab (:8443) already use this form.
4. puppet, modules/profile/manifests/phabricator/main.pp:183-184: the admin leg becomes host aphlict-admin.discovery.wmnet, port 30443, protocol => 'https'. The port currently defaults from profile::phabricator::aphlict::admin_port (main.pp:13), which will no longer exist. The admin host and port are already hiera-driven independently (main.pp:10-15, the aphlict_domain and profile::phabricator::aphlict::admin_port lookups); only the admin protocol, hardcoded to 'http', needs opening up. The client entry is hardcoded to $domain / 443 / https and should stay that way (see the SAN answer in Open questions).
5. Flip the catalog entries to state: production once probes are green (that is the gate for monitoring, modules/profile/manifests/prometheus/ops.pp:68).

hieradata/role/common/cache/text.yaml:161 (caching: 'websockets') stays as is. The public hostname does not change, so nothing in templates/wikimedia.org moves.

Rollback at any point before step 5 is reverting step 3.

Phase 5: decommission

ensure: absent on the VMs, run the decom cookbook, then delete:

• manifests/site.pp:131-133
• hieradata/role/common/aphlict.yaml, hieradata/hosts/aphlict1002.yaml, hieradata/hosts/aphlict2001.yaml
• modules/role/manifests/aphlict.pp, modules/profile/manifests/phabricator/aphlict.pp
• modules/phabricator/manifests/aphlict.pp and its templates: templates/aphlict-config.json.erb, templates/initscripts/aphlict.systemd.erb, files/logrotate_aphlict
• modules/admin/data/system_users.txt:83
• modules/profile/templates/cumin/aliases.yaml.erb:365-366 (phabricator-aphlict, and fix phabricator-all)
• modules/profile/data/profile/installserver/preseed.yaml:71
• modules/profile/files/sre/buster.yaml:454

Cautions:

• modules/profile/manifests/phabricator/main.pp:426-434 declares phabricator::aphlict locally on the phab host, gated on phabricator_aphlict_enabled (false everywhere today). Decide whether "aphlict on the phab host" stays a supported mode; if yes, modules/phabricator/manifests/aphlict.pp survives and only the profile and role go.
• Cloud VPS was checked: grep -rn aphlict hieradata/cloud* is empty and phabricator_aphlict_enabled is set nowhere, so no realm enables the module. The blocker is the line above instead: main.pp:426 declares the class unconditionally, so main.pp must change before the module can go.
• aphlict1001 is already gone: the site.pp regex matches aphlict100[12] but no hieradata/hosts/aphlict1001.yaml exists. Only aphlict1002 is active, so this is a migration from one live instance, not two.

Phase 6: monitoring (new capability)

• The service::catalog entries give blackbox probes once state: production.
• operations/alerts: a kube_deployment_status_replicas_available rule for the namespace. Model it on team-data-platform/turnilo-availability.yaml, including its # deploy-tag / # deploy-site headers.
• A check that exercises a websocket round trip. The catalog probe has to be type: tcp, because the client server answers every non-upgraded request with HTTP/501 Use Websockets, so a TCP probe cannot detect the one failure Phase 0 exists to prevent. Without a round-trip check, Phase 4 step 5 gates production on a signal that cannot see a broken upgrade path, and since nothing monitors aphlict today, a silent outage would go unnoticed.
• Update Wikitech: the Phabricator page (aphlict section and the DNS-swap failover runbook) must describe the k8s deployment, the image bump procedure and the new failover path.

Open questions

1. Admin endpoint authorization. Today the admin port is srange => [phab1005.eqiad.wmnet]. Behind the ingress, the connection's source is the ingressgateway pod, so mesh.restricted_listeners.allow_only cannot see phab1005's address, and no chart in deployment-charts uses AuthorizationPolicy today. Decided: we write the first one. It has five design constraints. It must be action: DENY, because an ALLOW selecting istio: ingressgateway would deny every other request that gateway serves (jaeger, kafka-ui, codesearch, zarcillo, miscweb, etherpad, redioscope, sophroid). It must live in istio-system and target the gateway: aux-k8s injects no istio sidecar (proxy.autoInject: disabled), and the mesh sidecar is a plain WMF envoy that does not implement AuthorizationPolicy, so a policy in the service namespace would validate, appear in kubectl get, and enforce nothing. It must use notIpBlocks, not remoteIpBlocks, because no numTrustedProxies is configured anywhere, so X-Forwarded-For is untrusted and the socket peer is the only meaningful source. It must use security.istio.io/v1beta1 with selector, because CI runs kubeconform -strict and only jsonschema/istio/authorizationpolicy_v1beta1.json is vendored. And it needs a wide hosts: ["aphlict-admin.*"], because in a DENY policy an over-broad host match is safe while an under-broad one fails open, silently. aux-k8s runs Istio 1.24.2-1, not 1.29. The policy goes in the existing istio-gateways-envoyfilters release in helmfile_istio-gateways.yaml, driven from values/aux-k8s.yaml, not in the aphlict chart. Two caveats. The ferm rule follows $phabricator_active_server on failover and this YAML cannot, so listing both phab hosts widens the allowlist from one host to two; that is deliberate, and better than a policy that breaks aphlict the day Phabricator fails over. And the whole design assumes the gateway sees phab1005's real address. That is expected (LVS uses ipip encapsulation, and the gateway Service is externalTrafficPolicy: Local, which is safe because it is a DaemonSet), but it is the one runtime assumption here, so measure downstream_remote_address in the gateway access log before writing the policy. See also T340169: consider moving aphlict admin port to https / envoy, which covers the same admin-leg problem on bare metal and is blocked on something k8s already solves.
2. Version coupling. See Risks.
3. Source restriction on the admin port inside the cluster. The AuthorizationPolicy guards the path through the gateway, but the pod's own NetworkPolicy opens the admin port with no source selector, so any pod on aux-k8s that can route to the pod IP can push a notification. Today's ferm rule is narrower. The source to allow would be the istio: ingressgateway pods in istio-system, not the Phabricator servers, since those arrive through the gateway and the pod sees the gateway as its peer. The mechanism exists but is not cheap. base.networkpolicy offers extraRules for egress only, so an ingress from: has to be hand-written. And because app.port is the admin port, the app.generic.networkpolicy_ingress include opens it unrestricted, so the whole ingress: block would have to be written by hand instead of composed from the modules. charts/flink-app/templates/networkpolicy.yaml is the only precedent in the repo. The telemetry port cannot be restricted the same way, since Prometheus scrapes it, not the gateway. Not done, deliberately: aux-k8s hosts a handful of SRE-operated services and the externally reachable path is already covered, so for now the cost of diverging from every other chart outweighs the gain.
4. codfw. Decided: both DCs from the start. It reproduces today's semantics most closely and costs a helmfile environment line. This constrains the DNS work: aphlict's two legs must land in the same DC, because if phab pushes to one DC while browsers hold connections in the other, notifications are lost silently. Both FQDNs therefore have to point at the same k8s-ingress-aux-rw record, which is a single metafo record and flips atomically. -ro is geoip and active/active, so it would also break notifications silently; do not let anyone substitute it for a browser-facing service.
5. phabricator.wikimedia.org SAN on the namespace cert: needed. The browser's wss URL keeps the public hostname (notification_servers[client].host is $domain at main.pp:171-180). ATS remaps the URL but preserves the Host header (pristine_host_hdr INT 1 in trafficserver/templates/records.config.erb, no sni_policy override anywhere) and validates the origin certificate (verify_server_policy: ENFORCED in trafficserver/backend.yaml). So ATS presents and checks phabricator.wikimedia.org even though the URL points at aphlict.discovery.wmnet. With sni_support: "no", the single cert is served for any SNI, and the SAN exists only for that name check. The etherpad and gitlab roles use the same pattern. Dropping the SAN breaks the client leg.

Risks

The main risk is version skew. Today scap deploy of phabricator/deployment updates the web tier and aphlict in the same operation, so the aphlict server and the aphlict client JS served to browsers can never disagree. After this migration they come from two independent pipelines: a GitLab-built image and a scap deploy. Aphlict has no protocol version check. The only version token is a display string in the admin /status/ blob ('version': 7), which the config UI reads as pht('Version %s', ...) and compares to nothing; grep -rn version src/applications/notification/ src/applications/aphlict/ is empty, and the client never sends a subprotocol. An unrecognised command is logged and dropped with the socket left open, so skew in either direction degrades notifications silently, and with no monitoring today, nobody would notice.

Two things reduce the real exposure. support/aphlict/server/lib has the same git tree hash on the 2026 tip as on the 2022 snapshot, so the wire protocol has not changed in four years. And the image tag can be pinned to the exact revision scap deploys (see Phase 1), which removes this failure mode.

A more concrete version risk is ws itself. Production runs ws 1.1.1 (2016), vendored in phabricator/deployment/aphlict/node_modules/, while the source package-lock.json pins ws 7.5.10. Nothing in the deploy path reads package.json. Containerising from the lockfile is therefore a silent jump of six major versions. The code tolerates both because of a shim (if ('upgradeReq' in ws)), which is why nobody has noticed. Decide deliberately which version the image ships.

To confirm on the host: scap/phabricator-targets lists aphlict2001 (the inactive host) and not aphlict1002. If that dsh list is current, aphlict already does not get its tree from the same scap operation as the web tier, and the atomicity this risk assumes does not exist today.

This would be the first websocket service on WMF Kubernetes. Phase 0 touches a module that every service in the cluster shares and another team owns, so keep it strictly opt-in.

Stop condition: both hops relay upgrades, so what remains is scheduling. Aphlict is not the only service waiting on the mesh change: T435509: Deploy etherpad-new to aux-k8s cluster, etherpad on aux-k8s, runs on long-polling until it lands, and both services already set profile::tlsproxy::envoy::websockets: true on bare metal. The case for the module owners is two blocked migrations and an opt-in flag that changes nothing for existing charts.

Phase 0 is also not strictly on aphlict's critical path. A pod that terminates TLS itself reaches the gateway without the sidecar, as charts/ratelimit does, and aphlict's config supports it. That is not the plan, because puppet already decided the other way and it would lose the envoy metrics Phase 6 wants. It does mean that if the module bump cannot be scheduled, aphlict still has a way forward. Phase 1, the image, depends on none of this and is the longest piece of work; it can start now.

Both the replicas > 1 trap and version skew degrade notifications without an error, so Phase 6 should land soon after Phase 4.

Out of scope

• Real aphlict HA (clustering across replicas). Separate task.
• Moving Phabricator itself to Kubernetes.
• Any change to the CDN-facing hostname or the varnish websocket policy.

Repos touched

• Image repo, Blubber config and CI (build, production-variant build, runtime smoke test): repos/phabricator/aphlict
• Mesh websocket support: out for review (1338959, 1338751)
• Mesh values.yaml documentation fix, standalone: merged (1338960)
• AuthorizationPolicy on the admin leg: out for review (1341190)
• aux-k8s namespace and certificate: out for review (1341855)
• Helm chart for aphlict: out for review (1341856)
• Deployment to both aux-k8s clusters: out for review (1341857)
• Puppet kubeconfig for aux-k8s: merged (1341711)
• Trusted runner access request: merged (gitlab-trusted-runner!190)
• Phase 3: parallel run and validation
• DNS CNAMEs (svc.eqiad.wmnet / svc.codfw.wmnet)
• service::catalog entries
• ATS traffic flip (Phase 4)
• operations/alerts rule
• Decommission (Phase 5)