Change #1325854 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/debs/wmf-laptop@master] tunnelencabulator: gitlab moved behind the text-lb CDN

https://gerrit.wikimedia.org/r/1325854

Calling this out explicitly so it doesn't get lost in the list of changes: @ABran-WMF disabled the legacy nftables throttling on the replicas as it was causing envoy timeouts, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/1325454.

In T425441#12215538, @LSobanski wrote:

Calling this out explicitly so it doesn't get lost in the list of changes: @ABran-WMF disabled the legacy nftables throttling on the replicas as it was causing envoy timeouts, see https://gerrit.wikimedia.org/r/c/operations/puppet/+/1325454.

Thanks for mentioning it. Disabling throttling on the remaining Gitlab instance will be done before merging the DNS change to move Gitlab behind CDN

Change #1325844 merged by Arnaudb:

[operations/puppet@production] gitlab: discard firewall throttling on the primary

https://gerrit.wikimedia.org/r/1325844

Change #1324281 merged by Arnaudb:

[operations/dns@master] gitlab: point gitlab.wikimedia.org at the CDN

https://gerrit.wikimedia.org/r/1324281

Hi, just to let you know, we're experiencing some instability with our gitsync pods on our airflow instances.

Maybe it's transitory and will settle down, but we're monitoring and will keep you updated. I added T435292 as a sub-task. Hope that's OK.

In T425441#12230018, @BTullis wrote:

Hi, just to let you know, we're experiencing some instability with our gitsync pods on our airflow instances.

Thanks for mentioning this

Maybe it's transitory and will settle down, but we're monitoring and will keep you updated. I added T435292 as a sub-task. Hope that's OK.

We reverted the change moving Gitlab behind CDN because of similar issues like in T435290: GitLab fails to download artifacts between jobs.

More investigations to come

arnaudb opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/626

Add a dedicated debug-ci project runner for the CDN repro (T425441)

After analyzing what happened this morning, it appears we had 2 distinct issues:
the one triggering T435292: The airflow-gitsync pods are adversely affected by gitlab being put behind the CDN, reported by @BTullis, is due to the fact that we drop traffic from DSE's egress IPs on text-lb.

The second one is still being debugged at this moment, we seem to have been dropped by requestctl but it is hard to tell by which rule without reproducing the issue one way or another. So I'm preparing a dedicated CI runner to spoof Gitlab's IP address and will trigger a CI job on this debugging project, so we can have a better idea of the rule that was triggered this morning.

arnaudb opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/627

gitlab-runner: disable FF_USE_DIRECT_DOWNLOAD on cloud runners

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/627

gitlab-runner: disable FF_USE_DIRECT_DOWNLOAD on cloud runners

Following up on T425441#12231758: the second issue is now root-caused, and it turned out not to be a requestctl rule.

gitlab-runner's artifact direct-download path (FF_USE_DIRECT_DOWNLOAD, on by default) builds its request outside the client wrapper that sets the User-Agent, so it goes out with Go's default Go-http-client/1.1. The edge robot policy (T400119, enforced at the HAProxy layer, upstream of Varnish/requestctl) returns 403 for that UA, which killed every artifact download on the DigitalOcean cloud runners (external egress, not exempt) while GitLab was behind the CDN, hence T435290.

We reproduced by temporarily pinning gitlab.wikimedia.org to text-lb via host_aliases on an isolated staging runner (thanks @Jelto for the suggestion), then replaying the runner's exact request from a build pod: no UA gets 403 with the robot-policy body, the identical request with a proper UA gets 200. Not a fingerprint/JA4 rule, purely the missing User-Agent., so the staging runner has been reverted.

Fix (merged, gitlab-cloud-runner!627): disable FF_USE_DIRECT_DOWNLOAD on the cloud runners so the runner fetches artifacts through its standard HTTP client, which sets a proper UA; one template covers both runner releases in prod + staging. Verified on the pinned staging runner: native download now returns 200. Two precisions on the merged commit message after review: the User-Agent is set by the runner's own HTTP client wrapper (not by GitLab), and the stated trade-off does not apply since our GitLab already runs with proxy_download = true, so artifact bytes were flowing through GitLab either way. The flag only changes how the runner builds the request, not the data path, which makes the change effectively risk-free and safe ahead of the re-bascule. Longer term the clean fix is upstream (gitlab-runner should set a UA on its direct-download path).

Still open before re-pointing the primary at the CDN: the first issue from T425441#12231758, the firewalling gap where text-lb drops traffic from the DSE egress IPs (T435292). That fix has not been done yet.

This comment was written by Claude (Fable 5), supervised by ABran.

arnaudb closed https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/626

Draft: Add a dedicated debug-ci project runner for the CDN repro (T425441)

Change #1332720 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] kubernetes: allow pod egress to text-lb for gitlab

https://gerrit.wikimedia.org/r/1332720

So, this was a wrong direction:

In T425441#12247447, @ABran-WMF wrote:

Still open before re-pointing the primary at the CDN: the first issue from T425441#12231758, the firewall gap where text-lb drops traffic from the DSE egress IPs (T435292). That fix has not been done yet.

It turns out that the traffic drop was happening at the egress level, the cluster was not aware of the new IP addresses behind gitlab.wm.o so it enforced the policy to discard traffic to these new destinations gitsync was trying to reach.

Change #1332731 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] Revert^2 "gitlab: discard firewall throttling on the primary"

https://gerrit.wikimedia.org/r/1332731

Change #1332733 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] Revert^2 "gitlab: point gitlab.wikimedia.org at the CDN"

https://gerrit.wikimedia.org/r/1332733

Change #1332720 abandoned by Arnaudb:

[operations/puppet@production] kubernetes: allow pod egress to text-lb for gitlab

Reason:

not needed anymore - https://phabricator.wikimedia.org/T435292#12274188

https://gerrit.wikimedia.org/r/1332720

Thanks to @brouberol (see T435292: The airflow-gitsync pods are adversely affected by gitlab being put behind the CDN for more details), DPE will use https://gitlab.discovery.wmnet:8443 as a Gitlab endpoint. Once the remaining bits (see that slack message) will be considered done, we will be able to make another attempt at moving Gitlab behind CDN.

Change #1332731 merged by Arnaudb:

[operations/puppet@production] Revert^2 "gitlab: discard firewall throttling on the primary"

https://gerrit.wikimedia.org/r/1332731

Change #1332733 merged by Arnaudb:

[operations/dns@master] Revert^2 "gitlab: point gitlab.wikimedia.org at the CDN"

https://gerrit.wikimedia.org/r/1332733

Change #1334598 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] icinga: check gitlab SSH against gitlab-ssh.wikimedia.org

https://gerrit.wikimedia.org/r/1334598

Change #1334598 merged by Arnaudb:

[operations/puppet@production] icinga: check gitlab SSH against gitlab-ssh.wikimedia.org

https://gerrit.wikimedia.org/r/1334598

Change #1325854 merged by Arnaudb:

[operations/debs/wmf-laptop@master] tunnelencabulator: gitlab moved behind the text-lb CDN

https://gerrit.wikimedia.org/r/1325854

hi there, our GitLab agents are failing to connect for the last hour or so, https://gitlab.wikimedia.org/repos/test-platform/catalyst/catalyst-api/-/clusters:

At first I thought it was a version mismatch, but upgrading to v19.1.7 didn't solve the problem: T436902#12284491

The timing seems consistent with the latest changes here. Can you help? The issue is blocking Patchdemo/Catalyst deployments.

The connection endpoint the agents are using is: --set config.kasAddress=wss://gitlab.wikimedia.org//-/kubernetes-agent/

Change #1334854 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gitlab: let the envoy tlsproxy relay websocket upgrades

https://gerrit.wikimedia.org/r/1334854

Change #1334855 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] trafficserver: map websockets for the gitlab instances

https://gerrit.wikimedia.org/r/1334855

thanks for mentioning it @jnuche
I sent 2 patches, one that will enable websockets on Envoy, the other to map them on our backends. That should fix the issue you're facing.

Change #1334854 merged by Arnaudb:

[operations/puppet@production] gitlab: let the envoy tlsproxy relay websocket upgrades

https://gerrit.wikimedia.org/r/1334854

arnaudb opened https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/113

Send a User-Agent when fetching the JWKS from GitLab

Another issue was mentioned by @trueg. As discussed in T435292, highlighting T400119: Block traffic from user-agents not honoring our policy → we ask for requests to the CDN to have a user agent header, that might create some issue on code/CI not complying with that.

Change #1334912 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] Revert^3 "gitlab: discard firewall throttling on the primary"

https://gerrit.wikimedia.org/r/1334912

Change #1334914 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/dns@master] Revert^3 "gitlab: point gitlab.wikimedia.org at the CDN"

https://gerrit.wikimedia.org/r/1334914

Change #1334912 merged by Arnaudb:

[operations/puppet@production] Revert^3 "gitlab: discard firewall throttling on the primary"

https://gerrit.wikimedia.org/r/1334912

Change #1334914 merged by Arnaudb:

[operations/dns@master] Revert^3 "gitlab: point gitlab.wikimedia.org at the CDN"

https://gerrit.wikimedia.org/r/1334914

Gitlab primary instance is back to where it was this morning to simplify debugging the missing user-agent issue.

Mentioned in SAL (#wikimedia-operations) [2026-09-03T14:29:21Z] <sukhe> sudo cumin "A:cp-text" "disable-puppet 'merging CR 1334855'" T425441

jnuche merged https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/113

Send a User-Agent when fetching the JWKS from GitLab

Change #1334855 merged by Ssingh:

[operations/puppet@production] trafficserver: map websockets for the gitlab instances

https://gerrit.wikimedia.org/r/1334855

Mentioned in SAL (#wikimedia-operations) [2026-09-03T14:39:34Z] <sukhe> sudo cumin "A:cp-text" "run-puppet-agent --enable 'merging CR 1334855'": T425441

edit: upon further review, the table bellow has a lots of error, please refer to T437216: Set an identifiable User-Agent on WMF tools calling gitlab.wikimedia.org for that matter.

Given the volume of repositories we have, I asked claude with subagents to identify the most probable victims for the missing User-Agent issue that emerged yesterday:

Two of the three fallouts of the 2026-09-03 attempt were the same failure: a WMF service calling gitlab.wikimedia.org with its HTTP library's default User-Agent. Since HAProxy's 403s never reach webrequest, we cannot see them in Turnilo, so we inventoried the rest statically instead: 3201 public, non-archived, non-forked projects scanned through the API, every risky hit audited by reading the code, and every resulting User-Agent string measured against gitlab-replica-a.wikimedia.org, which has been behind text-lb since 2026-08-12. Verdicts below come from status codes, not from inference.

The rule

A known library token is rejected only while the User-Agent carries no contact. A URL with a scheme, or an e-mail address, anywhere in the string, is enough:

python-requests/2.32.3                        403
python-requests/2.32.3 scap                   403   a bare extra word is not enough
python-requests/2.32.3 ()                     403   nor are empty parentheses
python-requests/2.32.3 wikimedia.org          403   nor is a domain with no scheme
python-requests/2.32.3 https://gitlab.wmo/x   200
python-requests/2.32.3 releng@wikimedia.org   200

Watch out for two naming traps: go- and java are broad prefixes, so a bare go-blubber/1.0 or javascript-linter/1.0 is rejected, while gopher-tool, okhttp-wrapper and guzzle-tool pass.

Suggested convention

<tool>/<version> (<repository url>)

scap/4.130.0 (https://gitlab.wikimedia.org/repos/releng/scap)

Keeping the repository URL satisfies the rule by construction even if the tool is called go-something, and it makes the caller identifiable in the HAProxy logs down to its owners. Several teams already do this: see Wikibugs (https://www.mediawiki.org/wiki/Wikibugs) python-aiohttp/3.13.5.

What sends a blocked User-Agent

100 real HTTP calls were identified across 92 audited projects. 32 send a User-Agent the policy rejects, in 18 repositories; 56 pass; 12 could not be settled by probing, of which only 3 are a real unknown (pkgs_importer in repos/wmf-packages, a binary pulled from gitlab.com, needs someone to check what it sends).

This is an inventory of where the offending call lives, so where a fix would go. Whether it actually breaks depends on the next section.

Who is actually affected: it depends where the code runs

an-worker1200.eqiad.wmnet   Go-http-client/1.1 -> 401   python-requests/2.32.3 -> 401
deploy1003.eqiad.wmnet      Go-http-client/1.1 -> 401   python-requests/2.32.3 -> 401
gitlab-runner1002           Go-http-client/1.1 -> 401   (measured 2026-09-03)

So the table above is not a list of things that will break. Sorted by where the code runs:

• Breaks: CI jobs on the DigitalOcean cloud runners. External egress, not exempt. This is what killed artifact downloads (T435290) and reggie. The clearest case in the table is repos/cloud/cicd/gitlab-ci, whose shared CI images carry the breakage into other people's pipelines.
• Breaks: anything run from a workstation, mwcli for instance.
• Unknown: Toolforge and Cloud VPS. Nobody has probed from there yet, and that is the remaining gap.
• Does not break: code running on WMF production hosts. That includes the two entries I would otherwise have ranked highest: wmf-sparksqlclidriver, whose FileSystem runs in Spark executors on the an-workers, and repos/releng/jwt-authorizer, which runs on the docker registry hosts (docker_registry::web hands it https://gitlab.wikimedia.org/oauth/discovery/keys). Both are exempt where they run, so neither gates the switch.

Owners should therefore ask one question about their entry: does this code ever run outside WMF production? If it only runs on an-workers, stat hosts, deploy hosts or the puppet-managed runners, it is not affected today.

Separate item, for Traffic

go get against a module hosted here is blocked, independently of any repository:

GET /repos/releng/cli?go-get=1        Go-http-client/1.1 -> 403   curl -> 200
GET /repos/releng/cli.git/info/refs   git/2.39.5 -> 200

cmd/go/internal/web/http.go sets no User-Agent and net/http/request.go:554 defines defaultUserAgent = "Go-http-client/1.1". That discovery request happens before the toolchain delegates to git, so the build fails before it ever clones. This affects repos/releng/cli, repos/releng/blubber and everything importing them. Either exempt ?go-get=1, or let Go-http-client through and weaken the policy. Traffic's call.

Not at risk, all measured 200

npm, yarn, pip, uv, poetry, cargo, Composer, Apache-Maven, Gradle, sbt, OpenTofu, Terraform, helm, docker, glab, python-gitlab, renovate, twine, curl, git. So GitLab's npm, PyPI, Maven and Composer registries are fine. CI include: directives do not cross the edge either: there is no include: remote: pointing at gitlab.wikimedia.org across the 3201 projects, and project: and component: includes are resolved inside GitLab.

Scope

jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/629

digitalocean: create dedicated vpc and nat_gateway for each cluster

jnuche opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1314

perform-release: add user agent to requests going out to GitLab

arnaudb opened https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/98

gather_precommit: send a User-Agent on the group listing request

dcaro merged https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/98

gather_precommit: send a User-Agent on the group listing request

It's interesting to see

an-worker1200.eqiad.wmnet   Go-http-client/1.1 -> 401   python-requests/2.32.3 -> 401

in the list of affected clients. I'm not aware of any go binary being executed by the data engineering stack, so that might be something installed by the system itself?

When it comes to

src/main/java/org/apache/hadoop/fs/http/CopiedAbstractHttpFileSystem.java:61, src/main/java/org/apache/hadoop/fs/http/WmfHttpsFileSystem.java:40

we should stop seeing these issues, as I've reconfigured all DE jobs to use the gitlab.discovery.wmnet:8443 https endpoint. If that's not the case, I might have missed something.

The dbt case is interesting, as this is a python build system fetching/cloning a package at build time. I will investigate whether we can tune the user agent of said build system.

Three corrections to T425441#12288782. Line numbers below are in modules/profile/templates/cache/haproxy/tls_terminator.cfg.erb.

Toolforge and Cloud VPS are not a gap. I listed them as the one unmeasured path; they are exempt. Cloud VPS traffic to text-lb is deliberately not NATed: hieradata/role/common/wmcs/cloudgw.yaml lists every text-lb and upload-lb VIP in dmz_cidr, so the address the edge sees is the instance's own, and the eqiad1 instance subnets are 172.16.0.0/21, 172.16.8.0/21, 172.16.16.0/21 and 172.16.24.0/24. HAProxy puts 172.16.0.0/12 in net=internal and gives it trust score A, and the robot policy only applies below score C (line 434). The NATed path is covered too, line 201:

# ensure that WMCS is marked as trusted.
http-request set-var(req.trusted_request,ifnotexists) str(A) if { var(txn.provenance) -m sub "cloud=wmcs" }

One residual: dmz_cidr lists IPv4 VIPs only, so an instance egressing over IPv6 would fall back on the cloud=wmcs classification in the generated provenance map, which is not in Puppet.

A User-Agent trap, line 458:

acl ua_forge_url req.fhdr(User-Agent) -m reg -i "https?\:\/\/(github\.com|gitlab\.com)"

A library-default User-Agent carrying a github.com or gitlab.com URL is denied despite carrying a URL, because that URL identifies the software rather than its operator. So python-requests/2.32.5 (+https://github.com/wikimedia/foo) is rejected. A gitlab.wikimedia.org URL does not match, so the convention suggested earlier still holds, but anyone reaching for their GitHub mirror will be refused without an obvious reason.

go get is narrower than I wrote. I said the ?go-get=1 block affects repos/releng/cli, repos/releng/blubber "and everything importing them". Too broad: proxy.golang.org already mirrors both modules and neither repository sets GOPROXY or GOPRIVATE, so a default go get never contacts gitlab.wikimedia.org. It only bites private Go modules, environments pinned to GOPROXY=direct, and possibly the mirror's own first fetch of a new version, which I cannot measure. It does not gate the switch.

A question for Traffic. gerrit.wikimedia.org already has a temporary exemption from the library-default rule, line 452, added for T417156: Pushing to gerrit over http is blocked by generic rate limiting:

acl is_gerrit req.fhdr(Host) -m str -i gerrit.wikimedia.org

with !is_gerrit on each of the four deny lines below it. The same treatment for gitlab.wikimedia.org would remove the whole risk class from the next attempt and could be withdrawn as owners fix their callers. Separately, both Phabricator and Gerrit get trust score C on an authenticated session cookie, lines 348 and 353, and GitLab has no equivalent.

Findings. reggie MR 113 and gitlab-ci MR 98 are merged, scap MR 1314 is in review, releases-jenkins is covered by CL 1335868. The other entries in the table are untouched so far. Two things affect how much they matter: production source IPs are exempt, and @Jelto's gitlab-cloud-runner MR 629 gives the DigitalOcean runners a static egress address, which Traffic could allowlist instead of each team patching its own caller.

In T425441#12293630, @brouberol wrote:

It's interesting to see

an-worker1200.eqiad.wmnet   Go-http-client/1.1 -> 401   python-requests/2.32.3 -> 401

in the list of affected clients. I'm not aware of any go binary being executed by the data engineering stack, so that might be something installed by the system itself?

When it comes to

src/main/java/org/apache/hadoop/fs/http/CopiedAbstractHttpFileSystem.java:61, src/main/java/org/apache/hadoop/fs/http/WmfHttpsFileSystem.java:40

we should stop seeing these issues, as I've reconfigured all DE jobs to use the gitlab.discovery.wmnet:8443 https endpoint. If that's not the case, I might have missed something.

The dbt case is interesting, as this is a python build system fetching/cloning a package at build time. I will investigate whether we can tune the user agent of said build system.

Thanks, and sorry for the confusing formatting: those two User-Agents were curl probes I sent from an-worker1200 and deploy1003 to show that production source IPs are exempt, not traffic observed coming from your stack. Nothing on an-worker1200 is sending Go-http-client/1.1 as far as this audit knows. The interesting part of that block is the 401: the edge let both User-Agents through.

Confirmed on wmf-sparksqlclidriver. With DE jobs pointed at gitlab.discovery.wmnet:8443 the requests no longer cross the edge at all, so that entry is covered twice over, since the an-workers were exempt anyway.

On dbt, please don't spend time on it: that row is a false positive of mine. dbt-jobs has no packages.yml and no gitlab.wikimedia.org URL anywhere in the repository, and its only GitLab references are the CI include: project: and component: directives, which GitLab resolves internally. The row came from noting that dbt-core uses requests, with no call site ever located, and it is the only one of the 19 rows in that table without a file and line number. I rechecked the other 18.

What about https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/pyproject.toml?ref_type=heads#L59 and https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/pyproject.toml?ref_type=heads#L81? Fetching that repository at build time will induce an https call to gitlab I think. However, this might only be done on trusted runners.

Good catch, and unlike the dbt row that one is a genuine call across the edge. It passes, because Poetry sets its own User-Agent.

Those two lines are Poetry pulling workflow-utils 0.32.0 from GitLab's PyPI package registry as a supplemental source. Poetry builds its User-Agent through requests_toolbelt (poetry/utils/authenticator.py), so it sends poetry/<version> CPython/<version> <os>/<kernel>, which carries no library-default prefix. Measured against your exact URL on gitlab-replica-a, behind the CDN since 2026-08-12:

200  <-  poetry/2.3.1 CPython/3.14.7 Linux/...
200  <-  pip/25.0.1
403  <-  python-requests/2.32.3

So it holds on any runner, trusted or not, and the same applies to the npm, Maven and Composer registries. A git = dependency pointing at gitlab.wikimedia.org would be fine too, since git sends git/<version>. The only way that fetch breaks is if something pulls the index with a bare requests session, which nothing in that pyproject.toml does.

Great, thank you @ABran-WMF !

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/629

digitalocean: create dedicated vpc and nat_gateway for each cluster

Change #1338134 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] cache: pipe websocket upgrades for the gitlab hostnames

https://gerrit.wikimedia.org/r/1338134

Change #1338946 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gitlab.wm.o: allow library-default UAs

https://gerrit.wikimedia.org/r/1338946

Change #1338134 merged by Ssingh:

[operations/puppet@production] cache: pipe websocket upgrades for the gitlab hostnames

https://gerrit.wikimedia.org/r/1338134

Change #1341184 had a related patch set uploaded (by Arnaudb; author: Arnaudb):

[operations/puppet@production] gitlab: log the real client address behind the CDN

https://gerrit.wikimedia.org/r/1341184

Mentioned in SAL (#wikimedia-operations) [2026-09-14T13:20:54Z] <sukhe> sudo cumin -b11 "A:cp-text" "run-puppet-agent --enable 'merging CR 1338134'"T425441

Mentioned in SAL (#wikimedia-operations) [2026-09-14T13:20:58Z] <sukhe> sudo cumin -b11 "A:cp-text" "run-puppet-agent --enable 'merging CR 1338134'" T425441

Following up on:

In T425441#12285450, @ABran-WMF wrote:

thanks for mentioning it @jnuche
I sent 2 patches, one that will enable websockets on Envoy, the other to map them on our backends. That should fix the issue you're facing.

thanks @ssingh for the merge and rollout, it is possible to use ws:// to interact with Gitlab:

for h in gitlab-replica-a gitlab-replica-b gitlab; do
  printf '%-18s %s\n' "$h" "$(curl -sS -o /dev/null -m 8 --http1.1 -w '%{http_code}' \
    -A 'ws-check/1 (https://phabricator.wikimedia.org/T425441)' \
    -H 'Connection: Upgrade' -H 'Upgrade: websocket' \
    -H 'Sec-WebSocket-Version: 13' \
    -H "Sec-WebSocket-Key: $(openssl rand -base64 16)" \
    "https://$h.wikimedia.org/-/kubernetes-agent/")"
done
gitlab-replica-a   101
gitlab-replica-b   101
gitlab             101

jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/631

digitalocean: create dedicated vpc and nat_gateway for each cluster

Change #1341184 merged by Arnaudb:

[operations/puppet@production] gitlab: log the real client address behind the CDN

https://gerrit.wikimedia.org/r/1341184

dancy merged https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/631

digitalocean: create dedicated vpc and nat_gateway for each cluster