Page MenuHomePhabricator
Create Task
Maniphest T392356

Replace ingress-nginx before upstream EOL date
Open, In Progress, HighPublic
Actions

Description

The ingress-nginx project that we heavily rely on for handling Toolforge traffic is now in maintenance mode: https://github.com/kubernetes/ingress-nginx/issues/13002

We have several options:

  1. Do nothing.
  2. Migrate to some other ingress provider.
  3. Once it's ready, migrate to InGate, which is a work-in-progress Ingress/Gateway API controller by the same people who maintain ingress-nginx
  4. Migrate to some other Gateway API implementation

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
toolforge: Remove support for ingress workerscloud/wmcs-cookbooksmain+3 -164
P:toolforge::prometheus: Stop monitoring ingress-nginxoperations/puppetproduction+0 -19
P:toolforge::k8s::haproxy: Remove old ingress nodesoperations/puppetproduction+2 -12
P:toolforge: k8s: haproxy: Fix istio-gateway health checksoperations/puppetproduction+3 -2
P:toolforge: k8s: haproxy: Use HTTP/1.1 for health checksoperations/puppetproduction+2 -1
P:toolforge: k8s: haproxy: Add option for sending traffic to Istiooperations/puppetproduction+10 -2
Customize query in gerrit
Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
kubernetes: Remove ingress-nginx specific alertsrepos/cloud/toolforge/alerts!61taavimain-I103ce32659250d1c56782534d4ca4de573821b09main
Undeploy ingress-nginxrepos/cloud/toolforge/toolforge-deploy!1222taavimain-I92a669407df35312cf6a62bccacbd23a5fc84b0fmain
kubernetes: Stop manipulating Ingress objectsrepos/cloud/toolforge/webservice-cli!100taavimain-Ieebd5be6d70c7f3ede4674e5f565d9c88fa865ccmain
istio-gateway: Expose metrics endpointrepos/cloud/toolforge/toolforge-deploy!1180taavimain-I239dad25031471ccbb3be1fd2e3b98f13121245fmain
Customize query in GitLab

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
dcaro moved this task from Backlog to Ready to be worked on on the Toolforge board.May 14 2025, 5:08 PM
• bd808 added a comment.Nov 18 2025, 4:23 PM

https://www.kubernetes.dev/blog/2025/11/12/ingress-nginx-retirement/

To prioritize the safety and security of the ecosystem, Kubernetes SIG Network and the Security Response Committee are announcing the upcoming retirement of Ingress NGINX. Best-effort maintenance will continue until March 2026. Afterward, there will be no further releases, no bugfixes, and no updates to resolve any security vulnerabilities that may be discovered.

• taavi renamed this task from toolforge: Investigate ingress-nginx replacements to Replace ingress-nginx before upstream EOL date.Jan 15 2026, 12:41 PM
• taavi raised the priority of this task from Medium to High.
• taavi created subtask T414674: Remove remaining uses of ingress-nginx specific annotations.Jan 15 2026, 12:44 PM
• bd808 mentioned this in T414836: Create a reusable container to replace nginx ingress anonymizing reverse proxy setups.Jan 16 2026, 6:29 PM
• bd808 added a subtask: T414836: Create a reusable container to replace nginx ingress anonymizing reverse proxy setups.Jan 16 2026, 10:22 PM
Arendpieter subscribed.Jan 20 2026, 4:09 PM

It looks like InGate is no option:

https://github.com/kubernetes-sigs/ingate/commit/7c0f10563c6ef12e97172b8ca40e1a2006f73a35

InGate is being retired (early 2026).
SIG Network and the Security Response Committee recommend that all users begin migration to Gateway API or another Ingress controller immediately. Many options are listed in the Kubernetes documentation: Gateway API and Ingress. Additional options may be available from vendors you work with.

• taavi removed a subtask: T414836: Create a reusable container to replace nginx ingress anonymizing reverse proxy setups.Jan 21 2026, 11:16 AM
• taavi updated the task description. (Show Details)Jan 21 2026, 12:54 PM
• taavi created subtask T415192: Move ingress annotations to an allowlist model.Jan 21 2026, 1:06 PM
• taavi closed subtask T415192: Move ingress annotations to an allowlist model as Resolved.Jan 28 2026, 4:07 PM
• taavi claimed this task.Feb 24 2026, 12:26 PM
• taavi created subtask T418274: Pick and install a Gateway API implementation for Toolforge.Feb 24 2026, 4:51 PM
• taavi created subtask T418276: Update access control logic for migration from Ingress to Gateway API objects.Feb 24 2026, 4:58 PM
• taavi created subtask T418349: Make webservice provision HTTPRoutes for each tool.Feb 25 2026, 9:05 AM
• taavi created subtask T418479: Port fourohfour tool to new Gateway system.Feb 26 2026, 1:48 PM
dcaro moved this task from Ready to be worked on to Toolforge iteration 25 on the Toolforge board.Mar 3 2026, 2:58 PM
dcaro edited projects, added: Toolforge (Toolforge iteration 25); removed: Toolforge.
dcaro changed the task status from Open to In Progress.Mar 3 2026, 3:25 PM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 25) board.
• taavi closed subtask T418276: Update access control logic for migration from Ingress to Gateway API objects as Resolved.Mar 4 2026, 4:04 PM
dcaro edited projects, added: Toolforge (Toolforge iteration 26); removed: Toolforge (Toolforge iteration 25).Mar 5 2026, 9:35 AM
dcaro moved this task from Next Up to In Progress on the Toolforge (Toolforge iteration 26) board.
• taavi closed subtask T418274: Pick and install a Gateway API implementation for Toolforge as Resolved.Mar 6 2026, 12:17 PM
• taavi created subtask T419231: Scale up Toolforge Istio deployment for production traffic levels.Mar 6 2026, 12:28 PM
• taavi closed subtask T418349: Make webservice provision HTTPRoutes for each tool as Resolved.Mar 9 2026, 3:35 PM
• taavi closed subtask T419231: Scale up Toolforge Istio deployment for production traffic levels as Resolved.Mar 9 2026, 5:15 PM
Arendpieter unsubscribed.Mar 10 2026, 7:46 PM
Stashbot added a comment.Mar 18 2026, 12:00 PM

Mentioned in SAL (#wikimedia-cloud) [2026-03-18T12:00:34Z] <taavi> restarting existing web services to backfill HTTPRoute resources T392356

fnegri moved this task from Inbox to FY2025/2026-Q3-Q4 on the cloud-services-team board.Mar 18 2026, 3:14 PM
fnegri edited projects, added: cloud-services-team (FY2025/2026-Q3-Q4); removed: cloud-services-team.
fnegri moved this task from Backlog to In progress on the cloud-services-team (FY2025/2026-Q3-Q4) board.Mar 19 2026, 4:13 PM
gerritbot added a comment.Mar 23 2026, 9:17 AM

Change #1258948 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: k8s: haproxy: Add option for sending traffic to Istio

https://gerrit.wikimedia.org/r/1258948

gerritbot added a project: Patch-For-Review.Mar 23 2026, 9:17 AM
gerritbot added a comment.Mar 23 2026, 10:02 AM

Change #1258948 merged by Majavah:

[operations/puppet@production] P:toolforge: k8s: haproxy: Add option for sending traffic to Istio

https://gerrit.wikimedia.org/r/1258948

gerritbot added a comment.Mar 23 2026, 10:15 AM

Change #1258980 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: k8s: haproxy: Use HTTP/1.1 for health checks

https://gerrit.wikimedia.org/r/1258980

gerritbot added a comment.Mar 23 2026, 10:20 AM

Change #1258980 merged by Majavah:

[operations/puppet@production] P:toolforge: k8s: haproxy: Use HTTP/1.1 for health checks

https://gerrit.wikimedia.org/r/1258980

CodeReviewBot added a comment.Mar 23 2026, 10:25 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1180

istio-gateway: Expose metrics endpoint

gerritbot added a comment.Mar 23 2026, 10:37 AM

Change #1259000 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: k8s: haproxy: Fix istio-gateway health checks

https://gerrit.wikimedia.org/r/1259000

CodeReviewBot added a comment.Mar 23 2026, 10:45 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1180

istio-gateway: Expose metrics endpoint

gerritbot added a comment.Mar 23 2026, 10:45 AM

Change #1259000 merged by Majavah:

[operations/puppet@production] P:toolforge: k8s: haproxy: Fix istio-gateway health checks

https://gerrit.wikimedia.org/r/1259000

Stashbot added a comment.Mar 23 2026, 10:53 AM

Mentioned in SAL (#wikimedia-cloud) [2026-03-23T10:53:31Z] <taavi> send 5% of traffic to istio T392356

Stashbot added a comment.Mar 23 2026, 11:16 AM

Mentioned in SAL (#wikimedia-cloud) [2026-03-23T11:16:07Z] <taavi> send 10% of traffic to istio T392356

Maintenance_bot removed a project: Patch-For-Review.Mar 23 2026, 11:31 AM
• taavi created subtask T421386: cadvisor-reported Istio network usage is way too high.Mar 26 2026, 2:22 PM
• taavi added a subtask: T422829: Toolforge HTML head links sometimes are issued as http://<tool>.toolforge:443.Apr 9 2026, 2:17 PM
• taavi edited projects, added: Toolforge; removed: Toolforge (Toolforge iteration 26).Apr 9 2026, 3:30 PM
• taavi closed subtask T422829: Toolforge HTML head links sometimes are issued as http://<tool>.toolforge:443 as Resolved.Apr 10 2026, 8:40 AM
Stashbot added a comment.Apr 13 2026, 7:33 AM

Mentioned in SAL (#wikimedia-cloud) [2026-04-13T07:33:03Z] <taavi> bump istio traffic percentage 10% -> 25% T392356

• taavi closed subtask T421386: cadvisor-reported Istio network usage is way too high as Resolved.Apr 13 2026, 7:55 AM
Stashbot added a comment.Apr 13 2026, 9:11 AM

Mentioned in SAL (#wikimedia-cloud) [2026-04-13T09:11:09Z] <taavi> bump istio traffic percentage 25% -> 50% T392356

Stashbot added a comment.Apr 15 2026, 10:45 AM

Mentioned in SAL (#wikimedia-cloud) [2026-04-15T10:45:34Z] <taavi> bump istio traffic percentage 50% -> 75% T392356

Stashbot added a comment.Apr 16 2026, 1:09 PM

Mentioned in SAL (#wikimedia-cloud) [2026-04-16T13:09:27Z] <taavi> bump istio traffic percentage 75% -> 100% T392356

gerritbot added a comment.Apr 16 2026, 1:27 PM

Change #1272714 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Remove old ingress nodes

https://gerrit.wikimedia.org/r/1272714

gerritbot added a project: Patch-For-Review.Apr 16 2026, 1:27 PM
gerritbot added a comment.Apr 16 2026, 2:53 PM

Change #1272714 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Remove old ingress nodes

https://gerrit.wikimedia.org/r/1272714

Maintenance_bot removed a project: Patch-For-Review.Apr 16 2026, 3:31 PM
JJMC89 mentioned this in T423652: uploadmap reverse proxy broken because of platform changes.Apr 16 2026, 8:18 PM
• bd808 added a comment.Apr 16 2026, 9:14 PM
In T392356#11830166, @gerritbot wrote:

Change #1272714 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Remove old ingress nodes

https://gerrit.wikimedia.org/r/1272714

This unannounced breaking change has caught at least T423652: uploadmap reverse proxy broken because of platform changes and likely a few more Ingress objects without matching HTTPRoute objects unaware.

I checked cloud-announce and was unable to find any message mentioning HTTPRoute.

CodeReviewBot added a project: Patch-For-Review.Apr 17 2026, 3:23 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/100

kubernetes: Stop manipulating Ingress objects

CodeReviewBot added a comment.Apr 17 2026, 3:25 PM

taavi updated https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/86

Remove tool ability to interact with Ingress objects

CodeReviewBot added a comment.Apr 17 2026, 3:26 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1222

Undeploy ingress-nginx

CodeReviewBot added a comment.Apr 21 2026, 12:51 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/100

kubernetes: Stop manipulating Ingress objects

CodeReviewBot added a comment.Apr 21 2026, 12:53 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/101

d/changelog: bump to 0.103.20

CodeReviewBot added a comment.Apr 21 2026, 12:58 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/webservice-cli/-/merge_requests/101

d/changelog: bump to 0.103.20

CodeReviewBot added a comment.Apr 21 2026, 1:35 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/86

Remove tool ability to interact with Ingress objects

CodeReviewBot added a comment.Apr 21 2026, 1:38 PM

group_203_bot_3c0afd0d9fd9529f3b7bc7e69a4a3bce opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1224

maintain-kubeusers: bump to 0.0.194-20260421133553-217c468a

CodeReviewBot added a comment.Apr 21 2026, 1:45 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1224

maintain-kubeusers: bump to 0.0.194-20260421133553-217c468a

Stashbot added a comment.Apr 21 2026, 2:06 PM

Mentioned in SAL (#wikimedia-cloud) [2026-04-21T14:06:10Z] <taavi> save backup of all ingress objects to ~taavi/ingresses-backup-2026-04-21.json T392356

Stashbot added a comment.Apr 21 2026, 2:10 PM

Mentioned in SAL (#wikimedia-cloud) [2026-04-21T14:10:20Z] <taavi> delete all ingress objects from toolsbeta T392356

Stashbot added a comment.Apr 23 2026, 8:08 AM

Mentioned in SAL (#wikimedia-cloud) [2026-04-23T08:08:18Z] <taavi> delete all ingress objects T392356

gerritbot added a comment.Apr 23 2026, 8:10 AM

Change #1276596 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::prometheus: Stop monitoring ingress-nginx

https://gerrit.wikimedia.org/r/1276596

gerritbot added a comment.Apr 23 2026, 8:18 AM

Change #1276596 merged by Majavah:

[operations/puppet@production] P:toolforge::prometheus: Stop monitoring ingress-nginx

https://gerrit.wikimedia.org/r/1276596

CodeReviewBot added a comment.Apr 23 2026, 9:11 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1222

Undeploy ingress-nginx

Stashbot added a comment.Apr 23 2026, 9:12 AM

Mentioned in SAL (#wikimedia-cloud) [2026-04-23T09:12:52Z] <taavi> uninstall ingress-nginx-gen2 from the cluster T392356

CodeReviewBot added a comment.Apr 23 2026, 9:17 AM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/61

kubernetes: Remove ingress-nginx specific alerts

gerritbot added a comment.Apr 23 2026, 9:40 AM

Change #1276631 had a related patch set uploaded (by Majavah; author: Majavah):

[cloud/wmcs-cookbooks@main] toolforge: Remove support for ingress workers

https://gerrit.wikimedia.org/r/1276631

CodeReviewBot added a comment.Apr 23 2026, 12:11 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/61

kubernetes: Remove ingress-nginx specific alerts

gerritbot added a comment.Apr 23 2026, 12:14 PM

Change #1276631 merged by jenkins-bot:

[cloud/wmcs-cookbooks@main] toolforge: Remove support for ingress workers

https://gerrit.wikimedia.org/r/1276631

• taavi mentioned this in rCCKB1a16e21a31c4: toolforge: Remove support for ingress workers.Apr 23 2026, 12:15 PM
LucasWerkmeister mentioned this in R2362:4f201ca05af1: Update gunicorn config.Apr 25 2026, 11:33 PM
HakanIST subscribed.EditedApr 28 2026, 1:46 PM

Issue in T424029 with a Python webservice (lingua-libre) where OAuth callbacks were failing with upstream connect error or disconnect/reset before headers after the Istio migration. Turned out the X-Envoy-Peer-Metadata header injected by the sidecar was exceeding uwsgi's default buffer size. Adding buffer-size = 65535 to uwsgi.ini seems to resolve it. Leaving this here in case others hit the same thing.

CodeReviewBot added a comment.Apr 28 2026, 3:45 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235

istio-system: Disable X-Envoy-Peer-Metadata headers

• taavi added a comment.Apr 28 2026, 4:01 PM
In T392356#11866031, @HakanIST wrote:

Issue in T424029 with a Python webservice (lingua-libre) where OAuth callbacks were failing with upstream connect error or disconnect/reset before headers after the Istio migration. Turned out the X-Envoy-Peer-Metadata header injected by the sidecar was exceeding uwsgi's default buffer size. Adding buffer-size = 65535 to uwsgi.ini seems to resolve it. Leaving this here in case others hit the same thing.

Thanks. I don't think that header is very useful for most tools, so I'll see if we can disable it to avoid breaking more things.

Arendpieter added a comment.Apr 28 2026, 10:41 PM

Just to make sure: this issue only affects Toolforge, i.e., Cloud Services, right? Is ingress-nginx also used on the production Kubernetes cluster that serves production traffic?

Arendpieter subscribed.Apr 29 2026, 11:04 AM
Yug subscribed.EditedApr 29 2026, 1:02 PM

@Arendpieter / FYI :

Yug mentioned this in T424029: Upstream connect error.Apr 29 2026, 1:11 PM
HakanIST mentioned this in T425229: istio-envoy enforcing strict RFC 7231 compliance for 204 status responses.May 2 2026, 9:13 PM
• bd808 added a subtask: T425229: istio-envoy enforcing strict RFC 7231 compliance for 204 status responses.May 14 2026, 3:58 PM
• taavi created subtask T428159: Istio X-Envoy-Peer-Metadata causes buffer size issues for uWSGI based tools.Thu, Jun 4, 12:28 PM
CodeReviewBot added a comment.Thu, Jun 4, 12:56 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1235

istio-gateway: Reduce Istio header verbosity

• taavi closed subtask T428159: Istio X-Envoy-Peer-Metadata causes buffer size issues for uWSGI based tools as Resolved.Thu, Jun 4, 1:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits