Page MenuHomePhabricator
Create Task
Maniphest T283948

Merge Toolforge Nginx front proxy into the existing K8s HAProxy setup
Closed, ResolvedPublic
Actions

Description

We want to eliminate the manual failover with the Toolforge dynamicproxy servers (SSL termination + grid engine routing). This is currently the only point on Toolforge k8s web requests that has a single point of failure, haproxy+ingress+worked pods will all automatically fail over on node failure. I essentially see two options here:

  • Completely remove that layer, and terminate TLS at either on haproxy or on the ingress, like on PAWS haproxy
  • Do automatic failover for Nginx (easy) and its Redis backend (complex for a "temporary" solution)

The first one is ideal in the long-term, but would require a workaround for grid engine tools (T282975) until the grid is deprecated and removed (a long time).

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
P:toolforge: Remove obsolete spec testoperations/puppetproduction+0 -14
P:toolforge: Remove separate proxy roleoperations/puppetproduction+0 -381
P:toolforge::prometheus: Drop separate front proxy scrape targetoperations/puppetproduction+0 -5
P:toolforge::k8s::haproxy: Drop proxy IP rate limit exemptionoperations/puppetproduction+1 -6
P:toolforge::k8s::haproxy: Add per-IP rate limitingoperations/puppetproduction+19 -3
P:toolforge::k8s::haproxy: Add banned IP listoperations/puppetproduction+23 -18
P:toolforge: Migrate default robots and favicon handlers to HAProxyoperations/puppetproduction+20 -30
P:toolforge: Migrate error page static assets to tools-staticoperations/puppetproduction+42 -11
P:toolforge::proxy: Remove config moved to HAProxyoperations/puppetproduction+0 -21
P:toolforge: Move http redirect rewrite to HAProxyoperations/puppetproduction+8 -8
P:toolforge: Move U-A/Referer blocks to HAProxyoperations/puppetproduction+14 -20
P:toolforge: Move ru_monuments backwards compat redirect to HAProxyoperations/puppetproduction+3 -5
P:toolforge::k8s::haproxy: Use http-after-response for headersoperations/puppetproduction+6 -6
P:toolforge::k8s::haproxy: Fix tool address in CSP headeroperations/puppetproduction+2 -1
P:toolforge::k8s::haproxy: Move per-tool rate limiting to HAProxyoperations/puppetproduction+30 -27
P:toolforge::k8s::haproxy: Use custom error pagesoperations/puppetproduction+29 -0
P:toolforge::k8s::haproxy: Handle API gateway external accessoperations/puppetproduction+17 -206
P:toolforge::k8s::haproxy: Drop old TCP listeneroperations/puppetproduction+0 -23
P:toolforge::proxy: Set backend to new HAProxy HTTPS serviceoperations/puppetproduction+8 -5
P:toolforge::k8s::haproxy: Add a HTTPS listeneroperations/puppetproduction+72 -3
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 28 2021, 6:28 PM
taavi added a subtask: T282975: Create Kubernetes ingress for tools running on the grid engine to remove dynamicproxy.May 28 2021, 6:28 PM
Frostly subscribed.May 28 2021, 7:13 PM
taavi added a parent task: T90534: Make toolforge reliable enough (tracking).Jun 12 2021, 5:02 PM
taavi mentioned this in T119936: Investigate alternatives to Dynamicproxy on Cloud VPS.Jun 13 2021, 11:15 AM
• nskaggs triaged this task as Medium priority.Aug 10 2021, 10:07 PM
Frostly unsubscribed.Aug 23 2021, 11:58 PM
taavi closed subtask T282975: Create Kubernetes ingress for tools running on the grid engine to remove dynamicproxy as Resolved.Jul 2 2022, 2:13 PM
Andrew closed subtask T317714: Include unique ip pageviews in the toolviews report as Resolved.Sep 19 2022, 3:21 PM
fnegri edited projects, added cloud-services-team; removed cloud-services-team (Kanban).Jan 18 2023, 6:51 PM
fnegri moved this task from Kanban to Inbox on the cloud-services-team board.
taavi removed a subtask: T317714: Include unique ip pageviews in the toolviews report.Jan 24 2024, 1:54 PM
taavi closed subtask T284564: Update webservicemonitor to work without dynamicproxy as Declined.Jan 24 2024, 1:57 PM
akosiaris subscribed.Sep 18 2025, 3:11 PM
gerritbot added a comment.Sep 19 2025, 9:18 AM

Change #1189801 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Add a HTTPS listener

https://gerrit.wikimedia.org/r/1189801

gerritbot added a project: Patch-For-Review.Sep 19 2025, 9:18 AM

Change #1189802 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::proxy: Set backend to new HAProxy HTTPS service

https://gerrit.wikimedia.org/r/1189802

taavi claimed this task.Sep 19 2025, 9:33 AM
gerritbot added a comment.Sep 19 2025, 10:00 AM

Change #1189801 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Add a HTTPS listener

https://gerrit.wikimedia.org/r/1189801

gerritbot added a comment.Sep 19 2025, 10:20 AM

Change #1189802 merged by Majavah:

[operations/puppet@production] P:toolforge::proxy: Set backend to new HAProxy HTTPS service

https://gerrit.wikimedia.org/r/1189802

Maintenance_bot removed a project: Patch-For-Review.Sep 19 2025, 10:31 AM
taavi edited projects, added tools-infrastructure-team; removed cloud-services-team.Sep 19 2025, 11:03 AM
gerritbot added a comment.Sep 19 2025, 11:34 AM

Change #1189840 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Drop old TCP listener

https://gerrit.wikimedia.org/r/1189840

gerritbot added a project: Patch-For-Review.Sep 19 2025, 11:34 AM

Change #1189841 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Handle API gateway external access

https://gerrit.wikimedia.org/r/1189841

gerritbot added a comment.Sep 19 2025, 12:06 PM

Change #1189840 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Drop old TCP listener

https://gerrit.wikimedia.org/r/1189840

gerritbot added a comment.Sep 22 2025, 9:40 AM

Change #1189841 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Handle API gateway external access

https://gerrit.wikimedia.org/r/1189841

Maintenance_bot removed a project: Patch-For-Review.Sep 22 2025, 10:30 AM
taavi renamed this task from Eliminate single point of failure from Toolforge front proxy to Merge Toolforge Nginx front proxy into the existing K8s HAProxy setup.Sep 25 2025, 9:17 AM
taavi updated the task description. (Show Details)
gerritbot added a comment.Sep 26 2025, 7:52 AM

Change #1191582 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Use custom error pages

https://gerrit.wikimedia.org/r/1191582

gerritbot added a project: Patch-For-Review.Sep 26 2025, 7:52 AM

Change #1191583 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Move per-tool rate limiting to HAProxy

https://gerrit.wikimedia.org/r/1191583

gerritbot added a comment.Sep 26 2025, 7:52 AM

Change #1191584 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Fix tool address in CSP header

https://gerrit.wikimedia.org/r/1191584

gerritbot added a comment.Sep 26 2025, 9:50 AM

Change #1191582 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Use custom error pages

https://gerrit.wikimedia.org/r/1191582

gerritbot added a comment.Sep 26 2025, 9:55 AM

Change #1191583 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Move per-tool rate limiting to HAProxy

https://gerrit.wikimedia.org/r/1191583

gerritbot added a comment.Sep 29 2025, 7:19 AM

Change #1191584 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Fix tool address in CSP header

https://gerrit.wikimedia.org/r/1191584

Maintenance_bot removed a project: Patch-For-Review.Sep 29 2025, 7:31 AM
taavi closed subtask T405078: Rebuild Toolforge HAProxies to support IPv6 as Resolved.Oct 3 2025, 3:05 PM
gerritbot added a comment.Oct 3 2025, 4:07 PM

Change #1193448 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: Move ru_monuments backwards compat redirect to HAProxy

https://gerrit.wikimedia.org/r/1193448

gerritbot added a project: Patch-For-Review.Oct 3 2025, 4:07 PM

Change #1193449 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: Move U-A/Referer blocks to HAProxy

https://gerrit.wikimedia.org/r/1193449

gerritbot added a comment.Oct 3 2025, 4:07 PM

Change #1193450 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: Move http redirect rewrite to HAProxy

https://gerrit.wikimedia.org/r/1193450

gerritbot added a comment.Oct 3 2025, 4:13 PM

Change #1193451 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Use http-after-response for headers

https://gerrit.wikimedia.org/r/1193451

gerritbot added a comment.Oct 3 2025, 4:28 PM

Change #1193454 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::proxy: Remove config moved to HAProxy

https://gerrit.wikimedia.org/r/1193454

Krinkle awarded a token.Oct 3 2025, 6:25 PM
gerritbot added a comment.Oct 6 2025, 9:54 AM

Change #1193451 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Use http-after-response for headers

https://gerrit.wikimedia.org/r/1193451

gerritbot added a comment.Oct 6 2025, 9:54 AM

Change #1193448 merged by Majavah:

[operations/puppet@production] P:toolforge: Move ru_monuments backwards compat redirect to HAProxy

https://gerrit.wikimedia.org/r/1193448

gerritbot added a comment.Oct 6 2025, 9:59 AM

Change #1193449 merged by Majavah:

[operations/puppet@production] P:toolforge: Move U-A/Referer blocks to HAProxy

https://gerrit.wikimedia.org/r/1193449

gerritbot added a comment.Oct 6 2025, 10:00 AM

Change #1193450 merged by Majavah:

[operations/puppet@production] P:toolforge: Move http redirect rewrite to HAProxy

https://gerrit.wikimedia.org/r/1193450

gerritbot added a comment.Oct 6 2025, 10:00 AM

Change #1193454 merged by Majavah:

[operations/puppet@production] P:toolforge::proxy: Remove config moved to HAProxy

https://gerrit.wikimedia.org/r/1193454

Maintenance_bot removed a project: Patch-For-Review.Oct 6 2025, 10:31 AM
gerritbot added a comment.Oct 8 2025, 7:47 AM

Change #1194532 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: Migrate error page static assets to tools-static

https://gerrit.wikimedia.org/r/1194532

gerritbot added a project: Patch-For-Review.Oct 8 2025, 7:47 AM

Change #1194533 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: Migrate default robots and favicon handlers to HAProxy

https://gerrit.wikimedia.org/r/1194533

gerritbot added a comment.Oct 8 2025, 8:54 AM

Change #1194532 merged by Majavah:

[operations/puppet@production] P:toolforge: Migrate error page static assets to tools-static

https://gerrit.wikimedia.org/r/1194532

gerritbot added a comment.Oct 8 2025, 8:57 AM

Change #1194533 merged by Majavah:

[operations/puppet@production] P:toolforge: Migrate default robots and favicon handlers to HAProxy

https://gerrit.wikimedia.org/r/1194533

CodeReviewBot added a comment.Oct 8 2025, 9:14 AM

taavi opened https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/15

Use assets from tools-static

CodeReviewBot added a comment.Oct 8 2025, 10:16 AM

taavi merged https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/15

Use assets from tools-static

Maintenance_bot removed a project: Patch-For-Review.Oct 8 2025, 10:34 AM
gerritbot added a comment.Oct 9 2025, 8:48 AM

Change #1194881 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Add banned IP list

https://gerrit.wikimedia.org/r/1194881

gerritbot added a project: Patch-For-Review.Oct 9 2025, 8:48 AM

Change #1194882 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Add per-IP rate limiting

https://gerrit.wikimedia.org/r/1194882

gerritbot added a comment.Oct 9 2025, 9:22 AM

Change #1194881 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Add banned IP list

https://gerrit.wikimedia.org/r/1194881

gerritbot added a comment.Oct 9 2025, 9:26 AM

Change #1194882 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Add per-IP rate limiting

https://gerrit.wikimedia.org/r/1194882

Maintenance_bot removed a project: Patch-For-Review.Oct 9 2025, 9:31 AM
CodeReviewBot added a project: Patch-For-Review.Oct 20 2025, 6:04 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/94

Allocate floating IP to HAProxy service VIP

CodeReviewBot added a comment.Oct 20 2025, 6:05 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/95

Allow inbound HTTPS traffic to HAProxy service VIP

CodeReviewBot added a comment.Oct 20 2025, 6:05 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/96

toolsbeta: Point web traffic to HAProxy directly

CodeReviewBot added a comment.Oct 21 2025, 2:08 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/94

Allocate floating IP to HAProxy service VIP

CodeReviewBot added a comment.Oct 21 2025, 2:18 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/95

Allow inbound HTTPS traffic to HAProxy service VIP

CodeReviewBot added a comment.Oct 21 2025, 3:41 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/97

tools: Point web traffic to HAProxy directly

CodeReviewBot added a comment.Oct 22 2025, 9:25 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/96

toolsbeta: Point web traffic to HAProxy directly

Stashbot added a comment.Oct 22 2025, 12:35 PM

Mentioned in SAL (#wikimedia-cloud) [2025-10-22T12:35:40Z] <taavi> moving toolforge traffic to haproxy directly T283948

CodeReviewBot added a comment.Oct 22 2025, 12:36 PM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/97

tools: Point web traffic to HAProxy directly

gerritbot added a comment.Oct 22 2025, 12:55 PM

Change #1198049 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::k8s::haproxy: Drop proxy IP rate limit exemption

https://gerrit.wikimedia.org/r/1198049

gerritbot added a comment.Oct 22 2025, 12:55 PM

Change #1198050 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: Remove separate proxy role

https://gerrit.wikimedia.org/r/1198050

CodeReviewBot added a comment.Oct 22 2025, 12:57 PM

taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/98

Remove old front proxy instances

gerritbot added a comment.Oct 22 2025, 2:24 PM

Change #1198049 merged by Majavah:

[operations/puppet@production] P:toolforge::k8s::haproxy: Drop proxy IP rate limit exemption

https://gerrit.wikimedia.org/r/1198049

gerritbot added a comment.Oct 22 2025, 4:05 PM

Change #1198105 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge::prometheus: Drop separate front proxy scrape target

https://gerrit.wikimedia.org/r/1198105

taavi closed subtask T284558: Pull toolviews data from Kubernetes HAProxy or ingress-nginx instead of the front nginx as Resolved.Oct 22 2025, 4:14 PM
gerritbot added a comment.Oct 22 2025, 4:17 PM

Change #1198105 merged by Majavah:

[operations/puppet@production] P:toolforge::prometheus: Drop separate front proxy scrape target

https://gerrit.wikimedia.org/r/1198105

Stashbot added a comment.Oct 23 2025, 10:11 AM

Mentioned in SAL (#wikimedia-cloud) [2025-10-23T10:11:11Z] <taavi> deleting old nginx front proxy instances T283948

CodeReviewBot added a comment.Oct 23 2025, 10:11 AM

taavi merged https://gitlab.wikimedia.org/repos/cloud/toolforge/tofu-provisioning/-/merge_requests/98

Remove old front proxy instances

gerritbot added a comment.Oct 23 2025, 2:50 PM

Change #1198050 merged by Majavah:

[operations/puppet@production] P:toolforge: Remove separate proxy role

https://gerrit.wikimedia.org/r/1198050

taavi closed this task as Resolved.Oct 23 2025, 2:52 PM
gerritbot added a comment.Oct 23 2025, 3:04 PM

Change #1198349 had a related patch set uploaded (by Majavah; author: Majavah):

[operations/puppet@production] P:toolforge: Remove obsolete spec test

https://gerrit.wikimedia.org/r/1198349

gerritbot added a comment.Oct 23 2025, 3:05 PM

Change #1198349 merged by Majavah:

[operations/puppet@production] P:toolforge: Remove obsolete spec test

https://gerrit.wikimedia.org/r/1198349

Maintenance_bot removed a project: Patch-For-Review.Oct 23 2025, 3:30 PM
taavi mentioned this in T407048: Can't log in to Luthor.Oct 23 2025, 8:43 PM
LucasWerkmeister mentioned this in T409008: Toolforge fourohfour tool (404 error) shows URL twice.Nov 2 2025, 6:36 PM
bd808 added a subtask: T409008: Toolforge fourohfour tool (404 error) shows URL twice.Nov 3 2025, 7:01 PM
taavi closed subtask T409008: Toolforge fourohfour tool (404 error) shows URL twice as Resolved.Nov 4 2025, 9:23 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits