Page MenuHomePhabricator
Create Task
Maniphest T331318

Find a sensible way to direct traffic to mw-on-k8s
Closed, ResolvedPublic
Actions

Description

We currently are copy/pasting blocks of configuration in the trafficserver configuration to direct test2wiki https://gerrit.wikimedia.org/r/c/862845 and testwikidata T331268: Migrate testwikidata to Kubernetes to wikikube.

We'd like to have a solution that is a bit more DRY and flexible, such as inserting a header at the HAProxy layer and use it at the second layer to forward the different URLs to their respective LVS endpoints (mw-web, mw-api-ext).

Seeking opinion from Traffic on how to best handle this, or if another better solution exists.

Details

SubjectRepoBranchLines +/-
trafficserver: make routing to mw on k8s more manageableoperations/puppetproduction+115 -61
Customize query in gerrit

Related Objects
Search...

Event Timeline

Clement_Goubert created this task.Mar 6 2023, 5:34 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 6 2023, 5:34 PM
Clement_Goubert triaged this task as Medium priority.Mar 6 2023, 5:34 PM
Clement_Goubert added a parent task: T290536: Serve production traffic via Kubernetes.
Clement_Goubert updated the task description. (Show Details)
Clement_Goubert moved this task from Incoming 🐫 to this.quarter 🍕 on the serviceops board.Mar 6 2023, 5:37 PM
Maintenance_bot added a project: SRE.Mar 6 2023, 5:45 PM
Clement_Goubert renamed this task from Insert a header for specific domains at the first ATS layer to redirect traffic to mw-on-k8s to Insert a header for specific domains at haproxy layer to redirect traffic to mw-on-k8s.Mar 8 2023, 12:10 PM
Clement_Goubert updated the task description. (Show Details)
Vgutierrez added a comment.Mar 8 2023, 12:31 PM

We traditionally perform that kind of header mangling in varnish rather than on the TLS termination layer as we try to keep the latter cluster agnostic.

But I'm not totally sold on the idea of adding an HTTP header for this. ATS should be able to handle this on its own leveraging regex_map rules. We should come with a sane way of producing those regex rather than hardcoding them on backend.yaml but I think this should be solved in ATS realm

Clement_Goubert renamed this task from Insert a header for specific domains at haproxy layer to redirect traffic to mw-on-k8s to Find a sensible way to redirect traffic to mw-on-k8s.Mar 8 2023, 12:40 PM

Changed the task title to reflect the direction of the discussion.

Joe added a comment.Mar 16 2023, 9:55 AM

After some thought, I think the most maintainable way to do this is to add an additional lua module to the maps for api/appservers.

Specifically, this would allow us to:

  • Define a list of domains we want to serve from k8s (what we want to do now)
  • select the backend (k8s or baremetal) based on traffic percentages or cookies (what we probably want to do in the future)
  • Separate traffic between our k8s clusters in aa more fine-grained way. I can see us in the future wanting to separate human-like traffic from pure-bot traffic.
gerritbot added a comment.Mar 17 2023, 5:49 PM

Change 900704 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/puppet@production] trafficserver: make routing to mw on k8s more manageable

https://gerrit.wikimedia.org/r/900704

gerritbot added a project: Patch-For-Review.Mar 17 2023, 5:49 PM
Krinkle renamed this task from Find a sensible way to redirect traffic to mw-on-k8s to Find a sensible way to direct traffic to mw-on-k8s.Mar 28 2023, 2:24 AM
Krinkle updated the task description. (Show Details)
Clement_Goubert moved this task from Backlog to In Progress on the MW-on-K8s board.Mar 28 2023, 9:24 AM
Clement_Goubert moved this task from this.quarter 🍕 to Doing 😎 on the serviceops board.
Stashbot added a comment.Mar 30 2023, 9:09 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-30T09:09:00Z] <claime> Merging mw-on-k8s ATS lua routing script - T331318

gerritbot added a comment.Mar 30 2023, 9:09 AM

Change 900704 merged by Clément Goubert:

[operations/puppet@production] trafficserver: make routing to mw on k8s more manageable

https://gerrit.wikimedia.org/r/900704

Maintenance_bot removed a project: Patch-For-Review.Mar 30 2023, 9:10 AM
Stashbot added a comment.Mar 30 2023, 9:12 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-30T09:12:38Z] <claime> puppet disabled for A:cp-text - T331318

Stashbot added a comment.Mar 30 2023, 9:15 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-30T09:15:55Z] <claime> puppet disabled for A:cp-upload - T331318

Stashbot added a comment.Mar 30 2023, 9:16 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-30T09:16:48Z] <claime> Running puppet on cp2028.codfw.wmnet (cp-upload noop test) - T331318

Stashbot added a comment.Mar 30 2023, 9:23 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-30T09:23:53Z] <claime> Re-enabling puppet for A:cp-upload - T331318

Stashbot added a comment.Mar 30 2023, 9:35 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-30T09:35:04Z] <claime> Re-enabling puppet for cp4037 - T331318

Stashbot added a comment.Mar 30 2023, 9:44 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-30T09:44:24Z] <claime> Re-enabling puppet for cp-text_ulsfo - T331318

Stashbot added a comment.Mar 30 2023, 11:03 AM

Mentioned in SAL (#wikimedia-operations) [2023-03-30T11:03:52Z] <claime> Re-enabling puppet for cp-text - T331318

Clement_Goubert closed this task as Resolved.Apr 3 2023, 9:20 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL