Page MenuHomePhabricator
Create Task
Maniphest T332027

Replace current L4LB with with Katran-based alternative
Open, MediumPublic
Actions

Description

This is the tracking task for all things L4LB: the WMF implementation of a L4LB based on katran named liberica

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Enable BGP Multipath for PyBal groupoperations/homer/publicmaster+1 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenVgutierrezT332027 Replace current L4LB with with Katran-based alternative
 ResolvedRobHT341992 Relocate lvs1013-lvs1016 to rows E & F
 ResolvedVgutierrezT342618 Perform katran load tests on lvs1013
 ResolvedVgutierrezT348837 Investigate IPVS IPIP encapsulation support
 ResolvedVgutierrezT351069 Enable IPIP encapsulation for ncredir
 ResolvedVgutierrezT352143 Firewall rules prevent IPIP/IP6IP6 encapsulated traffic from reaching realservers
 ResolvedVgutierrezT352160 RP filtering drops requests incoming via IPIP tunnels on ncredir realservers
 ResolvedVgutierrezT350462 Provide a TCP MSS clamping mechanism for real servers
 OpenNoneT350463 Provide an etcd schema to store real servers information
 ResolvedVgutierrezT353279 CI on gitlab for eBPF / networking heavy projects
 ResolvedVgutierrezT357257 Use IPIP encapsulation on lvs<-->upload cluster
 ResolvedVgutierrezT357258 Release tcp-mss-clamper for bullseye
 ResolvedVgutierrezT365101 MSS clamper check triggers false positives
 ResolvedVgutierrezT365354 rp_filter should be disabled on puppet apply
 ResolvedVgutierrezT365689 Provide a ferm based alternative to tcp-mss-clamper
 ResolvedCDobbinsT367204 LVSRealserverMSS alert is broken for ferm based hosts
 ResolvedVgutierrezT366466 Use IPIP encapsulation on lvs<-->text cluster
 ResolvedVgutierrezT367312 Migrate services behind high-traffic2 LVS to IPIP encapsulation
 ResolvedbkingT367511 Migrate Cloudelastic load balancing to IPIP encapsulation (LVS)
 ResolvedNoneT367861 Migrate ldap-ro and ldap-ro-ssl to IPIP encapsulation
 ResolvedBCornwallT368083 migrate all high-traffic1 and high-traffic2 services to maglev
 OpenVgutierrezT368544 IPIP encapsulation considerations for low-traffic services
 ResolvedVgutierrezT368545 weighted maglev viability for low-traffic services
 ResolvedVgutierrezT372731 liberica triggers one DNS query per HTTPCheck execution/realserver
 ResolvedVgutierrezT373020 Remove katran blockers for low-traffic non-k8s based services
 ResolvedVgutierrezT373027 Add support for mh-port scheduler flag on pybal
 ResolvedVgutierrezT385564 migrate swift/swift-https LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387290 Migrate apus LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387291 Migrate thanos-query LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387292 Migrate thanos-web LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387293 Migrate thanos-swift LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387294 Migrate docker-registry LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387295 Migrate jobrunner LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387296 Migrate videoscaler LB VIPs to IPIP encapsulation
 InvalidNoneT387297 Migrate kartotherian LB VIPs to IPIP encapsulation
 InvalidNoneT387298 Migrate kartotherian-ssl LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387299 Migrate restbase-backend LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387300 Migrate restbase-https LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387301 Migrate kibana7 LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387302 Migrate prometheus LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387303 Migrate prometheus-https LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387304 Migrate logs-api LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387305 Migrate labweb-ssl LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387306 Migrate datahubsearch LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387307 Migrate druid-public-broker LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387308 Migrate schema LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387313 Migrate wcqs LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387314 Migrate wdqs LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387315 Migrate wdqs-main LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387316 Migrate wdqs-scholarly LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387317 Migrate wdqs-heavy-queries LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387318 Migrate wdqs-internal LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387319 Migrate wdqs-internal-main LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387320 Migrate wdqs-internal-scholarly LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387321 Migrate wdqs-ssl LB VIPs to IPIP encapsulation
 ResolvedbkingT394062 Migrate CirrusSearch LVS pools to IPIP
 ResolvedbkingT387312 Migrate search-psi-https LB VIPs to IPIP encapsulation
 ResolvedbkingT387309 Migrate search LB VIPs to IPIP encapsulation
 ResolvedbkingT387569 Update Elastic puppet code to filter LVS config based on cluster membership
 ResolvedbkingT387311 Migrate search-omega-https LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT387310 Migrate search-https LB VIPs to IPIP encapsulation
 ResolvedVgutierrezT375464 Test liberica BGP support
 ResolvedVgutierrezT376600 Provide debian packages for liberica
 InvalidVgutierrezT376696 Sync liberica etcd library requirements with versions provided on debian bookworm
 ResolvedVgutierrezT377127 liberica puppetization
 ResolvedVgutierrezT378453 Testing liberica with ncredir@eqiad
 ResolvedVgutierrezT378341 harden liberica systemd service units
 ResolvedVgutierrezT379164 BGP settings for liberica
 ResolvedVgutierrezT379238 liberica should offer prometheus endpoints on both IPv4 and IPv6
 ResolvedVgutierrezT380450 bring katran to liberica
 ResolvedVgutierrezT393903 control plane should fetch default gateway MAC address dynamically
 ResolvedVgutierrezT384477 Replace pybal with liberica on the PoPs
 ResolvedVgutierrezT385001 missing ipip-multiqueue-optimizer prometheus metrics for liberica cluster
 ResolvedVgutierrezT386687 backport gobgp 3.33 from trixie
 ResolvedVgutierrezT387880 liberica fails to refresh liberica_cp_unhealthy_pooled_realservers_total metric
 ResolvedVgutierrezT388369 Provide cookbook(s) to operate liberica
 OpenJMeybohmT352956 Handling inbound IPIP traffic on low traffic LVS k8s based realservers
 ResolvedVgutierrezT395228 Test katran forwarding plane on lvs1013
 ResolvedVgutierrezT396227 unable to fetch forwarding plane metrics when using katran
 OpenNoneT367973 Replace ping offload servers with eBPF
 ResolvedVgutierrezT396561 Switch to katran as forwarding plane on non-core DCs
 ResolvedVgutierrezT396751 liberica forwarding plane fails to start on systems with 48 CPUs using katran
 ResolvedVgutierrezT397036 Provide connections stats for katran
 ResolvedVgutierrezT397053 seamless upgrade triggers dropped packets with katran
 ResolvedVgutierrezT400259 Stop using lvs1013 as a liberica canary

Event Timeline

Vgutierrez created this task.Mar 14 2023, 3:21 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 14 2023, 3:21 PM
CDanis subscribed.Mar 14 2023, 3:23 PM
Volans subscribed.Mar 14 2023, 3:28 PM
Maintenance_bot added a project: SRE.Mar 14 2023, 3:29 PM
ayounsi subscribed.Mar 14 2023, 3:31 PM
Vgutierrez moved this task from Backlog to Actively Servicing on the Traffic board.Mar 14 2023, 3:39 PM
MoritzMuehlenhoff subscribed.Mar 15 2023, 12:46 PM
BCornwall subscribed.Mar 15 2023, 6:07 PM
BCornwall mentioned this in T321309: Upgrade Traffic hosts to bullseye.Mar 15 2023, 6:10 PM
ssingh subscribed.Mar 15 2023, 6:40 PM
Ladsgroup subscribed.Mar 19 2023, 1:25 PM
taavi subscribed.Apr 5 2023, 2:55 PM
ssingh mentioned this in T334259: Upgrade lvs1013-1016 firmware.Apr 12 2023, 3:18 PM
KOfori subscribed.Apr 18 2023, 8:57 PM
BCornwall renamed this task from L4LB tracking task to Replace current L4LB with with Katran-based alternative.Apr 25 2023, 8:53 PM
BCornwall changed the task status from Open to In Progress.
BCornwall assigned this task to Vgutierrez.
BCornwall triaged this task as Medium priority.
BCornwall removed a project: SRE.
Fabfur subscribed.May 17 2023, 10:04 PM
Vgutierrez closed subtask T341992: Relocate lvs1013-lvs1016 to rows E & F as Resolved.Jul 31 2023, 1:08 PM
Vgutierrez mentioned this in T245059: depool / confctl commands should print warnings or errors if too many nodes from that service are already depooled.Oct 2 2023, 1:31 AM
cmooney mentioned this in T346428: Netbox: Add support for our complex host network setups in provision script.Oct 4 2023, 12:19 PM
Vgutierrez added a subtask: T348837: Investigate IPVS IPIP encapsulation support.Oct 13 2023, 8:15 AM
Jgreen subscribed.Nov 16 2023, 7:23 PM
Jgreen mentioned this in T344290: Fundraising environment support for Debian 12 (Bookworm).Nov 16 2023, 7:32 PM
Vgutierrez closed subtask T348837: Investigate IPVS IPIP encapsulation support as Resolved.Dec 7 2023, 8:35 AM
Vgutierrez changed the status of subtask T357257: Use IPIP encapsulation on lvs<-->upload cluster from Open to In Progress.May 10 2024, 10:04 AM
Vgutierrez closed subtask T350462: Provide a TCP MSS clamping mechanism for real servers as Resolved.May 23 2024, 9:38 AM
Vgutierrez closed subtask T357257: Use IPIP encapsulation on lvs<-->upload cluster as Resolved.May 23 2024, 1:00 PM
Vgutierrez closed subtask T365689: Provide a ferm based alternative to tcp-mss-clamper as Resolved.Jun 11 2024, 3:35 PM
Vgutierrez closed subtask T366466: Use IPIP encapsulation on lvs<-->text cluster as Resolved.Jun 12 2024, 2:03 PM
Vgutierrez changed the status of subtask T367312: Migrate services behind high-traffic2 LVS to IPIP encapsulation from Open to In Progress.Jun 20 2024, 4:16 PM
Vgutierrez closed subtask T367312: Migrate services behind high-traffic2 LVS to IPIP encapsulation as Resolved.Jun 25 2024, 1:35 PM
Vgutierrez closed subtask T342618: Perform katran load tests on lvs1013 as Resolved.Jun 27 2024, 10:09 AM
Vgutierrez closed subtask T353279: CI on gitlab for eBPF / networking heavy projects as Resolved.
BCornwall closed subtask T368083: migrate all high-traffic1 and high-traffic2 services to maglev as Resolved.Jul 10 2024, 4:09 PM
Vgutierrez closed subtask T368545: weighted maglev viability for low-traffic services as Resolved.Aug 21 2024, 12:20 PM
Vgutierrez closed subtask T372731: liberica triggers one DNS query per HTTPCheck execution/realserver as Resolved.Aug 22 2024, 7:48 AM
BCornwall moved this task from Actively Servicing to Scheduled work for this quarter on the Traffic board.Sep 9 2024, 8:57 PM
BCornwall moved this task from Scheduled work for this quarter to Actively Servicing on the Traffic board.Sep 9 2024, 10:28 PM
Vgutierrez closed subtask T377127: liberica puppetization as Resolved.Nov 6 2024, 2:41 PM
Vgutierrez changed the status of subtask T378341: harden liberica systemd service units from Open to In Progress.Nov 7 2024, 9:30 AM
Vgutierrez closed subtask T375464: Test liberica BGP support as Resolved.Nov 21 2024, 9:38 AM
Vgutierrez closed subtask T376600: Provide debian packages for liberica as Resolved.
Vgutierrez closed subtask T378341: harden liberica systemd service units as Resolved.
Aklapper mentioned this in PyBal.Dec 4 2024, 4:36 PM
Aklapper mentioned this in T200319: Migrate Pybal to Python 3.
Vgutierrez closed subtask T380450: bring katran to liberica as Resolved.Jan 24 2025, 10:12 AM
Vgutierrez changed the status of subtask T384477: Replace pybal with liberica on the PoPs from Open to In Progress.Jan 28 2025, 5:55 PM
Vgutierrez closed subtask T379164: BGP settings for liberica as Resolved.Feb 18 2025, 8:39 AM
Vgutierrez closed subtask T386687: backport gobgp 3.33 from trixie as Resolved.Feb 26 2025, 3:36 PM
gerritbot added a comment.Mar 7 2025, 4:28 PM

Change #1125471 had a related patch set uploaded (by Cathal Mooney; author: Cathal Mooney):

[operations/homer/public@master] Enable BGP Multipath for PyBal group

https://gerrit.wikimedia.org/r/1125471

gerritbot added a project: Patch-For-Review.Mar 7 2025, 4:28 PM
Vgutierrez changed the status of subtask T388369: Provide cookbook(s) to operate liberica from Open to In Progress.Mar 12 2025, 4:05 PM
Vgutierrez added a project: Liberica.Mar 14 2025, 4:09 PM
Vgutierrez closed subtask T384477: Replace pybal with liberica on the PoPs as Resolved.Mar 17 2025, 4:09 PM
Vgutierrez closed subtask T388369: Provide cookbook(s) to operate liberica as Resolved.Mar 20 2025, 10:32 AM
Vgutierrez closed subtask T379238: liberica should offer prometheus endpoints on both IPv4 and IPv6 as Resolved.Apr 10 2025, 9:17 AM
Vgutierrez closed subtask T387880: liberica fails to refresh liberica_cp_unhealthy_pooled_realservers_total metric as Resolved.Apr 11 2025, 4:04 PM
Aklapper changed the task status from In Progress to Open.Apr 11 2025, 10:15 PM

Resetting task status from "In Progress" to "Open" as this task has been "in progress" for more than one and a half years (see T380300).

cmooney removed a subtask: T352956: Handling inbound IPIP traffic on low traffic LVS k8s based realservers.May 14 2025, 10:30 AM
cmooney added a subtask: T394286: Avoid using codfw expansion cage for non-IPIP LVS-fronted services.
cmooney added a subtask: T352956: Handling inbound IPIP traffic on low traffic LVS k8s based realservers.May 14 2025, 10:41 AM
cmooney removed a subtask: T394286: Avoid using codfw expansion cage for non-IPIP LVS-fronted services.May 14 2025, 11:04 AM
Vgutierrez added a subtask: T367973: Replace ping offload servers with eBPF.May 26 2025, 8:40 AM
Vgutierrez closed subtask T395228: Test katran forwarding plane on lvs1013 as Resolved.Jun 11 2025, 8:02 AM
Vgutierrez closed subtask T396561: Switch to katran as forwarding plane on non-core DCs as Resolved.Jul 7 2025, 4:18 PM
Vgutierrez closed subtask T373020: Remove katran blockers for low-traffic non-k8s based services as Resolved.Jul 9 2025, 3:39 PM
Vgutierrez changed the status of subtask T400259: Stop using lvs1013 as a liberica canary from Open to In Progress.Jul 23 2025, 1:27 PM
Vgutierrez closed subtask T400259: Stop using lvs1013 as a liberica canary as Resolved.Jul 24 2025, 9:58 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits