Page MenuHomePhabricator

Remove katran blockers for low-traffic non-k8s based services
Closed, ResolvedPublic

Description

Right now we have the following blockers that would preclude from migrating low-traffic services to Katran:

  • Direct routing mode being used to forward traffic to realservers
  • Weighted round robin (wrr) scheduler

IPIP encapsulation can be adopted by non-k8s based services behind the low-traffic load balancer in the same way as it was done for high-traffic2 services on T367312.
Moving away from wrr is feasible using the mh-port flag as mentioned on T368545, a small patch is required on pybal to support the mh-port flag to use -b mh-port flag on ipvsadm.

This is the list of services that could benefit from this migration (k8s based services are excluded from this migration):

taskservicecontact
T387290apusData-Persistence
T385564swiftData-Persistence
T385564swift-httpsData-Persistence
T387293thanos-swiftData-Persistence
T387294docker-registryServiceOps new
T387295jobrunnerServiceOps new
T387296videoscalerServiceOps new
T387297kartotherianCore Platform / ServiceOps new
T387298kartotherian-sslCore Platform / ServiceOps new
T387299restbase-backendCore Platform / ServiceOps new
T387300restbase-httpsCore Platform / ServiceOps new
T387291thanos-queryobservability
T387292thanos-webobservability
T387301kibana7observability
T387302prometheusobservability
T387303prometheus-httpsobservability
T387304logs-apiobservability
T387305labweb-sslCloud-Services
T387306datahubsearchData-Platform-SRE
T387307druid-public-brokerData-Platform-SRE
T387308schemaData-Platform-SRE
T387309searchSearch Platform / Data-Platform-SRE
T387310search-httpsSearch Platform / Data-Platform-SRE
T387311search-omega-httpsSearch Platform / Data-Platform-SRE
T387312search-psi-httpsSearch Platform / Data-Platform-SRE
T387313wcqsSearch Platform / Data-Platform-SRE
T387314wdqsWikidata-Query-Service
T387315wdqs-mainWikidata-Query-Service
T387316wdqs-scholarlyWikidata-Query-Service
T387317wdqs-heavy-queriesWikidata-Query-Service
T387318wdqs-internalWikidata-Query-Service
T387319wdqs-internal-mainWikidata-Query-Service
T387320wdqs-internal-scholarlyWikidata-Query-Service
T387321wdqs-sslWikidata-Query-Service

This has been automatically generated using the following prompt:

sybil:puppet vgutierrez$ yq '.catalog
    | to_entries
    | map(select(.value.lvs.class != "high-traffic1" and .value.lvs.class != "high-traffic2"))
    | map(select(.value.lvs.conftool.cluster != "kubernetes"
                 and .value.lvs.conftool.cluster != "k8s"
                 and .value.lvs.conftool.cluster != "aux-k8s"
                 and .value.lvs.conftool.cluster != "dse-k8s"
                 and .value.lvs.conftool.cluster != "kubernetes-staging"))
    | map(select(.value.lvs.conftool.service != "kubesvc"
                 and .value.lvs.conftool.service != "kubemaster"))
    | map(select(.value.lvs.conftool.cluster != null))
    | map(.key)' hieradata/common/service.yaml | sed s/-/[]/

contact information obtained (partially) with this oneliner:

vgutierrez@carrot:~/wikimedia.org/operations/puppet/hieradata$ yq '.catalog
    | to_entries
    | map(select(.value.lvs.class != "high-traffic1" and .value.lvs.class != "high-traffic2"))
    | map(select(.value.lvs.conftool.cluster != "kubernetes"
                 and .value.lvs.conftool.cluster != "k8s"
                 and .value.lvs.conftool.cluster != "aux-k8s"
                 and .value.lvs.conftool.cluster != "dse-k8s"
                 and .value.lvs.conftool.cluster != "kubernetes-staging"))
    | map(select(.value.lvs.conftool.service != "kubesvc"
                 and .value.lvs.conftool.service != "kubemaster"))
    | map(select(.value.lvs.conftool.cluster != null))
    | map(.key)' common/service.yaml |egrep -v "^\[$|^\]$" | sed s/\"//g | sed s/,// | sed "s/  //" | xargs -I % sh -c 'git grep role_contacts | grep % || echo "contacts for % not found"'

Related Objects

StatusSubtypeAssignedTask
OpenVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
InvalidNone
InvalidNone
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
ResolvedVgutierrez
Resolvedbking
Resolvedbking
Resolvedbking
Resolvedbking
Resolvedbking
ResolvedVgutierrez

Event Timeline

Vgutierrez moved this task from Backlog to Scheduled incidental work on the Traffic board.

Change #1114015 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] site,swift: Use profile::lvs::realserver

https://gerrit.wikimedia.org/r/1114015

Change #1114015 merged by Vgutierrez:

[operations/puppet@production] site,swift: Use profile::lvs::realserver

https://gerrit.wikimedia.org/r/1114015

Change #1122097 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/puppet@production] hiera: Add scapy to cookbooks dependencies

https://gerrit.wikimedia.org/r/1122097

Change #1122097 merged by Vgutierrez:

[operations/puppet@production] hiera: Add scapy to cookbooks dependencies

https://gerrit.wikimedia.org/r/1122097

Change #1122152 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/cookbooks@master] sre.loadbalancer: Add migrate-service-ipip cookbook

https://gerrit.wikimedia.org/r/1122152

Change #1122194 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/software/spicerack@master] add scapy to setup.py dependencies

https://gerrit.wikimedia.org/r/1122194

Change #1122194 abandoned by Vgutierrez:

[operations/software/spicerack@master] add scapy to setup.py dependencies

https://gerrit.wikimedia.org/r/1122194

Change #1122152 merged by jenkins-bot:

[operations/cookbooks@master] sre.loadbalancer: Add migrate-service-ipip cookbook

https://gerrit.wikimedia.org/r/1122152

Change #1123611 had a related patch set uploaded (by Vgutierrez; author: Vgutierrez):

[operations/cookbooks@master] migrate-service-ipip: Allow picking realservers using a cumin alias

https://gerrit.wikimedia.org/r/1123611

Change #1123611 merged by jenkins-bot:

[operations/cookbooks@master] migrate-service-ipip: Allow picking realservers using a cumin alias

https://gerrit.wikimedia.org/r/1123611

bking changed the status of subtask T387312: Migrate search-psi-https LB VIPs to IPIP encapsulation from Stalled to In Progress.
bking changed the status of subtask T387309: Migrate search LB VIPs to IPIP encapsulation from Stalled to In Progress.
bking changed the status of subtask T387311: Migrate search-omega-https LB VIPs to IPIP encapsulation from Stalled to In Progress.