Create a cookbook for depooling one or all services from one kubernetes cluster
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Joe
	Aug 18 2020, 8:51 AM

Description

We want to have a cookbook we can use to completely depool all services running on a kubernetes cluster from the discovery system, or to depool just one.

A cli I could imagine would be:

# Depool a cluster from traffic for one or more services
# This will need to check that the service is available in some other cluster before acting
$ cookbook sre.k8s.depool-service <k8s-cluster> [SVC1,SVC2,...]
# Pool a cluster for one or more services
$ cookbook sre.k8s.pool-service <k8s-cluster> [SVC1,...]
# Check status of traffic to services
$ cookbook.sre.k8s.check-service-route [SVC1,...]

Details

Subject	Repo	Branch	Lines +/-
sre.k8s.pool-depool-cluster: Add new cookbook	operations/cookbooks	master	+125 -0
sre.discovery.service-route: Make the cookbook work	operations/cookbooks	master	+21 -16
sre.discovery: Refactor	operations/cookbooks	master	+266 -125

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	JMeybohm	T307943 Update Kubernetes clusters to v1.23
Open	None	T341984 Update Kubernetes clusters to >1.25
Open	elukey	T277677 Write a cookbook to set a k8s cluster in maintenance mode
Open	None	T203943 Spicerack cookbooks TODO list
Resolved	JMeybohm	T260663 Create a cookbook for depooling one or all services from one kubernetes cluster

Event Timeline

Joe created this task.Aug 18 2020, 8:51 AM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 18 2020, 8:51 AM

fgiunchedi triaged this task as Medium priority.Aug 19 2020, 9:05 AM

JMeybohm mentioned this in T258572: Refactor our helmfile.d dir structure for services.Aug 20 2020, 9:45 AM

The pool/depool logic is quite the same as in sre.discovery.pool/depool I guess.

For checking the service availability on some other cluster we could probably query the LVS monitoring commands from icinga or can we just rely on the fact that the other DC/cluster is pooled in confctl?

What do you expect sre.k8s.check-service-route output to be? Something like:

codfw: true
eqiad: false

In T260663#6399630, @JMeybohm wrote:
The pool/depool logic is quite the same as in sre.discovery.pool/depool I guess.

For checking the service availability on some other cluster we could probably query the LVS monitoring commands from icinga or can we just rely on the fact that the other DC/cluster is pooled in confctl?

What do you expect sre.k8s.check-service-route output to be? Something like:
codfw: true
eqiad: false

What we should actually do is something similar to what switchdc does when it changes discovery records.

Specifically,

check-route should read an expected state from etcd, then query the authdns servers and report discrepancies.
pool/depool should change the value in etcd, verify it's changed on the authdns servers, then purge the resolver caches (optionally)

btw, given some of the functionality is common, I would say we should revisit the CLI I proposed above:

cookbook sre.k8s.service-route ACTION [ACTION_ARGS]

Available actions:

pool all|eqiad|codfw SVC1,... pool the listed services in all datacenters, or in a single one
check [SVC1,...] returns the current routes for the listed services

example of output for check:

$ cookbook sre.k8s.service-route check mobileapps mathoid
Expected routes:
mathoid: all
mobileapps: eqiad

In case of problems (i.e. what is in etcd is not in all the authdns servers), an error should be reported.

Claiming as I would like us to have this for helmfile migration.

Change 621721 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/cookbooks@master] sre.k8s.service-route: New cookbook to pool/depool k8s servies

https://gerrit.wikimedia.org/r/621721

gerritbot added a project: Patch-For-Review.Aug 21 2020, 2:31 PM

As this is not k8s specific I decided to refactor sre.discovery instead of generating a new cookbook.
We can create an alias (via symlink) if we want to have that as sre.k8s.service-route as well.

The EDNS client subnet aware DNS functions are currently part of the cookbook but should be refactored into spicerack dns and dnsdisc after the DC switchover.

Change 621721 merged by jenkins-bot:
[operations/cookbooks@master] sre.discovery: Refactor

https://gerrit.wikimedia.org/r/621721

Merged the current version as is but the cookbook should be updated in short term with:

[03.09.20 14:05] <_joe_> in the meantime I cooked up a general solution to the dns name -> discovery record correspondence for another cookbook
[03.09.20 14:05] <_joe_> we can just adopt it here in a followup patch
[03.09.20 14:05] <jayme> sweet. Is it already in spicerack then?
[03.09.20 14:09] <_joe_> yes
[03.09.20 14:09] <_joe_> sre.switchdc.services

Maintenance_bot removed a project: Patch-For-Review.Sep 7 2020, 9:10 AM

JMeybohm moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.Sep 11 2020, 12:28 PM

JMeybohm added a parent task: T203943: Spicerack cookbooks TODO list.Oct 5 2020, 2:43 PM

elukey subscribed.Jan 13 2021, 9:58 AM

JMeybohm added a project: Prod-Kubernetes.Mar 17 2021, 4:55 PM

JMeybohm mentioned this in T277677: Write a cookbook to set a k8s cluster in maintenance mode.Mar 17 2021, 4:58 PM

The cookbook does not seem to work (tried during the kubernetes codfw reinit):

It did not allow multiple services as arguments
It did not actually depool services from a DC

JMeybohm mentioned this in T277191: Update Kubernetes cluster codfw to kubernetes 1.16.Mar 18 2021, 9:51 AM

Aklapper added a project: Infrastructure-Foundations.Jun 21 2021, 8:59 PM

Jelto added a project: Sustainability (Incident Followup).Nov 30 2021, 9:01 AM

Joe mentioned this in T300879: Add a kubernetes module to spicerack.Feb 3 2022, 4:04 PM

akosiaris added a parent task: T277677: Write a cookbook to set a k8s cluster in maintenance mode.May 4 2022, 2:28 PM

akosiaris subscribed.

JMeybohm mentioned this in T307943: Update Kubernetes clusters to v1.23.May 9 2022, 5:10 PM

joanna_borun added a project: Spicerack.Jun 15 2022, 10:49 AM

Change 823659 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/cookbooks@master] sre.discovery.service-route: Make the cookbook work

https://gerrit.wikimedia.org/r/823659

gerritbot added a project: Patch-For-Review.Aug 16 2022, 3:13 PM

Change 823659 merged by jenkins-bot:

[operations/cookbooks@master] sre.discovery.service-route: Make the cookbook work

https://gerrit.wikimedia.org/r/823659

Maintenance_bot removed a project: Patch-For-Review.Aug 17 2022, 1:33 PM

In T260663#6924279, @JMeybohm wrote:

The cookbook does not seem to work (tried during the kubernetes codfw reinit):

It did not allow multiple services as arguments

It did not actually depool services from a DC

These issues have now been fixed.
Next would be to add a k8s specific cookbook figuring out the services running on k8s and taking action on them https://doc.wikimedia.org/spicerack/master/api/spicerack.service.html?highlight=services%20yaml#spicerack.service.Catalog

Change 824485 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/cookbooks@master] sre.k8s.pool-depool-cluster: Add new cookbook

https://gerrit.wikimedia.org/r/824485

gerritbot added a project: Patch-For-Review.Aug 18 2022, 12:40 PM

Change 824485 merged by jenkins-bot:

[operations/cookbooks@master] sre.k8s.pool-depool-cluster: Add new cookbook

https://gerrit.wikimedia.org/r/824485

Maintenance_bot removed a project: Patch-For-Review.Sep 13 2022, 8:30 AM

Merged as sre.k8s.pool-depool-cluster

JMeybohm mentioned this in rCCKB9c9b3ba706d2: sre.discovery: Refactor.Dec 14 2022, 3:26 PM

JMeybohm mentioned this in rCCKBadc09e622193: sre.discovery.service-route: Make the cookbook work.Dec 14 2022, 3:31 PM