Change 862222 had a related patch set uploaded (by Btullis; author: Btullis):

[operations/deployment-charts@master] Remove chartid from matchlabels for eventstreams

https://gerrit.wikimedia.org/r/862222

Adding various Data Engineering planning and streaming tags for visibility.
Expediting into current sprint due to its being an unbreak now.

Change 862222 merged by jenkins-bot:

[operations/deployment-charts@master] Remove chartid from matchlabels for eventstreams

https://gerrit.wikimedia.org/r/862222

I have received guidance from @Clement_Goubert on the steps required to depool/destroy/deploy/repool eventstreams in each data centre: P41870

In codfw and eqiad we have canary releases though, so I can apply the change with additional steps based on adding --selector name=canary and --selector name=production to the helmfile commands.

I have merged the linked CR so I will begin this process now.

Mentioned in SAL (#wikimedia-analytics) [2022-11-30T11:11:04Z] <btullis> depooling codfw for eventstreams for T324074

New eventstreams clients in codfw have virtually stopped. Existing clients are draining slowly. I will wait until they have all gone.

eventstreams in codfw has been completely drained.

Mentioned in SAL (#wikimedia-analytics) [2022-11-30T11:32:14Z] <btullis> destroying the eventstreams deployment in codfw and reapplying for T324074

Mentioned in SAL (#wikimedia-analytics) [2022-11-30T11:34:55Z] <btullis> repooling codfw for eventstreams for T324074

eventreams in codfw is now handling traffic again nicely.

Mentioned in SAL (#wikimedia-analytics) [2022-11-30T11:59:20Z] <btullis> depooling eqiad for eventstreams for T324074

Depooled eqiad using the cookbook method, which was new to me.

btullis@cumin1001:~$ sudo cookbook sre.discovery.service-route depool eqiad eventstreams
START - Cookbook sre.discovery.service-route
TTL already set to 300, nothing to do
Setting pooled=False for tags: {'dnsdisc': '(eventstreams)', 'name': 'eqiad'}
Waiting 300.00 seconds for DNS changes to propagate
Expected routes:
eventstreams: codfw
Checking that eventstreams.discovery.wmnet records for eqiad matches eventstreams.svc.codfw.wmnet (10.2.1.34)
Checking that eventstreams.discovery.wmnet records for codfw matches eventstreams.svc.codfw.wmnet (10.2.1.34)
END (PASS) - Cookbook sre.discovery.service-route (exit_code=0)

Now awaiing full draining of eventstreams in eqiad.

Mentioned in SAL (#wikimedia-analytics) [2022-11-30T12:29:53Z] <btullis> repooling eqiad for eventstreams for T324074

All looks good with eventstreams in eqiad again.

Thank you both!