Page MenuHomePhabricator

Remove blubberoid LVS/k8s service
Closed, ResolvedPublic

Description

Blubberoid is being decommissioned and it is time to remove the running services in k8s eqiad/codfw/staging. Since blubberoid is an LVS service, it needs special attention according to https://wikitech.wikimedia.org/wiki/LVS#Remove_a_load_balanced_service.

I will likely need assistance from folks in traffic on this one as I already bungled the removal once.

Event Timeline

dduvall changed the task status from Open to In Progress.May 23 2024, 5:52 PM
dduvall triaged this task as Medium priority.

Change #1035533 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/dns@master] Remove blubberoid wmnet and wikimedia.org records

https://gerrit.wikimedia.org/r/1035533

Change #1035533 merged by BCornwall:

[operations/dns@master] Remove blubberoid wmnet and wikimedia.org records

https://gerrit.wikimedia.org/r/1035533

Change #1035543 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] service: Remove probes for blubberoid

https://gerrit.wikimedia.org/r/1035543

Change #1035589 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] service: Remove blubberoid from backend servers and load balancers

https://gerrit.wikimedia.org/r/1035589

Change #1035797 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] service: Remove blubberoid from service catalog and conftool

https://gerrit.wikimedia.org/r/1035797

Change #1035798 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/puppet@production] service: Remove remaining blubberoid related configuration

https://gerrit.wikimedia.org/r/1035798

Change #1035543 merged by Ssingh:

[operations/puppet@production] service: Remove probes for blubberoid

https://gerrit.wikimedia.org/r/1035543

Change #1035589 merged by Ssingh:

[operations/puppet@production] service: Remove blubberoid from backend servers and load balancers

https://gerrit.wikimedia.org/r/1035589

Mentioned in SAL (#wikimedia-operations) [2024-05-28T16:41:26Z] <sukhe> sudo cumin 'O:lvs::balancer' 'run-puppet-agent': T365742

Mentioned in SAL (#wikimedia-operations) [2024-05-28T16:47:13Z] <sukhe@cumin1002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad or A:lvs-secondary-codfw and A:lvs (T365742)

Mentioned in SAL (#wikimedia-operations) [2024-05-28T16:59:29Z] <sukhe@cumin1002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-eqiad or A:lvs-secondary-codfw and A:lvs (T365742)

Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:03:29Z] <sukhe> removing blubberoid's IP from ipvsadm: T365742

Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:08:05Z] <sukhe> sudo cumin 'A:lvs-secondary-eqiad or A:lvs-low-traffic-eqiad' 'ipvsadm --delete-service --tcp-service 10.2.2.31:4666': T365742

Change #1035797 merged by Ssingh:

[operations/puppet@production] service: Remove blubberoid from service catalog and conftool

https://gerrit.wikimedia.org/r/1035797

Change #1035798 merged by Ssingh:

[operations/puppet@production] service: Remove remaining blubberoid related configuration

https://gerrit.wikimedia.org/r/1035798

Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:21:29Z] <dduvall> removing blubberoid from staging, helmfile -e staging destroy (T365742)

Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:24:12Z] <dduvall> removing blubberoid from codfw, helmfile -e codfw destroy (T365742)

Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:24:18Z] <sukhe> sudo -i puppet cert clean blubberoid.discovery.wmnet: T365742

Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:25:15Z] <dduvall> removing blubberoid from eqiad, helmfile -e eqiad destroy (T365742)

Change #1036716 had a related patch set uploaded (by Dduvall; author: Dduvall):

[operations/deployment-charts@master] admin_ng: remove blubberoid namespace and helmfile.d files

https://gerrit.wikimedia.org/r/1036716

I will take care of the rest when I find some time, thank you!

Change #1036716 merged by jenkins-bot:

[operations/deployment-charts@master] admin_ng: remove blubberoid

https://gerrit.wikimedia.org/r/1036716