Blubberoid is being decommissioned and it is time to remove the running services in k8s eqiad/codfw/staging. Since blubberoid is an LVS service, it needs special attention according to https://wikitech.wikimedia.org/wiki/LVS#Remove_a_load_balanced_service.
I will likely need assistance from folks in traffic on this one as I already bungled the removal once.
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | dduvall | T318289 Deprecate Blubber's CLI and microservice (blubberoid) interfaces | |||
| Resolved | dduvall | T365742 Remove blubberoid LVS/k8s service |
Change #1035533 had a related patch set uploaded (by Dduvall; author: Dduvall):
[operations/dns@master] Remove blubberoid wmnet and wikimedia.org records
Change #1035533 merged by BCornwall:
[operations/dns@master] Remove blubberoid wmnet and wikimedia.org records
Change #1035543 had a related patch set uploaded (by Dduvall; author: Dduvall):
[operations/puppet@production] service: Remove probes for blubberoid
Change #1035589 had a related patch set uploaded (by Dduvall; author: Dduvall):
[operations/puppet@production] service: Remove blubberoid from backend servers and load balancers
Change #1035797 had a related patch set uploaded (by Dduvall; author: Dduvall):
[operations/puppet@production] service: Remove blubberoid from service catalog and conftool
Change #1035798 had a related patch set uploaded (by Dduvall; author: Dduvall):
[operations/puppet@production] service: Remove remaining blubberoid related configuration
Change #1035543 merged by Ssingh:
[operations/puppet@production] service: Remove probes for blubberoid
Change #1035589 merged by Ssingh:
[operations/puppet@production] service: Remove blubberoid from backend servers and load balancers
Mentioned in SAL (#wikimedia-operations) [2024-05-28T16:41:26Z] <sukhe> sudo cumin 'O:lvs::balancer' 'run-puppet-agent': T365742
Mentioned in SAL (#wikimedia-operations) [2024-05-28T16:47:13Z] <sukhe@cumin1002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on A:lvs-secondary-eqiad or A:lvs-secondary-codfw and A:lvs (T365742)
Mentioned in SAL (#wikimedia-operations) [2024-05-28T16:59:29Z] <sukhe@cumin1002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on A:lvs-secondary-eqiad or A:lvs-secondary-codfw and A:lvs (T365742)
Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:03:29Z] <sukhe> removing blubberoid's IP from ipvsadm: T365742
Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:08:05Z] <sukhe> sudo cumin 'A:lvs-secondary-eqiad or A:lvs-low-traffic-eqiad' 'ipvsadm --delete-service --tcp-service 10.2.2.31:4666': T365742
Change #1035797 merged by Ssingh:
[operations/puppet@production] service: Remove blubberoid from service catalog and conftool
Change #1035798 merged by Ssingh:
[operations/puppet@production] service: Remove remaining blubberoid related configuration
Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:21:29Z] <dduvall> removing blubberoid from staging, helmfile -e staging destroy (T365742)
Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:24:12Z] <dduvall> removing blubberoid from codfw, helmfile -e codfw destroy (T365742)
Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:24:18Z] <sukhe> sudo -i puppet cert clean blubberoid.discovery.wmnet: T365742
Mentioned in SAL (#wikimedia-operations) [2024-05-28T17:25:15Z] <dduvall> removing blubberoid from eqiad, helmfile -e eqiad destroy (T365742)
Change #1036716 had a related patch set uploaded (by Dduvall; author: Dduvall):
[operations/deployment-charts@master] admin_ng: remove blubberoid namespace and helmfile.d files
Change #1036716 merged by jenkins-bot:
[operations/deployment-charts@master] admin_ng: remove blubberoid