Page MenuHomePhabricator
Create Task
Maniphest T299700

Remove legacy ELK LVS entries
Closed, ResolvedPublic
Actions

Description

The last of the elk5 hosts are ready for decom, time to remove the old LVS instances for legacy logstash inputs and the old kibana dashboard.

From https://wikitech.wikimedia.org/wiki/LVS#Remove_a_load_balanced_service The procedure for removal of a service should more or less follow the inverse order of what gets done adding it. It is important to perform the following actions in order. Specifically:

Details

SubjectRepoBranchLines +/-
cleanup logstash and kibana svc recordsoperations/dnsmaster+0 -8
remove logstash and kibana entries from conftool-data discovery servicesoperations/puppetproduction+0 -2
remove elk5 related LVS servicesoperations/puppetproduction+0 -186
remove realserver_ips from legacy elk roles & set lvs state: service_setupoperations/puppetproduction+6 -13
remove kibana-disc from discovery-metafo-resourcesoperations/dnsmaster+0 -1
switch legacy elk LVS entries to state: lvs_setupoperations/puppetproduction+6 -6
remove kibana.discovery.wmnet recordoperations/dnsmaster+0 -1
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedherronT281266 Decommission old ELK5 Logstash cluster
 ResolvedherronT299700 Remove legacy ELK LVS entries

Event Timeline

herron created this task.Jan 20 2022, 8:39 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptJan 20 2022, 8:39 PM
herron updated the task description. (Show Details)Jan 20 2022, 8:40 PM
gerritbot added a comment.Jan 20 2022, 8:40 PM

Change 755480 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] remove elk5 related LVS services

https://gerrit.wikimedia.org/r/755480

gerritbot added a project: Patch-For-Review.Jan 20 2022, 8:40 PM
gerritbot added a comment.Jan 20 2022, 8:48 PM

Change 755789 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] switch legacy elk LVS entries to state: lvs_setup

https://gerrit.wikimedia.org/r/755789

herron updated the task description. (Show Details)Jan 20 2022, 8:48 PM
gerritbot added a comment.Jan 20 2022, 8:50 PM

Change 755790 had a related patch set uploaded (by Herron; author: Herron):

[operations/dns@master] remove kibana.discovery.wmnet record

https://gerrit.wikimedia.org/r/755790

herron updated the task description. (Show Details)Jan 20 2022, 8:51 PM
herron added a project: Traffic.Jan 20 2022, 8:57 PM
Stashbot added a comment.Jan 21 2022, 3:07 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-21T15:07:41Z] <herron> removing kibana.discovery.wmnet record and switching legacy elk LVS instances to state: lvs_setup T299700

gerritbot added a comment.Jan 21 2022, 3:09 PM

Change 755790 merged by Herron:

[operations/dns@master] remove kibana.discovery.wmnet record

https://gerrit.wikimedia.org/r/755790

gerritbot added a comment.Jan 21 2022, 3:10 PM

Change 755789 merged by Herron:

[operations/puppet@production] switch legacy elk LVS entries to state: lvs_setup

https://gerrit.wikimedia.org/r/755789

herron updated the task description. (Show Details)Jan 21 2022, 3:42 PM
herron added a comment.Jan 21 2022, 5:20 PM

Services have been moved to lvs_setup, but there are some pybal icinga alerts still open e.g.

lvs1015 PyBal IPVS diff check CRITICAL	2022-01-21 17:11:09	1d 21h 24m 8s	3/3	CRITICAL: Services known to PyBal but not to IPVS: set(['10.2.2.36:11514', '10.2.2.36:8324', '10.2.2.33:443', '10.2.2.33:80', '10.2.2.36:12201'])
lvs1015 PyBal backends health check PYBAL CRITICAL - CRITICAL - kibana-ssl_443: Servers logstash1009.eqiad.wmnet, logstash1008.eqiad.wmnet are marked down but pooled: logstash-json-tcp_11514: Servers logstash1009.eqiad.wmnet, logstash1008.eqiad.wmnet are marked down but pooled: kibana_80: Servers logstash1009.eqiad.wmnet, logstash1008.eqiad.wmnet are marked down but pooled

I'm assuming these will go away after a pybal restart, going to connect with Traffic before proceeding with that

gerritbot added a comment.Jan 21 2022, 5:55 PM

Change 756036 had a related patch set uploaded (by Herron; author: Herron):

[operations/dns@master] remove kibana-disc from discovery-metafo-resources

https://gerrit.wikimedia.org/r/756036

gerritbot added a comment.Jan 21 2022, 5:57 PM

Change 756036 merged by Herron:

[operations/dns@master] remove kibana-disc from discovery-metafo-resources

https://gerrit.wikimedia.org/r/756036

gerritbot added a comment.Jan 21 2022, 6:15 PM

Change 756038 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] remove realserver_ips from legacy elk roles & set lvs state: service_setup

https://gerrit.wikimedia.org/r/756038

gerritbot added a comment.Jan 21 2022, 6:26 PM

Change 756038 merged by Herron:

[operations/puppet@production] remove realserver_ips from legacy elk roles & set lvs state: service_setup

https://gerrit.wikimedia.org/r/756038

Stashbot added a comment.Jan 21 2022, 6:46 PM

Mentioned in SAL (#wikimedia-operations) [2022-01-21T18:46:42Z] <herron> restarting pybal on lvs1015,lvs1020,lvs2009,lvs2010 to remove legacy elk5 services T299700

gerritbot added a comment.Jan 21 2022, 7:05 PM

Change 755480 merged by Herron:

[operations/puppet@production] remove elk5 related LVS services

https://gerrit.wikimedia.org/r/755480

herron updated the task description. (Show Details)Jan 21 2022, 7:06 PM
gerritbot added a comment.Jan 21 2022, 7:10 PM

Change 756045 had a related patch set uploaded (by Herron; author: Herron):

[operations/dns@master] cleanup kibana.svc records

https://gerrit.wikimedia.org/r/756045

gerritbot added a comment.Jan 21 2022, 7:14 PM

Change 756046 had a related patch set uploaded (by Herron; author: Herron):

[operations/puppet@production] remove logstash and kibana entries from conftool-data discovery services

https://gerrit.wikimedia.org/r/756046

gerritbot added a comment.Jan 21 2022, 7:18 PM

Change 756046 merged by Herron:

[operations/puppet@production] remove logstash and kibana entries from conftool-data discovery services

https://gerrit.wikimedia.org/r/756046

gerritbot added a comment.Jan 21 2022, 8:09 PM

Change 756045 merged by Herron:

[operations/dns@master] cleanup logstash and kibana svc records

https://gerrit.wikimedia.org/r/756045

herron closed this task as Resolved.Jan 21 2022, 8:21 PM
herron added a subscriber: BBlack.

These have been removed with much help from @BBlack thank you!

lmata moved this task from Inbox to Done on the SRE Observability (FY2021/2022-Q3) board.Jan 21 2022, 8:29 PM
herron mentioned this in T281266: Decommission old ELK5 Logstash cluster.Jan 26 2022, 2:58 PM
Volans reopened this task as Open.Oct 11 2023, 11:52 AM
Volans subscribed.

FYI the service IPs are still allocated in Netbox:
https://netbox.wikimedia.org/ipam/ip-addresses/?q=kibana.svc
https://netbox.wikimedia.org/ipam/ip-addresses/?q=logstash.svc

I guess they should be removed. Once done make sure to run the sre.dns.netbox cookbook too.

Volans mentioned this in T270071: SVC DNS zonefiles and source of truth.Oct 11 2023, 12:21 PM
lmata edited projects, added SRE Observability (FY2023/2024-Q2); removed SRE Observability (FY2021/2022-Q3).Oct 16 2023, 11:38 PM
Stashbot added a comment.Dec 7 2023, 5:08 PM

Mentioned in SAL (#wikimedia-operations) [2023-12-07T17:08:41Z] <herron@cumin1001> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cleanup logstash/kibana records T299700 - herron@cumin1001"

Stashbot added a comment.Dec 7 2023, 5:09 PM

Mentioned in SAL (#wikimedia-operations) [2023-12-07T17:09:35Z] <herron@cumin1001> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cleanup logstash/kibana records T299700 - herron@cumin1001"

herron closed this task as Resolved.Dec 7 2023, 5:13 PM
In T299700#9242375, @Volans wrote:

FYI the service IPs are still allocated in Netbox:
https://netbox.wikimedia.org/ipam/ip-addresses/?q=kibana.svc
https://netbox.wikimedia.org/ipam/ip-addresses/?q=logstash.svc

I guess they should be removed. Once done make sure to run the sre.dns.netbox cookbook too.

Good catch thanks, this has been done

lmata moved this task from Inbox to Done on the SRE Observability (FY2023/2024-Q2) board.Jan 26 2024, 1:08 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits