Remove unused plain HTTP services from LVS
Open, Stalled, MediumPublic
Actions

Assigned To

Authored By

	• ema
	Oct 21 2019, 1:34 PM

Description

While working on T210411 we have added various HTTPS services to LVS. At the end of the ATS migration (T227432) we should go through the list of plain-HTTP services and remove those with no clients left. A good starting point to identify removal candidates is searching for -https in hieradata/common/lvs/configuration.yaml and checking the plain HTTP counterpart.

A list gathered from comments below of items to remove (will be actively updated with activity):

Items not to be touched:

ldap-ro (actively in use)
ncredir (:80 is for https redirection)
text (:80 is for https redirection)
upload (:80 is for https redirection)

Details

	Subject	Repo	Branch	Lines +/-
	hierdata/service: Remove HTTP-only LVS services	operations/puppet	production	+47 -257

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Stalled		BCornwall	T236065 Remove unused plain HTTP services from LVS
		Open		None	T317463 Rename references of labweb to cloudweb

Event Timeline

• ema created this task.Oct 21 2019, 1:34 PM

Restricted Application added a project: SRE. · View Herald TranscriptOct 21 2019, 1:34 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

• ema triaged this task as Medium priority.Oct 21 2019, 1:34 PM

• ema moved this task from Backlog to LoadBalancer on the Traffic board.

Can I ask how I can check if they are not used anymore? That file doesn't exist anymore and it's all in services.yaml

In there we have for example:

search:
  description: Elasticsearch search for MediaWiki
  encryption: false
  ip:
    codfw:
      default: 10.2.1.30
    eqiad:
      default: 10.2.2.30
  lvs:
    class: low-traffic
    conftool:
      cluster: elasticsearch
      service: elasticsearch
    depool_threshold: ".5"
    enabled: true
    monitors:
      IdleConnection:
        max-delay: 300
        timeout-clean-reconnect: 3
      ProxyFetch:
        url:
          - http://localhost/
    scheduler: wrr
  ...
search-https:
  description: Elasticsearch search for MediaWiki - HTTPS
  encryption: true
  ip:
    codfw:
      default: 10.2.1.30
    eqiad:
      default: 10.2.2.30
  lvs:
    class: low-traffic
    conftool:
      cluster: elasticsearch
      service: elasticsearch-ssl
    depool_threshold: ".5"
    enabled: true
    monitors:
      IdleConnection:
        max-delay: 300
        timeout-clean-reconnect: 3
      ProxyFetch:
        url:
          - https://localhost/
    scheduler: wrr
  ...

It seems they are basically duplicates, how can I find if "search" is being used somewhere else?

The swap of Traffic for Traffic-Icebox in this ticket's set of tags was based on a bulk action for all such tickets that haven't been updated in 6 months or more. This does not imply any human judgement about the validity or importance of the task, and is simply the first step in a larger task cleanup effort. Further manual triage and/or requests for updates will happen this month for all such tickets. For more detail, have a look at the extended explanation on the main page of Traffic-Icebox . Thank you!

Change 882722 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] hierdata/service: Remove HTTP-only LVS services

https://gerrit.wikimedia.org/r/882722

gerritbot added a project: Patch-For-Review.Jan 23 2023, 8:56 PM

I also am not sure of how to find out consumers of the HTTP-only services, but I've created a WIP patch that at least lists the candidates.

bblack has some ideas:

13:27 <bblack> we don't have a 100% reliable spot-check to know for sure
13:27 <bblack> but yeah, we can guestimate based on whether it seems to be in active use at all, even at one point in time, that's a useful filter
13:27 <bblack> since they're all pybal services, you can check the ipvs connection tables
13:28 <bblack> basicaly from services.yaml, you can figure out the site(s) and the cluster (e.g. eqiad/high-traffic1, or codfw/low-traffic, etc)
13:29 <bblack> map that to an lvs server like lvs1017
13:29 <bblack> and then knowing the destination ip:port (from service.yaml again), you can:
13:29 <bblack> root@lvs1017:~# ipvsadm -Lnt <external-ip>:80
13:29 <bblack> Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
13:29 <bblack> TCP  <external-ip>:80 wrr -> <internal-ip>:80               Route   10     23         209       -> 10.64.32.89:80               Route   10     26         213       
13:29 <bblack> if you can see activeconn and/or inactiveconn showing some activity, it's probably still being used
13:30 <bblack> although, if the numbers are very low, those might just be various monitoring checks, too
13:30 <bblack> but at least ones with decently-high numbers, you can rule those out as "definitely still really used"
13:31 <bblack> but yeah the devil's in the details: if there's like 2 connections showing up, that could be light real use, or could be some kind of icinga checks hitting it.
13:31 <bblack> and even if it's all-zeros... maybe something wakes up and hits it once every 4 hours, I donno :)
13:32 <bblack> but yeah, maybe start with making a list of all the ones that seem to be in pairs in service.yaml with both an https and http variant
13:33 <bblack> and check to see if the https looks like it's been used at all (maybe never finished setting it up), and whether the http looks very-active or not

The list seems to be:

kartotherian
labweb
ldap-ro
ncredir
restbase-backend
search
swift
text
upload
wdqs

And thus begins the quest of sir Brett the Beautiful...

root@lvs1019:/home/brett# ipvsadm -Lnt 10.2.2.13:6533  # kartotherian
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.2.2.13:6533 wrr
  -> <snip>:6533              Route   10     0          12
  -> <snip>:6533              Route   10     0          11
  -> <snip>:6533              Route   10     0          11
  -> <snip>:6533             Route   10     0          11
  -> <snip>:6533              Route   10     0          11


root@lvs1019:/home/brett# ipvsadm -Lnt 10.2.2.40:80  # labweb
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.2.2.40:80 wrr
  -> <snip>:80            Route   10     0          0
  -> <snip>:80            Route   10     0          0


root@lvs1018:/home/brett# ipvsadm -Lnt 208.80.154.252:389  # ldap-ro
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  208.80.154.252:389 wrr
  -> <snip>:389           Route   10     501        473       
  -> <snip>:389           Route   10     558        431       



root@lvs1017:/home/brett# ipvsadm -Lnt 208.80.154.232:80  #  ncredir
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  208.80.154.232:80 wrr
  -> <snip>:80               Route   10     23         151
  -> <snip>:80               Route   10     18         146

root@lvs1019:/home/brett# ipvsadm -Lnt 10.2.2.17:7233  # restbase-backend
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.2.2.17:7233 wrr
  -> <snip>:7233              Route   10     0          0
  -> <snip>:7233             Route   10     0          0
  -> <snip>:7233             Route   10     0          0
  -> <snip>:7233             Route   10     0          0
  -> <snip>:7233             Route   10     0          0
  -> <snip>:7233             Route   10     0          0
  -> <snip>:7233             Route   10     0          0
  -> <snip>:7233            Route   10     0          0
  -> <snip>:7233            Route   10     0          0
  -> <snip>:7233            Route   10     0          0
  -> <snip>:7233            Route   10     0          0
  -> <snip>:7233            Route   10     0          0
  -> <snip>:7233             Route   10     0          0
  -> <snip>:7233             Route   10     0          0
  -> <snip>:7233            Route   10     0          0
  -> <snip>:7233            Route   10     0          0
  -> <snip>:7233            Route   10     0          0
  -> <snip>:7233            Route   10     0          0


root@lvs1019:/home/brett# ipvsadm -Lnt 10.2.2.30:9200  # search
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.2.2.30:9200 wrr
  -> <snip>:9200              Route   10     1          0
  -> <snip>:9200              Route   10     0          0
  -> <snip>:9200              Route   10     0          0
  -> <snip>:9200              Route   10     0          0
  -> <snip>:9200              Route   10     0          0
  -> <snip>:9200              Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     1          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     1          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200            Route   10     1          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200            Route   10     1          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200            Route   10     0          0
  -> <snip>:9200             Route   10     1          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     1          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     1          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0
  -> <snip>:9200             Route   10     0          0


root@lvs1019:/home/brett# ipvsadm -Lnt 10.2.2.27:80  # swift
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.2.2.27:80 wrr
  -> <snip>:80                Route   40     2          8
  -> <snip>:80               Route   40     2          11
  -> <snip>:80               Route   40     1          7
  -> <snip>:80              Route   40     0          7
  -> <snip>:80               Route   40     3          0


root@lvs1017:/home/brett# ipvsadm -Lnt 208.80.154.224:80  # text
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  208.80.154.224:80 sh
  -> <snip>:80               Route   1      594        6489
  -> <snip>:80               Route   1      577        7675
  -> <snip>:80               Route   1      736        6951
  -> <snip>:80               Route   1      633        7521
  -> <snip>:80               Route   1      640        12366
  -> <snip>:80               Route   1      632        7744
  -> <snip>:80              Route   1      588        6459
  -> <snip>:80              Route   1      593        6097

root@lvs1018:/home/brett# ipvsadm -Lnt 208.80.154.240:80  # upload
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  208.80.154.240:80 sh
  -> <snip>:80               Route   1      65         183       
  -> <snip>:80               Route   1      77         211       
  -> <snip>:80               Route   1      59         268       
  -> <snip>:80               Route   1      77         150       
  -> <snip>:80               Route   1      67         151       
  -> <snip>:80               Route   1      80         134       
  -> <snip>:80              Route   1      88         142       
  -> <snip>:80              Route   1      64         190       

root@lvs1019:/home/brett# ipvsadm -Lnt 10.2.2.32:80  # wdqs
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.2.2.32:80 wrr
  -> <snip>:80                Route   10     0          5
  -> <snip>:80               Route   10     0          5
  -> <snip>:80               Route   10     0          4
  -> <snip>:80              Route   10     0          4
  -> <snip>:80              Route   10     0          4
  -> <snip>:80              Route   10     0          4
  -> <snip>:80               Route   10     0          5
  -> <snip>:80               Route   10     0          4
  -> <snip>:80               Route   10     0          5

From the above it looks like:

ldap-ro and upload both somehow don't have any service established any more and, assuming I haven't done anything wrong, can be removed.
Due to low connections, these can be removed:
- kartotherian
- labweb
- restbase-backend
- search
- swift
- wdqs