Page MenuHomePhabricator

Remove the legacy CNAME endpoints for the URL downloaders
Open, MediumPublic

Description

Starting with T429175 the URL downloaders are accessed via three LVS endpoints as documented here:
https://wikitech.wikimedia.org/wiki/Url-downloader#urldownloader_low-traffic_service

The legacy CNAMEs are still present and point one of the invidivual proxy hosts only. I fixed all uses in Puppet to use the new endpoints, but grepping through the deployment-charts repo there are still a few services which use the legacy endpoints and outdated comments:

jmm@pastis:~/git/deployment-charts$ rgrep url-downloader *
charts/chromium-render/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/mathoid/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/mobileapps/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/wikifeeds/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
helmfile.d/admin_ng/values/main.yaml:  # Allow all pods access to url-downloaders
helmfile.d/services/_mediawiki-common_/global.yaml:      # url-downloader is covered by the default egress rules
helmfile.d/services/mw-debug/values.yaml:      # url-downloader is covered by the default egress rules
helmfile.d/services/toolhub/values.yaml:      # -- url-downloader is covered by default egress rules
helmfile.d/services/toolhub/values.yaml:    http_proxy: http://url-downloader.eqiad.wikimedia.org:8080
helmfile.d/services/toolhub/values.yaml:    https_proxy: http://url-downloader.eqiad.wikimedia.org:8080
helmfile.d/services/toolhub/values-codfw.yaml:    http_proxy: http://url-downloader.codfw.wikimedia.org:8080
helmfile.d/services/toolhub/values-codfw.yaml:    https_proxy: http://url-downloader.codfw.wikimedia.org:8080
helmfile.d/services/miscweb/values-wmf-navigator.yaml:        HTTP_PROXY: "http://url-downloader.eqiad.wikimedia.org:8080"
helmfile.d/services/miscweb/values-wmf-navigator.yaml:        HTTPS_PROXY: "http://url-downloader.eqiad.wikimedia.org:8080"
helmfile.d/services/miscweb/values-wmf-navigator.yaml:        HTTP_PROXY: "http://url-downloader.eqiad.wikimedia.org:8080"
helmfile.d/services/miscweb/values-wmf-navigator.yaml:        HTTPS_PROXY: "http://url-downloader.eqiad.wikimedia.org:8080"
helmfile.d/services/miscweb/values-wmf-navigator-aux-k8s-codfw.yaml:        HTTP_PROXY: "http://url-downloader.codfw.wikimedia.org:8080"
helmfile.d/services/miscweb/values-wmf-navigator-aux-k8s-codfw.yaml:        HTTPS_PROXY: "http://url-downloader.codfw.wikimedia.org:8080"
helmfile.d/services/miscweb/values-wmf-navigator-aux-k8s-codfw.yaml:        HTTP_PROXY: "http://url-downloader.codfw.wikimedia.org:8080"
helmfile.d/services/miscweb/values-wmf-navigator-aux-k8s-codfw.yaml:        HTTPS_PROXY: "http://url-downloader.codfw.wikimedia.org:8080"
jmm@pastis:~/git/deployment-charts$

There might also be remaining uses in Cloud VPS, so that needs some syncronisation as well before we can remove the CNAMEs from the DNS repository.

Event Timeline

Change #1347421 had a related patch set uploaded (by Jelto; author: Jelto):

[operations/deployment-charts@master] miscweb wmf-navigator: update urldownloader URL to LVS endpoint

https://gerrit.wikimedia.org/r/1347421

Change #1347421 merged by jenkins-bot:

[operations/deployment-charts@master] miscweb wmf-navigator: update urldownloader URL to LVS endpoint

https://gerrit.wikimedia.org/r/1347421

The legacy CNAME url-downloader URL was removed from wmf-navigator with the patch above.

Change #1348555 had a related patch set uploaded (by Blake; author: Blake):

[operations/deployment-charts@master] mathoid: replace url-downloader cname with discovery name.

https://gerrit.wikimedia.org/r/1348555

MoritzMuehlenhoff triaged this task as Medium priority.

Change #1351840 had a related patch set uploaded (by Muehlenhoff; author: Muehlenhoff):

[operations/dns@master] Remove the url-downloader legacy CNAMEs

https://gerrit.wikimedia.org/r/1351840

I think this task can be resolved, no references are left (only comment lines):

deployment-charts (master) $ git grep url-downloader
charts/chromium-render/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/mathoid/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/mobileapps/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/wikifeeds/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
helmfile.d/admin_ng/values/main.yaml:  # Allow all pods access to url-downloaders
helmfile.d/services/_mediawiki-common_/global.yaml:      # url-downloader is covered by the default egress rules
helmfile.d/services/mw-debug/values.yaml:      # url-downloader is covered by the default egress rules

Change #1348555 merged by jenkins-bot:

[operations/deployment-charts@master] various: replace url-downloader cname.

https://gerrit.wikimedia.org/r/1348555

I think this task can be resolved, no references are left (only comment lines):

deployment-charts (master) $ git grep url-downloader
charts/chromium-render/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/mathoid/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/mobileapps/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/wikifeeds/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
helmfile.d/admin_ng/values/main.yaml:  # Allow all pods access to url-downloaders
helmfile.d/services/_mediawiki-common_/global.yaml:      # url-downloader is covered by the default egress rules
helmfile.d/services/mw-debug/values.yaml:      # url-downloader is covered by the default egress rules

These comments are addressed by https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1348555, but one significant part which still needs to happen is to announce the removal and then un-deploy the CNAMEs in our DNS repo. I'll take care of it in the next 1-2 weeks.

I think this task can be resolved, no references are left (only comment lines):

deployment-charts (master) $ git grep url-downloader
charts/chromium-render/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/mathoid/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/mobileapps/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
charts/wikifeeds/templates/_config.yaml:      #      proxy: http://url-downloader.eqiad.wikimedia.org:8080
helmfile.d/admin_ng/values/main.yaml:  # Allow all pods access to url-downloaders
helmfile.d/services/_mediawiki-common_/global.yaml:      # url-downloader is covered by the default egress rules
helmfile.d/services/mw-debug/values.yaml:      # url-downloader is covered by the default egress rules

These comments are addressed by https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1348555, but one significant part which still needs to happen is to announce the removal and then un-deploy the CNAMEs in our DNS repo. I'll take care of it in the next 1-2 weeks.

https://gerrit.wikimedia.org/r/c/operations/dns/+/1351840 is ready to be deployed, confirmed through codesearch as well.