Page MenuHomePhabricator
Create Task
Maniphest T253396

Upgrade all TLS enabled charts to v0.2 tls_helper
Closed, ResolvedPublic
Actions

Description

When all charts are migrated, we can:

  • discontinue maintenance for the "envoy-tls-local-proxy"
  • remove "envoy-tls-local-proxy" from git
  • maybe remove the images as well (so they are not used by accident)?

Charts using v0.1 _tls_helper:

  • blubberoid
  • citoid
  • cxserver
  • wikifeeds
  • chromium-render
  • mobileapps
  • recommendation-api

Special charts that don't use the common helper:

  • changeprop/templates/deployment.yaml: image: {{ .Values.docker.registry }}/envoy-tls-local-proxy:{{ .Values.tls.image_version }}
  • eventgate/templates/_tls_helpers.tpl: image: {{ .Values.docker.registry }}/envoy-tls-local-proxy:{{ .Values.tls.image_version }}
  • eventstreams/templates/_tls_helpers.tpl: image: {{ .Values.docker.registry }}/envoy-tls-local-proxy:{{ .Values.tls.image_version }}

To figure out if envoy-tls-local-proxyis still running somewhere:

kubectl get pods --all-namespaces -o go-template --template='{{range .items}}{{$n := .metadata.name}}{{range .spec.containers}}{{$n}}{{":\t"}}{{.image}}{{"\n"}}{{end}}{{end}}' | grep envoy-tls-local

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

JMeybohm created this task.May 22 2020, 5:05 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 22 2020, 5:05 PM
JMeybohm updated the task description. (Show Details)May 26 2020, 8:13 AM
gerritbot added a comment.May 26 2020, 3:10 PM

Change 598759 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] blubberoid: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598759

gerritbot added a project: Patch-For-Review.May 26 2020, 3:10 PM
gerritbot added a comment.May 26 2020, 3:14 PM

Change 598760 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] charts: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598760

gerritbot added a comment.May 26 2020, 3:19 PM

Change 598766 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] cxserver: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598766

gerritbot added a comment.May 26 2020, 3:21 PM

Change 598774 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] wikifeeds: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598774

gerritbot added a comment.May 26 2020, 3:22 PM

Change 598777 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] chromium-render: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598777

gerritbot added a comment.May 26 2020, 3:27 PM

Change 598779 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] mobileapps: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598779

gerritbot added a comment.May 26 2020, 3:29 PM

Change 598780 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] recommendation-api: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598780

JMeybohm added a comment.May 26 2020, 3:41 PM

I've used something like this to update the charts:

function update_helper() {
    CHART=$(basename $1)
    git checkout -b ${CHART}_tls_v0.2 master
    
    pushd $CHART

    version=$(yq r Chart.yaml version)
    a=( ${version//./ } )
    ((a[2]++))
    new_version="${a[0]}.${a[1]}.${a[2]}"
    yq w -i Chart.yaml version $new_version

    pushd templates
    find . -type l -exec bash -c 'rm $0; ln -s ../../../common_templates/0.2/$(basename $0)' {} \;
    popd

    yq w -i values.yaml tls.image_version 1.13.1-2
    yq w -i .fixtures/tls_enabled.yaml tls.image_version 1.13.1-2
    popd

    helm package $CHART
    helm repo index .
    
    git add -u .
    git add ${CHART}-${new_version}.tgz
    git commit -e -m "$(echo -e "${CHART}: Update to v0.2 helpers\n\nBug: T253396")"
}
JMeybohm added a parent task: T235411: Add TLS termination to services running on kubernetes.May 27 2020, 8:03 AM
gerritbot added a comment.May 28 2020, 8:49 AM

Change 598759 merged by jenkins-bot:
[operations/deployment-charts@master] blubberoid: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598759

gerritbot added a comment.May 28 2020, 9:18 AM

Change 599289 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] blubberoid: Use chart defaults for deployment

https://gerrit.wikimedia.org/r/599289

gerritbot added a comment.May 28 2020, 10:28 AM

Change 599289 merged by jenkins-bot:
[operations/deployment-charts@master] blubberoid: Use chart defaults for deployment

https://gerrit.wikimedia.org/r/599289

gerritbot added a comment.May 28 2020, 4:02 PM

Change 598780 merged by jenkins-bot:
[operations/deployment-charts@master] recommendation-api: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598780

gerritbot added a comment.May 28 2020, 4:08 PM

Change 598779 merged by jenkins-bot:
[operations/deployment-charts@master] mobileapps: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598779

gerritbot added a comment.May 28 2020, 4:13 PM

Change 598760 merged by jenkins-bot:
[operations/deployment-charts@master] citoid: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598760

gerritbot added a comment.May 28 2020, 4:15 PM

Change 598777 merged by jenkins-bot:
[operations/deployment-charts@master] chromium-render: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598777

gerritbot added a comment.May 28 2020, 4:17 PM

Change 598766 merged by jenkins-bot:
[operations/deployment-charts@master] cxserver: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598766

gerritbot added a comment.May 28 2020, 4:20 PM

Change 598774 merged by jenkins-bot:
[operations/deployment-charts@master] wikifeeds: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/598774

Maintenance_bot removed a project: Patch-For-Review.May 28 2020, 5:10 PM
JMeybohm updated the task description. (Show Details)May 29 2020, 11:23 AM
Restricted Application added a subscriber: Liuxinyu970226. · View Herald TranscriptMay 29 2020, 11:23 AM
gerritbot added a comment.May 31 2020, 1:35 PM

Change 600862 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] changeprop: Migrate to common_templates 0.2 tls_helper

https://gerrit.wikimedia.org/r/600862

gerritbot added a project: Patch-For-Review.May 31 2020, 1:35 PM
JMeybohm added a subscriber: Ottomata.Jun 3 2020, 7:37 AM

So eventgate and eventstream use forked tls_helpers (currently even the forks slightly differ).

As far as I can tell, this is due to types of reasons. One being diverging use of .metadata.labels.app and .metadata.labels.chart and the other one is the additional label .metadata.labels.routing_tag.

  • .metadata.labels.app is charts .Values.main_app.name. This could be replaced with the default value when using .Values.main_app.name as part of the helm release name (like main-production instead of production for example). Alternatively .Values.chartName could be used to overwrite the chart name with eventgate-main, keeping production as helm release name.
  • .metadata.labels.chart uses the wmf.chartname template. The common_templates have wmf.chartname in .metadata.labels.app.
  • .metadata.labels.routing_tag is set to .Release.Name by default and is used as additional selector for the services. I don't see why this is needed. As we run one release (except for canary) per namespace and the kubernetes service will always only consider pods in it's namespace it will only select the correct release anyways. So maybe this can be removed.

So, my suggestion would be to switch to common_helpers, use .Values.chartName instead of .Values.main_app.name and drop .metadata.labels.routing_tag completely.

@Ottomata what do you think?

JMeybohm added a comment.Jun 3 2020, 8:58 AM

Oh. I see that the current canary setup will not work with my suggestions and as I see it there is currently no way on how to do it with the default scaffold/templates.

JMeybohm added a comment.Jun 3 2020, 10:08 AM

And I now see T242861, so please ignore what I said (or at least what I was suggesting).
I'll evaluate the route of merging the common_templates v0.2 changes into eventgate/eventstream forks instead to not have this blocked.

gerritbot added a comment.Jun 3 2020, 12:01 PM

Change 602060 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] eventstreams: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/602060

gerritbot added a comment.Jun 3 2020, 12:03 PM

Change 602061 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/deployment-charts@master] eventgate: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/602061

Ottomata added a comment.Jun 3 2020, 1:29 PM

Thanks @JMeybohm yeah T242861 is complicated!

gerritbot added a comment.Jun 4 2020, 8:21 AM

Change 600862 merged by jenkins-bot:
[operations/deployment-charts@master] changeprop: Migrate to common_templates 0.2 tls_helper

https://gerrit.wikimedia.org/r/600862

JMeybohm updated the task description. (Show Details)Jun 4 2020, 9:22 AM
JMeybohm updated the task description. (Show Details)Jun 4 2020, 9:45 AM
JMeybohm moved this task from Incoming 🐫 to Doing 😎 on the serviceops board.Jun 5 2020, 6:35 AM
gerritbot added a comment.Jun 10 2020, 7:31 AM

Change 602060 merged by jenkins-bot:
[operations/deployment-charts@master] eventstreams: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/602060

JMeybohm updated the task description. (Show Details)Jun 10 2020, 9:02 AM
gerritbot added a comment.Jun 10 2020, 2:00 PM

Change 602061 merged by jenkins-bot:
[operations/deployment-charts@master] eventgate: Update to v0.2 helpers

https://gerrit.wikimedia.org/r/602061

Maintenance_bot removed a project: Patch-For-Review.Jun 10 2020, 2:10 PM
JMeybohm updated the task description. (Show Details)Jun 11 2020, 1:10 PM
JMeybohm added a comment.Jun 11 2020, 3:23 PM

All clusters clean from envoy-tls-local-proxy image!

gerritbot added a comment.Jun 29 2020, 9:53 AM

Change 608277 had a related patch set uploaded (by JMeybohm; owner: JMeybohm):
[operations/docker-images/production-images@master] Remove deprecated and unmaintained image: envoy-tls-local-proxy

https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/ /608277

gerritbot added a project: Patch-For-Review.Jun 29 2020, 9:53 AM
Stashbot added a comment.Jun 29 2020, 12:32 PM

Mentioned in SAL (#wikimedia-operations) [2020-06-29T12:32:50Z] <jayme> deleted all tags for docker-registry.wikimedia.org/envoy-tls-local-proxy from docker registry - T253396

JMeybohm added a comment.Jun 29 2020, 5:20 PM

I tried to delete the tags/image with the process described here[1] but unfortunately the tags can still be pulled after successful DELETE (another DELETE even returns HTTP 404).
I guess a garbage-collection[2] run is needed to actually remove the tags from the registry. I tried that (--dry-run) on registry2001 where is is running since 5 hours and still going. According to the output (which seems to go over all images in alphabetic order) it has reached the last image but it's still doing a lot of swift requests, so its probably not stuck...

JMeybohm closed this task as Resolved.Jun 30 2020, 9:42 AM

Seems it is requited to try to fetch the tag list while bypassing the caches once to have the lasting references removed:
curl https://docker-registry.wikimedia.org/v2/envoy-tls-local-proxy/tags/list?x=y

Unfortunately I did not manage to remove the "repository" itself (docker-registry.wikimedia.org/v2/envoy-tls-local-proxy).

For reference:
There is a tool called docker-registryctl in https://gerrit.wikimedia.org/g/operations/docker-images/docker-report which can be used to delete image tags (via the docker-registry API as well).

gerritbot added a comment.Jun 30 2020, 9:43 AM

Change 608277 merged by JMeybohm:
[operations/docker-images/production-images@master] Remove deprecated and unmaintained image: envoy-tls-local-proxy

https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/ /608277

Stashbot added a comment.Jun 30 2020, 11:31 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-30T11:31:58Z] <jayme> pushed a scratch docker image as docker-registry.discovery.wmnet/envoy-tls-local-proxy:dontuseme - T253396

Stashbot added a comment.Jun 30 2020, 11:32 AM

Mentioned in SAL (#wikimedia-operations) [2020-06-30T11:32:02Z] <jayme> restarted docker-reporter-base-images and docker-reporter-releng-images on deneb - T253396

JMeybohm reopened this task as Open.Jun 30 2020, 11:32 AM

This led to failing docker-reporter-base-images.service on deneb. I'm definitely missing something here...

JMeybohm closed this task as Resolved.Jul 2 2020, 8:18 AM
JMeybohm added a subscriber: Joe.

This is resolved now. For anyone passing along, please see: https://wikitech.wikimedia.org/wiki/Docker#Deleting_an_image_(from_registry) and T242604

Thanks @Joe

JMeybohm mentioned this in T271381: "envoy-tls-local-proxy" image on docker-registry.wikimedia.org has no tags.Jan 7 2021, 8:48 AM
Content licensed under Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) unless otherwise noted; code licensed under GNU General Public License (GPL) or other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL