Page MenuHomePhabricator

Move eventgate services to use TLS only
Closed, ResolvedPublic

Description

  • Add TLS support to the deployment chart
  • Enable TLS on k8s in production
  • Add Additional LVS endpoint configuration
  • Switch services to use the TLS LVS
  • Remove non-TLS LVS endpoint configuration
  • Remove the non-TLS k8s service

Event Timeline

JMeybohm triaged this task as Medium priority.Jul 21 2020, 7:53 AM

@Ottomata We would like the HTTP services to be decommissioned from Kubernetes after a service is switched to TLS only in LVS but I think that's currently not possible with eventgate. Could you add support for that, please?

Sure! Although I have to admit I don't know what this means. It already runs envoyproxy as a sidecar for TLS. Is this so that its outgoing HTTP connections also go through envoy? Are there some docs or examples I could follow?

Sure! Although I have to admit I don't know what this means. It already runs envoyproxy as a sidecar for TLS. Is this so that its outgoing HTTP connections also go through envoy? Are there some docs or examples I could follow?

Sorry for not being very precise.
What it needs is an option to disable the creation of the kubernestes service object "announcing" the HTTP port (https://github.com/wikimedia/operations-deployment-charts/blob/master/charts/eventgate/templates/service.yaml#L15-L55) in case TLS is enabled. So the eventgates would end up with only one kubernetes service (the TLS one) instead of two. The enventgate chart seems to have an option like that already because it creates the TLS service object only.

OH! yes...there was a reason we left HTTP on...I think it was before MW was using a local envoyproxy to do TLS, because PHP HTTPS is crappy. I think we can turn it off.

Change 710111 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/deployment-charts@master] eventgate - Disable http service if tls.enabled

https://gerrit.wikimedia.org/r/710111

Ottomata renamed this task from Move eventgate-analytics-external to use TLS only to Move eventgate services to use TLS only.Aug 4 2021, 8:22 PM
Ottomata added a project: Analytics-Kanban.
Ottomata moved this task from Next Up to In Code Review on the Analytics-Kanban board.

I think that will do it. helm template looks good locally.

@JMeybohm is it ok that I moved the debug ports to their own Service? That's rarely every used, but I didn't want to remove that feature.

I think that will do it. helm template looks good locally.

@JMeybohm is it ok that I moved the debug ports to their own Service? That's rarely every used, but I didn't want to remove that feature.

Sure, that is perfectly fine!

Change 710111 merged by Ottomata:

[operations/deployment-charts@master] eventgate - Disable http service if tls.enabled

https://gerrit.wikimedia.org/r/710111

@JMeybohm, I merged that and am trying to apply for eventgate-logging-external staging.

Diff looks good:

20:23:31 [@deploy1002:/srv/ … /helmfile.d/services/eventgate-logging-external] (master+)[368713f] ± helmfile -e staging diff
skipping missing values file matching "values-production.yaml"
Comparing release=production, chart=wmf-stable/eventgate
eventgate-logging-external, eventgate-logging-external-production, Service (v1) has been removed:
- # Source: eventgate/templates/service.yaml
- apiVersion: v1
- kind: Service
- metadata:
-   name: eventgate-logging-external-production
-   labels:
-     chart: eventgate
-     app: eventgate-logging-external
-     release: production
-     heritage: Tiller
- spec:
-   type: NodePort
-   selector:
-     chart: eventgate
-     app: eventgate-logging-external
-     routing_tag: eventgate-logging-external
-   ports:
-   - name: eventgate-logging-external-http
-     protocol: TCP
-     port: 8192
-     nodePort: 33192
+

But apply gives:

20:24:19 [@deploy1002:/srv/ … /helmfile.d/services/eventgate-logging-external] (master+)[368713f] ± helmfile -e staging apply
skipping missing values file matching "values-production.yaml"
Comparing release=production, chart=wmf-stable/eventgate
Listing releases matching ^canary$
in ./helmfile.yaml: 2 errors:
err 0: command "/usr/bin/helm" exited with non-zero status:

PATH:
  /usr/bin/helm

ARGS:
  0: helm (4 bytes)
  1: diff (4 bytes)
  2: upgrade (7 bytes)
  3: --reset-values (14 bytes)
  4: --allow-unreleased (18 bytes)
  5: production (10 bytes)
  6: wmf-stable/eventgate (20 bytes)
  7: --tiller-namespace (18 bytes)
  8: eventgate-logging-external (26 bytes)
  9: --namespace (11 bytes)
  10: eventgate-logging-external (26 bytes)
  11: --values (8 bytes)
  12: /tmp/values631491484 (20 bytes)
  13: --values (8 bytes)
  14: /tmp/values260155467 (20 bytes)
  15: --values (8 bytes)
  16: /tmp/values344461102 (20 bytes)
  17: --values (8 bytes)
  18: /tmp/values549597877 (20 bytes)
  19: --detailed-exitcode (19 bytes)
  20: --kubeconfig=/etc/kubernetes/eventgate-logging-external-staging.config (70 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  Error: forwarding ports: error upgrading connection:

COMBINED OUTPUT:
  Error: forwarding ports: error upgrading connection:
err 1: command "/usr/bin/helm" exited with non-zero status:

PATH:
  /usr/bin/helm

ARGS:
  0: helm (4 bytes)
  1: list (4 bytes)
  2: ^canary$ (8 bytes)
  3: --tiller-namespace (18 bytes)
  4: eventgate-logging-external (26 bytes)
  5: --deployed (10 bytes)
  6: --failed (8 bytes)
  7: --pending (9 bytes)
  8: --kubeconfig=/etc/kubernetes/eventgate-logging-external-staging.config (70 bytes)

ERROR:
  exit status 1

EXIT STATUS
  1

STDERR:
  Error: forwarding ports: error upgrading connection:

COMBINED OUTPUT:
  Error: forwarding ports: error upgrading connection:

@Ottomata that looks unrelated to your chance (but related to yours @Jelto ). We will take a look!

@Ottomata that looks unrelated to your chance (but related to yours @Jelto ). We will take a look!

Totally different issue, but we figured it out and I just deploy to staging successfully.

I see you've deployed all eventgates, thanks! Resolving this