Page MenuHomePhabricator

Deploy an updated eventgate-logging-external with NEL patches
Closed, ResolvedPublic

Description

We need a new eventgate-wikimedia deployed to eventgate-logging-external:

  • at least version 1.3.2 of eventgate itself
  • as schemas are packaged with the binary image, built at a time after the schemas were merged https://gerrit.wikimedia.org/r/c/schemas/event/primary/+/623067
  • CORS header output enabled in the helmfile configuration (currently this isn't parameterized in the config template and is always set as false, but that's probably okay as we should be able to override in values.yaml for the appropriate deployment?)

Event Timeline

Example request/responses of both preflight and actual request are in NDA'd paste P12494 (has my own PII in it)

Chrome sends an OPTIONS request to the endpoint URL with e.g.

Origin: https://mcnulty.nucleosynth.space:8443
Access-Control-Request-Method: POST
Access-Control-Request-Headers: content-type

and it expects to receive in the response:

HTTP/1.1 200 OK
access-control-allow-origin: *
access-control-allow-headers: accept, x-requested-with, content-type
access-control-expose-headers: etag
access-control-allow-methods: post

The allowed origin doesn't have to be *, it can just include the current origin, but I don't see much of a danger of making it be * in our configuration?

I discovered this by reading Chromium source: https://bit.ly/2YWcf6f

Change 625965 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-logging-external - set cors: '*'

https://gerrit.wikimedia.org/r/625965

Change 625966 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Bump schema repo version to latest in blubber.yaml

https://gerrit.wikimedia.org/r/625966

Change 625966 merged by Ottomata:
[eventgate-wikimedia@master] Bump schema repo version to latest in blubber.yaml

https://gerrit.wikimedia.org/r/625966

Change 625965 merged by Ottomata:
[operations/deployment-charts@master] eventgate-logging-external - set cors: '*'

https://gerrit.wikimedia.org/r/625965

Change 626166 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate - Set default cors only if not provided in values

https://gerrit.wikimedia.org/r/626166

Change 626166 merged by Ottomata:
[operations/deployment-charts@master] eventgate - Set default cors only if not provided in values

https://gerrit.wikimedia.org/r/626166

Ok! I think we are good to go!

We'll need to add a wgEventStreams stream config entry and then redeploy (or just restart) eventgate-logging-external pods to get the change.

Change 627353 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/mediawiki-config@master] modify wgEventStreams to reference NEL schema

https://gerrit.wikimedia.org/r/627353

Change 627353 merged by jenkins-bot:
[operations/mediawiki-config@master] modify wgEventStreams to reference NEL schema

https://gerrit.wikimedia.org/r/627353

Mentioned in SAL (#wikimedia-operations) [2020-09-14T20:26:00Z] <cdanis@deploy1001> Synchronized wmf-config/InitialiseSettings.php: a588eb0c6 T262087 modify wgEventStreams to reference NEL schema (duration: 00m 56s)

I believe the only thing left to do is to perform a rolling restart of the eventgate-logging-external pods (or the container within them).

I'd like @Joe or someone else on serviceops to walk me through that tomorrow when I'm awake, as I'm inexperienced but feel like I should know how, and also I found some things that seemed odd to me in the configuration:

17:54:28	<cdanis>	so what I see is, a ReplicaSet that says:
17:54:30	<cdanis>	Replicas:       1 current / 1 desired
17:54:30	<cdanis>	Pods Status:    1 Running / 0 Waiting / 0 Succeeded / 0 Failed
17:54:58	<cdanis>	and also, I see in `kubectl get pods` output: eventgate-logging-external-production-79f8b8bc48-dvpnh   3/3     Running   0          5d6h
17:55:28	<cdanis>	and I don't see where that 3 comes from except if we manually scaled it, in which case I have no idea if the replicaset is controlling it anymore

Change 627501 had a related patch set uploaded (by CDanis; owner: CDanis):
[operations/deployment-charts@master] eventgate-logging-external: +replicas & cache NEL

https://gerrit.wikimedia.org/r/627501

Change 627501 merged by CDanis:
[operations/deployment-charts@master] eventgate-logging-external: +replicas & cache NEL

https://gerrit.wikimedia.org/r/627501

Change 627539 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/deployment-charts@master] eventgate-logging-external - set schema_uri_query_param and stream_query_param

https://gerrit.wikimedia.org/r/627539

Change 627539 merged by Ottomata:
[operations/deployment-charts@master] eventgate-logging-external - set schema_uri_query_param and stream_query_param

https://gerrit.wikimedia.org/r/627539