Page MenuHomePhabricator

eventgate-wikimedia should fill in defaults for some important fields
Closed, ResolvedPublic

Description

Some fields we want set are either not available for clients to manually set in event data, or are inconvenient to do so:

  • client_ip (needed for downstream geocoding)
  • user_agent (already sent in HTTP header, annoying to have to duplicate it in event body from client)
  • id (uuid, could be computed server side)
  • dt (if browser client doesn't support ISOString )

And there might be more.

Doing event augmentation in EventGate isn't great. EventGate is meant to be a dumb validator and router only, not an event transformer/processor. However, since some of this data is only available during the HTTP request to EventGate, EventGate will have to do some processing after all.

Also related:
T240460: Clients need to generate an ISO 8601 formatted timestamp
T240461: Clients may need to generate a UUID for sending events.

Event Timeline

Ok, to do this we need to standardize and bikeshed the client_ip and user_agent fields, since this is not yet in any official common schema.

Thus far for the two schemas that have needed user_agent, we've put it into the http.request_headers map field. I'd be ok with doing the same here for user agent.

For client_ip...should we put this into meta? @Pchelolo thoughts?

+1 to including UA in http.request_headers field

Hm, actually, we get X-Client-IP as a request header from varnish VCL right now. I guess that will continue to be the case in the ATS world? If so, we can also stick it in the http.request_headers map. @ema can we continue counting on the fact that all requests that go through a Varnish/ATS frontend will have the X-Client-IP request header set?

Oh, we already made a an http.client_ip field for api/request and cirrusearch/request. I guess that's where it goes! Hm. I guess we still have to get it from X-Client-IP though, right?

Change 556484 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Set WMF specific event field values if they are not set by client

https://gerrit.wikimedia.org/r/556484

Change 556484 merged by Ottomata:
[eventgate-wikimedia@master] Set WMF specific event field values if they are not set by client

https://gerrit.wikimedia.org/r/556484