Page MenuHomePhabricator

eventgate-wikimedia should fill in defaults for some important fields
Open, Needs TriagePublic

Description

Some fields we want set are either not available for clients to manually set in event data, or are inconvenient to do so:

  • client_ip (needed for downstream geocoding)
  • user_agent (already sent in HTTP header, annoying to have to duplicate it in event body from client)
  • id (uuid, could be computed server side)
  • dt (if browser client doesn't support ISOString )

And there might be more.

Doing event augmentation in EventGate isn't great. EventGate is meant to be a dumb validator and router only, not an event transformer/processor. However, since some of this data is only available during the HTTP request to EventGate, EventGate will have to do some processing after all.

Also related:
T240460: Clients need to generate an ISO 8601 formatted timestamp
T240461: Clients may need to generate a UUID for sending events.

Details

Related Gerrit Patches:

Event Timeline

Ottomata created this task.Dec 11 2019, 4:33 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptDec 11 2019, 4:33 PM
Ottomata added a comment.EditedDec 11 2019, 8:00 PM

Ok, to do this we need to standardize and bikeshed the client_ip and user_agent fields, since this is not yet in any official common schema.

Thus far for the two schemas that have needed user_agent, we've put it into the http.request_headers map field. I'd be ok with doing the same here for user agent.

For client_ip...should we put this into meta? @Pchelolo thoughts?

Nuria added a comment.Dec 11 2019, 8:05 PM

+1 to including UA in http.request_headers field

Ottomata added a subscriber: ema.Dec 11 2019, 8:11 PM

Hm, actually, we get X-Client-IP as a request header from varnish VCL right now. I guess that will continue to be the case in the ATS world? If so, we can also stick it in the http.request_headers map. @ema can we continue counting on the fact that all requests that go through a Varnish/ATS frontend will have the X-Client-IP request header set?

Oh, we already made a an http.client_ip field for api/request and cirrusearch/request. I guess that's where it goes! Hm. I guess we still have to get it from X-Client-IP though, right?

Change 556484 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventgate-wikimedia@master] Set WMF specific event field values if they are not set by client

https://gerrit.wikimedia.org/r/556484

Change 556484 merged by Ottomata:
[eventgate-wikimedia@master] Set WMF specific event field values if they are not set by client

https://gerrit.wikimedia.org/r/556484

Ottomata moved this task from Backlog to Done on the Event-Platform board.Dec 17 2019, 9:01 PM
fdans moved this task from Incoming to Event Platform on the Analytics board.Dec 23 2019, 5:04 PM