Page MenuHomePhabricator

[EventGate] Perform GeoIP lookup at event ingestion time
Open, MediumPublic5 Estimated Story Points

Description

Background

Currently, the only way to perform GeoIP lookup on analytics events is to:

  1. Use a schema built with the fragment/http/client_ip schema fragment, which adds the http.client_ip field;
  2. Rely on Refine to perform the GeoIP lookup when it refines the event stream into the corresponding Hive table

Collecting IP addresses increases the risk of the data collection activity per the Data Collection Activity Guidelines. However, because the vast majority (all?) of geographic analysis of analytics events is done at the level of country, this increased risk is unnecessary.

TODO

  • Bikeshed the name of the agent_geo_ip_country contextual attribute

AC

  • When an instrument or experiment is configured to use the agent_geo_ip_country contextual attribute, then the field is set on the event
  • When an instrument or experiment sends an event with the agent.geo_ip_country value set to "awaiting", then the field is set on the event
  • The IP address isn't copied to the event

Notes

  1. The field is set on the event using the value of the X-Geo-IP-Country header

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
product_metrics/base 2.2.0repos/data-engineering/schemas-event-secondary!143bearlogaT436524master
Customize query in GitLab

Event Timeline

Hm...I wonder.

I recall something about geocoding happening on webrequests themselves.

For events that are submitted via the CDN (e.g. from external browsers/clients), it is possible the X-Geo-IP-Country header is set. If it is, you maybe be able to use the built in header hoisting config to pull it into the event, and never have to explicitly set client IP at all!

That is a really, really neat solution! I'll update the task description…

I've made this a parent task of T430541 as we will introduce a module to fetch instrument and experiment configs from Test Kitchen as part of that task.

JVanderhoop-WMF moved this task from Incoming to READY TO GROOM on the Test Kitchen board.
KReid-WMF set the point value for this task to 5.Thu, Aug 13, 3:51 PM