Page MenuHomePhabricator
Create Task
Maniphest T287121

Add geolocation information to Growth schemas
Closed, ResolvedPublic
Actions

Description

In the fiscal year that has just started, the Growth team's focus will be on the Newcomer Experience Pilots. This work has a geographic component to it, we're going to be interested in analyzing newcomer behaviour and usage of Growth team features on a per-country basis. To enable that analysis, we'll be needing to add geolocation information in the schemas the team uses (this was removed as part of the migration process for legacy schemas, and never added to Event Platform schemas). The team has a mix of legacy and Event Platform schemas that we need this added to.

Legacy schemas:

Event Platform schemas:

We are not looking to retain any geolocation data beyond 90 days, as none of the schemas have a structure that would allow for that. And by the way, the only schema of the ones listed that are on the allowlist is ServerSideAccountCreation.

The team also maintains the legacy NewcomerTask schema, but that schema does not need this information as it's only used to store tasks-specific information to be connected with HomepageModule, HelpPanel, and link_suggestion_interaction.

Details

SubjectRepoBranchLines +/-
Use serversideaccountcreation schema version 1.1.0 (client_ip added)mediawiki/extensions/Campaignsmaster+1 -1
Use updated schemas with client_ipmediawiki/extensions/GrowthExperimentsmaster+3 -3
Add client_ip to Growth schemasschemas/event/secondarymaster+2 K -12
Add client_ip to serversideaccountcreation schemaschemas/event/secondarymaster+456 -4
Add a link: Update schema to support edit mode and link inspector toggles; add client_ipschemas/event/secondarymaster+435 -5
Customize query in gerrit

Related Objects

Event Timeline

nettrom_WMF created this task.Jul 21 2021, 8:55 PM
nettrom_WMF added a subscriber: Ottomata.

From what I've been able to find, this is the first time this has been requested, and so I'm unsure what exactly to ask for and how to do this. I'm hoping someone from Data Engineering can help us out. I suspect @Ottomata is the right person to tag first, so adding him here.

Ottomata added a comment.Jul 22 2021, 2:04 PM

If you add the http.client_ip field to your schemas, EventGate will automatically populate it. If this field exists, the Hive Refine step will then automatically add the geocoded data field to the Hive table.

https://wikitech.wikimedia.org/wiki/Event_Platform/Schemas/Guidelines#http_information

These schemas are all migrated to Event Platform, so the steps are the same for each of them. :)

Note that there is a fragment/http/client_ip schema that should be $refed to add this field. See other schemas that $ref client_ip, like virtualpageview.

• mewoph edited projects, added Growth-Team (Sprint 0 (Growth Team)); removed Growth-Team.Jul 22 2021, 9:16 PM
• mewoph moved this task from Incoming to Ready for Development on the Growth-Team (Sprint 0 (Growth Team)) board.
odimitrijevic edited projects, added Analytics-Radar; removed Analytics.Jul 26 2021, 3:46 PM
nettrom_WMF mentioned this in T287210: EchoMail and EchoInteraction Event Platform Migration.Jul 26 2021, 9:14 PM
ldelench_wmf moved this task from Triage to Tracking on the Product-Analytics board.Jul 27 2021, 5:15 PM
nettrom_WMF added a subscriber: • mewoph.Jul 29 2021, 11:27 PM

@mewoph : Can we add the reference to client_ip to https://gerrit.wikimedia.org/r/c/schemas/event/secondary/+/704402 so we also get that done and only have one version change to the schema? Based on the code for VirtualPageview that @Ottomata mentions above, all that should be needed is to add - $ref: /fragment/http/client_ip/1.0.0# to the referenced fragments.

I'm not great at Gerrit, so I'm unsure how I'd go about adding it to that patch. If you give or point me to some instructions, I can probably do it, though.

• mewoph added a comment.Aug 2 2021, 3:28 PM

Hi @nettrom_WMF — I can add it to the existing patch

gerritbot added a comment.Aug 2 2021, 3:37 PM

Change 704402 had a related patch set uploaded (by MewOphaswongse; author: MewOphaswongse):

[schemas/event/secondary@master] Add a link: Update schema to support edit mode and link inspector toggles; add client_ip

https://gerrit.wikimedia.org/r/704402

gerritbot added a project: Patch-For-Review.Aug 2 2021, 3:37 PM
gerritbot added a comment.Aug 3 2021, 6:23 AM

Change 704402 merged by jenkins-bot:

[schemas/event/secondary@master] Add a link: Update schema to support edit mode and link inspector toggles; add client_ip

https://gerrit.wikimedia.org/r/704402

Maintenance_bot removed a project: Patch-For-Review.Aug 3 2021, 7:10 AM
• mewoph claimed this task.Aug 5 2021, 4:24 PM
• mewoph updated the task description. (Show Details)
• mewoph moved this task from Ready for Development to In Progress on the Growth-Team (Sprint 0 (Growth Team)) board.
gerritbot added a comment.Aug 5 2021, 5:20 PM

Change 710310 had a related patch set uploaded (by MewOphaswongse; author: MewOphaswongse):

[schemas/event/secondary@master] Add client_ip to Growth schemas

https://gerrit.wikimedia.org/r/710310

gerritbot added a project: Patch-For-Review.Aug 5 2021, 5:20 PM
gerritbot added a comment.Aug 5 2021, 5:22 PM

Change 710311 had a related patch set uploaded (by MewOphaswongse; author: MewOphaswongse):

[mediawiki/extensions/GrowthExperiments@master] Use updated schemas with client_ip

https://gerrit.wikimedia.org/r/710311

• mewoph added a comment.EditedAug 5 2021, 5:24 PM

Sample payload from updated schemas

HomepageVisit

"http": {
  "request_headers": {
    "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1"
  },
  "client_ip": "127.0.0.1"
},
"schema": "HomepageVisit",
"wiki": "my_wiki",
"webHost": "localhost:8080",
"$schema": "/analytics/legacy/homepagevisit/1.2.0",
"client_dt": "2021-08-05T17:11:51Z"

HomepageModule

"$schema": "/analytics/legacy/homepagemodule/1.3.0",
"client_dt": "2021-08-05T17:02:43.108Z",
"meta": {
  "stream": "eventlogging_HomepageModule",
  "domain": "localhost",
  "id": "a876ef03-b2d0-414b-a496-ff6a4d8b5774",
  "dt": "2021-08-05T17:02:44.116Z",
  "request_id": "f38ef720-f60e-11eb-8e57-dd5d9374b9c7"
},
"http": {
  "client_ip": "127.0.0.1",
  "request_headers": {
    "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1"
  }

HelpPanel

"$schema": "/analytics/legacy/helppanel/1.1.0",
"client_dt": "2021-08-05T17:04:23.849Z",
"meta": {
  "stream": "eventlogging_HelpPanel",
  "domain": "localhost",
  "id": "b8d0b7a0-dbe7-43ce-b92f-88bc72e1685b",
  "dt": "2021-08-05T17:04:24.914Z",
  "request_id": "2f9a5d40-f60f-11eb-905b-ba280dc5deb6"
},
"http": {
  "client_ip": "127.0.0.1",
  "request_headers": {
    "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1"
  }

ServerSideAccountCreation

event: {
  "meta": {
    "domain": "localhost",
    "stream": "eventlogging_ServerSideAccountCreation",
    "id": "7d2d859c-f535-4bb1-8a85-598423fad779",
    "dt": "2021-08-05T20:12:50.405Z",
    "request_id": "82137f60-f629-11eb-b3e5-dce138bec23e"
  },
  "http": {
    "request_headers": {
      "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36"
    },
    "client_ip": "127.0.0.1"
  },
  "event": {
    "userId": 25,
    "userName": "MewCampaign1",
    "isSelfMade": true,
    "campaign": "testCampaign",
    "displayMobile": false,
    "token": "",
    "userBuckets": "",
    "isApi": false
  },
  "schema": "ServerSideAccountCreation",
  "wiki": "my_wiki",
  "webHost": "localhost:8080",
  "$schema": "/analytics/legacy/serversideaccountcreation/1.1.0",
  "client_dt": "2021-08-05T20:12:50Z"
}
gerritbot added a comment.Aug 5 2021, 8:15 PM

Change 710349 had a related patch set uploaded (by MewOphaswongse; author: MewOphaswongse):

[schemas/event/secondary@master] Add client_ip to serversideaccountcreation schema

https://gerrit.wikimedia.org/r/710349

gerritbot added a comment.Aug 5 2021, 8:16 PM

Change 710350 had a related patch set uploaded (by MewOphaswongse; author: MewOphaswongse):

[mediawiki/extensions/Campaigns@master] Use serversideaccountcreation schema version 1.1.0 (client_ip added)

https://gerrit.wikimedia.org/r/710350

• mewoph moved this task from In Progress to Code Review on the Growth-Team (Sprint 0 (Growth Team)) board.Aug 5 2021, 8:18 PM
gerritbot added a comment.Aug 5 2021, 11:34 PM

Change 710349 merged by jenkins-bot:

[schemas/event/secondary@master] Add client_ip to serversideaccountcreation schema

https://gerrit.wikimedia.org/r/710349

Tgr moved this task from Code Review to QA on the Growth-Team (Sprint 0 (Growth Team)) board.Aug 5 2021, 11:45 PM
gerritbot added a comment.Aug 5 2021, 11:45 PM

Change 710310 merged by jenkins-bot:

[schemas/event/secondary@master] Add client_ip to Growth schemas

https://gerrit.wikimedia.org/r/710310

gerritbot added a comment.Aug 6 2021, 2:11 AM

Change 710311 merged by jenkins-bot:

[mediawiki/extensions/GrowthExperiments@master] Use updated schemas with client_ip

https://gerrit.wikimedia.org/r/710311

ReleaseTaggerBot added a project: MW-1.37-notes (1.37.0-wmf.18; 2021-08-09).Aug 6 2021, 3:00 AM
gerritbot added a comment.Aug 6 2021, 8:32 AM

Change 710350 merged by jenkins-bot:

[mediawiki/extensions/Campaigns@master] Use serversideaccountcreation schema version 1.1.0 (client_ip added)

https://gerrit.wikimedia.org/r/710350

Etonkovidova closed this task as Resolved.Aug 12 2021, 11:57 PM
Etonkovidova subscribed.

Checked in production (wmf.18)

    • schemas HomepageModule (/analytics/legacy/homepagemodule/1.3.0) and HelpPanel (/analytics/legacy/helppanel/1.1.0) have updated versions
  • the logstash doesn't records any errors for the schemas
nettrom_WMF mentioned this in T288853: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 .Aug 13 2021, 6:38 PM
Tgr added subscribers: Krinkle, Tgr.Aug 25 2021, 8:29 AM

@Krinkle pointed out on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventBus/+/713526 that collecting IPs when we only need high-level geolocation is unnecessarily privacy-invasive. It's possible to collect geodata manually, but adding that for five different schemas is a bit of a chore - @Ottomata is there any chance that will be supported in the foreseeable future? (Ie. a fragment similar to #client_ip that results in the geolocation data, but not the IP, being automatically added in the refine step)

Ottomata added a comment.Aug 30 2021, 5:05 PM

I guess that would be T290014: Source geolocation directly rather than using IP in schema? I'll comment over there.

Ottomata mentioned this in T290014: Source geolocation directly rather than using IP in schema.Aug 30 2021, 5:08 PM
MMiller_WMF mentioned this in T291016: Quick country-level analysis on homepage.Sep 14 2021, 10:03 PM
MNeisler mentioned this in T314178: Add geolocation information to EditAttemptStep.Jul 29 2022, 7:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits