Page MenuHomePhabricator

Make provisions for geodata in all MEP schemas
Closed, ResolvedPublic

Description

To have geocoded_data available on MEP schemas, they have to ref the client_ip fragment. This can be done by updating our existing wikipedia_android_app fragment to include it.
Follow the discussion for more details:
https://wikimedia.slack.com/archives/CSV483812/p1661282090889679?thread_ts=1661281570.613779&cid=CSV483812

Event Timeline

Change 826373 had a related patch set uploaded (by Sharvaniharan; author: Bearloga):

[schemas/event/secondary@master] Retroactively add http.client_ip to Android schemas

https://gerrit.wikimedia.org/r/826373

Change 826373 merged by jenkins-bot:

[schemas/event/secondary@master] Retroactively add http.client_ip to Android schemas

https://gerrit.wikimedia.org/r/826373

After schema merge, I manually ran a Refine of android_notification_interaction to make sure it would notice the new fields and evolve the Hive table. It did!

sudo -u analytics kerberos-run-command analytics refine_event --table_include_regex=android_notification_interaction --since=4 --until=1

In yarn app logs:

ALTER TABLE `event`.`android_notification_interaction`
ADD COLUMNS (
`geocoded_data` MAP<STRING, STRING>
)
ALTER TABLE `event`.`android_notification_interaction`
CHANGE COLUMN `http` `http` STRUCT<`has_cookies`: BOOLEAN, `method`: STRING, `protocol`: STRING, `request_headers`: MAP<STRING, STRING>, `response_headers`: MAP<STRING, STRING>, `status_code`: BIGINT, `client_ip`: STRING>

I will bounce eventgate-analytics-external following instructions in just a few minutes.

Mentioned in SAL (#wikimedia-operations) [2022-08-25T18:33:33Z] <ottomata> rolling restart of eventgate-analytics-external to pick up retroactive schema change for android schemas in T316047

Done, and I confirmed that client_ip is now being set in the android.article_toolbar_interaction stream.

Thank you so much for your help and for verifying this @Ottomata ! 🎉