Page MenuHomePhabricator

Include X-Client-IP in EventLogging data and geocode during Hive JSON Refinement
Closed, ResolvedPublic8 Story Points

Description

  • Send X-Client-IP from varnishkafka: %{X-Client-IP}o
  • EventLogging processors parse this into ip field: %{ip}i
  • EventLogging jrm.py (MySQL inserter) code blacklists ip field in NO_DB_PROPERTIES.
  • JSONRefine EventLogging job configured with geocode_ip --transform-functions that uses the analytics refinery geocode logic to add geocoded_data map to all EventLogging Hive tables.

(Tilman will be so happy.)

Details

Related Gerrit Patches:

Event Timeline

Ottomata triaged this task as Medium priority.Feb 8 2018, 9:27 PM
Ottomata created this task.
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Feb 9 2018, 2:48 PM

Change 409350 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventlogging@master] Add ip to EventCapsule, don't insert into MySQL

https://gerrit.wikimedia.org/r/409350

Change 409354 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] EventLogging: emit X-Client-IP and parse as ip field

https://gerrit.wikimedia.org/r/409354

Change 409350 merged by Ottomata:
[eventlogging@master] Add ip to EventCapsule, don't insert into MySQL

https://gerrit.wikimedia.org/r/409350

Change 407508 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] Add TransformFunctions for JsonRefine job

https://gerrit.wikimedia.org/r/407508

Mentioned in SAL (#wikimedia-operations) [2018-02-12T14:22:21Z] <otto@tin> Started deploy [eventlogging/analytics@01d5761]: T186833

Mentioned in SAL (#wikimedia-operations) [2018-02-12T14:22:35Z] <otto@tin> Finished deploy [eventlogging/analytics@01d5761]: T186833 (duration: 00m 04s)

Mentioned in SAL (#wikimedia-analytics) [2018-02-12T14:46:41Z] <ottomata> deploying eventlogging for T186833 with EventCapsule in code and IP NO_DB_PROPERTIES

Change 409354 merged by Ottomata:
[operations/puppet@production] EventLogging: emit X-Client-IP and parse as ip field

https://gerrit.wikimedia.org/r/409354

Mentioned in SAL (#wikimedia-operations) [2018-02-12T14:51:33Z] <ottomata> emitting IP field from varnishkafka-eventlogging instance T186833

Change 410240 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@jsonrefine] Add TransformFunctions for JsonRefine job

https://gerrit.wikimedia.org/r/410240

Change 407508 abandoned by Ottomata:
Add TransformFunctions for JsonRefine job

Reason:
in favor of https://gerrit.wikimedia.org/r/#/c/410240/

https://gerrit.wikimedia.org/r/407508

Change 410240 merged by Ottomata:
[analytics/refinery/source@jsonrefine] Add TransformFunctions for JsonRefine job

https://gerrit.wikimedia.org/r/410240

Ottomata updated the task description. (Show Details)Mar 6 2018, 8:41 PM

Change 417287 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Apply geocode and deuplicate transform function for refine jobs

https://gerrit.wikimedia.org/r/417287

Change 417287 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Apply geocode, deduplicate and monitoring for refine jobs

https://gerrit.wikimedia.org/r/417287

Change 417287 merged by Ottomata:
[operations/puppet@production] Apply geocode, deduplicate and monitoring for refine jobs

https://gerrit.wikimedia.org/r/417287

Change 418985 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add --queue opt default to production for Refine jobs

https://gerrit.wikimedia.org/r/418985

Change 418985 merged by Ottomata:
[operations/puppet@production] Add --queue opt default to production for Refine jobs

https://gerrit.wikimedia.org/r/418985

Nuria closed this task as Resolved.Mar 26 2018, 10:15 PM