Page MenuHomePhabricator
Create Task
Maniphest T186833

Include X-Client-IP in EventLogging data and geocode during Hive JSON Refinement
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

  • Send X-Client-IP from varnishkafka: %{X-Client-IP}o
  • EventLogging processors parse this into ip field: %{ip}i
  • EventLogging jrm.py (MySQL inserter) code blacklists ip field in NO_DB_PROPERTIES.
  • JSONRefine EventLogging job configured with geocode_ip --transform-functions that uses the analytics refinery geocode logic to add geocoded_data map to all EventLogging Hive tables.

(Tilman will be so happy.)

Details

SubjectRepoBranchLines +/-
Add --queue opt default to production for Refine jobsoperations/puppetproduction+3 -2
Apply geocode, deduplicate and monitoring for refine jobsoperations/puppetproduction+234 -180
Add TransformFunctions for JsonRefine jobanalytics/refinery/sourcejsonrefine+203 -29
Add TransformFunctions for JsonRefine jobanalytics/refinery/sourcemaster+203 -29
EventLogging: emit X-Client-IP and parse as `ip` fieldoperations/puppetproduction+2 -4
Add ip to EventCapsule, don't insert into MySQLeventloggingmaster+13 -4
Customize query in gerrit

Related Objects
Search...

Event Timeline

Ottomata triaged this task as Medium priority.Feb 8 2018, 9:27 PM
Ottomata created this task.
• Tbayer awarded a token.Feb 8 2018, 10:21 PM
Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.Feb 9 2018, 2:48 PM
gerritbot subscribed.Feb 9 2018, 3:19 PM

Change 409350 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[eventlogging@master] Add ip to EventCapsule, don't insert into MySQL

https://gerrit.wikimedia.org/r/409350

gerritbot added a project: Patch-For-Review.Feb 9 2018, 3:19 PM
gerritbot added a comment.Feb 9 2018, 3:32 PM

Change 409354 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] EventLogging: emit X-Client-IP and parse as ip field

https://gerrit.wikimedia.org/r/409354

gerritbot added a comment.Feb 9 2018, 3:47 PM

Change 409350 merged by Ottomata:
[eventlogging@master] Add ip to EventCapsule, don't insert into MySQL

https://gerrit.wikimedia.org/r/409350

Ottomata mentioned this in T186130: Hive EventLogging tables not updating since January 26.Feb 9 2018, 5:24 PM
gerritbot added a comment.Feb 9 2018, 7:29 PM

Change 407508 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] Add TransformFunctions for JsonRefine job

https://gerrit.wikimedia.org/r/407508

Stashbot subscribed.Feb 12 2018, 2:22 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T14:22:21Z] <otto@tin> Started deploy [eventlogging/analytics@01d5761]: T186833

Stashbot added a comment.Feb 12 2018, 2:22 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T14:22:35Z] <otto@tin> Finished deploy [eventlogging/analytics@01d5761]: T186833 (duration: 00m 04s)

Stashbot added a comment.Feb 12 2018, 2:46 PM

Mentioned in SAL (#wikimedia-analytics) [2018-02-12T14:46:41Z] <ottomata> deploying eventlogging for T186833 with EventCapsule in code and IP NO_DB_PROPERTIES

gerritbot added a comment.Feb 12 2018, 2:50 PM

Change 409354 merged by Ottomata:
[operations/puppet@production] EventLogging: emit X-Client-IP and parse as ip field

https://gerrit.wikimedia.org/r/409354

Stashbot added a comment.Feb 12 2018, 2:51 PM

Mentioned in SAL (#wikimedia-operations) [2018-02-12T14:51:33Z] <ottomata> emitting IP field from varnishkafka-eventlogging instance T186833

gerritbot added a comment.Feb 13 2018, 7:46 PM

Change 410240 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@jsonrefine] Add TransformFunctions for JsonRefine job

https://gerrit.wikimedia.org/r/410240

gerritbot added a comment.Feb 13 2018, 9:00 PM

Change 407508 abandoned by Ottomata:
Add TransformFunctions for JsonRefine job

Reason:
in favor of https://gerrit.wikimedia.org/r/#/c/410240/

https://gerrit.wikimedia.org/r/407508

Ottomata mentioned this in T187277: Should it be possible for a schema to override DNT in exceptional circumstances?.Feb 14 2018, 11:05 PM
gerritbot added a comment.Feb 15 2018, 2:44 PM

Change 410240 merged by Ottomata:
[analytics/refinery/source@jsonrefine] Add TransformFunctions for JsonRefine job

https://gerrit.wikimedia.org/r/410240

Ottomata moved this task from In Progress to In Code Review on the Analytics-Kanban board.Feb 21 2018, 5:08 PM
Ottomata updated the task description. (Show Details)Mar 6 2018, 8:41 PM
gerritbot added a comment.Mar 8 2018, 3:03 PM

Change 417287 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Apply geocode and deuplicate transform function for refine jobs

https://gerrit.wikimedia.org/r/417287

Ottomata moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Mar 8 2018, 3:08 PM
gerritbot added a comment.Mar 8 2018, 3:25 PM

Change 417287 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Apply geocode, deduplicate and monitoring for refine jobs

https://gerrit.wikimedia.org/r/417287

gerritbot added a comment.Mar 12 2018, 7:36 PM

Change 417287 merged by Ottomata:
[operations/puppet@production] Apply geocode, deduplicate and monitoring for refine jobs

https://gerrit.wikimedia.org/r/417287

gerritbot added a comment.Mar 12 2018, 7:50 PM

Change 418985 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] Add --queue opt default to production for Refine jobs

https://gerrit.wikimedia.org/r/418985

gerritbot added a comment.Mar 12 2018, 7:51 PM

Change 418985 merged by Ottomata:
[operations/puppet@production] Add --queue opt default to production for Refine jobs

https://gerrit.wikimedia.org/r/418985

Ottomata moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Mar 12 2018, 8:28 PM
• Nuria closed this task as Resolved.Mar 26 2018, 10:15 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits