Lookout for duplicates in EL refine, implement pluggable transform method config in JSONRefine
Closed, ResolvedPublic8 Estimated Story Points
Actions

Assigned To

Authored By

	• Nuria
	Jan 18 2018, 6:11 PM

Description

Lookout for duplicates in EL refine similar to how we remove possible kafka duplicates in webrequest

Details

Subject	Repo	Branch	Lines +/-
Apply geocode, deduplicate and monitoring for refine jobs	operations/puppet	production	+234 -180
Add TransformFunctions for JsonRefine job	analytics/refinery/source	jsonrefine	+203 -29
Add TransformFunctions for JsonRefine job	analytics/refinery/source	master	+203 -29
Add configurable transform function to JSONRefine	analytics/refinery/source	master	+146 -37

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Ottomata	T159170 Sunset MySQL data store for eventlogging
Resolved	Ottomata	T162610 Implement EventLogging Hive refinement
Resolved	Ottomata	T153328 Research Spike: Better support for Eventlogging data on hive
Resolved	JAllemandou	T161924 Write Spark schema differ / Hive DDL generator
Resolved	mforns	T166414 Explore NavigationTiming by faceted properties - EventLogging refine
Duplicate	mforns	T176426 Implement purging scheme for eventlogging data on top of eventlogging refine
Resolved	Ottomata	T178440 Refine should parse user agent field as it is done on refinery pipeline
Resolved	Ottomata	T179540 Timestamp format in Hive-refined EventLogging tables is incompatible with MySQL version
Resolved	Ottomata	T179625 Resolve EventCapsule / MySQL / Hive schema discrepancies
Resolved	mforns	T181064 Sanitize Hive EventLogging
Resolved	Ottomata	T185237 Lookout for duplicates in EL refine, implement pluggable transform method config in JSONRefine

Event Timeline

• Nuria triaged this task as Medium priority.Jan 18 2018, 6:11 PM

• Nuria created this task.

• Nuria edited projects, added Analytics; removed Analytics-Kanban.

elukey subscribed.Jan 18 2018, 6:12 PM

Ottomata claimed this task.Jan 18 2018, 6:19 PM

Ottomata renamed this task from Lookout for duplicates in EL refine to Lookout for duplicates in EL refine, implement pluggable transform method config in JSONRefine.Jan 22 2018, 4:03 PM

Ottomata added a parent task: T181064: Sanitize Hive EventLogging .

Ottomata changed the point value for this task from 0 to 8.

Ottomata edited projects, added Analytics-Kanban; removed Analytics.

Ottomata moved this task from Next Up to In Progress on the Analytics-Kanban board.

Change 405800 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] [WIP] Add configurable transform function to JSONRefine

https://gerrit.wikimedia.org/r/405800

gerritbot added a project: Patch-For-Review.Jan 22 2018, 9:21 PM

Change 407508 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@master] [WIP] Add TransformFunctions for JsonRefine job

https://gerrit.wikimedia.org/r/407508

Change 405800 merged by Ottomata:
[analytics/refinery/source@master] Add configurable transform function to JSONRefine

https://gerrit.wikimedia.org/r/405800

Change 410240 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[analytics/refinery/source@jsonrefine] Add TransformFunctions for JsonRefine job

https://gerrit.wikimedia.org/r/410240

Change 407508 abandoned by Ottomata:
Add TransformFunctions for JsonRefine job

Reason:
in favor of https://gerrit.wikimedia.org/r/#/c/410240/

https://gerrit.wikimedia.org/r/407508

Change 410240 merged by Ottomata:
[analytics/refinery/source@jsonrefine] Add TransformFunctions for JsonRefine job

https://gerrit.wikimedia.org/r/410240

Ottomata moved this task from In Progress to In Code Review on the Analytics-Kanban board.Feb 21 2018, 5:08 PM

Change 417287 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Apply geocode and deuplicate transform function for refine jobs

https://gerrit.wikimedia.org/r/417287

Ottomata moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Mar 8 2018, 3:08 PM

Change 417287 had a related patch set uploaded (by Ottomata; owner: Ottomata):
[operations/puppet@production] [WIP] Apply geocode, deduplicate and monitoring for refine jobs

https://gerrit.wikimedia.org/r/417287

Change 417287 merged by Ottomata:
[operations/puppet@production] Apply geocode, deduplicate and monitoring for refine jobs

https://gerrit.wikimedia.org/r/417287

Ottomata moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Mar 12 2018, 8:28 PM

Ottomata closed this task as Resolved.Mar 28 2018, 9:00 PM