Page MenuHomePhabricator
Create Task
Maniphest T321854

[Event Platform] Move Spark JsonSchemaConverter out of analytics/refinery/source and into wikimedia-event-utilities
Closed, ResolvedPublic
Actions

Description

In this patch as part of T310302, we created an abstracted interface for iterating over a JSONSchema and converting it into a different (Java based) type system. We used this to convert from JSONSchema to Flink's DataStream Row and Table API schema type systems.

This was based on our implementation of the same code in analytics/refinery/source in the Spark JsonSchemaConverter.

We should remove the Spark specific JsonSchemaConverter and instead implement the data type conversions interface for Spark, and put that into wikimedia-event-utilties. analytics/refinery/source can then use the JsonSchemaConverter code from wikimedia-event-utilities.

Details

SubjectRepoBranchLines +/-
Refinery job - bump jar versions for refine and test refineoperations/puppetproduction+2 -2
Bump eventutilities version to 1.3.2analytics/refinery/sourcemaster+5 -6
Fix JsonSchemaConverter Draft 3 required bugwikimedia-event-utilitiesmaster+32 -6
refine - bump to refinery version 0.2.25 to pick up JsonSchemaConverter changeoperations/puppetproduction+1 -1
test/refine - update refinery jar version for analytics test cluster refine joboperations/puppetproduction+1 -1
Use eventutilities-spark JsonSchemaSparkConverter in Refine and elsewhereanalytics/refinery/sourcemaster+242 -672
Customize query in gerrit

Related Objects

Event Timeline

Ottomata created this task.Oct 27 2022, 6:46 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 27 2022, 6:46 PM
Ottomata added a comment.Oct 27 2022, 6:48 PM

Along the way , we could consider implementing T278467: Use Hive/Spark timestamps in Refined event data.

Using the migrated converter would then require some manual Hive table migrations.

Ottomata added a subscriber: dcausse.Oct 27 2022, 6:48 PM
• EChetty edited projects, added Data-Engineering-Planning; removed Data-Engineering.Nov 7 2022, 10:27 AM
• EChetty moved this task from Backlog to Event Platform on the Data-Engineering-Planning board.Nov 7 2022, 10:31 AM
Milimetric subscribed.Jan 30 2023, 3:46 PM

Oooh, I'd love to work on this

Ottomata added a comment.Jan 30 2023, 3:52 PM

Patch away my friend! The code will be the easy/fun part. T278467: Use Hive/Spark timestamps in Refined event data would be really nice...but changes the way Hive tables auto-ingested, so requires a more careful migration. We could do that separately.

Ottomata mentioned this in T278467: Use Hive/Spark timestamps in Refined event data.May 31 2023, 2:45 PM
Ottomata mentioned this in T309717: Event Utilities partially downloads schemas.Jun 27 2023, 2:02 PM
gerritbot added a comment.Jun 27 2023, 8:24 PM

Change 933620 had a related patch set uploaded (by Ottomata; author: Ottomata):

[wikimedia-event-utilities@master] [WIP] Add eventutilities-spark module and implement schema conversions

https://gerrit.wikimedia.org/r/933620

gerritbot added a project: Patch-For-Review.Jun 27 2023, 8:24 PM
Ottomata added a subscriber: xcollazo.EditedJun 27 2023, 8:28 PM

WIP patch for doing this ^.

This does not implement conversion to the TimestampType, although perhaps it should? I realized that we won't really be able to use this until we are 100% off of old metawiki EventLogging schemas. Old schemas are JSONSchema Draft-3, which uses a different required field format. refinery-spark JsonSchemaConverter supports both. eventutilities-core JsonSchemaConverter only supports Draft-7 onward.

Alternatively we could implement Draft-3 required field support in eventutilities-core JsonSchemaConverter and do this soon.

Ottomata added a comment.Jun 28 2023, 9:22 PM

could implement Draft-3 required field support in eventutilities-core JsonSchemaConverter

Done in patch. This will allow us to use this class in refinery-source, and delete the one there.

This does not implement conversion to the TimestampType, although perhaps it should?

Also done in patch, but in a parameterized way. When we are ready for T278467, we'll be able to test Refine using TimestampTypes via a parameter instead of having to upgrade eventutilities in refinery-source.

JArguello-WMF removed a project: Data-Engineering-Planning.Jun 29 2023, 9:48 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptJun 29 2023, 9:48 PM
JArguello-WMF moved this task from Incoming (new tickets) to Event Platform Backlog on the Data-Engineering board.Jun 29 2023, 10:22 PM
JArguello-WMF added a project: Data Engineering and Event Platform Team.Jun 30 2023, 4:29 PM
JArguello-WMF moved this task from Data Eng Backlog to Event Platform Backlog on the Data Engineering and Event Platform Team board.Jun 30 2023, 4:38 PM
gerritbot added a comment.Jul 7 2023, 3:06 PM

Change 936293 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery/source@master] Use eventutilities-spark JsonSchemaSparkConverter

https://gerrit.wikimedia.org/r/936293

Ottomata mentioned this in T349121: [metrics platform] Event sanitization fails when processing monoschema events.Oct 17 2023, 5:28 PM
lbowmaker moved this task from Event Platform Backlog to Event Platform Maintenance (current quarter) on the Data Engineering and Event Platform Team board.Oct 19 2023, 8:01 PM
Ahoelzl renamed this task from Move Spark JsonSchemaConverter out of analytics/refinery/source and into wikimedia-event-utilities to [Event Platform] Move Spark JsonSchemaConverter out of analytics/refinery/source and into wikimedia-event-utilities.Oct 20 2023, 5:29 PM
Ottomata triaged this task as Medium priority.Oct 23 2023, 3:10 PM
Ottomata moved this task from Event Platform Maintenance (current quarter) to Sprint 4 on the Data Engineering and Event Platform Team board.Oct 25 2023, 7:37 PM
Ottomata edited projects, added Data Engineering and Event Platform Team (Sprint 4); removed Data Engineering and Event Platform Team.
Ottomata claimed this task.Oct 30 2023, 4:50 PM
Ottomata moved this task from Next Up to In Review on the Data Engineering and Event Platform Team (Sprint 4) board.
gerritbot added a comment.Nov 8 2023, 2:20 PM

Change 936293 merged by jenkins-bot:

[analytics/refinery/source@master] Use eventutilities-spark JsonSchemaSparkConverter in Refine and elsewhere

https://gerrit.wikimedia.org/r/936293

gerritbot added a comment.Nov 8 2023, 9:24 PM

Change 972894 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] test/refine - update refinery jar version for analytics test cluster refine job

https://gerrit.wikimedia.org/r/972894

Ottomata moved this task from In Review to Ready to Deploy on the Data Engineering and Event Platform Team (Sprint 4) board.Nov 8 2023, 9:34 PM
gerritbot added a comment.Nov 9 2023, 2:41 PM

Change 972894 merged by Ottomata:

[operations/puppet@production] test/refine - update refinery jar version for analytics test cluster refine job

https://gerrit.wikimedia.org/r/972894

Ottomata removed Ottomata as the assignee of this task.Nov 9 2023, 2:58 PM

This is deployed for Refine in the analytics-test hadoop cluster. Ran a refine job there and it worked, also ran EvolveHiveTable to manually create a test table, and it worked just fine!

I'll deploy this for production Refine on Monday.

Ottomata claimed this task.Nov 9 2023, 5:14 PM
lbowmaker edited projects, added Data-Engineering (Sprint 5); removed Data Engineering and Event Platform Team (Sprint 4), Data-Engineering.Nov 10 2023, 1:51 PM
Restricted Application edited projects, added Data-Engineering; removed Data-Engineering (Sprint 5). · View Herald TranscriptNov 10 2023, 1:51 PM
lbowmaker moved this task from Event Platform Backlog to Sprint 5 on the Data-Engineering board.Nov 10 2023, 1:52 PM
lbowmaker edited projects, added Data-Engineering (Sprint 5); removed Data-Engineering.
Restricted Application edited projects, added Data-Engineering; removed Data-Engineering (Sprint 5). · View Herald TranscriptNov 10 2023, 1:52 PM
lbowmaker edited projects, added Data-Engineering (Sprint 5); removed Data-Engineering.Nov 10 2023, 1:55 PM
Restricted Application edited projects, added Data-Engineering; removed Data-Engineering (Sprint 5). · View Herald TranscriptNov 10 2023, 1:55 PM
lbowmaker moved this task from Incoming (new tickets) to Sprint 5 on the Data-Engineering board.Nov 10 2023, 2:04 PM
lbowmaker edited projects, added Data-Engineering (Sprint 5); removed Data-Engineering.
lbowmaker moved this task from Next Up to Ready to Deploy on the Data-Engineering (Sprint 5) board.
gerritbot added a comment.Nov 13 2023, 4:52 PM

Change 973837 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] refine - bump to refinery version 0.2.25 to pick up JsonSchemaConverter change

https://gerrit.wikimedia.org/r/973837

Stashbot added a comment.Nov 13 2023, 4:57 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-13T16:57:58Z] <otto@deploy2002> Started deploy [analytics/refinery@25ef91f]: deploying refinery with refinery-source 0.2.25 jars for T321854 [analytics/refinery@25ef91f2]

Stashbot added a comment.Nov 13 2023, 5:04 PM

Mentioned in SAL (#wikimedia-operations) [2023-11-13T17:04:34Z] <otto@deploy2002> Finished deploy [analytics/refinery@25ef91f]: deploying refinery with refinery-source 0.2.25 jars for T321854 [analytics/refinery@25ef91f2] (duration: 06m 36s)

Stashbot added a comment.Nov 13 2023, 5:07 PM

Mentioned in SAL (#wikimedia-analytics) [2023-11-13T17:07:55Z] <ottomata> deploying refinery with refinery source 0.2.25 jars and using 0.2.25 for refine job - T321854

gerritbot added a comment.Nov 13 2023, 5:33 PM

Change 973837 merged by Ottomata:

[operations/puppet@production] refine - bump to refinery version 0.2.25 to pick up JsonSchemaConverter change

https://gerrit.wikimedia.org/r/973837

Ottomata moved this task from Ready to Deploy to Done on the Data-Engineering (Sprint 5) board.Nov 13 2023, 5:38 PM
gerritbot added a comment.Nov 13 2023, 8:35 PM

Change 973868 had a related patch set uploaded (by Ottomata; author: Ottomata):

[wikimedia-event-utilities@master] Fix JsonSchemaConverter Draft 3 required bug

https://gerrit.wikimedia.org/r/973868

Ottomata added a comment.Nov 13 2023, 8:42 PM

Reverting deployment for production refine jobs. There was an edge case bug for old EventLogging Draft-3 JSONSchemas where an object node that marked itself as required fails conversion.

Fix for bug (with tests) here: https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/973868

Will need to release eventutilities and refinery-source, and deploy to apply bug fix.

Ottomata moved this task from Done to In Review on the Data-Engineering (Sprint 5) board.Nov 13 2023, 8:58 PM
gerritbot added a comment.Nov 14 2023, 1:56 PM

Change 973868 merged by Ottomata:

[wikimedia-event-utilities@master] Fix JsonSchemaConverter Draft 3 required bug

https://gerrit.wikimedia.org/r/973868

gerritbot added a comment.Nov 14 2023, 5:54 PM

Change 974246 had a related patch set uploaded (by Ottomata; author: Ottomata):

[analytics/refinery/source@master] Bump eventutilities version to 1.3.2

https://gerrit.wikimedia.org/r/974246

gerritbot added a comment.Nov 14 2023, 6:25 PM

Change 974246 merged by jenkins-bot:

[analytics/refinery/source@master] Bump eventutilities version to 1.3.2

https://gerrit.wikimedia.org/r/974246

gerritbot added a comment.Nov 15 2023, 2:40 PM

Change 974545 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] Refinery job - bump jar versions for refine, test refine, and producecanaryevents

https://gerrit.wikimedia.org/r/974545

gerritbot added a comment.Nov 15 2023, 2:43 PM

Change 974545 merged by Ottomata:

[operations/puppet@production] Refinery job - bump jar versions for refine and test refine

https://gerrit.wikimedia.org/r/974545

Stashbot added a comment.Nov 15 2023, 2:51 PM

Mentioned in SAL (#wikimedia-analytics) [2023-11-15T14:51:12Z] <ottomata> deployed refine using refinery-job 0.2.26 JsonSchemaConverter from wikimedia-event-utilities - https://phabricator.wikimedia.org/T321854

Ottomata moved this task from In Review to Done on the Data-Engineering (Sprint 5) board.Nov 15 2023, 2:59 PM
lbowmaker closed this task as Resolved.Dec 1 2023, 1:03 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL