Page MenuHomePhabricator
Create Task
Maniphest T327381

Migrate RDF Tooling to Spark 3
Closed, ResolvedPublic5 Estimated Story Points
Actions

Details

SubjectRepoBranchLines +/-
Upgrade rdf-spark-tools to 0.3.121 (spark 3)wikimedia/discovery/analyticsmaster+14 -32
Migrate to spark 3wikidata/query/rdfmaster+216 -152
Debugging tests that fail after upgrade to spark 3wikidata/query/rdfmaster+36 -52
Customize query in gerrit

Related Objects
Search...

Event Timeline

pfischer created this task.Jan 19 2023, 8:51 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 19 2023, 8:51 AM
pfischer changed the task status from Open to In Progress.Jan 19 2023, 8:52 AM
pfischer added a parent task: T322905: [EPIC] Upgrade Search Platform spark jobs to spark 3.
pfischer added a project: Discovery-Search.
pfischer edited projects, added Discovery-Search (Current work); removed Discovery-Search.Jan 23 2023, 4:20 PM
pfischer moved this task from Incoming to In Progress on the Discovery-Search (Current work) board.
gerritbot added a comment.Jan 23 2023, 4:57 PM

Change 882691 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[wikidata/query/rdf@master] Migrate to spark 3

https://gerrit.wikimedia.org/r/882691

gerritbot added a project: Patch-For-Review.Jan 23 2023, 4:57 PM
• MPhamWMF set the point value for this task to 5.Jan 30 2023, 4:28 PM
gerritbot added a comment.Feb 1 2023, 5:44 PM

Change 885848 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[wikidata/query/rdf@master] Debugging tests that fail after upgrade to spark 3

https://gerrit.wikimedia.org/r/885848

gerritbot added a comment.Feb 2 2023, 4:00 PM

Change 885848 abandoned by Peter Fischer:

[wikidata/query/rdf@master] Debugging tests that fail after upgrade to spark 3

Reason:

no longer needed, fixed in https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/882691

https://gerrit.wikimedia.org/r/885848

gerritbot added a comment.Feb 15 2023, 2:56 PM

Change 882691 merged by jenkins-bot:

[wikidata/query/rdf@master] Migrate to spark 3

https://gerrit.wikimedia.org/r/882691

Maintenance_bot removed a project: Patch-For-Review.Feb 15 2023, 3:10 PM
gerritbot added a comment.Feb 15 2023, 5:16 PM

Change 889579 had a related patch set uploaded (by Peter Fischer; author: Peter Fischer):

[wikimedia/discovery/analytics@master] Upgrade rdf-spark-tools to 0.3.121 (spark 3)

https://gerrit.wikimedia.org/r/889579

gerritbot added a project: Patch-For-Review.Feb 15 2023, 5:16 PM
EBernhardson merged a task: T329879: Migrate rdf_streaming_updater_reconcile.py from airflow 1 to airflow 2.Feb 16 2023, 6:56 PM
EBernhardson subscribed.
gerritbot added a comment.Feb 17 2023, 1:15 PM

Change 889579 merged by jenkins-bot:

[wikimedia/discovery/analytics@master] Upgrade rdf-spark-tools to 0.3.121 (spark 3)

https://gerrit.wikimedia.org/r/889579

pfischer mentioned this in rWDAN3a947658bd73: Upgrade rdf-spark-tools to 0.3.121 (spark 3).Feb 17 2023, 1:25 PM
Maintenance_bot removed a project: Patch-For-Review.Feb 17 2023, 1:29 PM
Stashbot added a comment.Feb 17 2023, 1:33 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-17T13:33:19Z] <pfischer@deploy1002> Started deploy [wikimedia/discovery/analytics@3a94765]: T327381: rdf-spark-tools update

Stashbot added a comment.Feb 17 2023, 1:35 PM

Mentioned in SAL (#wikimedia-operations) [2023-02-17T13:35:58Z] <pfischer@deploy1002> Finished deploy [wikimedia/discovery/analytics@3a94765]: T327381: rdf-spark-tools update (duration: 02m 39s)

pfischer moved this task from In Progress to Needs review on the Discovery-Search (Current work) board.Feb 20 2023, 3:58 PM
pfischer moved this task from Needs review to To Be Deployed on the Discovery-Search (Current work) board.
• dcausse moved this task from To Be Deployed to Needs Reporting on the Discovery-Search (Current work) board.Feb 20 2023, 4:08 PM
Gehel closed this task as Resolved.Feb 24 2023, 2:07 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits