Page MenuHomePhabricator
Create Task
Maniphest T218164

Implementation and testing of glent m0 integration with wmf infra
Closed, ResolvedPublic
Actions

Description

Reimplement hive UDF from analytics-hive in spark:

  • get_main_search_request

Write tests for the handshake between wmf analytics infrastructure and glent

  • initSpark
  • readSparkDataset
  • writeDf
  • flattenWmfLog

Details

SubjectRepoBranchLines +/-
Reimplement GetMainSearchRequest UDF from refinery-hivesearch/glentmaster+258 -5
Customize query in gerrit

Event Timeline

EBernhardson created this task.Mar 12 2019, 10:11 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 12 2019, 10:11 PM
EBernhardson updated the task description. (Show Details)Mar 12 2019, 10:11 PM
EBernhardson added a project: Discovery-Search (Current work).
Jdforrester-WMF added a project: Analytics-Clusters.Mar 12 2019, 10:12 PM
Restricted Application added a project: Analytics. · View Herald TranscriptMar 12 2019, 10:12 PM
EBernhardson renamed this task from Impleentation and testing of glent m0 integration with wmf infra to Implementation and testing of glent m0 integration with wmf infra.Mar 12 2019, 10:22 PM
EBernhardson updated the task description. (Show Details)
gerritbot subscribed.Mar 14 2019, 4:17 PM

Change 496361 had a related patch set uploaded (by EBernhardson; owner: EBernhardson):
[search/glent@master] Reimplement GetMainSearchRequest UDF from refinery-hive

https://gerrit.wikimedia.org/r/496361

gerritbot added a project: Patch-For-Review.Mar 14 2019, 4:17 PM
gerritbot added a comment.Mar 18 2019, 9:29 AM

Change 496361 merged by jenkins-bot:
[search/glent@master] Reimplement GetMainSearchRequest UDF from refinery-hive

https://gerrit.wikimedia.org/r/496361

Milimetric moved this task from Incoming to Radar on the Analytics board.Mar 18 2019, 3:48 PM
EBernhardson moved this task from Incoming to Needs Reporting on the Discovery-Search (Current work) board.Apr 2 2019, 5:26 PM
debt closed this task as Resolved.Apr 5 2019, 10:33 PM
Aklapper edited projects, added Analytics-Radar; removed Analytics.Jun 10 2020, 6:44 AM
Maintenance_bot removed a project: Patch-For-Review.Jun 10 2020, 7:10 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits