Page MenuHomePhabricator
Create Task
Maniphest T193169

[reportupdater] Add a configurable hive client
Closed, ResolvedPublic
Actions

Description

Today, if you want reportupdater to query hive, you have to use a script which calls hive/beeline from the command line.
It would be better if you could have a type=hive that lets you write plain (templated) hql and reportupdater would connect to hive for you.

Details

Show related patches Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 DeclinedNoneT193167 reportupdater TLC
 ResolvedNoneT193169 [reportupdater] Add a configurable hive client

Event Timeline

mforns created this task.Apr 26 2018, 2:58 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 26 2018, 2:58 PM
mforns added a parent task: T193167: reportupdater TLC.Apr 26 2018, 2:58 PM
mforns triaged this task as Unbreak Now! priority.Apr 26 2018, 4:32 PM
mforns moved this task from Incoming to Smart Tools for Better Data on the Analytics board.
Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptApr 26 2018, 4:32 PM
mforns lowered the priority of this task from Unbreak Now! to Needs Triage.Apr 26 2018, 4:33 PM
mforns triaged this task as Medium priority.Apr 30 2018, 4:47 PM
• Vvjjkkii renamed this task from [reportupdater] Add a configurable hive client to i5daaaaaaa.Jul 1 2018, 1:13 AM
• Vvjjkkii raised the priority of this task from Medium to High.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed a subscriber: Aklapper.
AfroThundr3007730 renamed this task from i5daaaaaaa to [reportupdater] Add a configurable hive client.Jul 1 2018, 7:06 AM
AfroThundr3007730 lowered the priority of this task from High to Medium.
AfroThundr3007730 removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
AfroThundr3007730 updated the task description. (Show Details)
AfroThundr3007730 added a subscriber: Aklapper.
mforns raised the priority of this task from Medium to Needs Triage.Mar 25 2019, 5:33 PM
mforns triaged this task as Medium priority.
mforns raised the priority of this task from Medium to Needs Triage.Dec 9 2019, 5:46 PM
mforns triaged this task as Medium priority.
awight subscribed.Feb 26 2021, 12:30 PM
gerritbot added a comment.Feb 26 2021, 4:04 PM

Change 667192 had a related patch set uploaded (by Awight; owner: Awight):
[analytics/reportupdater@master] [WIP] Support explicit "hive" script type

https://gerrit.wikimedia.org/r/667192

gerritbot added a project: Patch-For-Review.Feb 26 2021, 4:04 PM
awight claimed this task.Feb 26 2021, 4:58 PM
awight added projects: WMDE-TechWish-Sprint-2021-02-17, Unplanned-Sprint-Work.
awight moved this task from Sprint Backlog to Doing on the WMDE-TechWish-Sprint-2021-02-17 board.
awight removed awight as the assignee of this task.Feb 26 2021, 6:06 PM
awight moved this task from Doing to Watching (Stalled) on the WMDE-TechWish-Sprint-2021-02-17 board.
gerritbot added a comment.Mar 29 2021, 7:43 PM

Change 667192 merged by Mforns:
[analytics/reportupdater@master] Support explicit "hive" script type

https://gerrit.wikimedia.org/r/667192

Maintenance_bot removed a project: Patch-For-Review.Mar 29 2021, 8:11 PM
gerritbot added a comment.Apr 1 2021, 10:38 AM

Change 676299 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater-queries@master] Validate the native "hive" report type

https://gerrit.wikimedia.org/r/676299

gerritbot added a project: Patch-For-Review.Apr 1 2021, 10:38 AM
gerritbot added a comment.Apr 7 2021, 10:21 AM

Change 677511 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater@master] Fix date range parameters for native hive scripts

https://gerrit.wikimedia.org/r/677511

gerritbot added a comment.Apr 19 2021, 1:35 PM

Change 677511 merged by Mforns:

[analytics/reportupdater@master] Fix date range parameters for native hive scripts

https://gerrit.wikimedia.org/r/677511

gerritbot added a comment.Apr 21 2021, 3:30 PM

Change 676299 merged by Mforns:

[analytics/reportupdater-queries@master] Validate the native "hive" report type

https://gerrit.wikimedia.org/r/676299

awight mentioned this in rARPQ96c2ec32f3bf: Validate the native "hive" report type.Apr 21 2021, 3:42 PM
gerritbot added a comment.Apr 21 2021, 3:43 PM

Change 681707 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater-queries@master] Rewrite WMDE Tech Wishes reports as native HiveQL

https://gerrit.wikimedia.org/r/681707

gerritbot added a comment.Apr 23 2021, 4:15 PM

Change 681707 merged by Mforns:

[analytics/reportupdater-queries@master] Rewrite WMDE Tech Wishes reports as native HiveQL

https://gerrit.wikimedia.org/r/681707

awight mentioned this in rARPQdf648123f204: Rewrite WMDE Tech Wishes reports as native HiveQL.Apr 23 2021, 4:17 PM
Maintenance_bot removed a project: Patch-For-Review.Apr 23 2021, 5:10 PM
awight mentioned this in T205296: Migrate all reportupdater queries to hive.Apr 23 2021, 6:14 PM
gerritbot added a comment.Apr 26 2021, 6:46 PM

Change 682730 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater-queries@master] Convert `browser` queries to native HiveHQL

https://gerrit.wikimedia.org/r/682730

gerritbot added a project: Patch-For-Review.Apr 26 2021, 6:46 PM
gerritbot added a comment.Apr 26 2021, 8:41 PM

Change 682742 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater-queries@master] Convert `cx` queries to native HiveHQL

https://gerrit.wikimedia.org/r/682742

gerritbot added a comment.Apr 26 2021, 8:44 PM

Change 682743 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater-queries@master] Convert `interlanguage` queries to native HiveHQL

https://gerrit.wikimedia.org/r/682743

gerritbot added a comment.Apr 26 2021, 8:47 PM

Change 682744 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater-queries@master] Convert `pingback` queries to native HiveHQL

https://gerrit.wikimedia.org/r/682744

gerritbot added a comment.Apr 26 2021, 8:54 PM

Change 682747 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater-queries@master] Convert `published_cx2_translations` to native HiveHQL

https://gerrit.wikimedia.org/r/682747

gerritbot added a comment.Apr 26 2021, 8:58 PM

Change 682748 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater-queries@master] Convert `reference-previews` to native HiveHQL

https://gerrit.wikimedia.org/r/682748

gerritbot added a comment.Apr 26 2021, 9:00 PM

Change 682749 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater-queries@master] Convert `structured-data` to native HiveHQL

https://gerrit.wikimedia.org/r/682749

gerritbot added a comment.Apr 26 2021, 9:07 PM

Change 682753 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater-queries@master] Convert `wmcs` to native HiveHQL

https://gerrit.wikimedia.org/r/682753

gerritbot added a comment.Apr 26 2021, 9:44 PM

Change 682761 had a related patch set uploaded (by Awight; author: Awight):

[analytics/reportupdater@master] [WIP] Provide to_year, to_month, and to_day

https://gerrit.wikimedia.org/r/682761

gerritbot added a comment.Sep 7 2021, 8:19 AM

Change 682748 merged by Awight:

[analytics/reportupdater-queries@master] Convert `reference-previews` to native HiveHQL

https://gerrit.wikimedia.org/r/682748

awight mentioned this in rARPQda5e5ef92be4: Convert `reference-previews` to native HiveHQL.Sep 7 2021, 8:20 AM
Milimetric closed this task as Declined.Oct 28 2021, 11:39 PM
Milimetric subscribed.

not to diminish the work already done here, just declining in favor of moving to Airflow and to clean up the logs

awight changed the task status from Declined to Resolved.Oct 29 2021, 8:52 AM

I would say the work is complete. Migrating format of existing jobs can be left up to the owners, or never.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL