Page MenuHomePhabricator
Create Task
Maniphest T251320

Augment Hive event data with normalized host info from meta.domain
Closed, ResolvedPublic
Actions

Description

We should add a Refine transform step to add normalized host info if meta.domain (and/or EventCapsule's webHost?) is set.

Details

SubjectRepoBranchLines +/-
refine - bump refinery version to pick up normalized_host transformoperations/puppetproduction+2 -2
Add Refine transform function to add normalized hostanalytics/refinery/sourcemaster+128 -5
Customize query in gerrit

Related Objects

Event Timeline

Ottomata created this task.Apr 28 2020, 6:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 28 2020, 6:49 PM
Ottomata renamed this task from Augment event data with normalized host info from meta.domain to Augment Hive event data with normalized host info from meta.domain .Apr 28 2020, 6:50 PM
• jlinehan removed a parent task: T251281: Clients may need to set wiki/domain/webHost in event metadata.Apr 28 2020, 6:50 PM
mpopov mentioned this in T251281: Clients may need to set wiki/domain/webHost in event metadata.Apr 29 2020, 3:08 PM
LGoto edited projects, added Product-Analytics; removed Product-Analytics (Kanban).May 4 2020, 6:08 PM
LGoto moved this task from Triage to Tracking on the Product-Analytics board.
Milimetric triaged this task as High priority.May 7 2020, 4:00 PM
Milimetric moved this task from Incoming to Event Platform on the Analytics board.
Ottomata added projects: Analytics-Kanban, Event-Platform.May 7 2020, 4:01 PM
Ottomata moved this task from Next Up to Paused on the Analytics-Kanban board.May 11 2020, 9:08 PM
Milimetric moved this task from Event Platform to Incoming on the Analytics board.Jun 8 2020, 4:42 PM
Milimetric removed a project: Analytics-Kanban.
• fdans subscribed.Jun 15 2020, 3:37 PM

there's already code for this in pageviews (the UDF that determines the project given a webhost).

• fdans moved this task from Incoming to Event Platform on the Analytics board.Jun 15 2020, 3:37 PM
Ottomata added a comment.Jun 16 2020, 1:59 PM

Ya we just need to add a Refine transform function for this.

Ottomata moved this task from Backlog to Estimated/ Discussed on the Event-Platform board.Jan 14 2021, 7:53 PM
LGoto moved this task from Inbox to To Do on the Better Use Of Data board.Feb 8 2021, 7:39 PM
• Mholloway subscribed.Apr 21 2021, 6:44 PM
ldelench_wmf moved this task from To Do to Tracking on the Better Use Of Data board.Apr 21 2021, 6:50 PM
mforns subscribed.Apr 21 2021, 6:50 PM

Should we, once this is done, remove code that does this at query time?
I.e. the session length intermediate table query using a UDF to normalize the meta.domain?

ldelench_wmf removed a project: Better Use Of Data.May 13 2021, 3:36 PM
Ottomata added a comment.Jun 14 2021, 8:43 PM

@Mholloway this might be a fun one to do.

gerritbot added a comment.Jul 16 2021, 8:44 PM

Change 705021 had a related patch set uploaded (by Mholloway; author: Michael Holloway):

[analytics/refinery/source@master] Add Refine transform function to add normalized host

https://gerrit.wikimedia.org/r/705021

gerritbot added a project: Patch-For-Review.Jul 16 2021, 8:44 PM
• Mholloway added a comment.Jul 16 2021, 8:55 PM

I gave this a try for 10% time today.

gerritbot added a comment.Aug 2 2021, 2:47 PM

Change 705021 merged by Ottomata:

[analytics/refinery/source@master] Add Refine transform function to add normalized host

https://gerrit.wikimedia.org/r/705021

Ottomata reassigned this task from Ottomata to • Mholloway.Aug 2 2021, 3:07 PM
Ottomata added a project: Analytics-Kanban.
Ottomata moved this task from Paused to Ready to Deploy on the Analytics-Kanban board.
Maintenance_bot removed a project: Patch-For-Review.Aug 2 2021, 3:10 PM
gerritbot added a comment.Aug 3 2021, 7:06 PM

Change 709810 had a related patch set uploaded (by Ottomata; author: Ottomata):

[operations/puppet@production] refine - bump refinery version to pick up normalized_host transform

https://gerrit.wikimedia.org/r/709810

gerritbot added a project: Patch-For-Review.Aug 3 2021, 7:06 PM
gerritbot added a comment.Aug 3 2021, 7:19 PM

Change 709810 merged by Ottomata:

[operations/puppet@production] refine - bump refinery version to pick up normalized_host transform

https://gerrit.wikimedia.org/r/709810

Stashbot added a comment.Aug 3 2021, 7:23 PM

Mentioned in SAL (#wikimedia-analytics) [2021-08-03T19:23:18Z] <ottomata> bump Refine to refinery version 0.1.16 to pick up normalized_host transform - now all event tables will have a new normalized_host field - T251320

Maintenance_bot removed a project: Patch-For-Review.Aug 3 2021, 8:10 PM
Ottomata moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Aug 4 2021, 4:05 PM
Ottomata moved this task from Estimated/ Discussed to To be Estimated/To be discussed on the Event-Platform board.Aug 30 2021, 1:31 PM
• Mholloway closed this task as Resolved.Sep 17 2021, 7:22 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL