Page MenuHomePhabricator
Create Task
Maniphest T206342

Finalize eventlogging to druid ingestion
Closed, ResolvedPublic13 Estimated Story Points
Actions

Description

  • Parametrize job with a whitelist instead of a blacklist
  • Modify puppet profile and jobs accordingly
  • Write a comprehensive documentation
  • Remove the confusing Count metric from the datasources in Turnilo, or at least uncheck it by default.
  • Try to add a new metric to the datasource, eventCountPercentage, that normalizes eventCount splits by the total aggregate, so that time measure buckets become percentage-of-total values, instead of frequencies. This way they will not vary with throughput changes or seasonality, and will be a lot easier to follow.

Details

SubjectRepoBranchLines +/-
Fine tune eventlogging_to_druid_job spark and druid parametersoperations/puppetproduction+20 -41
Fix bug in EventLoggingToDruid, add time measures as dimensionsanalytics/refinery/sourcemaster+2 -1
Fix broken default job_class for eventlogging_to_druid_job.ppoperations/puppetproduction+1 -1
Add druid_load jobs to analytics refineryoperations/puppetproduction+212 -0
Rename start_date and end_date to since until in EventLoggingToDruid.scalaanalytics/refinery/sourcemaster+10 -10
Refactor EventLoggingToDruid to use whitelists and ConfigHelperanalytics/refinery/sourcemaster+253 -255
Customize query in gerrit

Related Objects

Event Timeline

• Nuria created this task.Oct 5 2018, 4:17 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 5 2018, 4:17 PM
• Nuria moved this task from Next Up to In Progress on the Analytics-Kanban board.Oct 8 2018, 3:06 PM
mforns triaged this task as Medium priority.Oct 8 2018, 3:36 PM
mforns moved this task from Incoming to Smart Tools for Better Data on the Analytics board.
gerritbot subscribed.Oct 9 2018, 11:17 PM

Change 465532 had a related patch set uploaded (by Mforns; owner: Mforns):
[analytics/refinery/source@master] Refactor EventLoggingToDruid to use whitelists and ConfigHelper

https://gerrit.wikimedia.org/r/465532

gerritbot added a project: Patch-For-Review.Oct 9 2018, 11:17 PM
gerritbot added a comment.Oct 10 2018, 7:50 PM

Change 465692 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/puppet@production] Add druid_load jobs to analytics refinery

https://gerrit.wikimedia.org/r/465692

mforns moved this task from In Progress to In Code Review on the Analytics-Kanban board.Oct 11 2018, 9:43 AM
gerritbot added a comment.Oct 15 2018, 2:17 PM

Change 465532 merged by jenkins-bot:
[analytics/refinery/source@master] Refactor EventLoggingToDruid to use whitelists and ConfigHelper

https://gerrit.wikimedia.org/r/465532

gerritbot added a comment.Oct 15 2018, 5:43 PM

Change 467422 had a related patch set uploaded (by Mforns; owner: Mforns):
[analytics/refinery/source@master] Rename start_date and end_date to since until in EventLoggingToDruid.scala

https://gerrit.wikimedia.org/r/467422

gerritbot added a comment.Oct 15 2018, 5:52 PM

Change 467422 merged by Ottomata:
[analytics/refinery/source@master] Rename start_date and end_date to since until in EventLoggingToDruid.scala

https://gerrit.wikimedia.org/r/467422

mforns moved this task from In Code Review to Done on the Analytics-Kanban board.Oct 17 2018, 3:15 PM
Milimetric raised the priority of this task from Medium to High.Oct 18 2018, 5:17 PM
gerritbot added a comment.Oct 18 2018, 6:15 PM

Change 465692 merged by Ottomata:
[operations/puppet@production] Add druid_load jobs to analytics refinery

https://gerrit.wikimedia.org/r/465692

gerritbot added a comment.Oct 18 2018, 6:28 PM

Change 468374 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/puppet@production] Fix broken default job_class for eventlogging_to_druid_job.pp

https://gerrit.wikimedia.org/r/468374

gerritbot added a comment.Oct 18 2018, 6:30 PM

Change 468374 merged by Ottomata:
[operations/puppet@production] Fix broken default job_class for eventlogging_to_druid_job.pp

https://gerrit.wikimedia.org/r/468374

gerritbot added a comment.Oct 19 2018, 11:22 AM

Change 468550 had a related patch set uploaded (by Mforns; owner: Mforns):
[analytics/refinery/source@master] Fix bug in EventLoggingToDruid, add time measures as dimensions

https://gerrit.wikimedia.org/r/468550

gerritbot added a comment.Oct 19 2018, 2:41 PM

Change 468588 had a related patch set uploaded (by Mforns; owner: Mforns):
[operations/puppet@production] Fine tune eventlogging_to_druid_job spark and druid parameters

https://gerrit.wikimedia.org/r/468588

mforns moved this task from Done to In Code Review on the Analytics-Kanban board.Oct 19 2018, 2:41 PM
gerritbot added a comment.Oct 19 2018, 6:26 PM

Change 468550 merged by jenkins-bot:
[analytics/refinery/source@master] Fix bug in EventLoggingToDruid, add time measures as dimensions

https://gerrit.wikimedia.org/r/468550

gerritbot added a comment.Oct 22 2018, 5:29 PM

Change 468588 merged by Ottomata:
[operations/puppet@production] Fine tune eventlogging_to_druid_job spark and druid parameters

https://gerrit.wikimedia.org/r/468588

mforns renamed this task from Parametize eventlogging to druid ingestion with a whitelist instead of a blacklist to Finalize eventlogging to druid ingestion with a whitelist instead of a blacklist.Oct 23 2018, 2:00 PM
mforns updated the task description. (Show Details)
mforns set the point value for this task to 13.
mforns mentioned this in T166414: Explore NavigationTiming by faceted properties - EventLogging refine.
mforns mentioned this in T202751: Ingest data from PageIssues EventLogging schema into Druid.
mforns mentioned this in T205562: Ingest data aggregate ReadingDepth data into Druid .
mforns renamed this task from Finalize eventlogging to druid ingestion with a whitelist instead of a blacklist to Finalize eventlogging to druid ingestion.Oct 23 2018, 2:02 PM
mforns moved this task from In Code Review to In Progress on the Analytics-Kanban board.
• Tbayer subscribed.Oct 23 2018, 6:59 PM
mforns updated the task description. (Show Details)Jan 25 2019, 6:08 PM
mforns updated the task description. (Show Details)Feb 8 2019, 3:45 PM

The docs are here: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Hive_to_Druid
Moving to done!

mforns moved this task from In Progress to Done on the Analytics-Kanban board.Feb 8 2019, 3:47 PM
• Nuria closed this task as Resolved.Feb 8 2019, 6:00 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL