Page MenuHomePhabricator
Create Task
Maniphest T150915

[Dashboard] Migrate golden to Reportupdater infrastructure
Closed, ResolvedPublic50 Estimated Story Points
Actions

Description

We need our golden data retriever codebase to be more modular and have better support for addition of new modules, which will be very helpful when we start doing ZRR calculation for "well-behaved searches" (see T150370 & T150901). Testing if a new script works and backfilling missing data is a huge pain right now.

After talking with Analytics, Chelsy and I decided to migrate our codebase to use their Reportupdater infrastructure as it seems to meet our needs. This will require the following steps:

  • Rewrite as many EventLogging (EL) based scripts to be pure SQL
  • Rewrite current pure-R scripts be shell scripts + R and use Reportupdater conventions
  • Update column names in current datasets
  • Finalize (test the heck out of) Reportupdater-based codebase
  • Code review by @chelsyx
  • Deploy & schedule for daily execution
  • Prepare dashboards for new formats/naming conventions (all in CR)
  • Deploy dashboards after Reportupdater-based refactor of golden has completed at least one successful run

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

mpopov created this task.Nov 16 2016, 11:49 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 16 2016, 11:49 PM
mpopov triaged this task as Medium priority.Nov 16 2016, 11:57 PM
mpopov added a project: Discovery-Analysis.
mpopov moved this task from Needs triage to Current work on the Discovery-Analysis board.Nov 17 2016, 12:02 AM
mpopov edited projects, added Discovery-Analysis (Current work); removed Discovery-Analysis.
mpopov moved this task from Backlog to In progress on the Discovery-Analysis (Current work) board.
mpopov added a subtask: T150370: [EPIC][Search][Dashboard] Add "well-behaved searchers" filter.
mpopov removed a subtask: T150370: [EPIC][Search][Dashboard] Add "well-behaved searchers" filter.
mpopov added a parent task: T150370: [EPIC][Search][Dashboard] Add "well-behaved searchers" filter.
mpopov added a subtask: T150901: [Search][Dashboard] Well-behaved searchers filter: ZRR.
gerritbot subscribed.Nov 17 2016, 4:41 AM

Change 322051 had a related patch set uploaded (by Bearloga):
Easier backfilling [WIP]

https://gerrit.wikimedia.org/r/322051

gerritbot added a project: Patch-For-Review.Nov 17 2016, 4:41 AM
mpopov moved this task from In progress to Backlog on the Discovery-Analysis (Current work) board.Nov 28 2016, 7:35 PM
mpopov added a comment.EditedNov 28 2016, 10:32 PM

Side-update: Chelsy and I spoke with Nuria and Dan from Analytics about possibly migrating to their Reportupdater infrastructure, but had the following concerns that we're waiting to hear back on:

  1. A lot of our datasets that we generate are in long format rather than wide format, with multiple rows for each date. For example, zero results rate on a project-language pair basis, with project as a column, language as a column, and rate as a column -- which yields hundreds of data points for any particular date. We can't force all of our datasets to be wide (which seems to be what Reportupdater requires), especially datasets with ever-changing user agent breakdowns.

Dan: The funnel setting is meant to address reports that can't easily be wide. Bad name, I know, but we use it and it works well.

  1. Some of our metrics require us to only keep a rolling N-day window. For example, that zero results rate broken down by project-language pairs since there are so many of those pairs that keeping hundreds of observations per day would make our dashboards very slow to use. Can Reportupdater be made to recognize when we want to delete rows where date is older than N days?

Dan: Yes, we have a max data points setting that cuts off data after a certain amount is collected. Takeaway for us would be to make sure it works as expected with the funnel reports (by counting distinct dates not rows)

  1. You mentioned that the stats user on stat1002 could have a personal library of R packages like Chelsy and I do. If we add a new script/dataset to a reportupdater repo that requires a new package or a new version of an existing package, how simple/fast is the process for installing/updating R packages in stats user's library. Right now it's super easy and fast for me/us to start using new packages when we add new metrics, since all our scripts run under my account on stat1002.

Dan: The process to install a new R package can vary a lot depending on what it takes to build the deb package (I think). This is better addressed by the opsier folks.

mpopov removed a project: Patch-For-Review.Dec 1 2016, 12:31 AM
mpopov updated the task description. (Show Details)
gerritbot added a comment.Dec 3 2016, 1:24 AM

Change 322051 abandoned by Bearloga:
Easier backfilling [WIP]

Reason:
Abandoning in favor of migrating codebase to Analytics' Reportupdater infrastructure.

https://gerrit.wikimedia.org/r/322051

mpopov moved this task from Backlog to In progress on the Discovery-Analysis (Current work) board.Dec 5 2016, 5:52 PM
mpopov renamed this task from [Dashboard] Make backfilling data easier to [Dashboard] Migrate golden to Reportupdater infrastructure.Dec 5 2016, 8:20 PM
mpopov updated the task description. (Show Details)
mpopov added a subtask: T152617: Wikipedia portal: update eventlogging for sister project clickthroughs.Dec 7 2016, 7:29 PM
mpopov added a subtask: T112170: Model user behavior and detect when reality heavily deviated from expectation.
mpopov added a subtask: T149440: [EPIC] Detect bots from searches (and learn C++/Rcpp).
mpopov added a subtask: T150410: [Dashboard] Implement the wiki/language selector in Search Metrics.Dec 7 2016, 10:57 PM
gerritbot added a comment.Dec 8 2016, 2:06 AM

Change 325870 had a related patch set uploaded (by Bearloga):
[WIP] Migrate to Reportupdater framework

https://gerrit.wikimedia.org/r/325870

gerritbot added a project: Patch-For-Review.Dec 8 2016, 2:06 AM
mpopov updated the task description. (Show Details)Dec 15 2016, 8:59 PM

Running the test utility right now to check that everything is working OK before I officially ask Chelsy for CR.

• chelsyx moved this task from In progress to Stalled/Waiting on the Discovery-Analysis (Current work) board.Dec 17 2016, 12:04 AM
• chelsyx moved this task from Stalled/Waiting to In progress on the Discovery-Analysis (Current work) board.Dec 19 2016, 6:23 PM
mpopov moved this task from In progress to Needs review on the Discovery-Analysis (Current work) board.Dec 21 2016, 9:39 PM
mpopov added a subtask: T153936: Create usage stats for WDQS LDF endpoint.Jan 3 2017, 9:30 PM
mpopov mentioned this in T153856: Add lint/CI to all wikimedia/discovery analytics repositories.Jan 3 2017, 10:07 PM
mpopov moved this task from Needs review to In progress on the Discovery-Analysis (Current work) board.Jan 6 2017, 1:55 AM
mpopov updated the task description. (Show Details)Jan 9 2017, 7:08 PM
mpopov changed the point value for this task from 10 to 50.
mpopov added a subscriber: • chelsyx.
mpopov added a subtask: T154634: Wikipedia.org: add app links to Portal dashboard.Jan 18 2017, 6:41 PM
mpopov mentioned this in T156523: Investigate adding user-friendly testing functionality to Reportupdater.Jan 27 2017, 8:18 PM
debt created subtask T156686: Search metrics: add sister projects to dashboard.Jan 30 2017, 10:00 PM
gerritbot added a comment.Feb 2 2017, 12:30 AM

Change 335575 had a related patch set uploaded (by Bearloga):
Point to new datasets

https://gerrit.wikimedia.org/r/335575

gerritbot added a comment.Feb 2 2017, 12:38 AM

Change 335576 had a related patch set uploaded (by Bearloga):
Point to new datasets

https://gerrit.wikimedia.org/r/335576

gerritbot added a comment.Feb 2 2017, 1:00 AM

Change 335579 had a related patch set uploaded (by Bearloga):
Point to new datasets and add LDF

https://gerrit.wikimedia.org/r/335579

mpopov updated the task description. (Show Details)Feb 2 2017, 10:23 PM
gerritbot added a comment.Feb 3 2017, 2:45 AM

Change 335746 had a related patch set uploaded (by Bearloga):
Point to new datasets

https://gerrit.wikimedia.org/r/335746

gerritbot added a comment.Feb 7 2017, 1:11 AM

Change 336350 had a related patch set uploaded (by Bearloga):
Point to new datasets & add new metrics

https://gerrit.wikimedia.org/r/336350

mpopov updated the task description. (Show Details)Feb 7 2017, 1:13 AM
mpopov moved this task from In progress to Needs review on the Discovery-Analysis (Current work) board.
debt closed subtask T156686: Search metrics: add sister projects to dashboard as Declined.Feb 7 2017, 4:29 PM
gerritbot added a comment.Feb 7 2017, 11:27 PM

Change 335575 merged by Bearloga:
Point to new datasets

https://gerrit.wikimedia.org/r/335575

gerritbot added a comment.Feb 7 2017, 11:27 PM

Change 335579 merged by Bearloga:
Point to new datasets and add LDF

https://gerrit.wikimedia.org/r/335579

gerritbot added a comment.Feb 7 2017, 11:28 PM

Change 335576 merged by Bearloga:
Point to new datasets

https://gerrit.wikimedia.org/r/335576

gerritbot added a comment.Feb 7 2017, 11:28 PM

Change 335746 merged by Bearloga:
Point to new datasets

https://gerrit.wikimedia.org/r/335746

gerritbot added a comment.Feb 7 2017, 11:28 PM

Change 336350 merged by Bearloga:
Point to new datasets & add new metrics

https://gerrit.wikimedia.org/r/336350

gerritbot added a comment.Mar 6 2017, 7:38 PM

Change 341373 had a related patch set uploaded (by chelsyx):
[wikimedia/discovery/prince] Fixed bug in tab country_breakdown

https://gerrit.wikimedia.org/r/341373

gerritbot added a comment.Mar 6 2017, 7:51 PM

Change 341373 merged by Bearloga:
[wikimedia/discovery/prince] Fixed bug in tab country_breakdown

https://gerrit.wikimedia.org/r/341373

gerritbot added a comment.Mar 8 2017, 2:06 AM

Change 325870 merged by Bearloga:
[wikimedia/discovery/golden] Migrate to Reportupdater framework

https://gerrit.wikimedia.org/r/325870

gerritbot added a comment.Mar 8 2017, 3:31 AM

Change 341746 had a related patch set uploaded (by Bearloga):
[wikimedia/discovery/twilightsparql] Annotate Reportupdater migration on graphs

https://gerrit.wikimedia.org/r/341746

gerritbot added a comment.Mar 8 2017, 3:31 AM

Change 341745 had a related patch set uploaded (by Bearloga):
[wikimedia/discovery/wetzel] Annotate Reportupdater migration on graphs

https://gerrit.wikimedia.org/r/341745

gerritbot added a comment.Mar 8 2017, 3:31 AM

Change 341744 had a related patch set uploaded (by Bearloga):
[wikimedia/discovery/wonderbolt] Annotate Reportupdater migration on graphs

https://gerrit.wikimedia.org/r/341744

gerritbot added a comment.Mar 8 2017, 3:32 AM

Change 341743 had a related patch set uploaded (by Bearloga):
[wikimedia/discovery/prince] Annotate Reportupdater migration on graphs

https://gerrit.wikimedia.org/r/341743

gerritbot added a comment.Mar 8 2017, 3:32 AM

Change 341742 had a related patch set uploaded (by Bearloga):
[wikimedia/discovery/rainbow] Annotate Reportupdater migration on graphs

https://gerrit.wikimedia.org/r/341742

gerritbot added a comment.Mar 8 2017, 6:55 PM

Change 341742 merged by Chelsyx:
[wikimedia/discovery/rainbow] Annotate Reportupdater migration on graphs

https://gerrit.wikimedia.org/r/341742

gerritbot added a comment.Mar 8 2017, 6:57 PM

Change 341743 merged by Chelsyx:
[wikimedia/discovery/prince] Annotate Reportupdater migration on graphs

https://gerrit.wikimedia.org/r/341743

gerritbot added a comment.Mar 8 2017, 6:58 PM

Change 341744 merged by Chelsyx:
[wikimedia/discovery/wonderbolt] Annotate Reportupdater migration on graphs

https://gerrit.wikimedia.org/r/341744

gerritbot added a comment.Mar 8 2017, 6:58 PM

Change 341745 merged by Chelsyx:
[wikimedia/discovery/wetzel] Annotate Reportupdater migration on graphs

https://gerrit.wikimedia.org/r/341745

gerritbot added a comment.Mar 8 2017, 6:59 PM

Change 341746 merged by Chelsyx:
[wikimedia/discovery/twilightsparql] Annotate Reportupdater migration on graphs

https://gerrit.wikimedia.org/r/341746

gerritbot added a comment.Mar 8 2017, 11:46 PM

Change 341935 had a related patch set uploaded (by Bearloga):
[wikimedia/discovery/dashboard] Deploy post-Reportupdater dashboards

https://gerrit.wikimedia.org/r/341935

gerritbot added a comment.Mar 8 2017, 11:46 PM

Change 341935 merged by Bearloga:
[wikimedia/discovery/dashboard] Deploy post-Reportupdater dashboards

https://gerrit.wikimedia.org/r/341935

mpopov updated the task description. (Show Details)Mar 8 2017, 11:49 PM
mpopov moved this task from Needs review to Done on the Discovery-Analysis (Current work) board.Mar 9 2017, 12:03 AM

stat1002:/a/discovery/golden has been updated

All the dashboards have been deployed. They use the new datasets located in stat1002:/a/aggregate-datasets/discovery

The pre-migration datasets have been archived in /a/discovery/legacy-datasets/pre-reportupdater

I will send an announcement email to discovery-l, since there are several new metrics being tracked on the WDQS and Portal dashboards.

debt subscribed.Mar 9 2017, 12:32 AM

Nice job! :)

debt closed subtask T152617: Wikipedia portal: update eventlogging for sister project clickthroughs as Resolved.Mar 16 2017, 12:35 PM
debt closed subtask T154634: Wikipedia.org: add app links to Portal dashboard as Resolved.Mar 16 2017, 12:38 PM
debt closed this task as Resolved.Mar 16 2017, 12:41 PM
debt closed subtask T153936: Create usage stats for WDQS LDF endpoint as Resolved.

...and wrapping this up! :) Good work!

debt closed subtask T149440: [EPIC] Detect bots from searches (and learn C++/Rcpp) as Invalid.Mar 28 2017, 8:15 PM
mpopov removed a subtask: T112170: Model user behavior and detect when reality heavily deviated from expectation.May 9 2017, 6:35 PM
debt closed subtask T150410: [Dashboard] Implement the wiki/language selector in Search Metrics as Resolved.Jun 7 2017, 3:56 PM
mpopov mentioned this in T174110: Private data access for non-person user that calculates metrics.Aug 24 2017, 10:05 PM
debt closed subtask T150901: [Search][Dashboard] Well-behaved searchers filter: ZRR as Declined.Aug 30 2017, 11:13 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL