Page MenuHomePhabricator

Move backend for current search dashboard to pull data from Hadoop
Closed, ResolvedPublic

Description

Goal: transition Search dashboards to systems that are maintainable/configurable by engineers on the Search team. This is also a prerequisite for improving sampling.

The backend of current dashboards (for some reports) should be moved to pull data from hadoop via reportupdater rather than from mysql. For now, we leave frontend as is for dashboards but similar to https://analytics.wikimedia.org/dashboards/browsers/#all-sites-by-os the dashboards pull data from tsvs that come from hadoop instead of mysql.

Event Timeline

kzimmerman triaged this task as Normal priority.

Analyst support of the transition is required for T197129.

@Nuria @TJones FYI on timing/priorities: @mpopov will be taking on this task after he has wrapped up his work relating to T212386 (ensuring the MaridDB sharding solution works with our R package for querying MySQL). Other tasks he'll be working on (possibly in parallel) that will affect timelines: pulling preliminary data for the Android Notifications project (T213458), completing reports for the SEO tests (end of February deadline).

mpopov moved this task from Triage to Backlog on the Product-Analytics board.Feb 14 2019, 7:27 PM
Nuria added a comment.Mar 12 2019, 5:53 PM

@kzimmerman do we have an idea when this work can be started, cc @mpopov ?

Nuria renamed this task from Move backend for current dashboard to pull data from Hadoop to Move backend for current search dashboard to pull data from Hadoop.Mar 15 2019, 5:35 PM
Nuria added a comment.Mar 15 2019, 5:39 PM

Spreadsheet with dashboards triage: https://docs.google.com/spreadsheets/d/1Znl8Tu_OlmOoah4PwSKhK09ZoGJzv9Wnc2X4Oo_Mgug/edit#gid=0

Also, this is happening and once migration of cirrus logging to the new MEP pipeline is completed we will be importing this data into druid: https://phabricator.wikimedia.org/T218347

mpopov moved this task from Backlog to Next Up on the Product-Analytics board.Mar 25 2019, 5:34 PM

I'm currently finishing up an important analysis for the Android team (T213458) and once that's done (today or tomorrow) I will resume work on this. I've already started but have done a little bit. Based on that, I expect to be done with this by the end of the week.

mpopov moved this task from Next Up to Doing on the Product-Analytics board.Mar 28 2019, 10:46 PM

Change 499938 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/golden@master] Switch metrics::portal::app_link_clicks SQL=>Hive

https://gerrit.wikimedia.org/r/499938

Change 499939 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/golden@master] [WIP] Switch Search Platform metrics from MySQL to Hadoop/Hive

https://gerrit.wikimedia.org/r/499939

@mpopov are there any of these metrics we want to remove in the light of the classification that @TJones did on spreadsheet?

@mpopov are there any of these metrics we want to remove in the light of the classification that @TJones did on spreadsheet?

I don't think we'd made any decisions about what to do based on the metrics spreadsheet. The discussion died down without any real resolution or firm plans. I don't think we should be dropping metrics as part of this task based on the current state of discussion related to the spreadsheet.

We really need to discuss the dashboard/metric distinction, what we can get out of Turnilo or similar for our metrics needs, and what we can get from a new platform for a smaller number of proper dashboard metrics before we start dropping things.

The rating system on the spreadsheet is impressionistic and was intended to spark discussion, not drive mechanical decision making about what to remove. Nonetheless, I've added min/max/median calculations for the scores and manually coded the "crucial" ones (median ≥ 2) and the "useless" ones (max ≤ 0) by color. Those are the only judgments I'm comfortable making deterministically.

The "useless" category is only 5 items out of almost 100 for current metrics, and they are all lower level features or ways of slicing data that shouldn't affect or be affected by where we pull our data from (which is what this task is about).

@mpopov, If some source of data turns out to be much more difficult to get than others, let Search know and we can figure out whether we can live without it for the short or long term.

ping @EBernhardson so he knows when this work is final we can change the sampling ratios for all languages, before that happens yours truly needs to remove the data from it being persisted to mysql.

Change 500076 had a related patch set uploaded (by Nuria; owner: Nuria):
[operations/puppet@production] Removing TestSearchSatisfaction from it being persisted to MySQL

https://gerrit.wikimedia.org/r/500076

Change 499938 merged by Chelsyx:
[wikimedia/discovery/golden@master] Switch metrics::portal::app_link_clicks SQL=>Hive

https://gerrit.wikimedia.org/r/499938

Change 499939 abandoned by Bearloga:
Switch Search Platform metrics from MySQL to Hadoop/Hive

Reason:
can't figure out how to resolve the merge conflict because I messed up the commits so just gonna upload a new patch

https://gerrit.wikimedia.org/r/499939

Change 502873 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/golden@master] Switch Search Platform metrics from MySQL to Hadoop/Hive

https://gerrit.wikimedia.org/r/502873

Change 502873 merged by Chelsyx:
[wikimedia/discovery/golden@master] Switch Search Platform metrics from MySQL to Hadoop/Hive

https://gerrit.wikimedia.org/r/502873

Change 500076 merged by Ottomata:
[operations/puppet@production] Removing TestSearchSatisfaction from it being persisted to MySQL

https://gerrit.wikimedia.org/r/500076

Nuria added a comment.Apr 11 2019, 5:48 PM

Ok, puppet code merged, next step is for @EBernhardson to change sampling ratios

Change 504649 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/golden@master] Spring cleaning and fixes

https://gerrit.wikimedia.org/r/504649

Change 504649 merged by Bearloga:
[wikimedia/discovery/golden@master] Spring cleaning and fixes

https://gerrit.wikimedia.org/r/504649

Change 505251 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/golden@master] metrics::search::sister_search_prevalence: Fix weird bug

https://gerrit.wikimedia.org/r/505251

Change 505251 merged by Bearloga:
[wikimedia/discovery/golden@master] metrics::search::sister_search_prevalence: Fix weird bug

https://gerrit.wikimedia.org/r/505251

Nuria added a comment.Apr 23 2019, 5:29 PM

ping @mpopov are all changes done to move dashboard backend? If so, can we close ticket?

TJones added a comment.Tue, Jun 4, 5:19 PM

ping @mpopov are all changes done to move dashboard backend? If so, can we close ticket?

mpopov added a subscriber: debt.Wed, Jun 5, 9:12 PM

ping @mpopov are all changes done to move dashboard backend? If so, can we close ticket?

Yes, all the changes that need to be done are done, with the exception of Wikipedia.org Portal metrics. There were a few issues that I didn't and don't have time to fix, so I disabled the "portal" module in April to focus on getting external traffic, WDQS, Maps, and Search Platform modules 100% working. Especially since work on the portal completed years ago.

Since disabling those reports, I haven't heard of anyone missing that portal dashboard or the metrics related to the portal page. If it's not being used to drive decisions, I think we should remove it from the codebase to reduce maintenance overhead.

@debt what do you think? If you agree, then I think we can close this ticket.

@mpopov, sure, we can close this ticket. I have mixed feelings, but ok.

kzimmerman closed this task as Resolved.Tue, Jun 18, 6:33 PM