Page MenuHomePhabricator
Create Task
Maniphest T131849

Fix limn-mobile-data mobile-options-last-3-months report after RU changes {lama}
Closed, ResolvedPublic13 Estimated Story Points
Actions

Description

After RU changes to fix 1-week lag issue and remove unused code, one report needs to be reconfigured to continue working. It's limn-mobile-data's mobile-options-last-3-months report. This report was previously using the timeboxed=False option to hack RU to build a report with the last 90 days only.

A more correct way is having RU accept a 'max_data_points=N' config option that trims the reports to a max of N data points (for example 90 days). And let the report be timeboxed, which is more computationally efficient, and simplifies the corresponding query.

Another option that could optionally be added to RU is reusing a query or script in different reports. This could be done by a config option 'execute=query_file_name'. With this, we could remove duplicated queries of limn-mobile-data that are virtually the same.

So, TODOs:

  1. add 'max_data_points' option to RU
  2. add 'execute' option to RU
  3. fix the configuration of the report in limn-mobile-data/mobile/config.yaml
  4. remove the duplicate query: limn-mobile-data/mobile/mobile-options-last-3-months.sql

Details

SubjectRepoBranchLines +/-
Fix bug in get_date_thresholdanalytics/reportupdatermaster+22 -5
Make code compatible with older versions of dateutilanalytics/reportupdatermaster+15 -7
Add max_data_points optionanalytics/reportupdatermaster+144 -74
Add 'execute' option to RU configanalytics/reportupdatermaster+41 -2
Reconfigure mobile-options-last-3-months queryanalytics/limn-mobile-datamaster+3 -28
Customize query in gerrit

Event Timeline

mforns created this task.Apr 5 2016, 2:50 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 5 2016, 2:50 PM
mforns moved this task from Next Up to In Progress on the Analytics-Kanban board.Apr 5 2016, 2:50 PM
mforns updated the task description. (Show Details)Apr 5 2016, 2:54 PM
gerritbot subscribed.Apr 5 2016, 4:36 PM

Change 281682 had a related patch set uploaded (by Mforns):
Add 'execute' option to RU config

https://gerrit.wikimedia.org/r/281682

gerritbot added a project: Patch-For-Review.Apr 5 2016, 4:36 PM
gerritbot added a comment.Apr 5 2016, 9:21 PM

Change 281815 had a related patch set uploaded (by Mforns):
Add max_data_points option

https://gerrit.wikimedia.org/r/281815

gerritbot added a comment.Apr 5 2016, 9:30 PM

Change 281818 had a related patch set uploaded (by Mforns):
Reconfigure mobile-options-last-3-months query

https://gerrit.wikimedia.org/r/281818

mforns moved this task from In Progress to In Code Review on the Analytics-Kanban board.Apr 5 2016, 9:31 PM
gerritbot added a comment.Apr 6 2016, 2:35 AM

Change 281818 merged by Milimetric:
Reconfigure mobile-options-last-3-months query

https://gerrit.wikimedia.org/r/281818

mforns added a comment.Apr 7 2016, 11:21 AM

Deployment plan for this:

  1. Merge the limn-mobile-data changes [ALREADY DONE]
  2. Merge both RU changes
  3. Wait for puppet to apply changes
  4. In stat1003, wait until limn-mobile-data's reportupdater is not running and remove the report file: /a/limn-public-data/mobile/datafiles/mobile-options-last-3-months.tsv
  5. Wait until RU backfills the report
  6. Check the newly build report is OK
  7. Check the dashboard is OK http://mobile-reportcard.wmflabs.org/
gerritbot added a comment.Apr 7 2016, 8:11 PM

Change 281682 merged by Milimetric:
Add 'execute' option to RU config

https://gerrit.wikimedia.org/r/281682

gerritbot added a comment.Apr 7 2016, 8:20 PM

Change 281815 merged by Milimetric:
Add max_data_points option

https://gerrit.wikimedia.org/r/281815

mforns added a comment.Apr 8 2016, 3:21 PM

I found a bug in RU, in the method get_date_threshold, used by the mobile-options-last-3-months report, that is breaking the execution of this report. Will create another patch for this.

gerritbot added a comment.Apr 11 2016, 3:36 PM

Change 282712 had a related patch set uploaded (by Mforns):
Make code compatible with older versions of dateutil

https://gerrit.wikimedia.org/r/282712

JAllemandou set the point value for this task to 13.Apr 11 2016, 4:17 PM
gerritbot added a comment.Apr 12 2016, 3:47 PM

Change 282712 merged by Mforns:
Make code compatible with older versions of dateutil

https://gerrit.wikimedia.org/r/282712

mforns moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Apr 12 2016, 3:50 PM
Milimetric triaged this task as Medium priority.Apr 12 2016, 4:14 PM
gerritbot added a comment.Apr 12 2016, 11:03 PM

Change 283101 had a related patch set uploaded (by Mforns):
Fix bug in get_date_threshold

https://gerrit.wikimedia.org/r/283101

mforns moved this task from Ready to Deploy to In Code Review on the Analytics-Kanban board.Apr 12 2016, 11:03 PM
gerritbot added a comment.Apr 14 2016, 2:13 PM

Change 283101 merged by Milimetric:
Fix bug in get_date_threshold

https://gerrit.wikimedia.org/r/283101

mforns moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Apr 14 2016, 2:20 PM
mforns moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Apr 15 2016, 10:53 AM
• Nuria closed this task as Resolved.Apr 18 2016, 3:57 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL