Can we use golden to collect those data that are not on dashboard and keep them in /srv/published-datasets/discovery?
For those on the dashboard but with a max_data_points limit, can we just create extra reports and remove the max_data_points limit?
@mpopov Any other ideas? ;)

In T172453#3517537, @chelsyx wrote:

Can we use golden to collect those data that are not on dashboard and keep them in /srv/published-datasets/discovery?
For those on the dashboard but with a max_data_points limit, can we just create extra reports and remove the max_data_points limit?
@mpopov Any other ideas? ;)

Without patching Reportupdater to output two reports (a limited one and an unlimited one), that's what I was thinking too. This means calculating the same thing twice, but it's the simplest solution.

According to config.yaml files in golden, the following reports have a max_data_points limit, and we are going to create duplicated reports with max_data_points removed for tracking purpose:

Search:

• app_event_counts_langproj_breakdown
• mobile_event_counts_langproj_breakdown
• desktop_event_counts_langproj_breakdown
• paulscore_approximations_fulltext_langproj_breakdown
• search_threshold_pass_rate_langproj_breakdown
• cirrus_langproj_breakdown_no_automata
• cirrus_langproj_breakdown_with_automata

Portal:

• all_country_data
• last_action_country
• most_common_country
• first_visits_country

The following metrics haven't been tracked by our dashboards and we are going to create new reports for them in golden:

• wikipedia portal pageview by device (desktop vs mobile)
• wikipedia portal clickthrough rate by device (desktop vs mobile)
• proportion of wikipedia portal on mobile devices in US vs elsewhere
• pageviews from full-text search (desktop vs mobile)
• search return rate
• SERPs by access method

Change 374900 had a related patch set uploaded (by Chelsyx; owner: Chelsyx):
[wikimedia/discovery/golden@master] Duplicate reports without max data points limit to keep data longer

https://gerrit.wikimedia.org/r/374900

Change 374900 merged by Bearloga:
[wikimedia/discovery/golden@master] Duplicate reports without max data points limit to keep data longer

https://gerrit.wikimedia.org/r/374900

Change 375408 had a related patch set uploaded (by Chelsyx; owner: Chelsyx):
[wikimedia/discovery/golden@master] Add new datasets in search and portal

https://gerrit.wikimedia.org/r/375408

Change 375408 merged by Bearloga:
[wikimedia/discovery/golden@master] Add new datasets in search and portal

https://gerrit.wikimedia.org/r/375408

Deployed to prod. Good job, @chelsyx!