Page MenuHomePhabricator

[Dashboards] Move Discovery's public datasets
Closed, ResolvedPublic1 Story Points

Description

WMF Analytics is archiving/retiring stat1002:/a/aggregate-datasets, so we need to make the following changes:

  • /a/aggregate-datasets/discovery => /a/published-datasets/discovery/metrics
  • /a/aggregate-datasets/discovery-forecasts => /a/published-datasets/discovery/forecasts

When our dashboards need to download those datasets: https://datasets.wikimedia.org/aggregate-datasets/ => https://analytics.wikimedia.org/datasets/

Event Timeline

mpopov created this task.May 31 2017, 7:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 31 2017, 7:51 PM

Change 356460 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/golden@master] Change paths

https://gerrit.wikimedia.org/r/356460

Change 356460 merged by Bearloga:
[wikimedia/discovery/golden@master] Change paths

https://gerrit.wikimedia.org/r/356460

Change 356468 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/experimental@master] Use new path

https://gerrit.wikimedia.org/r/356468

Change 356469 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/wonderbolt@master] Use new path

https://gerrit.wikimedia.org/r/356469

Change 356470 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/polloi@master] Update dataset path, downloaded prefixes, and documentation

https://gerrit.wikimedia.org/r/356470

Change 356472 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/wetzel@master] Use new path

https://gerrit.wikimedia.org/r/356472

Change 356473 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/twilightsparql@master] Use new path

https://gerrit.wikimedia.org/r/356473

Change 356474 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/prince@master] Use new path & add README

https://gerrit.wikimedia.org/r/356474

Change 356475 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/rainbow@master] Use new path

https://gerrit.wikimedia.org/r/356475

Change 356469 merged by Bearloga:
[wikimedia/discovery/wonderbolt@master] Use new path

https://gerrit.wikimedia.org/r/356469

Change 356472 merged by Bearloga:
[wikimedia/discovery/wetzel@master] Use new path

https://gerrit.wikimedia.org/r/356472

Change 356470 merged by Bearloga:
[wikimedia/discovery/polloi@master] Update dataset path, downloaded prefixes, and documentation

https://gerrit.wikimedia.org/r/356470

Change 356474 merged by Bearloga:
[wikimedia/discovery/prince@master] Use new path & add README

https://gerrit.wikimedia.org/r/356474

Change 356473 merged by Bearloga:
[wikimedia/discovery/twilightsparql@master] Use new path

https://gerrit.wikimedia.org/r/356473

Change 356468 merged by Bearloga:
[wikimedia/discovery/experimental@master] Use new path

https://gerrit.wikimedia.org/r/356468

Change 356475 merged by Bearloga:
[wikimedia/discovery/rainbow@master] Use new path

https://gerrit.wikimedia.org/r/356475

Change 356476 had a related patch set uploaded (by Bearloga; owner: Bearloga):
[wikimedia/discovery/dashboard@master] Deploy dashboards that use new dataset paths

https://gerrit.wikimedia.org/r/356476

Change 356476 merged by Bearloga:
[wikimedia/discovery/dashboard@master] Deploy dashboards that use new dataset paths

https://gerrit.wikimedia.org/r/356476

mpopov added subscribers: chelsyx, Milimetric.

@chelsyx: okie dokie, the files have been moved/copied, golden and polloi have been updated, dashboards have been updated/tested/deployed

@Milimetric: the following directories are safe to delete from /a/aggregate-datasets:

  • discovery-forecasts/
  • discovery/
  • discovery_legacy/
  • experimental/
  • external_traffic/
  • maps/
  • portal/
  • search/
  • wdqs/

That's great! Thank you @mpopov !

Thank you both very much, I'm actually very excited to be cleaning this up :)

I'll be doing this soon (keeping backups for a while just in case other people can't live without your precious data). I'll announce it on the analytics and research lists when it's done.

debt closed this task as Resolved.Jun 7 2017, 3:56 PM