Page MenuHomePhabricator

Rename last_access_uniques to per-domain uniques
Closed, ResolvedPublic5 Estimated Story Points

Description

For consistency, we rename last_access_uniques to unique_devices/per-domain (to match project-wide).

Things to do to deploy in production (order is somehow important):

  • Stop oozie jobs
      • last_access_uniques_daily and `monthly
      • cassandra_unique_devices_daily and monthly loading job
      • last_access_uniques_daily druid loading jobs
    • Change hive datasets
      • create the new hive tables unique_devices_per_domain_daily and unique_devices_per_domain_monthly
      • copy from last_access_uniques_daily and monthly tables to unique_devices_per_domain_daily and monthly, using uri_host to populate domain
    • Change archive folder and filenames in HDFS
      • create new HDFS archive folder /wmf/data/archive/unique_devices/per_domain
      • copy with renaming the HDFS files /wmf/data/archive/unique_devices/YYYY/YY-MM/unique_devices_daily-YYYY-MM-DD.gz and /wmf/data/archive/unique_devices/YYYY/YY-MM/unique_devices_monthly-YYYY-MM.gz to /wmf/data/archive/unique_devices/per_domain/YYYY/YYYY-MM/unique_devices_per_domain_...
    • Restart oozie jobs
      • unique_devices_per_domain_daily and monthly jobs (from last last_access_uniques job)
      • Cassandra unique_devices_loading_daily and monthlyas coordinators(from last unique_devices loading job, and keep a calendar reminder to restart the bundle at the beginning of next month)
      • unique_devices_per_domain_daily-druid - From beginning of uniques (2015-12-17), since schema change, full reload is needed.
  • Drop previous druid data (datasource and fieldnames change) (see our druid doc)
    • Disable the unique_devices_daily datasource (from druid coordinator UI)
    • Ask druid to delete deep storage data
  • Update documentation for names

Event Timeline

JAllemandou set the point value for this task to 5.
JAllemandou moved this task from Next Up to In Code Review on the Analytics-Kanban board.

Change 358017 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/refinery@master] Correct typo bug of unique-devices-per-domain

https://gerrit.wikimedia.org/r/358017

Change 358017 merged by Ottomata:
[analytics/refinery@master] Correct typo bug of unique-devices-per-domain

https://gerrit.wikimedia.org/r/358017

Change 358386 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] Rename unique devices daily endpoint

https://gerrit.wikimedia.org/r/358386

Change 358386 merged by Elukey:
[operations/puppet@production] pivot: rename unique devices daily endpoint

https://gerrit.wikimedia.org/r/358386

Change 358389 had a related patch set uploaded (by Joal; owner: Joal):
[operations/puppet@production] pivot: rename unique devices daily endpoint

https://gerrit.wikimedia.org/r/358389

Change 358389 merged by Elukey:
[operations/puppet@production] pivot: rename unique devices daily endpoint

https://gerrit.wikimedia.org/r/358389

Milimetric triaged this task as Medium priority.Jun 22 2017, 3:05 PM
Milimetric moved this task from Ready to Deploy to In Code Review on the Analytics-Kanban board.