Page MenuHomePhabricator

Final steps to expose project family unique devices data
Closed, ResolvedPublic8 Story Points

Description

Last things to are:

  • Copy archived data from /user/joal/wmf/data/archive/unique_devices/project_wide into /wmf/data/archive/unique_devices/project_wide to be visible for the outside world.
  • Relaunch oozie unique_device_project_wide daily and monthly jobs to use the production archive folder
  • Ensure documentation is complete in wikitech (and maybe README
  • Send an email to anounce !
  • We should expose three numbers: uniques_offset, uniques_underestimate , uniques_estimate (the sum of the two others)

https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Unique_Devices/Last_access_solution

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 9 2017, 6:43 PM
Nuria renamed this task from Finalise unique_device_project_wide to Final steps to expose project wide unique devices data .Jun 9 2017, 6:45 PM
Nuria edited projects, added Analytics-Kanban; removed Analytics.
Nuria set the point value for this task to 5.
JAllemandou removed JAllemandou as the assignee of this task.
JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.
JAllemandou moved this task from In Progress to Next Up on the Analytics-Kanban board.
Nuria renamed this task from Final steps to expose project wide unique devices data to Final steps to expose project family unique devices data .Sep 14 2017, 4:21 PM
Nuria moved this task from Next Up to Paused on the Analytics-Kanban board.Jan 22 2018, 4:50 PM
mforns triaged this task as Low priority.May 7 2018, 3:59 PM
mforns edited projects, added Analytics; removed Analytics-Kanban.
mforns moved this task from Incoming to Analytics Query Service on the Analytics board.
Nuria assigned this task to fdans.Nov 3 2018, 4:36 AM
Nuria added a project: Analytics-Kanban.
Nuria updated the task description. (Show Details)
Nuria added a subscriber: Nuria.

Let's go over this on our next grosking session, once this work is done we can retake work on wikistats UI to be able to select project families.

Nuria changed the point value for this task from 5 to 8.Nov 3 2018, 4:37 AM
Nuria moved this task from Paused to Next Up on the Analytics-Kanban board.
Nuria updated the task description. (Show Details)Nov 5 2018, 5:48 PM
fdans moved this task from Next Up to In Progress on the Analytics-Kanban board.Nov 8 2018, 2:35 PM
fdans added a comment.Nov 12 2018, 5:16 PM

Before adding project families we have to backfill the other two fields for per domain data. Then we'll load data to cassandra and move on to T205665.

Change 473228 had a related patch set uploaded (by Fdans; owner: Fdans):
[operations/puppet@production] Add info about new fields in the uniques dump

https://gerrit.wikimedia.org/r/473228

Change 473228 merged by Elukey:
[operations/puppet@production] Add info about new fields in the uniques dump

https://gerrit.wikimedia.org/r/473228

Change 473708 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/aqs@master] Add offset and underestimate to uniques table schema

https://gerrit.wikimedia.org/r/473708

Change 473746 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/refinery@master] [wip] Add offset and underestimate to uniques loading job

https://gerrit.wikimedia.org/r/473746

Change 473708 merged by Fdans:
[analytics/aqs@master] Add offset and underestimate to uniques table schema

https://gerrit.wikimedia.org/r/473708

Mentioned in SAL (#wikimedia-analytics) [2018-11-19T13:27:53Z] <fdans> deploying aqs to add new fields to uniques dataset (T167539)

Change 474701 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/aqs@master] Remove underestimate and offset before sending uniques response

https://gerrit.wikimedia.org/r/474701

Change 474701 merged by Milimetric:
[analytics/aqs@master] Remove underestimate and offset before sending uniques response

https://gerrit.wikimedia.org/r/474701

Change 473746 merged by Joal:
[analytics/refinery@master] Add offset and underestimate to uniques loading job

https://gerrit.wikimedia.org/r/473746

Change 476220 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/refinery@master] Add project families to uniques loading job

https://gerrit.wikimedia.org/r/476220

Change 476297 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/aqs@master] Add offset and underestimate to uniques v1 spec

https://gerrit.wikimedia.org/r/476297

Change 476297 merged by Milimetric:
[analytics/aqs@master] Add offset and underestimate to uniques v1 spec

https://gerrit.wikimedia.org/r/476297

Change 477251 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/aqs/deploy@master] Update insert-test-data script for unique change

https://gerrit.wikimedia.org/r/477251

Change 477251 merged by Fdans:
[analytics/aqs/deploy@master] Update insert-test-data script for unique change

https://gerrit.wikimedia.org/r/477251

Change 476220 merged by Fdans:
[analytics/refinery@master] Add project families to uniques loading job

https://gerrit.wikimedia.org/r/476220

fdans moved this task from In Code Review to Done on the Analytics-Kanban board.Dec 12 2018, 2:52 PM
Nuria added a comment.Dec 12 2018, 7:45 PM

@fdans please update docs with new per family and the addition of offset and underestimate

Nuria closed this task as Resolved.Dec 14 2018, 3:52 PM