Page MenuHomePhabricator
Create Task
Maniphest T167539

Final steps to expose project family unique devices data
Closed, ResolvedPublic8 Estimated Story Points
Actions

Description

Last things to are:

  • Copy archived data from /user/joal/wmf/data/archive/unique_devices/project_wide into /wmf/data/archive/unique_devices/project_wide to be visible for the outside world.
  • Relaunch oozie unique_device_project_wide daily and monthly jobs to use the production archive folder
  • Ensure documentation is complete in wikitech (and maybe README
  • Send an email to anounce !
  • We should expose three numbers: uniques_offset, uniques_underestimate , uniques_estimate (the sum of the two others)

https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Unique_Devices/Last_access_solution

Details

SubjectRepoBranchLines +/-
Add project families to uniques loading jobanalytics/refinerymaster+142 -30
Update insert-test-data script for unique changeanalytics/aqs/deploymaster+2 -2
Add offset and underestimate to uniques v1 specanalytics/aqsmaster+8 -0
Add offset and underestimate to uniques loading jobanalytics/refinerymaster+24 -16
Remove underestimate and offset before sending uniques responseanalytics/aqsmaster+13 -0
Add offset and underestimate to uniques table schemaanalytics/aqsmaster+4 -2
Add info about new fields in the uniques dumpoperations/puppetproduction+4 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

JAllemandou created this task.Jun 9 2017, 6:43 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 9 2017, 6:43 PM
• Nuria renamed this task from Finalise unique_device_project_wide to Final steps to expose project wide unique devices data .Jun 9 2017, 6:45 PM
• Nuria edited projects, added Analytics-Kanban; removed Analytics.
• Nuria set the point value for this task to 5.
JAllemandou claimed this task.Jun 12 2017, 9:32 AM
JAllemandou removed JAllemandou as the assignee of this task.
JAllemandou moved this task from Next Up to In Progress on the Analytics-Kanban board.
JAllemandou moved this task from In Progress to Next Up on the Analytics-Kanban board.
• Nuria renamed this task from Final steps to expose project wide unique devices data to Final steps to expose project family unique devices data .Sep 14 2017, 4:21 PM
• Nuria added a parent task: T169550: Final Vetting of Family Wide unique devices data .
Milimetric added a subtask: T168477: Update html language for per-domain uniques.Oct 3 2017, 10:20 PM
• Nuria moved this task from Next Up to Paused on the Analytics-Kanban board.Jan 22 2018, 4:50 PM
mforns triaged this task as Low priority.May 7 2018, 3:59 PM
mforns edited projects, added Analytics; removed Analytics-Kanban.
mforns moved this task from Incoming to Analytics Query Service on the Analytics board.
• Nuria assigned this task to • fdans.Nov 3 2018, 4:36 AM
• Nuria added a project: Analytics-Kanban.
• Nuria updated the task description. (Show Details)
• Nuria subscribed.

Let's go over this on our next grosking session, once this work is done we can retake work on wikistats UI to be able to select project families.

• Nuria changed the point value for this task from 5 to 8.Nov 3 2018, 4:37 AM
• Nuria moved this task from Paused to Next Up on the Analytics-Kanban board.
• Nuria updated the task description. (Show Details)Nov 5 2018, 5:48 PM
• fdans moved this task from Next Up to In Progress on the Analytics-Kanban board.Nov 8 2018, 2:35 PM
• fdans added a comment.Nov 12 2018, 5:16 PM

Before adding project families we have to backfill the other two fields for per domain data. Then we'll load data to cassandra and move on to T205665.

gerritbot subscribed.Nov 13 2018, 3:04 PM

Change 473228 had a related patch set uploaded (by Fdans; owner: Fdans):
[operations/puppet@production] Add info about new fields in the uniques dump

https://gerrit.wikimedia.org/r/473228

gerritbot added a project: Patch-For-Review.Nov 13 2018, 3:04 PM
gerritbot added a comment.Nov 13 2018, 5:25 PM

Change 473228 merged by Elukey:
[operations/puppet@production] Add info about new fields in the uniques dump

https://gerrit.wikimedia.org/r/473228

gerritbot added a comment.Nov 15 2018, 10:13 AM

Change 473708 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/aqs@master] Add offset and underestimate to uniques table schema

https://gerrit.wikimedia.org/r/473708

gerritbot added a comment.Nov 15 2018, 2:35 PM

Change 473746 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/refinery@master] [wip] Add offset and underestimate to uniques loading job

https://gerrit.wikimedia.org/r/473746

• Nuria added a subtask: T164201: AQS unique devices api should report offset/underestimate separately .Nov 15 2018, 4:46 PM
gerritbot added a comment.Nov 19 2018, 1:17 PM

Change 473708 merged by Fdans:
[analytics/aqs@master] Add offset and underestimate to uniques table schema

https://gerrit.wikimedia.org/r/473708

Stashbot subscribed.Nov 19 2018, 1:27 PM

Mentioned in SAL (#wikimedia-analytics) [2018-11-19T13:27:53Z] <fdans> deploying aqs to add new fields to uniques dataset (T167539)

gerritbot added a comment.Nov 19 2018, 2:52 PM

Change 474701 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/aqs@master] Remove underestimate and offset before sending uniques response

https://gerrit.wikimedia.org/r/474701

gerritbot added a comment.Nov 19 2018, 3:08 PM

Change 474701 merged by Milimetric:
[analytics/aqs@master] Remove underestimate and offset before sending uniques response

https://gerrit.wikimedia.org/r/474701

gerritbot added a comment.Nov 21 2018, 12:47 PM

Change 473746 merged by Joal:
[analytics/refinery@master] Add offset and underestimate to uniques loading job

https://gerrit.wikimedia.org/r/473746

gerritbot added a comment.Nov 28 2018, 9:32 AM

Change 476220 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/refinery@master] Add project families to uniques loading job

https://gerrit.wikimedia.org/r/476220

gerritbot added a comment.Nov 28 2018, 3:54 PM

Change 476297 had a related patch set uploaded (by Fdans; owner: Fdans):
[analytics/aqs@master] Add offset and underestimate to uniques v1 spec

https://gerrit.wikimedia.org/r/476297

gerritbot added a comment.Nov 28 2018, 3:55 PM

Change 476297 merged by Milimetric:
[analytics/aqs@master] Add offset and underestimate to uniques v1 spec

https://gerrit.wikimedia.org/r/476297

• fdans moved this task from In Progress to In Code Review on the Analytics-Kanban board.Nov 29 2018, 5:02 PM
gerritbot added a comment.Dec 3 2018, 12:30 PM

Change 477251 had a related patch set uploaded (by Joal; owner: Joal):
[analytics/aqs/deploy@master] Update insert-test-data script for unique change

https://gerrit.wikimedia.org/r/477251

gerritbot added a comment.Dec 3 2018, 12:34 PM

Change 477251 merged by Fdans:
[analytics/aqs/deploy@master] Update insert-test-data script for unique change

https://gerrit.wikimedia.org/r/477251

gerritbot added a comment.Dec 4 2018, 3:52 PM

Change 476220 merged by Fdans:
[analytics/refinery@master] Add project families to uniques loading job

https://gerrit.wikimedia.org/r/476220

• fdans moved this task from In Code Review to Done on the Analytics-Kanban board.Dec 12 2018, 2:52 PM
• Nuria added a comment.Dec 12 2018, 7:45 PM

@fdans please update docs with new per family and the addition of offset and underestimate

• Nuria closed subtask T164201: AQS unique devices api should report offset/underestimate separately as Resolved.Dec 12 2018, 7:49 PM
• Nuria added a comment.Dec 14 2018, 5:44 AM

Docs to be updated located here: https://wikitech.wikimedia.org/wiki/Analytics/AQS/Unique_Devices

• fdans added a comment.Dec 14 2018, 8:14 AM

Examples updated in that page. Also took a look at:

But all info seems to be up to date there.

• Nuria closed this task as Resolved.Dec 14 2018, 3:52 PM
• Nuria closed subtask T168477: Update html language for per-domain uniques as Resolved.Jan 14 2019, 5:53 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL