Page MenuHomePhabricator
Create Task
Maniphest T327828

AQS 2.0: Edit Analytics: Implement endpoints
Closed, ResolvedPublic1 Estimated Story Points
Actions

Description

Implement endpoints for the Edit Analytics service. These endpoints are Druid-based.

Completion criteria: the following endpoints are implemented:

  • edits/aggregate
  • edits/per-page
  • bytes_difference/net/aggregate
  • bytes_difference/net/per-page
  • bytes_difference/absolute/aggregate
  • bytes_difference/absolute/per-page
  • edited_pages/new
  • edited_pages/aggregate
  • edited_pages/top-by-edits
  • edited_pages/top-by-net-bytes-difference
  • edited_pages/top-by-absolute-bytes-difference

See the parent task for discussion on a reusable package for commonalities between endpoints in this service and endpoints in the Editor Analytics service. There is additional related discussion in T288301: AQS 2.0:Wikistats 2 service.

Extremely rough proof-of-concept code for querying Druid can be found here.

Keep in mind that editors analytics (the other Druid-base service) endpoints have been already implemented in T327829, so that service can be a good source of knowledge about how to query Druid. As far as we have explored query types are pretty similar in both services.

Useful information about the Druid schema can be found here

Some information about which data is available on Druid test environment can be found in T336405

Details

SubjectRepoBranchLines +/-
Passing activity-level value to the query for edited_pages_aggregategenerated-data-platform/aqs/edit-analyticsmain+5 -4
All endpoints are done and seem to work using reusable Druid functions to reduce code duplication. The purpose of this change is review the reusable functions for querying Druid to see if they are ok or could be improved. Or even if code duplication could be reduce more. I'll keep working on it but I think it's the right time to listen to other opinions.generated-data-platform/aqs/edit-analyticsmain+2 K -299
Customize query in gerrit

Related Objects
Search...

Event Timeline

BPirkle created this task.Jan 24 2023, 11:24 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJan 24 2023, 11:24 PM
BPirkle added a parent task: T327817: Edit Analytics Service.Jan 24 2023, 11:25 PM
BPirkle mentioned this in T317729: AQS 2.0: Wikistats2: Implement endpoints.Jan 24 2023, 11:29 PM
VirginiaPoundstone moved this task from Incoming to Must do now on the API Platform board.Feb 1 2023, 2:27 AM
VirginiaPoundstone moved this task from Must do now to Needs Grooming on the API Platform board.Feb 16 2023, 11:16 PM
JArguello-WMF moved this task from Needs Grooming to Incoming on the API Platform board.Feb 21 2023, 7:06 PM
JArguello-WMF removed a project: API Platform.Apr 18 2023, 4:17 PM
JArguello-WMF moved this task from Incoming to AQS 2.0 Backlog on the AQS2.0 board.May 1 2023, 1:28 PM
BPirkle mentioned this in T336378: Edit Analytics: Create Go Service.May 10 2023, 1:39 PM
Sfaci updated the task description. (Show Details)Jun 2 2023, 3:47 PM
Sfaci updated the task description. (Show Details)Jun 19 2023, 3:17 PM
Sfaci updated the task description. (Show Details)Jun 19 2023, 3:26 PM
Sfaci updated the task description. (Show Details)
Sfaci assigned this task to FGoodwin.Jun 28 2023, 7:33 AM
Sfaci removed FGoodwin as the assignee of this task.
Sfaci triaged this task as Medium priority.
Sfaci edited projects, added AQS2.0 (Sprint 10); removed AQS2.0.
Sfaci added a subscriber: FGoodwin.
Sfaci claimed this task.Jul 3 2023, 2:05 PM
Sfaci moved this task from Next Up to In Progress on the AQS2.0 (Sprint 10) board.
Sfaci removed a subscriber: FGoodwin.
Sfaci updated the task description. (Show Details)Jul 4 2023, 8:13 AM
Sfaci updated the task description. (Show Details)Jul 5 2023, 1:52 PM
Sfaci updated the task description. (Show Details)Jul 5 2023, 3:32 PM
Sfaci updated the task description. (Show Details)Jul 10 2023, 12:54 PM
Sfaci updated the task description. (Show Details)Jul 11 2023, 8:11 AM
gerritbot added a comment.Jul 11 2023, 8:56 AM

Change 937043 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[generated-data-platform/aqs/edit-analytics@main] [DNM] All endpoints are done and seem to work using reusable Druid functions to reduce code duplication. The purpose of this DNM is review the reusable functions for querying Druid to see if they are ok or could be improved. Or even if code duplication could be reduce more. I'll keep working on it but I think it's the right time to listen to other opinions.

https://gerrit.wikimedia.org/r/937043

gerritbot added a project: Patch-For-Review.Jul 11 2023, 8:56 AM
Sfaci moved this task from In Progress to Ready for Code Review/ Ready for Tech input on the AQS2.0 (Sprint 10) board.Jul 13 2023, 10:42 AM
Sfaci moved this task from Ready for Code Review/ Ready for Tech input to In code Review/ Tech Input on the AQS2.0 (Sprint 10) board.Jul 17 2023, 8:12 AM
SGupta-WMF claimed this task.Jul 19 2023, 10:26 AM
SGupta-WMF added a subscriber: Sfaci.
gerritbot added a comment.Jul 20 2023, 8:16 PM

Change 937043 merged by BPirkle:

[generated-data-platform/aqs/edit-analytics@main] All endpoints are done and seem to work using reusable Druid functions to reduce code duplication. The purpose of this change is review the reusable functions for querying Druid to see if they are ok or could be improved. Or even if code duplication could be reduce more. I'll keep working on it but I think it's the right time to listen to other opinions.

https://gerrit.wikimedia.org/r/937043

Maintenance_bot removed a project: Patch-For-Review.Jul 20 2023, 8:31 PM
Sfaci removed SGupta-WMF as the assignee of this task.Jul 21 2023, 7:10 AM
Sfaci added a subscriber: SGupta-WMF.
Sfaci moved this task from In code Review/ Tech Input to Ready for Testing on the AQS2.0 (Sprint 10) board.Jul 21 2023, 7:16 AM
Sfaci added a comment.Jul 21 2023, 10:14 AM

After the last merge (that includes endpoints using the definitive Druid code) some refactoring about common Druid code must be done to move it to aqsassist to be shared by this and editor service. Moving to "Next Up" to keep working on that.

Sfaci moved this task from Ready for Testing to Next Up on the AQS2.0 (Sprint 10) board.Jul 21 2023, 10:16 AM
Sfaci claimed this task.Aug 9 2023, 1:10 PM
Sfaci removed Sfaci as the assignee of this task.Aug 9 2023, 1:14 PM
Sfaci added a comment.Aug 22 2023, 11:47 AM

There is a bug in the edited_pages_aggregate_data.go file in the ProcessEditedPagesAggregateQuery function:

  • activityLevel was not considered when building the parameters map to pass later to the ProcessTimeseriesQuery function. Just the following code must be added when populating the query parameters:
var parameters = DruidQueryParams{
    . . .
    ActivityLevel:         activityLevel,
    . . .
}

Just keep in mind to fix it when we refactor all endpoints for this service

VirginiaPoundstone moved this task from Sprint 10 to To be discussed/To be estimated on the AQS2.0 board.Oct 1 2023, 5:40 PM
VirginiaPoundstone edited projects, added AQS2.0; removed AQS2.0 (Sprint 10).
Sfaci claimed this task.Dec 21 2023, 8:17 AM
Sfaci added a project: API Platform (Sprint 05).
Sfaci edited projects, added Data Products (Data Products Sprint 05); removed API Platform (Sprint 05).
Sfaci moved this task from Sprint Backlog to In Process on the Data Products (Data Products Sprint 05) board.
gerritbot added a comment.Dec 21 2023, 9:06 AM

Change 984802 had a related patch set uploaded (by Santiago Faci; author: Santiago Faci):

[generated-data-platform/aqs/edit-analytics@main] Passing activity-level value to the query for edited_pages_aggregate

https://gerrit.wikimedia.org/r/984802

gerritbot added a project: Patch-For-Review.Dec 21 2023, 9:06 AM
Sfaci added a comment.Dec 21 2023, 9:07 AM

The last MR intends to fix the bug I commented in my previous comment

Sfaci reassigned this task from Sfaci to SGupta-WMF.Dec 21 2023, 9:08 AM
Sfaci moved this task from In Process to Code Review / Tech Input on the Data Products (Data Products Sprint 05) board.
Sfaci set the point value for this task to 1.Dec 21 2023, 9:13 AM
Sfaci added a comment.Dec 21 2023, 10:52 AM

The current bug will be tracked in T353872: Edited pages aggregate endpoint does not filter on activity levels

Sfaci removed SGupta-WMF as the assignee of this task.Dec 21 2023, 10:52 AM
Sfaci removed a project: Data Products (Data Products Sprint 05).
Sfaci moved this task from AQS 2.0 Backlog to DONE on the AQS2.0 board.
gerritbot added a comment.Dec 21 2023, 10:55 AM

Change 984802 merged by jenkins-bot:

[generated-data-platform/aqs/edit-analytics@main] Passing activity-level value to the query for edited_pages_aggregate

https://gerrit.wikimedia.org/r/984802

Maintenance_bot removed a project: Patch-For-Review.Dec 21 2023, 11:30 AM
WDoranWMF closed subtask T353872: Edited pages aggregate endpoint does not filter on activity levels as Resolved.Jan 24 2024, 4:48 PM
VirginiaPoundstone closed this task as Resolved.Mar 1 2024, 9:22 PM
VirginiaPoundstone claimed this task.
VirginiaPoundstone moved this task from DONE to RESOLVED on the AQS2.0 board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL