Monthly aggregate endpoint returns unexpected results and invalid timestamp
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	MusikAnimal
	Jan 25 2017, 10:04 PM

Description

Example per-article monthly data for Jan and Feb 2016:
https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/user/Barack_Obama/monthly/2016010100/2016030100

The response gives timestamps with the 1st day of the month in YYYYMMDDHH format, e.g. 2016010100 and 2016020100

Example aggregate monthly data for Jan and Feb 2016:
https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/fr.wikipedia.org/all-access/user/monthly/2016010100/2016030100

The response gives back an invalid timestamp, with zero as the day of the month, 2016010000 and 2016020000. The response timestamp should match what is considered valid for the request, e.g. if I use 2016010000 and 2016020000 I get an error:
https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/fr.wikipedia.org/all-access/user/monthly/2016010000/2016020000

Details

Subject	Repo	Branch	Lines +/-
Change keyspace name in per project cassandra oozie job	analytics/refinery	master	+3 -3
Change keyspace name to project_v2 in fake data script	analytics/aqs/deploy	master	+1 -1
Switch table pointer to v2 in per project endpoint	analytics/aqs	master	+2 -28
Use v2 table in Cassandra, switch to padded day timestamp	analytics/refinery	master	+2 -2
Add secondary table endpoint to populate Cassandra with correct timestamps	analytics/aqs	master	+29 -3

Customize query in gerrit

Related Objects
Search...

		Status	Subtype	Assigned	Task
		Resolved		JAllemandou	T156312 Monthly aggregate endpoint returns unexpected results and invalid timestamp
		Open		None	T148610 Add hourly stats to Siteviews

Event Timeline

MusikAnimal created this task.Jan 25 2017, 10:04 PM

Restricted Application added a project: Analytics. · View Herald TranscriptJan 25 2017, 10:04 PM

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

MusikAnimal renamed this task from Invalid timestamp returned from monthly aggregate endpoint to Monthly aggregate endpoint returns invalid timestamp.Jan 25 2017, 10:04 PM

And actually, it looks like the aggregate monthly endpoint is also returning the wrong months. I used January 1st through March 1st to get data for January and February. Here's that on per-article:
https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/user/Barack_Obama/monthly/2016010100/2016030100

But for aggregate I get February and March:
https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/fr.wikipedia.org/all-access/user/monthly/2016010100/2016030100

MusikAnimal renamed this task from Monthly aggregate endpoint returns invalid timestamp to Monthly aggregate endpoint returns unexpected results and invalid timestamp.Jan 25 2017, 10:12 PM

Thanks for ping, Will look into it, hopefully next week.

• Nuria edited projects, added Analytics-Kanban; removed Analytics.Jan 26 2017, 4:52 PM

MusikAnimal mentioned this in T151373: Implement a monthly granularity view for Pageviews.Jan 26 2017, 11:21 PM

MusikAnimal added a subtask: T148610: Add hourly stats to Siteviews.

• Nuria claimed this task.Jan 27 2017, 1:19 AM

First thing I'll do is to load test data on beta so we can actually test fix.

• Nuria reassigned this task from • Nuria to • fdans.Jan 30 2017, 6:25 PM

Let's translate: 201001000000 (not really a date) to 201001010000

create new namespace
load table in new name space
swap name space on aqs

Change 338898 had a related patch set uploaded (by Fdans):
Add secondary sys endpoint to populate Cassandra with correct timestamps

https://gerrit.wikimedia.org/r/338898

gerritbot added a project: Patch-For-Review.Feb 20 2017, 11:05 PM

• fdans moved this task from In Progress to In Code Review on the Analytics-Kanban board.Feb 22 2017, 4:05 PM

Change 338898 merged by Fdans:
Add secondary table endpoint to populate Cassandra with correct timestamps

https://gerrit.wikimedia.org/r/338898

Change 340093 had a related patch set uploaded (by Fdans):
Use v2 table in Cassandra, switch to padded day timestamp

https://gerrit.wikimedia.org/r/340093

Change 340093 merged by Joal:
[analytics/refinery] Use v2 table in Cassandra, switch to padded day timestamp

https://gerrit.wikimedia.org/r/340093

Change 342205 had a related patch set uploaded (by Fdans):
[analytics/refinery] Change keyspace name in per project cassandra oozie job

https://gerrit.wikimedia.org/r/342205

• fdans moved this task from In Code Review to Ready to Deploy on the Analytics-Kanban board.Mar 10 2017, 2:06 PM

Change 342486 had a related patch set uploaded (by Fdans):
[analytics/aqs] Switch table pointer to v2 in per project endpoint

https://gerrit.wikimedia.org/r/342486

Change 342486 merged by Fdans:
[analytics/aqs] Switch table pointer to v2 in per project endpoint

https://gerrit.wikimedia.org/r/342486

Change 342876 had a related patch set uploaded (by Fdans):
[analytics/aqs/deploy] Change keyspace name to project_v2 in fake data script

https://gerrit.wikimedia.org/r/342876

Change 342876 merged by Joal:
[analytics/aqs/deploy] Change keyspace name to project_v2 in fake data script

https://gerrit.wikimedia.org/r/342876

• fdans reassigned this task from • fdans to JAllemandou.Mar 16 2017, 11:08 AM

• fdans moved this task from Ready to Deploy to In Progress on the Analytics-Kanban board.

• fdans subscribed.

Change 342205 merged by Joal:
[analytics/refinery] Change keyspace name in per project cassandra oozie job

https://gerrit.wikimedia.org/r/342205

@MusikAnimal : we are still doing some code on hive to load data with appropriate dates but on your end you should see the date issue having been fixed. Please do check, note that for monthly data dates need to be inclusive so https://wikimedia.org/api/rest_v1/metrics/pageviews/aggregate/fr.wikipedia.org/all-access/user/monthly/2016010100/2016030100 will not return data for March.

JAllemandou moved this task from In Progress to Ready to Deploy on the Analytics-Kanban board.Mar 17 2017, 10:52 AM

JAllemandou moved this task from Ready to Deploy to Done on the Analytics-Kanban board.Mar 20 2017, 3:43 PM

@Nuria Works great, thank you! :)

• Nuria closed this task as Resolved.Mar 22 2017, 7:45 PM

BPirkle mentioned this in T305502: Audit/review pageviews test cases.Apr 19 2022, 12:58 AM