Page MenuHomePhabricator

Load API request count and latency data from Hadoop to a dashboard
Open, Needs TriagePublic

Description

Action API traffic data (counts, user agents, errors, backend latency) are collected in the ApiAction tables in Hadoop. Currently the only way to use them is by logging in to the stats box and manually running Hive queries, which is not too useful for product management. We should expose them somehow.

This is probably although not necessarily blocked on T137321: Run ETL for wmf_raw.ActionApi into wmf.action_* aggregate tables (making the data collection more production-like).

Related Objects

StatusAssignedTask
ResolvedQgil
ResolvedKeegan
DeclinedNone
ResolvedDeskana
ResolvedAnomie
ResolvedQgil
ResolvedQgil
InvalidNone
InvalidNone
ResolvedNone
DeclinedQgil
ResolvedQgil
OpenNone
ResolvedAnomie
OpenNone
ResolvedTgr
ResolvedAnomie
OpenNone
OpenNone
ResolvedTgr
OpenNone
OpenNone
OpenTgr
ResolvedArielGlenn

Event Timeline

Tgr created this task.Aug 8 2015, 12:41 AM
Tgr claimed this task.
Tgr raised the priority of this task from to Needs Triage.
Tgr updated the task description. (Show Details)
Tgr added subscribers: Tgr, bd808.
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 8 2015, 12:41 AM
Anomie moved this task from Unsorted to Non-Code on the MediaWiki-API board.Aug 9 2015, 10:19 PM

Let us know when you figure out the metric / get it measured and we can help you make a dashboard.

Restricted Application added a subscriber: StudiesWorld. · View Herald TranscriptDec 7 2015, 6:18 PM
Milimetric moved this task from Incoming to Radar on the Analytics board.Dec 7 2015, 6:18 PM
mforns renamed this task from Load API request count and latency data from Hadoop to a dashboard (limn?) to Load API request count and latency data from Hadoop to a dashboard.May 30 2016, 4:41 PM
mforns added a subscriber: mforns.

@Tgr
This can be easily done with reportupdater and it will show on a dashiki instance.
We can help you with that.

Jhernandez triaged this task as Low priority.Feb 27 2019, 4:50 PM
Jhernandez added a subscriber: Jhernandez.

@Tgr Can you add a full description about what this is and move to backlog if it is ours? Thanks

Tgr updated the task description. (Show Details)Feb 27 2019, 6:45 PM
Tgr updated the task description. (Show Details)Feb 27 2019, 6:57 PM

@Jhernandez, added some description. This originally came about when Developer Relations was planning a pivot towards external developers (ie. people who use Wikimedia APIs for mashups but don't use Wikimedia code directly) and was interested in API usage / usability data (hence, T102079: Metrics about the use of the Wikimedia web APIs). The pivot eventually did not happen; exposing API usage data still seems like the sensible thing to do, but I guess today the potentially interested party would be @EvanProdromou, as API PM? Also, back then Reading Infrastructure was the team closest to owning the API so the ApiAction work was done by us. I have no idea how responsibilities are split today within the teams participating in the Better Use of Data CDP.

Anomie moved this task from Non-Code to Non-core-API stuff on the MediaWiki-API board.EditedFeb 27 2019, 9:26 PM
Anomie added a subscriber: Anomie.

Also, back then Reading Infrastructure was the team closest to owning the API so the ApiAction work was done by us. I have no idea how responsibilities are split today within the teams participating in the Better Use of Data CDP.

As far as I can tell, the Action API "ownership" went with me when I moved to the MediaWiki Platform team, and then that team became part of the Core Platform Team. Just like it came with me from MediaWiki Core to Reading Infrastructure (with a brief stop in the Wikimedia MediaWiki API Team) during the Reorg of Doom.

Evan is part of CPT too, and will presumably take over some of the Product Manager aspects of that ownership eventually.

On the other hand, this particular task is more "about" the API than actually within the scope of MediaWiki-API. I don't know who might own WMF-specific dashboards that are done outside of MediaWiki. I don't know anything about the "Better Use of Data CDP".

Tgr added a subscriber: kzimmerman.EditedFeb 27 2019, 9:33 PM

That's Better use of data. I guess @kzimmerman would be the other person who might be able to help prioritize this and decide on ownership.

This is relevant to recent discussions about tracking content consumption, but Product Analytics hasn't dug into API use (yet).

Who are the key stakeholders associated with this task?

Jhernandez raised the priority of this task from Low to Needs Triage.Feb 28 2019, 10:53 AM

Thanks for clarifying @Tgr @Anomie. I've moved it to tracking for reading infrastructure and reset the priority, it seems clear that it we shouldn't currently own this, and the appropriate followup teams have been pinged into the task.

kzimmerman moved this task from Triage to Icebox on the Product-Analytics board.Apr 17 2019, 12:43 AM
Restricted Application added a project: Core Platform Team. · View Herald TranscriptFri, Dec 6, 6:17 PM