⚓ T320537 <spike> Usage and traffic data for WMF APIs

Status	Assigned	Task
In Progress	None	T306043 <API Platform> API Gateway MVP
Open	None	T326697 Harden API Gateway in preparation for RESTbase deprecation service migration
Resolved	VirginiaPoundstone	T326698 Write One-Pager PRD for Harden API Gateway
Open	None	T317673 <API Platform> Documentation for API Platform Value Stream Team
Resolved	VirginiaPoundstone	T317677 Update API Platform Phab description and links
Declined	VirginiaPoundstone	T317678 Create MediaWiki Knowledge Hub Wiki for API Platform Team
Open	None	T319330 Document current WMF service usage across the WMF ecosystem
Open	pmiazga	T326701 Draft Diagrams of current state of API ecosystem
Declined	VirginiaPoundstone	T318805 Collate various API Catalogs
Open	pmiazga	T329419 Architect potential API Gateway Patterns in preparation for services migrated off RESTbase
Resolved	VirginiaPoundstone	T329668 Figure out how to get data on clients calling individual services through RESTbase
Resolved	• EChetty	T285561 API Usage Metrics
Open	None	T333217 Service & API product monitoring instrumentation
Open	VirginiaPoundstone	T285674 Baseline API Usage Metrics
Invalid	VirginiaPoundstone	T320537 <spike> Usage and traffic data for WMF APIs

VirginiaPoundstone reassigned this task from VirginiaPoundstone to hnowlan.Oct 11 2022, 4:40 PM

VirginiaPoundstone updated the task description. (Show Details)

VirginiaPoundstone updated the task description. (Show Details)Oct 11 2022, 4:43 PM

VirginiaPoundstone moved this task from Incoming to Must do now on the API Platform board.Oct 11 2022, 6:10 PM

For most/all of these dashboards be aware of the variable selectors at the top of the page - data will frequently be shown per datacentre, for holistic numbers take measurements from both eqiad and codfw. For many of these services the numbers are not fully representative - for example a page might be requested from mobileapps once but served by restbase many times over a period of time.

Action API/REST API - not sure how we differentiate between these, will need to do a bit of digging

Restbase dashboard: https://grafana-rw.wikimedia.org/d/000000068/restbase?orgId=1

WDQS: https://grafana-rw.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&refresh=1m

For Wikidata API I am assuming this means Wikibase: https://grafana.wikimedia.org/d/MR93RkVWk/wikibase-api?orgId=1&refresh=5m

Eventgate: https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1

Eventstreams: https://grafana.wikimedia.org/d/znIuUcsWz/eventstreams?orgId=1&refresh=1m

Maps - this is a bit of a vague one, tiles per second might be of interest on this dashboard? https://grafana-rw.wikimedia.org/d/000000305/maps-performances?orgId=1

Push Notifications: https://grafana.wikimedia.org/d/NQO_pqvMk/push-notifications

Mobileapps: https://grafana.wikimedia.org/d/5CmeRcnMz/mobileapps?orgId=1

Which recommendation service?

Kask/Sessionstore: https://grafana.wikimedia.org/d/000001590/sessionstore?orgId=1

ORES: https://grafana.wikimedia.org/d/vAN_bQemz/ores-advanced-metrics?orgId=1&refresh=1m

Proton: https://grafana.wikimedia.org/d/U4TuF-lMk/proton?orgId=1

Trending edits: Not sure what this is

Thumbor: qps panels here https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor?orgId=1&refresh=30s

image-suggestion: https://grafana.wikimedia.org/d/SUZQ6rWVz/image-suggestion?orgId=1

hnowlan added a project: Platform Team Workboards (Platform Engineering Reliability).Oct 13 2022, 1:59 PM

hnowlan moved this task from Backlog to Doing 🔨 on the Platform Team Workboards (Platform Engineering Reliability) board.

hnowlan moved this task from Doing 🔨 to In review on the Platform Team Workboards (Platform Engineering Reliability) board.

VirginiaPoundstone closed this task as a duplicate of T319330: Document current WMF service usage across the WMF ecosystem.Oct 14 2022, 11:08 PM

VirginiaPoundstone reopened this task as In Progress.Oct 14 2022, 11:10 PM

VirginiaPoundstone triaged this task as Medium priority.

VirginiaPoundstone moved this task from Must do now to Backlog on the API Platform board.

VirginiaPoundstone moved this task from Backlog to QA/Review on the API Platform board.Oct 25 2022, 4:24 PM

Thanks for all this @hnowlan
Is there a way to see who is driving what traffic? I need to know what bots, apps, and other caller/consumers are using what. This will help me gather requirements and strategize in general about what we can do to be useful to the movement. If there is a way to see the who in these dashboards, then maybe I just need a 15 minute meeting for a tutorial.

Also recommendations was Image recommendations. I'll see if I can learn more about trending edits.

VirginiaPoundstone updated the task description. (Show Details)Oct 25 2022, 7:07 PM

VirginiaPoundstone moved this task from QA/Review to Sprint 01 on the API Platform board.Oct 25 2022, 7:09 PM

VirginiaPoundstone edited projects, added API Platform (Sprint 01); removed API Platform.

JArguello-WMF edited projects, added API Platform (Sprint 02); removed API Platform (Sprint 01).Dec 1 2022, 3:03 PM

JArguello-WMF moved this task from Next Up to In code Review/ Tech Input on the API Platform (Sprint 02) board.

@VirginiaPoundstone How are we doing with this one? Is there anything else you need from @hnowlan? Do you have the traffic info you needed?

JArguello-WMF added a parent task: T285674: Baseline API Usage Metrics.Dec 5 2022, 8:03 PM

JArguello-WMF reassigned this task from hnowlan to VirginiaPoundstone.Dec 8 2022, 5:48 PM

JArguello-WMF added a subscriber: hnowlan.

JArguello-WMF updated the task description. (Show Details)Dec 8 2022, 5:52 PM

hnowlan moved this task from In review to Watching on the Platform Team Workboards (Platform Engineering Reliability) board.Dec 12 2022, 2:35 PM