Analytics Query Service (AQS) is the software behind the /metrics family of endpoints in RESTBase. It is a read-only HTTP proxy to results served from Cassandra and Druid. It is currently based on a very outdated fork of RESTBase, and has received little updates over the years.
As a part of the goal to sunset RESTBase, AQS needs to be migrated to a bespoke service exposed via the API Gateway.
We propose to break down the rewrite largely along dataset boundaries — similar to the module structure in RESTBase — with a separate project used to implement each.
The services were renamed during development. The names as of Jan 2023 are:
- Page Analytics (was pageviews)
- Device Analytics (was unique devices)
- Edit Analytics (the subset of endpoints previously considered under "wikistats2" that pertain to edits)
- Editor Analytics (the subset of endpoints previously considered under "wikistats2" that pertain to editors)
- Media Analytics (was mediarequests)
- Geo Analytics (was called both geoeditors and editors in different contexts)
The breakdown of endpoints by service can be found here. The remainder of this task description has been left unedited, for comparison.
The resulting services will be proxied by RESTBase and/or the API Gateway (the former to eventually be deprecated in favor of the latter) in order to maintain complete compatibility with the existing API.
The target language for these implementations is Go. While a complete comparison of Javascript/NodeJS and Go is out of scope for this issue, the (simplified) rationale is:
- Strong, static typing; Statically typed languages eliminate entire classes of bugs common to dynamic languages, improve security, and making code easier to reason about
- Ease of use; Go is more obvious, more explicit, and easier to understand. Complicated concepts like concurrency are easier to get right
- Performance; Service latency can be expected to be both lower, but more importantly, more predictable with Go
Overview
- Implement the new, stand-alone AQS service(s)
- Deploy to k8s
- Expose the /metrics hierarchy from the new service(s) using the API Gateway
- Switch RESTBase to proxying requests from the old AQS service, to the new k8s-based one
Solving this will make us progress on multiple fronts: T198901 T262315
Follow-up deprecation work to be done in T390065: AQS 2.0: follow-up deprecation work
See also:
https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS
https://github.com/wikimedia/analytics-aqs
https://github.com/wikimedia/restbase
https://wikimedia.org/api/rest_v1/
https://gitlab.wikimedia.org/eevans/aqs (work-in-progress AQS 2.0 code repository)