Page MenuHomePhabricator

AQS 2.0
Open, In Progress, HighPublic1 Estimated Story Points

Description

Analytics Query Service (AQS) is the software behind the /metrics family of endpoints in RESTBase. It is a read-only HTTP proxy to results served from Cassandra and Druid. It is currently based on a very outdated fork of RESTBase, and has received little updates over the years.

As a part of the goal to sunset RESTBase, AQS needs to be migrated to a bespoke service exposed via the API Gateway.


We propose to break down the rewrite largely along dataset boundaries — similar to the module structure in RESTBase — with a separate project used to implement each.

The services were renamed during development. The names as of Jan 2023 are:

  • Page Analytics (was pageviews)
  • Device Analytics (was unique devices)
  • Edit Analytics (the subset of endpoints previously considered under "wikistats2" that pertain to edits)
  • Editor Analytics (the subset of endpoints previously considered under "wikistats2" that pertain to editors)
  • Media Analytics (was mediarequests)
  • Geo Analytics (was called both geoeditors and editors in different contexts)

The breakdown of endpoints by service can be found here. The remainder of this task description has been left unedited, for comparison.


The resulting services will be proxied by RESTBase and/or the API Gateway (the former to eventually be deprecated in favor of the latter) in order to maintain complete compatibility with the existing API.

The target language for these implementations is Go. While a complete comparison of Javascript/NodeJS and Go is out of scope for this issue, the (simplified) rationale is:

  • Strong, static typing; Statically typed languages eliminate entire classes of bugs common to dynamic languages, improve security, and making code easier to reason about
  • Ease of use; Go is more obvious, more explicit, and easier to understand. Complicated concepts like concurrency are easier to get right
  • Performance; Service latency can be expected to be both lower, but more importantly, more predictable with Go

Overview

  1. Implement the new, stand-alone AQS service(s)
  2. Deploy to k8s
  3. Expose the /metrics hierarchy from the new service(s) using the API Gateway
  4. Switch RESTBase to proxying requests from the old AQS service, to the new k8s-based one
  5. Deprecate the http://{project}/api/rest_v1/metrics resources
  6. Eventually phase out the RESTBase /metrics hierarchy

Solving this will make us progress on multiple fronts: T198901 T262315


NOTE: This will be picked up by Platform Engineering, with support from Analytics.

See also:
https://wikitech.wikimedia.org/wiki/Analytics/Systems/AQS
https://github.com/wikimedia/analytics-aqs
https://github.com/wikimedia/restbase
https://wikimedia.org/api/rest_v1/
https://gitlab.wikimedia.org/eevans/aqs (work-in-progress AQS 2.0 code repository)

Related Objects

View Standalone Graph
This task is connected to more than 200 other tasks. Only direct parents and subtasks are shown here. Use View Standalone Graph to show more of the graph.
StatusSubtypeAssignedTask
In ProgressNone
In ProgressDAbad
InvalidNone
ResolvedNone
ResolvedNone
ResolvedSGupta-WMF
InvalidBPirkle
ResolvedNone
ResolvedSGupta-WMF
ResolvedNone
ResolvedNone
Openapaskulin
ResolvedBPirkle
ResolvedBPirkle
ResolvedBPirkle
DeclinedNone
ResolvedBPirkle
ResolvedBPirkle
Resolvedcodebug
ResolvedSGupta-WMF
DuplicateSpikeVirginiaPoundstone
Resolvedcodebug
ResolvedNone
ResolvedNone
ResolvedSpikeVirginiaPoundstone
Opendaniel
ResolvedBPirkle
ResolvedBPirkle

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Eevans updated the task description. (Show Details)

We can use https://gitlab.wikimedia.org/eevans/aqs to get started, and open merge-requests for code review there. Depending on the state of the Gitlab rollout as we get nearer completion, we can either move it to a dedicated project, or set it up in Gerrit in the usual way.

If there are no objections, we'll implement the new service Go.

DAbad added subscribers: nnikkhoui, BPirkle, DAbad.

AQS 2.0 to API Platform

  • as per team meeting AQS 2.0 work will begin to fall under the API Platform workstream
  • @Eevans will start to include @nnikkhoui and @BPirkle in code reviews & in 2x weekly standups for AQS 2.0
DAbad changed the task status from Open to In Progress.Apr 26 2022, 2:22 PM
DAbad claimed this task.
DAbad raised the priority of this task from Medium to High.

August 30, 2022

BPirkle added a subscriber: Unknown Object (User).Sep 7 2022, 1:43 PM