As we're thinking and exploring ways to improve monitoring of the API modules and the ability to enable teams to create their alerts when things break, we need to make sure we're providing a sensible way to get specific monitoring so we can create specific alarms.
The problem
Prometheus labels can by utilized in grafana for dynamic graphs, data "cuts" and focused alerts; most of those are done through targeted label values and regular expressions for targeting pieces of the label properly. The main property for this at the moment is the "path" property, that reflect the endpoint paths, and as a result, they aren't always consistent in allowing us to "cut" the data consistently with information we need.
For example, we currently have prometheus the "path" label that present the REST module paths, and are therefore inconsistent with the way they display the name and version of the module:
- campaignevents_v0_event_registration_id and site_v1_sitemap_indexId (version after module name)
- flaggedrevs_internal_diffheader_oldId_newId (no version at all)
- v1_file_title (version as the module name)
etc.
There are multiple tickets trying to resolve the problem of endpoint path consistency, and while we should resolve this issue, the question of endpoint paths structure (while related to monitoring) is not the same as the structure of monitoring labels that are meant to give us information about what is happening and where in the code.
Reframing the problem
I offer we separate this problem -- endpoints paths consistency vs prometheus representation of endpoints -- and first tackle the latter (prometheus labels and grafana graphs) so we can consistently have enough standardized information to allow for powerful graphs AND alerts.
For this, we need to understand and decide not quite what path/structure the labels should have, but rather what information/data we want to have in them. This should potentially be a little less complicated than thinking about the full endpoint pathways because unlike the endpoint paths, prometheus labels aren't meant for public consumption -- and we "just" need to use them for data management, monitoring, graphs and alerts.
The proposed solution
The solution offered here will solve the need to monitor API modules and provide focused alerts in multiple "cuts" of the data (module types, module names, status codes, etc) across the Wikimedia API systems. It is also designed to give API owners the power to monitor specific code-paths on top of the "basic" request/response monitoring that we currently have.
These types of metrics will be agnostic to the endpoint paths and are aimed to reflect module groupings and behaviors, which means they should not be affected by specific endpoint changes or even where the API module is (in REST or ACTION) api, and can therefore allow feature teams to produce targeted Grafana graphs and alerts.
Details
The patch (attached below) provides the following behavior:
Metrics representation
It creates new metrics:
- rest_api_modules_hits_total (counter)
- rest_api_modules_latency_seconds (observable)
A followup patch creates two more for Action API:
- action_api_modules_hits_total (counter)
- action_api_modules_latency_seconds (observable)
These new metrics share these labels:
- api_type: The type of API endpoint that the metric reflects; currently REST_API or ACTION_API (and can be extended further if needed)
- api_module: The module name (a logical collection of endpoints or behaviors, defined by the API module owner)
- api_scope: Whether the module is public or private (internal)
- path: (existing label) will provide a string that reflects data about the module and sub-behavior, if the API owner decided to monitor a code-path
- method (existing label) the method of the request (GET/PUT etc)
- status: The status code of the result (200/400/etc)
These details will be automatically collected on execution of the endpoint, and use the programmatic details we already have, though those details can also be overridden in the code.
Providing consistency in the data
This approach will ensure we get at least closer to providing consistent data representation so we can "cut" elements in grafana to produce consistent dashboards and alerts.
- The new labels provide easier way to fetch module-related information (rather than having to use regex on inconsistent "path"-like values that exist today)
- The label values are as consistent as possible between modules and endpoints
- The label values are as conceptually consistent as possible between REST and ACTION apis.
Assumptions and decisions
We are aiming to provide a consistent experience for the monitoring of our API ecosystem. However, there are inherent inconsistencies -- especially conceptually -- between REST and ACTION that we had to make some assumptions and decisions on how to provide a helpful consolidation:
Conceptual meaning of "Module"
The concept of "module" is different between Action and Rest APIs. In REST, a module is a collection of endpoints (Handlers) and in ActionAPI a "module" is a (for the most part) a single endpoint. Action API does not have a concept of 'groups' of related endpoints for the moment, though there are some attempts to define module information that provides some helpful groupings for documentation and behavior.
Given that, and the fact that we would like to consolidate on the conceptual meaning that an API Module is a *collection* of endpoints, the labels would be represented as:
| label | REST API | ACTION API |
|---|---|---|
| api_module | The module name | The NAMESPACE of the module class |
| api_endpoint | The handler ID (from OpenAPISpec) | The module name |
The NAMESPACE isn't an ideal representation, but when we fetch the metrics, it can give us a general grouping of where the API module comes from within extensions, and so can still provide some basic grouping while we figure out a more systemic way to document and define ActionAPI module groupings later in the system.
Status code inconsistency
The label status already exists in other metrics, and will be reused here to represent the status code of the response. However, there is another inconsistency here between REST and ACTION APIs, both practically and conceptually.
REST API represents the response fairly straightforward. However, Action API usually converts failures into a success so it can send a user a successful JSON response with a failure message. We can't quite rely on the response status, then, but the action API code does collect any failure codes it encountered for the logs -- and that can be used for the status label.
The only caveat here, is that ActionAPI seems to enable (and expect?) multiple status codes with an underscore separator. We will store those in the status label. This will still allow us to extract all metrics that have a certain code (using RegEx) but will still result in a slight mismatch between Rest and Action in the raw values.