===== Goal
Collect all inconsistencies found in AQS 1.0 services and some existing issues in AQS 2.0 services for possible future improvements
===== Current issues
- **Date range**: **[Breaking change]** Which range specify a full month? e.g.: 20230501/20230601 vs 20230501/20230531 between edit-analytics and device-analytics. "One month" has different considerations depending on the service.
- https://wikimedia.org/api/rest_v1/metrics/edits/aggregate/ab.wikipedia/user/content/monthly/20230201/20230229
```
{
"type": "https://mediawiki.org/wiki/HyperSwitch/errors/invalid_request",
"method": "get",
"detail": [
"end timestamp is invalid, must be a valid date in YYYYMMDD format"
],
"uri": "/analytics.wikimedia.org/v1/edits/aggregate/ab.wikipedia/user/content/monthly/20230201/20230229"
}
```
- https://wikimedia.org/api/rest_v1/metrics/unique-devices/en.wikipedia.org/all-sites/monthly/20200201/20200229
```
{
"items": [
{
"project": "en.wikipedia",
"access-site": "all-sites",
"granularity": "monthly",
"timestamp": "20200201",
"devices": 875114780,
"offset": 309778523,
"underestimate": 565336257
}
]
}
```
- **Response structure conventions**: **[Breaking change]** e.g.: lower_snake_case vs kebab-case between media-analytics & editor-analytics (sometimes even in the same service: editor/top-by-edits). Some services use lower_snake_case and others kebab-case. Some times even both conventions can be considered in the same response for the same service
- https://wikimedia.org/api/rest_v1/metrics/mediarequests/aggregate/all-referers/all-media-types/all-agents/monthly/20200101/20200330
```
{
"items": [
{
"referer": "all-referers",
"media_type": "all-media-types",
"agent": "all-agents",
"granularity": "monthly",
"timestamp": "2020010100",
"requests": 77922346640
},
{
"referer": "all-referers",
"media_type": "all-media-types",
"agent": "all-agents",
"granularity": "monthly",
"timestamp": "2020020100",
"requests": 74434493109
}
]
}
```
- https://wikimedia.org/api/rest_v1/metrics/editors/aggregate/ab.wikipedia/user/non-content/1..4-edits/daily/2023040112/2023050112
```
{
"items": [
{
"project": "ab.wikipedia",
"editor-type": "user",
"page-type": "non-content",
"activity-level": "1..4-edits",
"granularity": "daily",
"results": [
{
"timestamp": "2023-04-01T00:00:00.000Z",
"editors": 0
},
```
- **Error response structure**: **[Breaking change]** e.g.: detail[] vs detail between editor-analytics and media-analytics (this issue was already discussed and fixed. detail field is no longer an array in AQS 2.0). We don't have to do anything because it's already considered and fixed in all the AQS 2.0 services
- https://wikimedia.org/api/rest_v1/metrics/editors/aggregate/ab.wikipedia/user/non-constent/1..4-edits/daily/2024040112/2024050112
```
{
"type": "https://mediawiki.org/wiki/HyperSwitch/errors/bad_request",
"title": "Invalid parameters",
"method": "get",
"detail": "data.params['page-type'] should be equal to one of the allowed values: [all-page-types, content, non-content]",
"uri": "/wikimedia.org/v1/metrics/editors/aggregate/ab.wikipedia/user/non-constent/1..4-edits/daily/2024040112/2024050112"
}
```
- https://wikimedia.org/api/rest_v1/metrics/mediarequests/aggregate/all-referers/all-media-types/all-agents/daily/20200101/2023053000a
```
{
"type": "https://mediawiki.org/wiki/HyperSwitch/errors/invalid_request",
"method": "get",
"detail": [
"end timestamp is invalid, must be a valid date in YYYYMMDD format"
],
"uri": "/analytics.wikimedia.org/v1/mediarequests/aggregate/all-referers/all-media-types/all-agents/daily/20200101/2023053000a"
}
```
- **No data found error**: **[Breaking change]** e.g.: empty 200 OK vs 404 between edit-analytics & device-analytics. Some services consider an empty results array as a valid response. Others respond with a 404 Error. Both cases could be consider as ok but we should decide the same for all the services
- https://wikimedia.org/api/rest_v1/metrics/edits/aggregate/ab.wikipedia/user/content/daily/20240301/20240302
```
{
"items": [
{
"project": "ab.wikipedia",
"editor-type": "user",
"page-type": "content",
"granularity": "daily",
"results": []
}
]
}
```
- https://wikimedia.org/api/rest_v1/metrics/unique-devices/en.wikipedia.org/all-sites/daily/20240101/20240101
```
{
"type": "https://mediawiki.org/wiki/HyperSwitch/errors/not_found",
"title": "Not found.",
"method": "get",
"detail": "The date(s) you used are valid, but we either do not have data for those date(s), or the project you asked for is not loaded yet. Please check https://wikimedia.org/api/rest_v1/?doc for more information.",
"uri": "/analytics.wikimedia.org/v1/unique-devices/en.wikipedia.org/all-sites/daily/20240101/20240101"
}
```
- Documentation: **[Non breaking change]** kebab-case vs lower_snake_case when specifying path parameters in some services (see AQS 1.0 OpenAPI Spec). This inconsistency only affects to the OpenAPI specification document and it's being fixed while writing the docs for new AQS 2.0 services
- **Projects in personal repos**: Some AQS related projects are hosted in personal repos. We should move them to a non-personal one:
- aqsassist https://gitlab.wikimedia.org/frankie/aqsassist
- aqs-docker-test-env https://gitlab.wikimedia.org/frankie/aqs-docker-test-env
- aqs-druid-test-env: https://github.com/bpirkle/aqs-docker-druid-test-env
- **Lack of data in //mediarequest_top_files//**: There is a lack of data for the following dates: 2019.3, 2019.4 and 2019.5 when using the "all-days" value as "day" in //mediarequest_top_files// dataset (media-analytics). For instance: https://wikimedia.org/api/rest_v1/metrics/mediarequests/top/all-referers/all-media-types/2019/03/all-days
- **Add device-analytics to the aqs-docker-test-env-qa**: Starting from geo-analytics, we made some changes to every service's repository to be able to dockerize them and run the QA test suite using docker (some changes were done as well in [[https://gitlab.wikimedia.org/frankie/aqs-docker-test-env|aqs-docker-test-env]] project to create a docker-compose project with cassandra and all its services). Device analytics was deployed before that so it has not been included yet.
- See {T344373} for details
- **Project, referer and page-title validation/normalization**: During first part of the development we missed some validation/normalization process that was done by AQS 1.0 using some common functions. This work will be done for page, edit and editor but it will be pending for geo, device and media. We are working on some new aqsassist functions to do these tasks and the proposal is to call them via a middleware function in each service project. It could be useful to take a look to page, edit or editor to see how was addressed for these services, and also at {T346598}
- `aqsassist.ValidateProject`: Validate and normalize a project
- `aqassist.ValidateReferer`: Validate and normalized a referer
- `aqsassist.NormalizePageTitle`: Replace all occurrences of spaces with underscores
- **aqssassist common code for messaging, problems and loggers**: During AQS 2.0 development some common code was added to aqsassist to manage messages, problems and loggers (some details at {T343907}). Some services were deployed before completing that work so it's something we should do later to unify all services. This tasks is pending for device-analytics, geo-analytics and media-analytics.
===== Documentation/References
- [[ https://wikimedia.org/api/rest_v1/ | AQS 1.0 OpenAPI Spec ]]