I much prefer option C, but I'm not sure difficult that might be, or if is setting to strong of a new precedent. @HCoplin-WMF should have some good ideas here!

In order to expedite this work, I am considering implementing versioning in the AQS service handlers now, and then working with serviceops on the full routing choice and implementation based on that. So, https://wikimedia.org/api/rest_v1/metrics/v3/ will work now, and if/when we want to get rid of the rest_v1 part we can handle that in frontend routing rules.

I tried ^, and I can make it work, but I cannot get AQS to auto-generate swagger docs for new API version. I can manually add a /metrics/v3 route, but the generated API docs will always have.

basePath: /api/rest_v1/metrics
info:
  version: "1"

It might be possible to restructure AQS code to make it possible to host more than one base API endpoint per service, but I do not have time for this 🙀 !

I'm going to just add the /metrics/v3 path routing manually. version in the Open API docs will be wrong, but I'll add a note.

Regarding the options to version AQS, I would vote for Option C. The rest thing doesn't add any value.

It might be possible to restructure AQS code to make it possible to host more than one base API endpoint per service, but I do not have time for this 🙀 !

Not fully sure about this but I would say that Swagger 2.0 only allows one base path for all the endpoints. And the library we use to generate the specification, swag (https://github.com/swaggo/swag), only supports Swagger 2.0. We would need something that generates OAS 3.0 where several servers can be defined (anyway I'm not sure if that approach would work here)

...I am more and more leaning towards in the future just implementing APIs in MediaWiki REST API and relying on MW Interfaces team support...

I just deployed edit-analytics with the metrics/v3/edits/per_editor endpoint. It does not work externally at https://wikimedia.org/api/rest_v1/metrics/v3/edits/per_editor.

It looks like routing to the backend AQS service maps from e.g. metrics/edits -> edit-analytics, so having a URL like metrics/v3/edits will not work.

We could

• A. make a new routing rule for metrics/v3/edits
• B. change the versioned path to be metrics/edits/v3

@mforns and I originally discussed Option B in slack but decided to go with metrics/v3/edits. We waffled quite a bit on this.

I think this simplest thing to do right now is option B.

...

Hey -- sorry for the delay on my side! Still getting caught up on a few things, so I just got a chance to read through this. I have a few thoughts and questions.

Starting with questions!

First off, could you confirm that the version change is going to be applied for all AQS endpoints? Like, you're doing general clean up activity, and it' s reasonable to expect that all existing endpoints/capabilities will be migrated to v3? Or are you planning to only implement new endpoints as v3?

Second, do you have a deprecation plan for v1? Like, any idea of how long you intend to support it after v3 is out? Regarding deprecation of v1, you might want to read our new/draft deprecation process wiki (https://www.mediawiki.org/wiki/API/Deprecation). While AQS does state that y'all follow a semantic versioning policy, I didn't see anything specifically related to what users should expect when deprecation occurs. At minimum, we need to be firm on sunsetting timelines, migration messaging, and how old version deprecation shows up in your docs. @BPirkle also just created some really great mechanisms for module deprecation and headers firing from MediaWiki, so we should chat about what that could mean for AQS.

Third, wikimedia.org/api/metrics/v3/ would be my preferred routing pattern -- I'm a little bit confused on where y'all are landing based on the phab comments though. Could you clarify that as the intent real quick? It sounds like the plan is Version C, or including rest_v1? Could you clarify a bit on why the rest_v1 pattern needs to continue/did you confirm with the traffic SRE folks?

Fourth, I'm like 99% sure the answer is yes, but can someone confirm that the existing AQS endpoints are already being routed through the REST/Common Gateway? I assume yes since they're under the rest_v1 path, but want to confirm. (I also know I've asked this before, but I'm having a minor brain fart on it)

Now comments!

I 10000% agree with Bill's desire to drop the RESTBase remnants and make the routing match our desired module pattern as much as possible. The "Version C" in the phab ticket/suggestion in the third item above is definitely my recommendation for the new pattern as well.

Similarly, the leading proposal for the new canonical routing structure matches y'all's proposal, where we are hoping to utilize /api/{module}/vX as the main pattern for routing across the board. If that's the case, there will be no additional breaking change for y'all, which is a good thing. However, will add a HUGE DISCLAIMER that we haven't fully aligned or settled on that pattern, so there is a risk of a future breaking change in the name of path consistency.

The one thing I think is slightly odd from an end user perspective is to bump from v1 to v3, but I understand the reasoning behind it. I also think that we can explain it in the launch announcement, and that it doesn't reeeally matter if folks are going to have to change stuff anyway.

FINALLY (and somewhat tangentially related) we need to start chatting about getting AQS specs into the API Sandbox/Explorer! I saw the comment that y'all are on Swagger/OpenAPI v2.0. We would greatly prefer OpenAPI v3 to ensure compliance with our Swagger UI version and in-progress linting tools. A new API version could be a good time to start the transition :)

As an additional aside, heck yes to this! But we also need to improve our support for non-MW APIs, since it makes sense for some things to exist outside of MediaWiki! So thank you for helping us figure out where we still have gaps, while also acknowledging the great work that folks on the team are doing to improve the MW API experience :)

...I am more and more leaning towards in the future just implementing APIs in MediaWiki REST API and relying on MW Interfaces team support...

could you confirm that the version change is going to be applied for all AQS endpoints?

This would be nice. T342018: compile list of known issues for triage post AQS 2.0 launch lists a lot of breaking changes. However I doubt that we will actually prioritize doing this work.

Second, do you have a deprecation plan for v1?

No, we are just building new endpoints in T405041: Global Editor Metrics - HTTP API endpoints for now. Even though these are brand new, we wanted to version them at the same API version that we would theoretically bump the existing endpoints to if we ever fixed them. This way, we wouldn't also have to migrate these new endpoints.

A new API version also helps us be more explicit that the conventions the new endpoints use are different than the 'v1' ones.

Could you clarify that as the intent real quick

I don't like rest_v1 either, but as is, we can't host multiple API versions from the same Go microservice.  

So, in order to do it right, we'd have to:

• deploy a brand new AQS microservice (new repo, new k8s deployment, etc.)
• modify API Gateway(?) routing rules

We thought that forcing people to do all that just to bump API versions was a bit onerous, and that being able to manage API versions in the service's own endpoint path handlers was better. We are going with e.g. metrics/edits/v3/per_editor to make this possible now.

We've got some strict deadlines for this thing (mid November), so we are go go go!

can someone confirm that the existing AQS endpoints are already being routed through the REST/Common Gateway?

Someone other than me please! ;)

hoping to utilize /api/{module}/vX as the main pattern

Ya but what is the 'api module' in AQS here? Is it 'metrics'? Or 'metrics/edits'? Or is it 'metrics/edits/per_editor'? If so, where does the vX part go? I don't think we should have to edit anything but the service to version the service's APIs, and IIUC, the only possible way to do that (right now) with external routing rules is to do metrics/edits/v3/per_editor. So the module would be 'metrics/edits' and that is what we are versioning, right?

If so, that would mean that each 'AQS' / Analytics API microservice is its own 'module' and may be upgraded independently?

slightly odd from an end user perspective is to bump from v1 to v3

Agree its odd, but yeah I think it is easier to explain than versionless (v1) is AQS 2.0 and v3 is ... AQS 2.1?

Even though these are brand new, we wanted to version them at the same API version that we would theoretically bump the existing endpoints to if we ever fixed them. This way, we wouldn't also have to migrate these new endpoints.

Makes sense in theory. Another thing to consider is if the new/future version will follow the same pathing. If the end goal is to get rid of rest_v1 in the future canonical URLs, then it will require a secondary rerouting and breaking change at minimum regardless. If you want the new version to actually be future proof, then I think we do need to dig into the module definitions and gateway rerouting to remove the rest_v1 element to [hopefully] avoid a future breaking change.

As a related question, are you planning to address some of the issues in the v3? For example, are you using the new date formats recommended in https://phabricator.wikimedia.org/T342018 ? If not, it might make more sense to leave the new capabilities in the v1, then update everything for consistency. There is a stronger argument for making it v3 coming out of the gate if the new patterns will be reflected in the new endpoints, though.

Ya but what is the 'api module' in AQS here? Is it 'metrics'? Or 'metrics/edits'?

Great question! It's sort of up to you guys, to a degree. Would you want to increment edit version separately, or metrics overall? Do you want to manage their specs independently, as well? I was originally thinking more that AQS/metrics would be a single module, but I also didn't realize that each type of metric is basically its own microservice. If that's the case, it might make sense to version them independently, possibly with shared elements for things like error messages.

each type of metric is basically its own microservice

Which begs the question: should it be? ;)

are you planning to address some of the issues in the v3? For example, are you using the new date formats

We did a lot, but not all. Date formats were more annoying than it sounds (there are shared libraries with rules). We decided that while the new APIs may support the old date format yyyyMMdd, if/when we added support for the desired format yyyy-MM-dd we'd keep backwards compatible support for the old ones. Not the best, but it seemed less confusing than keeping the version the same but different response timestamp formats.

If the end goal is to get rid of rest_v1 in the future canonical URLs, then it will require a secondary rerouting and breaking change at minimum regardless. If you want the new version to actually be future proof, then I think we do need to dig into the module definitions and gateway rerouting to remove the rest_v1 element to [hopefully] avoid a future breaking change.

If we can do this in < 2 weeks, then I'm all for it! But I cannot work on it as I still have tons to do to try to hit the deadline for T403660: WE3.3.7 Year in Review and Activity Tab Services - Global Editor Metrics.

Your point is very valid, but I think that doing a suboptimal route with both rest_v1 & v3 now and adding custom re-routing rules later is better than not versioning now at all. Not sure though!