(This is technically not a "service-deployment-request" because the service is already deployed, but we plan to change significantly how it is used, so maybe having a conversation with the people watching that tag seems like a good idea. Overall, I tried to take inspiration from https://wikitech.wikimedia.org/wiki/SRE/Service_Operations/Intake_Standards for creating this task.)

Thanks for this writeup. Couple of comments below.

• If not already, DB entries should be pruned periodically, probably based on an individual TTL, to keep the size of the table from becoming unbounded.
• Related to that, are there any very rough guesstimations about what this will mean for the size(s) of the DB tables? Depending on those numbers, we might want to involve DBAs.
• Since I am on the guesstimations part, same things for requests to the internal linkrecommendation API. Depending on numbers, we might want to run a capacity planning exercise at some point. Given the gradual rollout approach, this can also wait a bit, so that you can have some data to base guessestimations on.
• To mitigate an easy DDoS abuse vector, and if you indeed intend MediaWiki to handle the API request, only expose it to logged-in users. An interesting question is about temporary users, which are being rolled out gradually. There is a chance it could also be fine, what we don't want is a fully open to the internet expensive API. It doesn't look like there is any wider reason, related to the mission, to serve recommendations to the wider internet, correct me if I am wrong.
• The above should also, at least partly, obviate the need for aggressive rate limiting. However, it can be done as well and depending on the UX for the user, the limits you set might be OK.
• 💯 to the progressive rollout based on feature flags per wiki.

Moving to serviceops-radar since there isn't something specific actionable for the team for now (at least).

Thank you for the very quick response!

In T382404#10412181, @akosiaris wrote:

• If not already, DB entries should be pruned periodically, probably based on an individual TTL, to keep the size of the table from becoming unbounded.
• Related to that, are there any very rough guesstimations about what this will mean for the size(s) of the DB tables? Depending on those numbers, we might want to involve DBAs.

An upper limit for rows would be the number of articles in the main namespace, likely much less. Currently, rows are removed when the page is edited and there should usually be only 1 row per page (for the latest revision).
We are also considering an alternative approach, where the idea would be to extend the limits of the maintenance script to generate recommendations for all articles as well (T378527).
Another idea to limit the load on the service would be to also store a row when we have no recommendations for a page so that we don't have to retry unless that page is edited. That might also result in the number of rows to be somewhere around the number of pages in the main namespace.

There were conversations with DBAs about this some years ago (T307902) and it was deemed fine, but maybe it could be good to check in about this again.

• Since I am on the guesstimations part, same things for requests to the internal linkrecommendation API. Depending on numbers, we might want to run a capacity planning exercise at some point. Given the gradual rollout approach, this can also wait a bit, so that you can have some data to base guessestimations on.

That sounds good to me. We are already planning to collect some data about this, but are there any specific metrics you have in mind?

• To mitigate an easy DDoS abuse vector, and if you indeed intend MediaWiki to handle the API request, only expose it to logged-in users. An interesting question is about temporary users, which are being rolled out gradually. There is a chance it could also be fine, what we don't want is a fully open to the internet expensive API. It doesn't look like there is any wider reason, related to the mission, to serve recommendations to the wider internet, correct me if I am wrong.

Yes, the current approach would be to only enable this for named (=non-temporary) accounts with 0 edits. Though, I can see this being extended in various ways in the future. I agree that it is probably not a good idea to expose this functionality to anon-users.

• The above should also, at least partly, obviate the need for aggressive rate limiting. However, it can be done as well and depending on the UX for the user, the limits you set might be OK.
• 💯 to the progressive rollout based on feature flags per wiki.

👍

Since I am on the guesstimations part, same things for requests to the internal linkrecommendation API. Depending on numbers, we might want to run a capacity planning exercise at some point. Given the gradual rollout approach, this can also wait a bit, so that you can have some data to base guessestimations on.

@akosiaris Is there any difference in ServiceOps eyes from directing the API requests to the internal linkrecommendation service (which is only used by MediaWiki, and its use cannot be triggered externally), versus directing the request to the external linkrecommendation service (which is already exposed via the API gateway)? Please correct me if that is not correct, but I think those two services have different SLOs attached (more or less for this reason). Happy to defer to you and your team though :).

It doesn't look like there is any wider reason, related to the mission, to serve recommendations to the wider internet, correct me if I am wrong.

FWIW, the external release of linkrecommendation is already released as https://api.wikimedia.org/service/linkrecommendation/apidocs/, and in its basic form, doesn't require any authentication.

Related to that, are there any very rough guesstimations about what this will mean for the size(s) of the DB tables? Depending on those numbers, we might want to involve DBAs.

I already briefly talked with @Ladsgroup about the impact on the DB, who said it wouldn't be a problem even if we maintained list of suggestions for all articles. Given this approach would only mean storing recommendations for selected/visited articles, that should be fine as well. We might want to put those conversations in writing somewhere though :).

I think we need an overarching or at least some best practices on integration of data intensive applications and services with mediawiki. Including but not limited to storage. But that's a much longer-term and broader conversation. In this specific case, since it's in x1, DBAs don't have any major concerns.

In T382404#10412551, @Urbanecm_WMF wrote:

Since I am on the guesstimations part, same things for requests to the internal linkrecommendation API. Depending on numbers, we might want to run a capacity planning exercise at some point. Given the gradual rollout approach, this can also wait a bit, so that you can have some data to base guessestimations on.

@akosiaris Is there any difference in ServiceOps eyes from directing the API requests to the internal linkrecommendation service (which is only used by MediaWiki, and its use cannot be triggered externally), versus directing the request to the external linkrecommendation service (which is already exposed via the API gateway)? Please correct me if that is not correct, but I think those two services have different SLOs attached (more or less for this reason). Happy to defer to you and your team though :).

The external one doesn't have an SLO. This is documented here: https://wikitech.wikimedia.org/wiki/SLO/linkrecommendation#Service_Level_Objectives where it is stated The external-facing service is not covered by the SLO. This alone is enough to indeed make the 2 things separate. It is however a symptom, not the cause. The separation happened for operational reasons (separation of concerns). The external one is mostly for testing and dev purposes, the internal is indeed the one used by production. This has also other repercussions, e.g. the external one has not had capacity planning exercises (because for a testing/dev thing does that it even make sense to spend the time to do it?). If we want to direct end user traffic to the externally exposed one, we should be having all of these conversations first.

It doesn't look like there is any wider reason, related to the mission, to serve recommendations to the wider internet, correct me if I am wrong.

FWIW, the external release of linkrecommendation is already released as https://api.wikimedia.org/service/linkrecommendation/apidocs/, and in its basic form, doesn't require any authentication.

Yes, but again, no SLOs, no capacity planning (and thus minimal capacity allocated), no defined use cases (that has been communicated to SRE at least) aside from dev/testing.

In T382404#10412293, @Michael wrote:

Thank you for the very quick response!

In T382404#10412181, @akosiaris wrote:

• If not already, DB entries should be pruned periodically, probably based on an individual TTL, to keep the size of the table from becoming unbounded.
• Related to that, are there any very rough guesstimations about what this will mean for the size(s) of the DB tables? Depending on those numbers, we might want to involve DBAs.

An upper limit for rows would be the number of articles in the main namespace, likely much less. Currently, rows are removed when the page is edited and there should usually be only 1 row per page (for the latest revision).

That's pretty nicely set, thanks!

We are also considering an alternative approach, where the idea would be to extend the limits of the maintenance script to generate recommendations for all articles as well (T378527).
Another idea to limit the load on the service would be to also store a row when we have no recommendations for a page so that we don't have to retry unless that page is edited. That might also result in the number of rows to be somewhere around the number of pages in the main namespace.

If you need the service to handle more load, that is possible. We can do some basic capacity planning and increase the amount of traffic the service can handle (to a certain degree of course).

There were conversations with DBAs about this some years ago (T307902) and it was deemed fine, but maybe it could be good to check in about this again.

• Since I am on the guesstimations part, same things for requests to the internal linkrecommendation API. Depending on numbers, we might want to run a capacity planning exercise at some point. Given the gradual rollout approach, this can also wait a bit, so that you can have some data to base guessestimations on.

That sounds good to me. We are already planning to collect some data about this, but are there any specific metrics you have in mind?

I assume you are going to collecting some data regarding requests per second and probably latency. Those should suffice on our side.

• To mitigate an easy DDoS abuse vector, and if you indeed intend MediaWiki to handle the API request, only expose it to logged-in users. An interesting question is about temporary users, which are being rolled out gradually. There is a chance it could also be fine, what we don't want is a fully open to the internet expensive API. It doesn't look like there is any wider reason, related to the mission, to serve recommendations to the wider internet, correct me if I am wrong.

Yes, the current approach would be to only enable this for named (=non-temporary) accounts with 0 edits. Though, I can see this being extended in various ways in the future. I agree that it is probably not a good idea to expose this functionality to anon-users.

Perfect, thanks!

In T382404#10412657, @Ladsgroup wrote:

I think we need an overarching or at least some best practices on integration of data intensive applications and services with mediawiki. Including but not limited to storage. But that's a much longer-term and broader conversation. In this specific case, since it's in x1, DBAs don't have any major concerns.

Perfect, thanks!