The rate limiting classes seem like a great way to think about the traffic/audience/caller types. I'm not convinced that using the rate limiting service to do the actual blocking is the right choice, but I'll let folks closer to that work weigh in.

If MediaWiki REST APIs are involved (aka rest.php), then the MediaWiki-API-Platform-Team has been talking about "restricted" endpoints. We don't have a ready-to-go solution yet - we're just at the stage of putting in some of the machinery that would allow REST modules to identify themselves as restricted. Doing the actual restriction in MediaWiki itself would likely be the wrong choice anyway. But we'd be happy to chat to see if there's any overlap between your needs and the things we're considering.

Update: based on a discussion with Mediawiki Interfaces and ServiceOps

Important thing to note: The discussion and the request for authorization and different rate limits apply only to the new LLM endpoints, while existing LiftWing endpoints stay as is. WMCS is the priority right now (with toolforge being a direct use case).

Key takeaways:

• Changing the response code to 403/401 for rate limits that are set to 0 is possible without an extra rate-limiting policy
• We want to avoid policy sprawl, since we're already at around five policies.
• In the short term, we're comfortable with a 403/401 override or tagging the pathway as restricted.
• In the long term, the strategy is the API Gateway (which already checks JWTs for user classes) to be combined with cost-based rate limiting which is well aligned with the plans of the ML team as rate limiting and API performance for LLMs is based on the size of the request+response (measured by the number of tokens) as requests per hours/minute/second doesn't tell the whole story. Explorations are scheduled for Q1 next FY.
• ML team will need to pre-pend these e.g. lw/inference/v1/models/llm-{model-name}/ so that they can easily be grouped. An alternative is to pattern match all endpoints that include openai/v1/chat/completions or openai/v1/completions
• Mediawiki Interfaces is exploring restricted endpoints in the future. Both internal and restricted endpoints are things that the ML team needs https://www.mediawiki.org/wiki/Wikimedia_APIs/Stability_policy#Module_audiences

Next steps:

• ML team can use rate limiting in the short term to block all external access except WMCS and revisit gateway auth and cost-based limiting in Q1. ML can explore the current work and ask for help from Mediawiki interfaces and ServiceOps if needed.
• Teams will continue to coordinate as the current plans and explorations as documented in https://www.mediawiki.org/wiki/Wikimedia_APIs/Stability_policy#Module_audiences are well aligned with ML needs.

If there's anything I've left out or misrepresented, please chime in. Thank you for the fruitful discussion!

@Clement_Goubert one thing I have missed in my notes above is something that you mentioned regarding future plans on restricted endpoints that are available through MediaWIki. The case that we would particularly interested in is making an endpoint available through VisualEditor. If you can add 1-2 sentences about this and a reference (if exists) that would be great!

In T426749#12017627, @isarantopoulos wrote:

@Clement_Goubert one thing I have missed in my notes above is something that you mentioned regarding future plans on restricted endpoints that are available through MediaWIki. The case that we would particularly interested in is making an endpoint available through VisualEditor. If you can add 1-2 sentences about this and a reference (if exists) that would be great!

I think @HCoplin-WMF may have more information than I do at this point, I haven't seen a proposal for technical implementation yet. From the above Restricted Stability Policy, it would seem for now to be a mix of different types of authorization, possibly signed requests.

I've looked into the current rate limiting setup for LiftWing and have questions and ideas on how we could approach it. We'd love to push this work forwards so I'd love to hear inputs from ServiceOps and MW-Interfaces.
Pinging @daniel and @Clement_Goubert I got a suggestion to include you here :)

Current state

Our new LLM services (some already here, some more coming) are exposed externally through REST Gateway and they currently sit on the relaxed rate-limit policy LiftWing in helmfile.d/services/rest-gateway/values.yaml , which we use for all our services. This policy is very generous and gives fully anonymous clients 50k req/hour. This gives a risk of anonymous clients to flood our LLM services if they would use this policy, which we would like to avoid.

What we want

Based on the discussions in this thread, the short-term plan would be to use existing rate-limit framework to block everything except WMCS and known clients, without per-service policy sprawl.

We want a single, reusable way to say "this endpoint is restricted, allow only WMCS + known client, rate limit everyone else to 0". This should be applied only to selected endpoints and leave other routes untouched. Ideally, we want to avoid adding a new policy per model and editing the gateway every time we add an LLM.

Possible approach

I see that user classes from description map onto what gateway computes in charts/api-gateway/lua/restgw_ratelimits.lua as:

• WMCS / Toolforge -> known-network, based on x-trusted-request: A set at CDN
• known clients -> known-client, based on x-trusted-request: B. Another could be approved-bot
• authenticated -> authed-user / established-user / highlimits-user
• user-agent-only -> unauthed-bot / unauthed-mediawiki
• unidentified -> anon* classes

We could take advantage of this to create a simple default-deny rate limiting policy, which we would reuse across all endpoints we want to restrict:

"RestrictedLiftWing":
  limits:
    "*":             { HOUR: 0 }
    "known-network": { HOUR: 9999999 }
    "known-client":  { HOUR: 9999999 }
    "approved-bot":  { HOUR: 9999999 }

The wildcard would block every class not in allow-list, which we can easily extend to other clients. The downside is that we are adding a new policy, but it could be reused for all the services we need to and we re-use the existing rate-limiting machinery existing in the chart.
We could also convert the 429 to 403 in the gateway's response Lua to better convey the meaning of "not allowed" rather than "slow down".

Open questions

1. Is the suggested approach above sound or is there a better way to approach it?

2. What is the way we should target the endpoints?

I see that we discussed prefixing the service names with llm-* such that we would block all lw/inference/v1/models/llm-{model-name}/ endpoints.

Another alternative that I see and would lean towards to is to have a dedicated namespace (e.g. restricted), where we would deploy the models we want to restrict access to. I lean towards it, because this would introduce a clear separation by where we deploy, rather than what name we use, and namespace feels like a good medium to convey an isolation level. However, since the gateway currently routes by path (builds the backend host from URL and can't see namespace directly), we'd need to surface the namespace as dedicated path prefix that maps 1:1 to it (e.g. expose as /service/lw/restricted/...).

3. Dependency on the CDN trust tag

The approach I presented above hinges heavily on WMCS traffic reaching the gateway with x-trusted-request: A, which translates to known-network. Could we confirm that the Toolforge/WMCS egress is reliably classified as such at the CDN level?

4. Authenticated users

We say that we would evaluate them case-by-case so I assume it's okay to start with blocking them as WMCS is the priority use case. We could add authed-user/established-user to allow-list later if a need shows up. Would this work for us?

Open questions

1. Is the suggested approach above sound or is there a better way to approach it?

Yes, one new policy sounds good.

2. What is the way we should target the endpoints?

Anything in the path that we can reliably match with a regular expression is fine wrt rate limiting.

3. Dependency on the CDN trust tag

The approach I presented above hinges heavily on WMCS traffic reaching the gateway with x-trusted-request: A, which translates to known-network. Could we confirm that the Toolforge/WMCS egress is reliably classified as such at the CDN level?

We haven't had complains from users on WMCS about API rate limits, so I'd assume so. Mis-classifying traffic is probelmatic, so if it does happen, I'd expect it to be fixed quickly.

4. Authenticated users

We say that we would evaluate them case-by-case so I assume it's okay to start with blocking them as WMCS is the priority use case. We could add authed-user/established-user to allow-list later if a need shows up. Would this work for us?

That's frine from teh gateway's perspective.

After giving it some thought and discussing with a couple of folks we concluded that it would be best to also enable also public access with some strict rate limits so that we don't block all users by default.
We want to enable technical contributors to build things and although building tools on toolforge will be the general recommendation it will be helpful for development purposes to have public access to the APIs. My understanding is that this could be handled in the new rate limit class we're adding for LLMs once we figure out what these numbers should be. Does that sound correct?

In T426749#12053061, @isarantopoulos wrote:

After giving it some thought and discussing with a couple of folks we concluded that it would be best to also enable also public access with some strict rate limits so that we don't block all users by default.
We want to enable technical contributors to build things and although building tools on toolforge will be the general recommendation it will be helpful for development purposes to have public access to the APIs. My understanding is that this could be handled in the new rate limit class we're adding for LLMs once we figure out what these numbers should be. Does that sound correct?

Yes, that would work. It just boils down to having a LiftWing-LLM policy in addition to the LiftWing policy, and finding out what path pattern to use to apply it.

Change #1305621 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] rest-gateway: Add LiftWingLLM rate limit policy for LLM endpoints

https://gerrit.wikimedia.org/r/1305621

I've drafted the REST Gateway changes in here https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1305621.
I went for the llm-* path matching routing to llm namespace. So all services matching the llm-* regex inside llm namespace would be automatically exposed with the LiftWingLLM rate limiting policy. I included conservative 100 requests per hour limit for public traffic according to the discussion above.

Change #1305621 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Add LiftWingLLM rate limit policy for LLM endpoints

https://gerrit.wikimedia.org/r/1305621

Change #1306272 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Move qwen3-14b to llm namespace.

https://gerrit.wikimedia.org/r/1306272

Change #1306286 had a related patch set uploaded (by Bartosz Wójtowicz; author: Bartosz Wójtowicz):

[operations/deployment-charts@master] ml-services: Add qwen3-14b deployment to llm namespace.

https://gerrit.wikimedia.org/r/1306286

Change #1306272 abandoned by Bartosz Wójtowicz:

[operations/deployment-charts@master] ml-services: Move qwen3-14b to llm namespace.

Reason:

New patch https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1306286

https://gerrit.wikimedia.org/r/1306272

Change #1306286 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: Add qwen3-14b deployment to llm namespace.

https://gerrit.wikimedia.org/r/1306286

The changes to the gateway (https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1305621) are merged and deployed. We've settled on this rate limiting policy:

"LiftWingLLM":
  shadow_mode: false
  limits:
    "*": # strict shared limit for public access (anon*, unauthed*, authed-*)
      HOUR: 100
    "known-network": # network under our control: WMCS / Toolforge (x-trusted-request: A)
      HOUR: 9999999
    "known-client": # network associated with a known client (x-trusted-request: B)
      HOUR: 9999999
    "approved-bot": # community approved bot, based on jwt auth
      HOUR: 9999999

I've tested the changes by deploying llm-qwen3-14b model server in the llm namespace, which correctly exposed the model through the REST Gateway with the 100req/hour limit for anonymous users.
Now, the same process can be used for other LiftWing LLM services we would like to expose 🎉