Page MenuHomePhabricator
Create Task
Maniphest T422804

Reroute LiftWing endpoints
Closed, ResolvedPublic
Actions

Description

  • T418145: Configure ATS to allow fractional routing for api.wikimedia.org
  • Configure the REST gateway to correctly route /service/lw API routes to the right liftwing services
  • Add liftwing-specific rate limits T413448: [5.2.2c Epic]: Support Higher Rate Limits for Lift Wing
  • Progressive rollout by liftwing model
    • revscoring_articletopic
    • revscoring_articletopic_outlink
    • revscoring_articlequality
    • articlequality_language_agnostic
    • article_country_language_agnostic
    • revscoring_drafttopic
    • revscoring_draftquality
    • revscoring_editquality_damaging
    • revscoring_editquality_goodfaith
    • revscoring_editquality_reverted
    • revertrisk
    • readability
    • langid
    • articledesc
    • reference_need
    • reference_risk
    • editcheck
    • recommendation-api-ng

We can use sample calls from https://api.wikimedia.org/wiki/Lift_Wing_API for testing purposes, summarized in the table below

The general procedure to follow is:

  • Make a CR that changes the endpoint's load_fraction to 1 in gateway-check.lua.conf
  • Disable puppet on all A:cp-text nodes
  • Submit and merge the CR
  • Enable and run puppet on one cp-text node (for example cp6009.drmrs.wmnet)
  • Check trafficserver reloaded the lua correctly sudo journalctl -u trafficserver.service
  • Test that the cp node curl works correctly for this endpoint (returns 200 rest-gateway)
  • Enable puppet on all A:cp-text nodes
  • Monitor grafana for this route on both eqiad and codfw https://grafana.wikimedia.org/goto/ffjnfdmjs7bwgf?orgId=1

LiftWing Route Tests

Common headers used in all commands:

  • Content-Type: application/json
  • User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)
  • Host: api.wikimedia.org

Returns only HTTP status code and the via header if it exists

Route NamePathReferencecp node curlrest-gateway curl
revscoring_articletopic/service/lw/inference/v1/models/enwiki-articletopic:predictGet revscoring articletopic predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-articletopic:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/enwiki-articletopic:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
revscoring_articletopic_outlink/service/lw/inference/v1/models/outlink-topic-model:predictGet articletopic outlink predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/outlink-topic-model:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"page_title":"Douglas_Adams","lang":"en"}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/outlink-topic-model:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"page_title":"Douglas_Adams","lang":"en"}'
revscoring_articlequality/service/lw/inference/v1/models/enwiki-articlequality:predictGet revscoring articlequality predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-articlequality:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/enwiki-articlequality:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
articlequality_language_agnostic/service/lw/inference/v1/models/articlequality:predictGet language agnostic articlequality predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/articlequality:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/articlequality:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'
article_country_language_agnostic/service/lw/inference/v1/models/article-country:predictGet article countrycurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/article-country:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"lang":"en","title":"Toni_Morrison"}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/article-country:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"lang":"en","title":"Toni_Morrison"}'
revscoring_drafttopic/service/lw/inference/v1/models/enwiki-drafttopic:predictGet revscoring drafttopic predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-drafttopic:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/enwiki-drafttopic:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
revscoring_draftquality/service/lw/inference/v1/models/enwiki-draftquality:predictGet revscoring draftquality predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-draftquality:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/enwiki-draftquality:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
revscoring_editquality_damaging/service/lw/inference/v1/models/enwiki-damaging:predictGet revscoring damaging predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-damaging:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/enwiki-damaging:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
revscoring_editquality_goodfaith/service/lw/inference/v1/models/enwiki-goodfaith:predictGet revscoring goodfaith predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-goodfaith:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/enwiki-goodfaith:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
revscoring_editquality_reverted/service/lw/inference/v1/models/viwiki-reverted:predictGet revscoring reverted predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/viwiki-reverted:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/viwiki-reverted:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
revertrisk/service/lw/inference/v1/models/revertrisk-multilingual:predictGet reverted risk multilingual predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/revertrisk-multilingual:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/revertrisk-multilingual:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'
revertrisk/service/lw/inference/v1/models/revertrisk-language-agnostic:predictGet reverted risk language agnostic predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/revertrisk-language-agnostic:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/revertrisk-language-agnostic:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'
readability/service/lw/inference/v1/models/readability:predictGet readability predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/readability:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/readability:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'
langid/service/lw/inference/v1/models/langid:predictGet language identification predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/langid:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"text":"Some sample text in any language that we want to identify"}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/langid:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"text":"Some sample text in any language that we want to identify"}'
articledesc/service/lw/inference/v1/models/article-descriptions:predictGet article descriptionscurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/article-descriptions:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"lang":"en","title":"Clandonald","num_beams":2,"debug":1}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/article-descriptions:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"lang":"en","title":"Clandonald","num_beams":2,"debug":1}'
reference_need/service/lw/inference/v1/models/reference-need:predictGet reference need predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/reference-need:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/reference-need:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'
reference_risk/service/lw/inference/v1/models/reference-risk:predictGet reference risk predictioncurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/reference-risk:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":1242378206,"lang":"en"}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/reference-risk:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":1242378206,"lang":"en"}'
editcheck/service/lw/inference/v1/models/edit-check:predicteditcheck httpbb testcurl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/edit-check:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"instances":[{"original_text":"","modified_text":"this is a great example of work","check_type":"tone","lang":"en","page_title":"Test_Page"}]}'curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST "https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/edit-check:predict" -H "Host: api.wikimedia.org" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"instances":[{"original_text":"","modified_text":"this is a great example of work","check_type":"tone","lang":"en","page_title":"Test_Page"}]}'
recommendation-api-ng/service/lw/recommendation/api/v1/translationGet content translation recommendationcurl -si -o /dev/null -w "%{http_code} %header{via}\n" -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/recommendation/api/v1/translation?source=en&target=fr&count=3&seed=Apple" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)"curl -si -o /dev/null -w "%{http_code} %header{via}\n" "https://rest-gateway.discovery.wmnet:4113/service/lw/recommendation/api/v1/translation?source=en&target=fr&count=3&seed=Apple" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)"

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Clement_Goubert created this task.Apr 9 2026, 10:33 AM
Clement_Goubert changed the task status from Open to In Progress.
Clement_Goubert triaged this task as High priority.
Clement_Goubert moved this task from Inbox to In Progress on the ServiceOps new board.
gerritbot added a comment.Apr 9 2026, 11:02 AM

Change #1269398 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] service-catalog: Add recommendation-api-ng

https://gerrit.wikimedia.org/r/1269398

gerritbot added a project: Patch-For-Review.Apr 9 2026, 11:02 AM

Change #1269401 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Add liftwing listeners and network policies

https://gerrit.wikimedia.org/r/1269401

gerritbot added a comment.Apr 9 2026, 11:20 AM

Change #1269403 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Add revscoring_articletopic routing

https://gerrit.wikimedia.org/r/1269403

kevinbazira subscribed.Apr 9 2026, 12:57 PM
KartikMistry subscribed.Apr 9 2026, 1:59 PM
gerritbot added a comment.Apr 9 2026, 2:50 PM

Change #1269398 merged by Clément Goubert:

[operations/puppet@production] service-catalog: Add recommendation-api-ng

https://gerrit.wikimedia.org/r/1269398

Clement_Goubert added a project: ServiceOps-SharedInfra.Apr 10 2026, 1:21 PM
gerritbot added a comment.Apr 13 2026, 12:36 PM

Change #1270434 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] rest-gateway: Add liftwing recommendation-api-ng routes

https://gerrit.wikimedia.org/r/1270434

Clement_Goubert added projects: Machine-Learning-Team, Lift-Wing.Apr 13 2026, 3:58 PM
gerritbot added a comment.Apr 15 2026, 4:21 PM

Change #1271804 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] gateway-check: Add matchers for liftwing and recommendation-api-ng

https://gerrit.wikimedia.org/r/1271804

gerritbot added a comment.Apr 16 2026, 3:18 PM

Change #1269401 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Add liftwing listeners and network policies

https://gerrit.wikimedia.org/r/1269401

gerritbot added a comment.Apr 16 2026, 3:19 PM

Change #1269403 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Add liftwing inference routes

https://gerrit.wikimedia.org/r/1269403

gerritbot added a comment.Apr 16 2026, 3:19 PM

Change #1270434 merged by jenkins-bot:

[operations/deployment-charts@master] rest-gateway: Add liftwing recommendation-api-ng routes

https://gerrit.wikimedia.org/r/1270434

Clement_Goubert added a comment.Apr 16 2026, 4:35 PM

Routes merged into the rest-gateway, initial tests look good:

$ curl -H 'Host: api.wikimedia.org' https://rest-gateway.discovery.wmnet:4113/service/lw/inference/v1/models/enwiki-goodfaith:predict -X POST -d '{"rev_id": 12345}' -H "Content-type: application/json"
{"enwiki":{"models":{"goodfaith":{"version":"0.5.1"}},"scores":{"12345":{"goodfaith":{"score":{"prediction":true,"probability":{"false":0.07396339218373627,"true":0.9260366078162637}}}}}}}

$ curl -H 'Host: api.wikimedia.org' "https://rest-gateway.discovery.wmnet:4113/service/lw/recommendation/api/v1/translation?source=en&target=fr&count=3&seed=Apple"
{"recommendations":[{"title":"Aymak Djangaliev","pageviews":0,"wikidata_id":"Q12527473","rank":492,"langlinks_count":2,"size":8114,"lead_section_size":null,"collection":null},{"title":"Summerland Research and Development Centre","pageviews":0,"wikidata_id":"Q60742547","rank":465,"langlinks_count":1,"size":7857,"lead_section_size":null,"collection":null},{"title":"List of national fruits","pageviews":0,"wikidata_id":"Q3351254","rank":374,"langlinks_count":8,"size":9942,"lead_section_size":null,"collection":null}],"continue_offset":null,"continue_seed":null}

Now I'll compile the examples from https://api.wikimedia.org/wiki/Lift_Wing_API/Reference into a test plan for the rest-gateway, and for external testing through ATS.
Once the rate limits are in, we can start moving traffic.

Clement_Goubert updated the task description. (Show Details)Apr 16 2026, 4:35 PM
KartikMistry added a comment.Apr 17 2026, 2:26 PM

@Clement_Goubert Hi, do we need to change anything on the recommendation-api side for this change?

Clement_Goubert added a comment.Apr 20 2026, 7:47 AM
In T422804#11833678, @KartikMistry wrote:

@Clement_Goubert Hi, do we need to change anything on the recommendation-api side for this change?

Hi Kartik, I don't think so, my test above looks ok to me but I admittedly don't know the service. It should be transparent for the backend services, but just to be sure, do you use any specific headers that would have been injected by the api-gateway?

Clement_Goubert updated the task description. (Show Details)Apr 20 2026, 9:51 AM
Clement_Goubert updated the task description. (Show Details)Apr 20 2026, 10:02 AM
Clement_Goubert updated the task description. (Show Details)Apr 20 2026, 10:20 AM
Clement_Goubert updated the task description. (Show Details)
Stashbot added a comment.Apr 21 2026, 10:07 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-21T10:07:41Z] <claime> Disabling puppet on A:cp to merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/1271804 - T422804

gerritbot added a comment.Apr 21 2026, 10:11 AM

Change #1271804 merged by Clément Goubert:

[operations/puppet@production] gateway-check: Add matchers for liftwing and recommendation-api-ng

https://gerrit.wikimedia.org/r/1271804

Clement_Goubert updated the task description. (Show Details)Apr 21 2026, 10:14 AM
Maintenance_bot removed a project: Patch-For-Review.Apr 21 2026, 10:31 AM
gerritbot added a comment.Apr 21 2026, 11:00 AM

Change #1275852 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/puppet@production] gateway-check: Escape - in lua patterns

https://gerrit.wikimedia.org/r/1275852

gerritbot added a project: Patch-For-Review.Apr 21 2026, 11:00 AM
gerritbot added a comment.Apr 21 2026, 11:03 AM

Change #1275852 merged by Clément Goubert:

[operations/puppet@production] gateway-check: Escape - in lua patterns

https://gerrit.wikimedia.org/r/1275852

Stashbot added a comment.Apr 21 2026, 11:11 AM

Mentioned in SAL (#wikimedia-operations) [2026-04-21T11:11:07Z] <claime> Enabling puppet on A:cp to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/1271804 - T422804

Clement_Goubert updated the task description. (Show Details)Apr 21 2026, 11:16 AM
Clement_Goubert changed the task status from In Progress to Stalled.Apr 21 2026, 11:21 AM
Clement_Goubert moved this task from In Progress to Needs Info / Blocked on the ServiceOps new board.

After some finaggling with lua patterns, everything on our side is now in place to start migrating traffic once rate limits for liftwing are implemented. Marking stalled pending resolution of T413448: [5.2.2c Epic]: Support Higher Rate Limits for Lift Wing.

gerritbot added a comment.Fri, May 8, 8:47 AM

Change #1285307 had a related patch set uploaded (by Blake; author: Blake):

[operations/puppet@production] gateway-check.lua: Route articletopic through the REST gateway.

https://gerrit.wikimedia.org/r/1285307

gerritbot added a comment.Fri, May 8, 9:03 AM

Change #1285307 merged by Blake:

[operations/puppet@production] gateway-check.lua: Route articletopic through the REST gateway.

https://gerrit.wikimedia.org/r/1285307

Blake added a comment.Fri, May 8, 9:18 AM

All of the traffic for revscoring_articletopic has been migrated, and things look good. I don't really want to migrate many endpoints on a Friday, so we'll leave this to cook over the weekend, and I'll start migrating in larger batches on Monday :)

blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-articletopic:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
200 rest-gateway
Maintenance_bot removed a project: Patch-For-Review.Fri, May 8, 9:30 AM
Blake updated the task description. (Show Details)Fri, May 8, 9:40 AM
gerritbot added a comment.Mon, May 11, 8:47 AM

Change #1285732 had a related patch set uploaded (by Blake; author: Blake):

[operations/puppet@production] gateway-check.lua: Route some LiftWing endpoints through the REST gateway.

https://gerrit.wikimedia.org/r/1285732

gerritbot added a project: Patch-For-Review.Mon, May 11, 8:47 AM
gerritbot added a comment.Mon, May 11, 8:56 AM

Change #1285732 merged by Blake:

[operations/puppet@production] gateway-check.lua: Route some LiftWing endpoints through the REST gateway.

https://gerrit.wikimedia.org/r/1285732

Blake changed the task status from Stalled to In Progress.Mon, May 11, 8:59 AM
Blake added a comment.Mon, May 11, 9:02 AM

Next set complete:

blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/outlink-topic-model:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"page_title":"Douglas_Adams","lang":"en"}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-articlequality:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/articlequality:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/article-country:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"lang":"en","title":"Toni_Morrison"}'
200 rest-gateway

I'll monitor this, then migrate another batch later today.

Blake updated the task description. (Show Details)Mon, May 11, 9:06 AM
Maintenance_bot removed a project: Patch-For-Review.Mon, May 11, 9:31 AM
gerritbot added a comment.Mon, May 11, 2:32 PM

Change #1285827 had a related patch set uploaded (by Blake; author: Blake):

[operations/puppet@production] gateway-check.lua: Route some LiftWing endpoints through the REST gateway.

https://gerrit.wikimedia.org/r/1285827

gerritbot added a project: Patch-For-Review.Mon, May 11, 2:32 PM
gerritbot added a comment.Mon, May 11, 2:55 PM

Change #1285827 merged by Blake:

[operations/puppet@production] gateway-check.lua: Route some LiftWing endpoints through the REST gateway.

https://gerrit.wikimedia.org/r/1285827

MLechvien-WMF moved this task from Needs Info / Blocked to In Progress on the ServiceOps new board.Mon, May 11, 2:57 PM
Blake added a comment.Mon, May 11, 3:03 PM

Next set complete:

blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-drafttopic:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-draftquality:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-damaging:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/enwiki-goodfaith:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/viwiki-reverted:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":12345}'
200 rest-gateway

I'll monitor this, and migrate the last set in 2 batches tomorrow.

Blake updated the task description. (Show Details)Mon, May 11, 3:03 PM
Maintenance_bot removed a project: Patch-For-Review.Mon, May 11, 3:32 PM
gerritbot added a comment.Tue, May 12, 8:39 AM

Change #1286284 had a related patch set uploaded (by Blake; author: Blake):

[operations/puppet@production] gateway-check.lua: Route some LiftWing endpoints through the REST gateway.

https://gerrit.wikimedia.org/r/1286284

gerritbot added a project: Patch-For-Review.Tue, May 12, 8:39 AM
gerritbot added a comment.Tue, May 12, 9:21 AM

Change #1286284 merged by Blake:

[operations/puppet@production] gateway-check.lua: Route some LiftWing endpoints through the REST gateway.

https://gerrit.wikimedia.org/r/1286284

Blake added a comment.Tue, May 12, 9:26 AM

Next set complete:

blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/revertrisk-multilingual:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/revertrisk-language-agnostic:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/readability:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/langid:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"text":"Some sample text in any language that we want to identify"}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/article-descriptions:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"lang":"en","title":"Clandonald","num_beams":2,"debug":1}'
200 rest-gateway

I'll monitor this, and migrate the last batch later today.

Blake updated the task description. (Show Details)Tue, May 12, 9:26 AM
Maintenance_bot removed a project: Patch-For-Review.Tue, May 12, 9:31 AM
gerritbot added a comment.Tue, May 12, 9:55 AM

Change #1286298 had a related patch set uploaded (by Blake; author: Blake):

[operations/puppet@production] gateway-check.lua: Route some LiftWing endpoints through the REST gateway.

https://gerrit.wikimedia.org/r/1286298

gerritbot added a project: Patch-For-Review.Tue, May 12, 9:55 AM
gerritbot added a comment.Tue, May 12, 1:34 PM

Change #1286298 merged by Blake:

[operations/puppet@production] gateway-check.lua: Route some LiftWing endpoints through the REST gateway.

https://gerrit.wikimedia.org/r/1286298

Blake added a comment.Tue, May 12, 1:40 PM

Final set complete:

blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/reference-need:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":123456,"lang":"en"}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/reference-risk:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"rev_id":1242378206,"lang":"en"}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -X POST -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/inference/v1/models/edit-check:predict" -H "Content-Type: application/json" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)" --data '{"instances":[{"original_text":"","modified_text":"this is a great example of work","check_type":"tone","lang":"en","page_title":"Test_Page"}]}'
200 rest-gateway
blake@cp6009:~$ curl -si -o /dev/null -w "%{http_code} %header{via}\n" -H "Host: api.wikimedia.org" "http://api.wikimedia.org:3128/service/lw/recommendation/api/v1/translation?source=en&target=fr&count=3&seed=Apple" -H "User-Agent: ServiceOpsTest (sre-serviceops@wikimedia.org)"
200 rest-gateway

I'll keep an eye on these, but things look good so far. I'll follow up with Claime when they're back to see if we can close this out.

Blake updated the task description. (Show Details)Tue, May 12, 1:51 PM
Maintenance_bot removed a project: Patch-For-Review.Tue, May 12, 2:31 PM
Clement_Goubert closed this task as Resolved.Tue, May 19, 10:56 AM

Thanks to @Blake for powering through the migration. This is done, we can move on to T422937: Cleanup ATS configuration for API paths

Clement_Goubert mentioned this in T426323: CX dashboard can't load page collections on some wikis (blocked by CORS).Tue, May 19, 4:31 PM
Clement_Goubert mentioned this in T413449: [5.2.2c Epic]: API Gateway Rerouting Testing.Wed, Jun 3, 10:39 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits