Page MenuHomePhabricator

Create external endpoint for recommendation-api-ng hosted on LiftWing
Closed, ResolvedPublic1 Estimated Story Points

Description

In T347015#9188042, we deployed the recommendation-api on LiftWing staging and set the internal endpoint below which can only be accessed by tools that run within the WMF infrastructure.

https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/spec

We have worked on deploying the recommendation-api to LiftWing production in T347015#9194064. The objective of this task is to create an external endpoint to enable tools outside the WMF infrastructure to access the rec-api hosted on LiftWing production.

Event Timeline

Hi @Isaac, @santhosh, and @Seddon. The ML team was assigned the task of migrating the recommendation-api from wmflabs to LiftWing. We have successfully deployed the recommendation-api on LiftWing.

An internal endpoint has been created which can only be accessed by tools that run within the WMF infrastructure. The endpoint can be accessed at https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/spec.

We're planning to create an external endpoint to allow tools outside the WMF infrastructure to access the API. However, before we proceed with this, we would like to confirm with you whether the internal endpoint will be sufficient.

Just to be clear this ticket is about the translation recommendation-api.

The mobile apps recommendation-api the apps rely on is https://gerrit.wikimedia.org/g/mediawiki/services/recommendation-api/+/refs/heads/master which I believe @Isaac and @elukey are coordinating on as a follow up.

This will need to be accessed via a public endpoint.

@Seddon my understanding is that this version of the recommendation API is the one that we want to progress from now on, deprecating the one that the apps are using). We need to consolidate the work into one single API, and the recommendation-api that the apps are currently using is already exposed via Restbase.

I think that is my understanding of our goals here, it was just more that the repo that was linked is not the repo for the recommendation-api that the mobile apps team leverages and so I'm assuming (perhaps incorrectly) that this hasn't yet to be included but that remains the destination we are aiming for.

I think that is my understanding of our goals here, it was just more that the repo that was linked is not the repo for the recommendation-api that the mobile apps team leverages and so I'm assuming (perhaps incorrectly) that this hasn't yet to be included but that remains the destination we are aiming for.

Ack! I just wanted to verify that the final goal was to use the recommendation-api-ng external endpoint (for the apps use case I mean). Thanks!

calbon reassigned this task from kevinbazira to klausman.
calbon moved this task from Unsorted to In Progress on the Machine-Learning-Team board.

Thanks @kevinbazira ! API spec worked great when I first tested. Maybe not turned on yet but I tried to test the actual predictions (following Wikitech instructions) and was having trouble getting them. Specifically, tried a few things but maybe I'm missing something small (host or a curl option or perhaps just not the right path):

isaacj@stat1008:~$ curl "https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/?s=en&t=fr&n=3&article=Apple" -A "isaac@wikimedia.org - test" --http1.1

(first time, just hung for ~10 seconds before I killed; second time, now I get "no healthy upstream" immediately for any call so apologies if that was my doing)

Reconstructing based on memory because I killed my shell history while testing:

isaacj@stat1008:~$ cat input.json
{
    "s": "en",
    "t": "fr",
    "n": 6
    "article": "Apple"
}
isaacj@stat1008:~$ curl "https://recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443/api/" -X POST -d @input.json -A "isaac@wikimedia.org - test" --http1.1

(got something about POST not being allowed I think)

Thank you for testing the internal endpoint @Isaac. We are investigating the cause of this issue in T347475 and a possible solution for it.

Thanks for the update @kevinbazira ! Don't hesitate to ask if I can help brainstorm what might be going on etc. if it turns out to be more than just a resources problem.

Internal endpoint available @kevinbazira:

elukey@stat1004:~$ curl "https://recommendation-api-ng.discovery.wmnet:31443/api/spec" -i --http1.1
HTTP/1.1 200 OK
content-type: application/json
content-length: 1908
access-control-allow-origin: *
access-control-allow-headers: Content-Type,Authorization
access-control-allow-methods: GET
x-envoy-upstream-service-time: 8
date: Tue, 03 Oct 2023 15:51:09 GMT
server: istio-envoy

{"basePath":"/api","consumes":["application/json"],"definitions":{"Article":{"properties":{"pageviews":{"description":"pageviews","type":"integer"},"rank":{"description":"rank","type":"number"},"title":{"description":"title","type":"string"},"wikidata_id":{"description":"wikidata_id","type":"string"}},"required":["rank","title","wikidata_id"],"type":"object"}},"info":{"title":"API","version":"1.0"},"paths":{"/":{"get":{"deprecated":true,"description":"Gets recommendations of source articles that are missing in the target","operationId":"get_legacy_article","parameters":[{"description":"Source wiki project language code","in":"query","name":"s","required":true,"type":"string"},{"description":"Target wiki project language code","in":"query","name":"t","required":true,"type":"string"},{"default":12,"description":"Number of recommendations to fetch","in":"query","maximum":24,"minimum":0,"name":"n","type":"integer"},{"description":"Seed article for personalized recommendations that can also be a list separated by \"|\"","in":"query","name":"article","pattern":"^([^|]+(\\|[^|]+)*)?$","type":"string"},{"default":true,"description":"Whether to include pageview counts","in":"query","name":"pageviews","type":"boolean"},{"collectionFormat":"multi","default":"morelike","description":"Which search algorithm to use if a seed is specified","enum":["morelike","wiki"],"in":"query","name":"search","type":"string"}],"responses":{"200":{"description":"Success","schema":{"items":{"$ref":"#/definitions/Article"},"type":"array"}}},"tags":["default"]}},"/spec":{"get":{"operationId":"get_spec","responses":{"200":{"description":"Success"}},"tags":["default"]}}},"produces":["application/json"],"responses":{"MaskError":{"description":"When any error occurs on mask"},"ParseError":{"description":"When a mask can't be parsed"}},"swagger":"2.0","tags":[{"description":"Default namespace","name":"default"}]}

Next and last step - API gateway config

calbon set the point value for this task to 1.Nov 2 2023, 7:11 PM
calbon moved this task from In Progress to Ready To Go on the Machine-Learning-Team board.
achou triaged this task as Medium priority.Nov 2 2023, 7:28 PM

Change 980865 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/deployment-charts@master] api-gateway: Add entry for recommendation-api-ng on Lift Wing

https://gerrit.wikimedia.org/r/980865

Change 980865 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: Add entry for recommendation-api-ng on Lift Wing

https://gerrit.wikimedia.org/r/980865

Change 981342 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/deployment-charts@master] API GW: Add ingress endpoints on Lift Wing to allowed destinations

https://gerrit.wikimedia.org/r/981342

Change 981342 merged by jenkins-bot:

[operations/deployment-charts@master] API GW: Add ingress endpoints on Lift Wing to allowed destinations

https://gerrit.wikimedia.org/r/981342

Change 981344 had a related patch set uploaded (by Klausman; author: Klausman):

[operations/deployment-charts@master] APIGW: add missing /32 to egress rules

https://gerrit.wikimedia.org/r/981344

Change 981344 merged by jenkins-bot:

[operations/deployment-charts@master] APIGW: add missing /32 to egress rules and fix port

https://gerrit.wikimedia.org/r/981344

Change 981359 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] api-gateway: set host and ingress for recommendation-api-ng

https://gerrit.wikimedia.org/r/981359

Change 981359 merged by jenkins-bot:

[operations/deployment-charts@master] api-gateway: set host and ingress for recommendation-api-ng

https://gerrit.wikimedia.org/r/981359

From my workstation at home, both the API endpoint and the spec query work, but there are still things that could improve (more on that below):

$ time curl -s "https://api.wikimedia.org/service/lw/recommendation/v1/api/?s=en&t=fr&n=3&article=France"  |jq .
[
  {
    "pageviews": 1831,
    "title": "Turks_in_Europe",
    "wikidata_id": "Q4821600",
    "rank": 498.0
  },
  {
    "pageviews": 3271,
    "title": "European_emigration",
    "wikidata_id": "Q3819571",
    "rank": 496.0
  },
  {
    "pageviews": 790,
    "title": "History_of_France_(1900–present)",
    "wikidata_id": "Q290313",
    "rank": 476.0
  }
]
real	0m4.985s
user	0m0.072s
sys	0m0.008s

And:

~ $ curl -s "https://api.wikimedia.org/service/lw/recommendation/v1/api/spec" |jq .

{
  "basePath": "/api",
  "consumes": [
    "application/json"
  ],
  "definitions": {
    "Article": {
      "properties": {
        "pageviews": {
          "description": "pageviews",
          "type": "integer"
        },
        "rank": {
          "description": "rank",
          "type": "number"
        },
        "title": {
          "description": "title",
          "type": "string"
        },
        "wikidata_id": {
          "description": "wikidata_id",
          "type": "string"
        }
      },
      "required": [
        "rank",
        "title",
        "wikidata_id"
      ],
      "type": "object"
    }
  },
  "info": {
    "title": "API",
    "version": "1.0"
  },
  "paths": {
    "/": {
      "get": {
        "deprecated": true,
        "description": "Gets recommendations of source articles that are missing in the target",
        "operationId": "get_legacy_article",
        "parameters": [
          {
            "description": "Source wiki project language code",
            "in": "query",
            "name": "s",
            "required": true,
            "type": "string"
          },
          {
            "description": "Target wiki project language code",
            "in": "query",
            "name": "t",
            "required": true,
            "type": "string"
          },
          {
            "default": 12,
            "description": "Number of recommendations to fetch",
            "in": "query",
            "maximum": 24,
            "minimum": 0,
            "name": "n",
            "type": "integer"
          },
          {
            "description": "Seed article for personalized recommendations that can also be a list separated by \"|\"",
            "in": "query",
            "name": "article",
            "pattern": "^([^|]+(\\|[^|]+)*)?$",
            "type": "string"
          },
          {
            "default": true,
            "description": "Whether to include pageview counts",
            "in": "query",
            "name": "pageviews",
            "type": "boolean"
          },
          {
            "collectionFormat": "multi",
            "default": "morelike",
            "description": "Which search algorithm to use if a seed is specified",
            "enum": [
              "morelike",
              "wiki"
            ],
            "in": "query",
            "name": "search",
            "type": "string"
          }
        ],
        "responses": {
          "200": {
            "description": "Success",
            "schema": {
              "items": {
                "$ref": "#/definitions/Article"
              },
              "type": "array"
            }
          }
        },
        "tags": [
          "default"
        ]
      }
    },
    "/spec": {
      "get": {
        "operationId": "get_spec",
        "responses": {
          "200": {
            "description": "Success"
          }
        },
        "tags": [
          "default"
        ]
      }
    }
  },
  "produces": [
    "application/json"
  ],
  "responses": {
    "MaskError": {
      "description": "When any error occurs on mask"
    },
    "ParseError": {
      "description": "When a mask can't be parsed"
    }
  },
  "swagger": "2.0",
  "tags": [
    {
      "description": "Default namespace",
      "name": "default"
    }
  ]
}

Things that still need improvement:

  • the slash after api is mandatory, i.e. https://api.wikimedia.org/service/lw/recommendation/v1/api/?s=en&t=fr&n=3&article=France works, but https://api.wikimedia.org/service/lw/recommendation/v1/api?s=en&t=fr&n=3&article=France results in a redirect to the internal endpoint. I think we should either fix the redirect (this may be tricky) or make the API gateway automatically rewrite the missing slash. I will see how much work the latter would be
  • There are 404ing requests in the logs, for /metrics. This is Prometheus trying to get metrics from the service. I think we should think about what metrics would be useful and export them there.

This is complete. I'll track the slash-vs-no-slash matter mentioned above in a separate ticket.

Things that still need improvement:

  • the slash after api is mandatory, i.e. https://api.wikimedia.org/service/lw/recommendation/v1/api/?s=en&t=fr&n=3&article=France works, but https://api.wikimedia.org/service/lw/recommendation/v1/api?s=en&t=fr&n=3&article=France results in a redirect to the internal endpoint. I think we should either fix the redirect (this may be tricky) or make the API gateway automatically rewrite the missing slash. I will see how much work the latter would be

Thank you for reporting this issue @klausman, we have fixed it in: T354601