Page MenuHomePhabricator
Create Task
Maniphest T348607

Configure envoy settings to enable rec-api-ng container to access endpoints external to k8s/LiftWing
Closed, ResolvedPublic
Actions

Description

In T347475, we discovered that the rec-api-ng container hosted on LiftWing was not able to access endpoints external to kubernetes.

To resolve this, we shall configure envoy settings to enable the rec-api-ng container hosted on LiftWing to access wikimedia, wikidata, and wikipedia apis.

Below are the steps we are going to take:

  1. add envoy listener(s) to the rec-api-ng deployment settings
  2. update the specified api urls to use localhost:<PORT> in recommendation_liftwing.ini
  3. configure requests to the above endpoints to send their respective Host headers (see example)

Details

Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

kevinbazira created this task.Oct 11 2023, 7:15 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 11 2023, 7:15 AM
kevinbazira mentioned this in T347475: Investigate recommendation-api-ng internal endpoint failure.Oct 11 2023, 7:58 AM
elukey moved this task from Unsorted to In Progress on the Machine-Learning-Team board.Oct 12 2023, 9:51 AM
gerritbot added a comment.Oct 12 2023, 12:45 PM

Change 965142 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[research/recommendation-api@master] Use envoy proxy to access endpoints external to k8s/LiftWing

https://gerrit.wikimedia.org/r/965142

gerritbot added a project: Patch-For-Review.Oct 12 2023, 12:45 PM
gerritbot added a comment.Oct 17 2023, 1:37 PM

Change 965142 merged by jenkins-bot:

[research/recommendation-api@master] Use envoy proxy to access endpoints external to k8s/LiftWing

https://gerrit.wikimedia.org/r/965142

Maintenance_bot removed a project: Patch-For-Review.Oct 17 2023, 2:10 PM
gerritbot added a comment.Oct 18 2023, 7:15 AM

Change 965585 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: update the recommendation-api-ng image

https://gerrit.wikimedia.org/r/965585

gerritbot added a project: Patch-For-Review.Oct 18 2023, 7:15 AM
gerritbot added a comment.Oct 18 2023, 7:47 AM

Change 965585 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: update the recommendation-api-ng image

https://gerrit.wikimedia.org/r/965585

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2023, 8:10 AM
gerritbot added a comment.Oct 18 2023, 9:46 AM

Change 966826 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[research/recommendation-api@master] Update external endpoint ports used on LiftWing

https://gerrit.wikimedia.org/r/966826

gerritbot added a project: Patch-For-Review.Oct 18 2023, 9:46 AM
gerritbot added a comment.Oct 18 2023, 9:48 AM

Change 966826 merged by jenkins-bot:

[research/recommendation-api@master] Update external endpoint ports used on LiftWing

https://gerrit.wikimedia.org/r/966826

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2023, 10:10 AM
gerritbot added a comment.Oct 18 2023, 10:17 AM

Change 966827 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add listeners for cxserver and eventgate-analytics to rec-api-ng

https://gerrit.wikimedia.org/r/966827

gerritbot added a project: Patch-For-Review.Oct 18 2023, 10:17 AM
gerritbot added a comment.Oct 18 2023, 10:22 AM

Change 966827 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: add listeners for cxserver and eventgate-analytics to rec-api-ng

https://gerrit.wikimedia.org/r/966827

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2023, 10:30 AM
kevinbazira added a comment.Oct 19 2023, 8:47 AM

We configured envoy settings and the rec-api-ng container hosted on LiftWing is able to access endpoints external to k8s/LiftWing. Below are the settings we configured:

endpoint nameenpoint host headerenvoy listenerport
language_pairscxserver.wikimedia.orgcxserver6015
pageviewswikimedia.orgmw-api-int-async-ro6500
wikipedia{source}.wikipedia.orgmw-api-int-async-ro6500
wikidatawww.wikidata.orgmw-api-int-async-ro6500
event_loggerintake-analytics.wikimedia.orgeventgate-analytics6004
gerritbot added a comment.Oct 20 2023, 9:35 AM

Change 967401 had a related patch set uploaded (by Elukey; author: Elukey):

[research/recommendation-api@master] Fix pageviews base endpoint

https://gerrit.wikimedia.org/r/967401

gerritbot added a project: Patch-For-Review.Oct 20 2023, 9:35 AM
gerritbot added a comment.Oct 20 2023, 9:40 AM

Change 967401 merged by Elukey:

[research/recommendation-api@master] Fix pageviews base endpoint

https://gerrit.wikimedia.org/r/967401

gerritbot added a comment.Oct 20 2023, 9:47 AM

Change 966836 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: update the recommendation-api-ng image

https://gerrit.wikimedia.org/r/966836

gerritbot added a comment.Oct 20 2023, 9:56 AM

Change 966836 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: update the recommendation-api-ng image

https://gerrit.wikimedia.org/r/966836

Maintenance_bot removed a project: Patch-For-Review.Oct 20 2023, 10:10 AM
kevinbazira added a comment.EditedOct 20 2023, 11:20 AM

We discovered that the pageviews endpoint does not use mw-api-int-async-ro but rather aqs listener. The full settings that enabled the rec-api-ng hosted on LiftWing to access endpoints external to k8s/LiftWing through the envoy proxy are:

endpoint nameenpoint host headerenpoint urienvoy listener
language_pairscxserver.wikimedia.orghttp://localhost:6015/v1/languagepairscxserver
pageviewswikimedia.orghttp://localhost:6033/wikimedia.org/v1/metrics/pageviewsrest-gateway
wikipedia{source}.wikipedia.orghttp://localhost:6500/w/api.phpmw-api-int-async-ro
wikidatawww.wikidata.orghttp://localhost:6500/w/api.phpmw-api-int-async-ro
event_loggerintake-analytics.wikimedia.orghttp://localhost:6004/v1/events?hasty=trueeventgate-analytics

Update 1: Following T348607#9275348, the pageviews uri has been updated from http://localhost:6020/analytics.wikimedia.org/v1/metrics/pageviews to http://localhost:6020/analytics.wikimedia.org/v1/pageviews

Update 2: Following T348607#9283681, the pageviews uri has been updated from http://localhost:6020/analytics.wikimedia.org/v1/pageview to http://localhost:6033/wikimedia.org/v1/metrics/pageviews and envoy listener from aqs to rest-gateway

kevinbazira added a comment.Oct 23 2023, 7:33 AM

To help others in future who might have challenges to resolve the issue above, I have added a note to the envoy proxy docs that shows the same host could use different envoy listeners based on the endpoint being accessed:
https://wikitech.wikimedia.org/w/index.php?title=Envoy&oldid=2121603#Example_(calling_mw-api)

gerritbot added a comment.Oct 24 2023, 7:42 AM

Change 967916 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[research/recommendation-api@master] Update pageviews external endpoint uri used on LiftWing

https://gerrit.wikimedia.org/r/967916

gerritbot added a project: Patch-For-Review.Oct 24 2023, 7:42 AM
kevinbazira added a comment.Oct 24 2023, 8:32 AM

As reported in T347475#9269007, the pageviews endpoint is failing. I reproduced this issue on staging and here are the logs showing that the endpoint fails because the URI:

2023-10-24 05:09:12,783 recommendation.utils.event_logger log_api_request():37 INFO -- Logging event: {"schema": "TranslationRecommendationAPIRequests", "$schema": "/analytics/legacy/$translationrecommendationapirequests/1.0.0", "revision": 16261139, "event": {"timestamp": 1698124152, "sourceLanguage": "en", "targetLanguage": "es", "searchAlgorithm": "related_articles"}, "webHost": "recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443", "client_dt": "2023-10-24T05:09:12.783157", "meta": {"stream": "eventlogging_TranslationRecommendationAPIRequests", "domain": "recommendation-api-ng.k8s-ml-staging.discovery.wmnet:31443"}}
2023-10-24 05:09:12,801 recommendation.api.external_data.fetcher get():26 INFO -- Request failed: {"url": "http://localhost:6020/analytics.wikimedia.org/v1/metrics/pageviews/top/en.wikipedia/all-access/2023/10/22", "error": "404 Client Error: Not Found for url: http://localhost:6020/analytics.wikimedia.org/v1/metrics/pageviews/top/en.wikipedia/all-access/2023/10/22"}
2023-10-24 05:09:12,801 recommendation.api.external_data.fetcher get_most_popular_articles():143 INFO -- pageview query failed
2023-10-24 05:09:12,847 recommendation.api.types.translation.translation process_request():195 INFO -- Request processed in 0.064753 seconds
[pid: 135|app: 0|req: 7360/10465] 127.0.0.1 () {44 vars in 2182 bytes} [Tue Oct 24 05:09:12 2023] GET /types/translation/v1/articles?source=en&target=es&seed=&search=related_articles&application=CX => generated 3 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 195 bytes (1 switches on core 0)
gerritbot added a comment.Oct 24 2023, 8:53 AM

Change 967916 merged by jenkins-bot:

[research/recommendation-api@master] Update pageviews external endpoint uri used on LiftWing

https://gerrit.wikimedia.org/r/967916

gerritbot added a comment.Oct 24 2023, 9:00 AM

Change 967917 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: update recommendation-api-ng image

https://gerrit.wikimedia.org/r/967917

gerritbot added a comment.Oct 24 2023, 9:43 AM

Change 967917 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: update recommendation-api-ng image

https://gerrit.wikimedia.org/r/967917

kevinbazira added a comment.Oct 24 2023, 10:02 AM

Since we couldn't experiment and check the correct pageviews uri from k8s, an SRE had to log into one of the AQS nodes and tested the uri with curl until we got the correct one: http://localhost:6020/analytics.wikimedia.org/v1/pageviews. We've added it to the rec-api-ng, deployed it on staging, tested that it works, and then deployed it to prod.

Maintenance_bot removed a project: Patch-For-Review.Oct 24 2023, 10:10 AM
gerritbot added a comment.Oct 25 2023, 10:10 AM

Change 967921 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[research/recommendation-api@master] Update pageviews endpoint to match rest-gateway envoy listener port

https://gerrit.wikimedia.org/r/967921

gerritbot added a project: Patch-For-Review.Oct 25 2023, 10:10 AM
gerritbot added a comment.Oct 25 2023, 12:34 PM

Change 967921 merged by jenkins-bot:

[research/recommendation-api@master] Update pageviews endpoint to match rest-gateway envoy listener port

https://gerrit.wikimedia.org/r/967921

gerritbot added a comment.Oct 25 2023, 12:43 PM

Change 967922 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: add rest-gateway listener for rec-api-ng

https://gerrit.wikimedia.org/r/967922

gerritbot added a comment.Oct 25 2023, 12:58 PM

Change 967922 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: add rest-gateway listener for rec-api-ng

https://gerrit.wikimedia.org/r/967922

Maintenance_bot removed a project: Patch-For-Review.Oct 25 2023, 1:10 PM
gerritbot added a comment.Oct 26 2023, 7:27 AM

Change 967925 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[research/recommendation-api@master] Update pageviews endpoint to match rest-gateway envoy listener

https://gerrit.wikimedia.org/r/967925

gerritbot added a project: Patch-For-Review.Oct 26 2023, 7:27 AM
gerritbot added a comment.Oct 26 2023, 8:11 AM

Change 967925 merged by jenkins-bot:

[research/recommendation-api@master] Update pageviews endpoint to match rest-gateway envoy listener

https://gerrit.wikimedia.org/r/967925

gerritbot added a comment.Oct 26 2023, 8:15 AM

Change 968966 had a related patch set uploaded (by Kevin Bazira; author: Kevin Bazira):

[operations/deployment-charts@master] ml-services: update recommendation-api-ng image

https://gerrit.wikimedia.org/r/968966

gerritbot added a comment.Oct 26 2023, 8:30 AM

Change 968966 merged by jenkins-bot:

[operations/deployment-charts@master] ml-services: update recommendation-api-ng image

https://gerrit.wikimedia.org/r/968966

Maintenance_bot removed a project: Patch-For-Review.Oct 26 2023, 9:10 AM
kevinbazira added a comment.Oct 26 2023, 11:17 AM

Folks from wikimedia-analytics notified us that they are migrating the pageviews endpoint as shown in the screenshot below:

notice - pageviews endpoint migration from retbase to rest-gateway : Screenshot from 2023-10-24 18-06-29.png (459×942 px, 83 KB)

We have updated the rec-api-ng envoy settings for the pageviews endpoint from:

uri: http://localhost:6020/analytics.wikimedia.org/v1/pageview
listener: aqs

to

uri: http://localhost:6033/wikimedia.org/v1/metrics/pageviews
listener: rest-gateway
calbon triaged this task as Medium priority.Nov 2 2023, 7:25 PM
kevinbazira mentioned this in T348156: Goal: Increase the number of models hosted on Lift Wing.Nov 14 2023, 4:01 PM
kevinbazira closed this task as Resolved.Dec 19 2023, 6:39 AM

The rec-api-ng container is now able to access endpoints external to k8s/LiftWing even after the wikimedia-analytics migrated the pageviews endpoint.

kevinbazira moved this task from In Progress to 2023-2024 Q3 Done on the Machine-Learning-Team board.Dec 19 2023, 7:06 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits