Page MenuHomePhabricator
Create Task
Maniphest T362316

Migrate ml-services to mw-api-int
Open, MediumPublic5 Estimated Story Points
Actions

Description

As these services don't use the envoy service mesh to reach out to mediawiki, they were not taken into account for the migration of internal services.

The following services reach out directly to the bare-metal cluster url api-ro.discovery.wmnet

servicedoneCR
article-descriptionsstaging prod
articletopic-outlinkstaging prod
experimentalprod
readabilitystaging prod
revertriskstaging prod
revscoring-articlequalitystaging prod
revscoring-articletopicstaging prod
revscoring-draftqualitystaging prod
revscoring-drafttopicstaging prod
revscoring-editquality-damagingstaging prod
revscoring-editquality-goodfaithstaging prod
revscoring-editquality-revertedstaging prod

Details

SubjectRepoBranchLines +/-
articletopic-outlink: Switch prod to mw-api-int-rooperations/deployment-chartsmaster+1 -1
revscoring-editquality-reverted: Switch prod to mw-api-int-rooperations/deployment-chartsmaster+1 -3
revscoring-editquality-goodfaith: Switch prod to mw-api-int-rooperations/deployment-chartsmaster+2 -3
revscoring-articletopic: Switch prod to mw-api-int-rooperations/deployment-chartsmaster+0 -2
revscoring-articlequality: Switch prod to mw-api-int-rooperations/deployment-chartsmaster+0 -2
revertrisk: Switch prod to mw-api-int-rooperations/deployment-chartsmaster+2 -2
readability: Switch prod to mw-api-int-rooperations/deployment-chartsmaster+1 -1
article-description: Switch prod to mw-api-int-rooperations/deployment-chartsmaster+1 -2
revscoring-editquality-damaging: Switch prod to mw-api-int-rooperations/deployment-chartsmaster+3 -4
revscoring-drafttopic: Switch prod to mw-api-int-rooperations/deployment-chartsmaster+0 -2
revscoring-draftquality: Switch prod to mw-api-int-rooperations/deployment-chartsmaster+0 -2
admin_ng: move Istio configs to mw-api-int-ro for ml-serveoperations/deployment-chartsmaster+89 -154
revertrisk: Switch staging to mw-api-int-rooperations/deployment-chartsmaster+23 -1
experimental: Switch to mw-api-int-rooperations/deployment-chartsmaster+9 -1
readability: Switch staging to mw-api-int-rooperations/deployment-chartsmaster+15 -0
revscoring-editquality-reverted: Switch staging to mw-api-int-rooperations/deployment-chartsmaster+9 -1
revscoring-editquality-goodfaith: Switch staging to mw-api-int-rooperations/deployment-chartsmaster+9 -1
revscoring-drafttopic: Switch staging to mw-api-int-rooperations/deployment-chartsmaster+9 -1
revscoring-draftquality: Switch staging to mw-api-int-rooperations/deployment-chartsmaster+9 -1
revscoring-articletopic: Switch staging to mw-api-int-rooperations/deployment-chartsmaster+9 -1
revscoring-articlequality: Switch staging to mw-api-int-rooperations/deployment-chartsmaster+9 -1
articletopic-outlink: Switch staging to mw-api-int-rooperations/deployment-chartsmaster+11 -0
admin_ng: add port 4680 in istio configs for ml-stagingoperations/deployment-chartsmaster+4 -0
article-description: Switch staging to mw-api-int-rooperations/deployment-chartsmaster+19 -1
revscoring-editquality-damaging: Switch staging to mw-api-int-rooperations/deployment-chartsmaster+9 -1
ml-staging-codfw: Override mediawiki-app-vsoperations/deployment-chartsmaster+92 -0
ml-serve: Add istio config for mw-api-int-rooperations/deployment-chartsmaster+52 -0
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Clement_Goubert created this task.Thu, Apr 11, 10:36 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptThu, Apr 11, 10:36 AM
Clement_Goubert triaged this task as Medium priority.Thu, Apr 11, 10:36 AM
gerritbot added a comment.Thu, Apr 11, 11:26 AM

Change #1018959 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] article-description: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018959

gerritbot added a project: Patch-For-Review.Thu, Apr 11, 11:26 AM

Change #1018960 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] article-description: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018960

gerritbot added a comment.Thu, Apr 11, 11:27 AM

Change #1018961 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] articletopic-outlink: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018961

gerritbot added a comment.Thu, Apr 11, 11:27 AM

Change #1018962 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] articletopic-outlink: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018962

gerritbot added a comment.Thu, Apr 11, 11:27 AM

Change #1018963 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] experimental: Switch to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018963

gerritbot added a comment.Thu, Apr 11, 11:27 AM

Change #1018964 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] readability: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018964

gerritbot added a comment.Thu, Apr 11, 11:28 AM

Change #1018965 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] readability: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018965

gerritbot added a comment.Thu, Apr 11, 11:28 AM

Change #1018986 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revertrisk: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018986

gerritbot added a comment.Thu, Apr 11, 11:28 AM

Change #1018987 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revertrisk: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018987

gerritbot added a comment.Thu, Apr 11, 11:29 AM

Change #1018988 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-articlequality: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018988

gerritbot added a comment.Thu, Apr 11, 11:29 AM

Change #1018989 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-articlequality: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018989

gerritbot added a comment.Thu, Apr 11, 11:29 AM

Change #1018990 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-articletopic: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018990

gerritbot added a comment.Thu, Apr 11, 11:29 AM

Change #1018991 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-articletopic: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018991

gerritbot added a comment.Thu, Apr 11, 11:30 AM

Change #1018992 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-draftquality: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018992

gerritbot added a comment.Thu, Apr 11, 11:30 AM

Change #1018993 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-draftquality: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018993

gerritbot added a comment.Thu, Apr 11, 11:30 AM

Change #1018994 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-drafttopic: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018994

gerritbot added a comment.Thu, Apr 11, 11:31 AM

Change #1018995 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-drafttopic: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018995

gerritbot added a comment.Thu, Apr 11, 11:31 AM

Change #1018996 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-editquality-damaging: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018996

gerritbot added a comment.Thu, Apr 11, 11:31 AM

Change #1018997 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-editquality-damaging: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018997

gerritbot added a comment.Thu, Apr 11, 11:32 AM

Change #1018998 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-editquality-goodfaith: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018998

gerritbot added a comment.Thu, Apr 11, 11:32 AM

Change #1018999 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-editquality-goodfaith: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018999

gerritbot added a comment.Thu, Apr 11, 11:32 AM

Change #1019000 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-editquality-reverted: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1019000

gerritbot added a comment.Thu, Apr 11, 11:32 AM

Change #1019001 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] revscoring-editquality-reverted: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1019001

Clement_Goubert updated the task description. (Show Details)Thu, Apr 11, 11:43 AM
Clement_Goubert added a comment.Thu, Apr 11, 11:48 AM

Aaaand I just realized they all use http and not https, so now I can change them all.

Clement_Goubert mentioned this in T362323: Move 100% of external traffic to Kubernetes (excluding Votewiki and Commons).Thu, Apr 11, 12:46 PM
gerritbot added a comment.Thu, Apr 11, 2:50 PM

Change #1019061 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] ml-serve: Add istio config for mw-api-int-ro

https://gerrit.wikimedia.org/r/1019061

gerritbot added a comment.Thu, Apr 11, 4:08 PM

Change #1019074 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] ml-staging-codfw: Override mediawiki-app-vs

https://gerrit.wikimedia.org/r/1019074

gerritbot added a comment.Fri, Apr 12, 12:40 PM

Change #1019061 merged by Elukey:

[operations/deployment-charts@master] ml-serve: Add istio config for mw-api-int-ro

https://gerrit.wikimedia.org/r/1019061

gerritbot added a comment.Fri, Apr 12, 2:13 PM

Change #1019074 merged by Elukey:

[operations/deployment-charts@master] ml-staging-codfw: Override mediawiki-app-vs

https://gerrit.wikimedia.org/r/1019074

gerritbot added a comment.Fri, Apr 12, 2:14 PM

Change #1018996 merged by Elukey:

[operations/deployment-charts@master] revscoring-editquality-damaging: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018996

gerritbot added a comment.Fri, Apr 12, 2:48 PM

Change #1019288 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: override port 80 with 4680 in istio configs for ml-staging

https://gerrit.wikimedia.org/r/1019288

gerritbot added a comment.Fri, Apr 12, 2:57 PM

Change #1018959 merged by Elukey:

[operations/deployment-charts@master] article-description: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018959

gerritbot added a comment.Fri, Apr 12, 2:57 PM

Change #1019288 merged by Elukey:

[operations/deployment-charts@master] admin_ng: add port 4680 in istio configs for ml-staging

https://gerrit.wikimedia.org/r/1019288

gerritbot added a comment.Fri, Apr 12, 3:18 PM

Change #1018961 merged by Elukey:

[operations/deployment-charts@master] articletopic-outlink: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018961

gerritbot added a comment.Fri, Apr 12, 3:40 PM

Change #1018988 merged by Elukey:

[operations/deployment-charts@master] revscoring-articlequality: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018988

gerritbot added a comment.Fri, Apr 12, 3:40 PM

Change #1018990 merged by Elukey:

[operations/deployment-charts@master] revscoring-articletopic: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018990

gerritbot added a comment.Fri, Apr 12, 3:40 PM

Change #1018992 merged by Elukey:

[operations/deployment-charts@master] revscoring-draftquality: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018992

gerritbot added a comment.Fri, Apr 12, 3:41 PM

Change #1018994 merged by Elukey:

[operations/deployment-charts@master] revscoring-drafttopic: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018994

gerritbot added a comment.Fri, Apr 12, 3:41 PM

Change #1018998 merged by Elukey:

[operations/deployment-charts@master] revscoring-editquality-goodfaith: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018998

gerritbot added a comment.Fri, Apr 12, 3:41 PM

Change #1019000 merged by Elukey:

[operations/deployment-charts@master] revscoring-editquality-reverted: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1019000

gerritbot added a comment.Fri, Apr 12, 3:52 PM

Change #1018964 merged by Elukey:

[operations/deployment-charts@master] readability: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018964

gerritbot added a comment.Fri, Apr 12, 3:52 PM

Change #1018963 merged by jenkins-bot:

[operations/deployment-charts@master] experimental: Switch to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018963

gerritbot added a comment.Fri, Apr 12, 3:56 PM

Change #1018986 merged by Elukey:

[operations/deployment-charts@master] revertrisk: Switch staging to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018986

elukey subscribed.Fri, Apr 12, 3:58 PM

Current status:

  • all services deployed in ml-staging, need to double check that all the pods are running but so far I didn't notice a problem.
  • plan the work for prod (we'll likely need to depool one DC at the time when we do it)
elukey removed Clement_Goubert as the assignee of this task.Fri, Apr 12, 3:58 PM
klausman subscribed.Mon, Apr 15, 7:23 AM
isarantopoulos subscribed.Mon, Apr 15, 7:45 AM
elukey mentioned this in T353622: Improve Istio's mesh traffic transparent proxy capabilities for external domains accessed by Lift Wing.Wed, Apr 17, 3:45 PM

Added some thoughts to T353622#9723070, I found out a big can of worms while testing staging :) The upgrade is more complex than anticipated, but we should be able to do it this or next week maximum.

gerritbot added a comment.Thu, Apr 18, 2:27 PM

Change #1021490 had a related patch set uploaded (by Elukey; author: Elukey):

[operations/deployment-charts@master] admin_ng: move Istio configs to mw-api-int-ro for ml-serve

https://gerrit.wikimedia.org/r/1021490

gerritbot added a comment.Fri, Apr 26, 12:28 PM

Change #1021490 merged by Elukey:

[operations/deployment-charts@master] admin_ng: move Istio configs to mw-api-int-ro for ml-serve

https://gerrit.wikimedia.org/r/1021490

elukey added a comment.Mon, Apr 29, 4:01 PM

After a lot of tests and config changes, we are almost ready to proceed with prod. Hopefully we'll get to it on April 2nd.

elukey added a comment.Tue, Apr 30, 1:52 PM

All changes rebased and ready to go (for prod). The main idea is the following:

  • Remove WIKI_URL for revscoring isvcs, so we'll use the transparent proxy functionality.
  • Move the rest to the new mw-api-int-ro discovery endpoint, and slowly migrate to transparent proxy later on.

Overall procedure to upgrade:

  1. Depool codfw from inference.discovery.wmnet
  2. Wait some time for traffic to drain, confirm via Grafana metrics before proceeding.
  3. Apply admin_ng changes for knative-serving (new Istio configs, already ready to go)
  4. Merge one of the revscoring prod endpoint changes (see task description), and test the isvc
  5. Merge one of the non-revscoring endpoint changes, and check the isvc.
  6. Merge and rollout all the changes.
  7. Use httpbb from deploy1002 to check all the isvcs
  8. Once ready, repool traffic.

I'd say to wait a couple of days to observe anomalies/errors/etc.., and then proceed with eqiad. The idea is that if anything is really wrong with codfw, we can depool very quickly and investigate.

Timeline:

  • Move codfw to the new endpoint on April 2nd
  • Move eqiad to the new endpoint on April 6th
elukey claimed this task.Tue, Apr 30, 2:13 PM
elukey moved this task from Unsorted to In Progress on the Machine-Learning-Team board.
gerritbot added a comment.Thu, May 2, 11:56 AM

Change #1018997 merged by Elukey:

[operations/deployment-charts@master] revscoring-editquality-damaging: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018997

gerritbot added a comment.Thu, May 2, 12:12 PM

Change #1018960 merged by Elukey:

[operations/deployment-charts@master] article-description: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018960

gerritbot added a comment.Thu, May 2, 12:13 PM

Change #1018965 merged by Elukey:

[operations/deployment-charts@master] readability: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018965

gerritbot added a comment.Thu, May 2, 12:13 PM

Change #1018987 merged by Elukey:

[operations/deployment-charts@master] revertrisk: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018987

gerritbot added a comment.Thu, May 2, 12:13 PM

Change #1018989 merged by Elukey:

[operations/deployment-charts@master] revscoring-articlequality: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018989

gerritbot added a comment.Thu, May 2, 12:13 PM

Change #1018991 merged by Elukey:

[operations/deployment-charts@master] revscoring-articletopic: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018991

gerritbot added a comment.Thu, May 2, 12:14 PM

Change #1018999 merged by Elukey:

[operations/deployment-charts@master] revscoring-editquality-goodfaith: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018999

gerritbot added a comment.Thu, May 2, 12:14 PM

Change #1019001 merged by Elukey:

[operations/deployment-charts@master] revscoring-editquality-reverted: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1019001

gerritbot added a comment.Thu, May 2, 12:14 PM

Change #1018962 merged by Elukey:

[operations/deployment-charts@master] articletopic-outlink: Switch prod to mw-api-int-ro

https://gerrit.wikimedia.org/r/1018962

kevinbazira mentioned this in T363449: Configure the logo-detection model-server hosted on LiftWing to process images from Wikimedia Commons.Fri, May 3, 8:48 AM
elukey added a comment.Fri, May 3, 1:36 PM

Status: Lift Wing codfw has been migrated successfully, we are going to do eqiad on Monday 6th.

elukey added a comment.Mon, May 6, 1:30 PM

And eqiad migrated as well, all done :)

elukey set the point value for this task to 5.Mon, May 6, 1:30 PM
elukey moved this task from In Progress to 2023-2024 Q4 Done on the Machine-Learning-Team board.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL