Page MenuHomePhabricator
Create Task
Maniphest T343388

Split the monolithic function-evaluator service up in production so we have differently-scalable pods for python vs. node
Closed, ResolvedPublic
Actions

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Jdforrester-WMF created this task.Aug 2 2023, 9:53 PM
Jdforrester-WMF added a parent task: T343389: Provide different builds of the function-evaluator service up in production for different language versions, so we have differently-scalable pods for python 3.11 vs. python 3.12 vs. ….Aug 2 2023, 10:01 PM
Jdforrester-WMF mentioned this in T286752: Re-design the Wikifunctions system config to be more like k8s and less encoded inside the MW app layer.Aug 2 2023, 10:04 PM
Jdforrester-WMF triaged this task as Medium priority.Aug 7 2023, 6:11 PM
Jdforrester-WMF moved this task from To Triage to Backlog on the Abstract Wikipedia team board.
Clement_Goubert moved this task from Incoming 🐫 to 🌻Mediawiki on the serviceops-deprecated board.Sep 18 2023, 11:39 AM
gerritbot added a comment.Oct 2 2023, 10:26 PM

Change 962716 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Begin split of function-evaluator into js and python services

https://gerrit.wikimedia.org/r/962716

gerritbot added a project: Patch-For-Review.Oct 2 2023, 10:26 PM

Change 962717 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Switch execution from main to language-specific evaluators

https://gerrit.wikimedia.org/r/962717

gerritbot added a comment.Oct 2 2023, 10:26 PM

Change 962718 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Drop references to legacy main evaluator

https://gerrit.wikimedia.org/r/962718

gerritbot added a comment.Oct 2 2023, 10:26 PM

Change 962719 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Drop lgeacy main evaluator

https://gerrit.wikimedia.org/r/962719

Jdforrester-WMF closed subtask T343459: Split the wikifunctions k8s pod up in production so we have differently-scalable pods for the orchestrator vs. the evaluator as Resolved.Oct 2 2023, 10:29 PM
Jdforrester-WMF changed the task status from Open to In Progress.Oct 3 2023, 6:16 PM
Jdforrester-WMF claimed this task.
Jdforrester-WMF moved this task from Backlog to In Progress on the Abstract Wikipedia team board.
JMeybohm subscribed.Oct 4 2023, 8:17 AM

I must admit I've not read the patches in full but on first glance it seems you're removing the evaluator chart and create two new (identical?) ones that only differ by name? If that's the case then it is not required or useful to do so. We can install the evaluator chart multiple times with different configuration (e.g. different images) which I assume should be good enough to run multiple, language specific evaluators.

gerritbot added a comment.Oct 10 2023, 7:22 PM

Change 962719 abandoned by Jforrester:

[operations/deployment-charts@master] wikifunctions: Drop lgeacy main evaluator

Reason:

https://gerrit.wikimedia.org/r/962719

gerritbot added a comment.Oct 11 2023, 5:54 PM

Change 962716 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Begin split of function-evaluator into js and python services

https://gerrit.wikimedia.org/r/962716

gerritbot added a comment.Oct 11 2023, 6:01 PM

Change 965234 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Define different ports for different service releases

https://gerrit.wikimedia.org/r/965234

gerritbot added a comment.Oct 11 2023, 6:03 PM

Change 965234 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Define different ports for different service releases

https://gerrit.wikimedia.org/r/965234

gerritbot added a comment.Oct 11 2023, 6:14 PM

Change 965237 had a related patch set uploaded (by Jforrester; author: Jforrester):

[operations/deployment-charts@master] wikifunctions: Move orchestrator config from chart to service values

https://gerrit.wikimedia.org/r/965237

gerritbot added a comment.Oct 11 2023, 6:18 PM

Change 965237 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Move orchestrator config from chart to service values

https://gerrit.wikimedia.org/r/965237

gerritbot added a comment.Oct 11 2023, 6:28 PM

Change 962717 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Switch execution from main to language-specific evaluators

https://gerrit.wikimedia.org/r/962717

gerritbot added a comment.Oct 11 2023, 6:35 PM

Change 962718 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Drop legacy main (all languages) evaluator

https://gerrit.wikimedia.org/r/962718

Jdforrester-WMF moved this task from In Progress to Verify in production on the Abstract Wikipedia team board.Oct 11 2023, 6:35 PM

Deployed.

Maintenance_bot removed a project: Patch-For-Review.Oct 11 2023, 7:10 PM
Jdforrester-WMF added a parent task: T343458: Allow logged-out users to run (community-approved) implementations on wikifunctions.org via the `wikilambda-execute` right.Oct 12 2023, 4:34 PM
Jdforrester-WMF closed this task as Resolved.Oct 12 2023, 5:18 PM
JMeybohm reopened this task as Open.Oct 13 2023, 9:28 AM

Copying stuff from slack so we don't forget:

Removing the release from helmfile does apparently not mean it gets actually uninstalled (as helmfile just no longer knows of it's existence). The correct way to do this would be to helmfile -e staging --selector name=main-evaluator destroy before merging https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/962718

gerritbot added a comment.Oct 13 2023, 9:30 AM

Change 965225 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Revert "wikifunctions: Drop legacy main (all languages) evaluator"

https://gerrit.wikimedia.org/r/965225

gerritbot added a project: Patch-For-Review.Oct 13 2023, 9:30 AM
gerritbot added a comment.Oct 13 2023, 11:05 AM

Change 965225 merged by jenkins-bot:

[operations/deployment-charts@master] Revert "wikifunctions: Drop legacy main (all languages) evaluator"

https://gerrit.wikimedia.org/r/965225

Maintenance_bot removed a project: Patch-For-Review.Oct 13 2023, 11:11 AM
gerritbot added a comment.Oct 13 2023, 11:12 AM

Change 965689 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Revert "Revert "wikifunctions: Drop legacy main (all languages) evaluator""

https://gerrit.wikimedia.org/r/965689

gerritbot added a project: Patch-For-Review.Oct 13 2023, 11:12 AM
JMeybohm added a comment.Oct 13 2023, 11:14 AM
In T343388#9249170, @JMeybohm wrote:

Copying stuff from slack so we don't forget:

Removing the release from helmfile does apparently not mean it gets actually uninstalled (as helmfile just no longer knows of it's existence). The correct way to do this would be to helmfile -e staging --selector name=main-evaluator destroy before merging https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/962718

Worked nicely and is now documented https://wikitech.wikimedia.org/wiki/Kubernetes/Deployments#Undeploy/delete_a_release

gerritbot added a comment.Oct 13 2023, 11:15 AM

Change 965689 merged by jenkins-bot:

[operations/deployment-charts@master] Revert "Revert "wikifunctions: Drop legacy main (all languages) evaluator""

https://gerrit.wikimedia.org/r/965689

gerritbot added a comment.Oct 13 2023, 11:15 AM

Change 965716 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] Add new version for base.helper (1.1.1)

https://gerrit.wikimedia.org/r/965716

gerritbot added a comment.Oct 13 2023, 11:15 AM

Change 965717 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] base.helper: Allow to use ClusterIP services

https://gerrit.wikimedia.org/r/965717

gerritbot added a comment.Oct 13 2023, 11:15 AM

Change 965718 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] wikifunctions: Use ClusterIP services for evaluators

https://gerrit.wikimedia.org/r/965718

JMeybohm added a comment.Oct 13 2023, 11:33 AM

I've added some patches that should make handling more evaluators easier as it removes the requirement to assign (and use) node ports for all those. As the evaluators will only ever be accessed by the cluster local orchestrator, it's safe to use ClusterIP services and the same TCP port for all evaluators.

gerritbot added a comment.Oct 16 2023, 2:57 PM

Change 966219 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/extensions/WikiLambda@master] README/etc.: Update instructions for local development to have two evaluator services

https://gerrit.wikimedia.org/r/966219

gerritbot added a comment.Oct 16 2023, 4:17 PM

Change 965716 merged by jenkins-bot:

[operations/deployment-charts@master] Add new version for base.helper (1.1.1)

https://gerrit.wikimedia.org/r/965716

gerritbot added a comment.Oct 16 2023, 4:18 PM

Change 965717 merged by jenkins-bot:

[operations/deployment-charts@master] base.helper: Allow to use ClusterIP services

https://gerrit.wikimedia.org/r/965717

gerritbot added a comment.Oct 16 2023, 6:01 PM

Change 966219 merged by jenkins-bot:

[mediawiki/extensions/WikiLambda@master] README/etc.: Update instructions for local development to have two evaluator services

https://gerrit.wikimedia.org/r/966219

ReleaseTaggerBot added a project: MW-1.42-notes (1.42.0-wmf.1; 2023-10-17).Oct 16 2023, 7:00 PM
gerritbot added a comment.Oct 18 2023, 8:28 AM

Change 965718 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Use ClusterIP services for evaluators

https://gerrit.wikimedia.org/r/965718

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2023, 8:31 AM
gerritbot added a comment.Oct 18 2023, 8:53 AM

Change 966816 had a related patch set uploaded (by JMeybohm; author: JMeybohm):

[operations/deployment-charts@master] wikifunctions: Make app and mesh port different

https://gerrit.wikimedia.org/r/966816

gerritbot added a project: Patch-For-Review.Oct 18 2023, 8:53 AM
gerritbot added a comment.Oct 18 2023, 9:09 AM

Change 966816 merged by jenkins-bot:

[operations/deployment-charts@master] wikifunctions: Make app and mesh port different

https://gerrit.wikimedia.org/r/966816

Maintenance_bot removed a project: Patch-For-Review.Oct 18 2023, 9:10 AM
JMeybohm added a comment.Oct 18 2023, 9:29 AM

I just merged and deployed the changes mentioned in my last comment. So from my end this is good to resolve.

Jdforrester-WMF closed this task as Resolved.Oct 18 2023, 1:18 PM
In T343388#9261111, @JMeybohm wrote:

I just merged and deployed the changes mentioned in my last comment. So from my end this is good to resolve.

Thanks! There's some non-prod follow-up for Beta Cluster and local dev, but I'll call this closed.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits