- Add a .pipeline/blubber.yaml to the repo
- Enable the pipeline for that repo. A good example would be 07351ff6bc8252cf4876298b91126
- Get an image and create the helm chart. https://gerrit.wikimedia.org/g/operations/deployment-charts has a README.md for how to that takes most of the complexity away (do read the generated files though, you probably want to change some stuff)
- Get the chart reviewed and merged.
- Ask ServiceOps SRE to create the kubernetes namespaces/tokens/network policies to allow deploying the service
- Deploy
- Switch traffic from the old instance to the pipeline one. That last part is again SRE actions only (but should be done in cooperation with research).
Description
Description
Details
Details
Related Changes in Gerrit:
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | akosiaris | T198901 Migrate production services to kubernetes using the pipeline | |||
| Resolved | Jdforrester-WMF | T248355 Archive mediawiki/services/recommendation-api/deploy once it's no longer used | |||
| Resolved | akosiaris | T241230 Migrate recommendation-api to kubernetes |
Event Timeline
Comment Actions
@Pchelolo Hey, any documentation on how to move the service to node-js 10?
Release-Engineering-Team any documentation on how to move the service to the deployment pipeline?
Comment Actions
Change 560454 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/puppet@production] Recommendation API: upgrade node to version 10
Comment Actions
Change 563185 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/docker-images/production-images@master] nodejs10: Add buster image
Comment Actions
Change 563185 merged by Alexandros Kosiaris:
[operations/docker-images/production-images@master] nodejs10: Add buster image
Comment Actions
Images for buster and nodejs10 have been been created and are present in the registry
Comment Actions
@bmansurov, off the top of my head (these belong in a page under wikitech, will get to that)
- Add a .pipeline/blubber.yaml to the repo
- Enable the pipeline for that repo. A good example would be 07351ff6bc8252cf4876298b91126
- Get an image and create the helm chart. https://gerrit.wikimedia.org/g/operations/deployment-charts has a README.md for how to that takes most of the complexity away (do read the generated files though, you probably want to change some stuff)
- Get the chart reviewed and merged.
- Ask ServiceOps SRE to create the kubernetes namespaces/tokens/network policies to allow deploying the service
- Deploy
- Switch traffic from the old instance to the pipeline one. That last part is again SRE actions only (but should be done in cooperation with research).
Comment Actions
Change 565788 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/deployment-charts@master] Add recommendation-api chart
Comment Actions
@akosiaris thanks! The first two points were already done. I've created a chart and uploaded a patch. Would you please review it.
Also, do I need to create a separate ticket to ask "ServiceOps SRE to create the kubernetes namespaces/tokens/network"? Or is this ticket good for that purpose too?
Comment Actions
Done.
Also, do I need to create a separate ticket to ask "ServiceOps SRE to create the kubernetes namespaces/tokens/network"? Or is this ticket good for that purpose too?
Yes please do create one.
Comment Actions
Change 577627 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/recommendation-api@master] Blubber: Use nodejs 10 versions of nodejs base images
Comment Actions
Change 577627 merged by jenkins-bot:
[mediawiki/services/recommendation-api@master] Blubber: Use nodejs 10 versions of nodejs base images
Comment Actions
Change 580294 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[labs/private@master] Add k8s dummy tokens for 3 new services.
Comment Actions
Change 580295 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Kubernetes: Create token stanzas for some new services
Comment Actions
@akosiaris could you please re-review this patch: https://gerrit.wikimedia.org/r/#/c/operations/deployment-charts/+/565788/
I'd also appreciate it if you could review/merge this: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/560454/
Thank you.
Comment Actions
Change 565788 merged by jenkins-bot:
[operations/deployment-charts@master] Add recommendation-api chart
Comment Actions
Change 599812 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Create namespaces/calico rules for new services
Comment Actions
Change 580294 merged by Alexandros Kosiaris:
[labs/private@master] Add k8s dummy tokens for 3 new services.
Comment Actions
Change 580295 merged by Alexandros Kosiaris:
[operations/puppet@production] Kubernetes: Create token stanzas for some new services
Comment Actions
Change 599812 merged by jenkins-bot:
[operations/deployment-charts@master] Create namespaces/calico rules for new services
Comment Actions
@bmansurov namespaces, rules, tokens have been created. Chart has been merged and publish. You are free to deploy. You will require a change like 968132909b4d24192b2f69a657c14bb30acd7a42 in order to instantiate the first deploy, feel free to add me as a reviewer.
Comment Actions
Change 602527 had a related patch set uploaded (by Bmansurov; owner: Bmansurov):
[operations/deployment-charts@master] Add recommendation-api helmfile stanzas
Comment Actions
PI will want to track this since Recommendation-API contains a couple of endpoints we maintain and the apps consume.
Comment Actions
hi, is that still worked on? Asking cause CI still has to maintain a Jessie based image / NodeJS 6.
Comment Actions
Change 560454 abandoned by Hashar:
[operations/puppet@production] Recommendation API: upgrade node to version 10
Reason:
Per Alexandros, the only path is to migrate to a container / k8s. T241230
Comment Actions
@akosiaris Thanks for reviewing my patchsets. I was wondering if you've seen my last comment. My understanding is that the patch is very close to being merged, which would be great. Could you please take a look at my benchmark numbers and let me know what you think? Thanks. Here's the patch: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/602527
Comment Actions
Change 641439 had a related patch set uploaded (by Alexandros Kosiaris; owner: Bmansurov):
[operations/deployment-charts@master] recommendation-api: Supply more configuration
Comment Actions
Change 641439 merged by jenkins-bot:
[operations/deployment-charts@master] recommendation-api: Supply more configuration
Comment Actions
Change 602527 merged by jenkins-bot:
[operations/deployment-charts@master] Add recommendation-api helmfile stanzas
Comment Actions
Change 641713 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] recommendation-api: Open up access to 3306
Comment Actions
Change 641713 merged by jenkins-bot:
[operations/deployment-charts@master] recommendation-api: Open up access to 3306
Comment Actions
Change 641742 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] recommendation-api: Allow overriding mysql_tables
Comment Actions
Change 641742 merged by jenkins-bot:
[operations/deployment-charts@master] recommendation-api: Allow overriding mysql_tables
Comment Actions
Change 641749 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] conftool: Add recommendation-api to kubernetes nodes
Comment Actions
Change 641750 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] lvs: Add new TLS enabled recommendation-api service
Comment Actions
Change 641753 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] recommendation-api: Switch to using envoy based discovery
Comment Actions
Change 641749 merged by Alexandros Kosiaris:
[operations/puppet@production] conftool: Add recommendation-api to kubernetes nodes
Comment Actions
Change 641753 merged by jenkins-bot:
[operations/deployment-charts@master] recommendation-api: Switch to using envoy based discovery
Comment Actions
Mentioned in SAL (#wikimedia-operations) [2020-11-18T17:13:56Z] <akosiaris> T241230 pool codfw kubernetes for recommendation-api at a very low weight
Comment Actions
Change 641750 merged by Alexandros Kosiaris:
[operations/puppet@production] lvs: Add new TLS enabled recommendation-api service
Comment Actions
The service has been deployed yesterday, and the traffic switch happened today. Per https://grafana.wikimedia.org/d/Y5wk80oGk/recommendation-api?orgId=1&var-dc=thanos&var-site=eqiad&var-service=recommendation-api&var-prometheus=k8s&var-container_name=All&from=now-3h&to=now traffic (alas there is no corresponding dashboard for the legacy infrastructure) is flowing now to the kubernetes based deployment. There is some cleanup work to happen, but otherwise this is done. I am gonna resolve it successfully, but feel free to reopen. Thanks to @bmansurov for working through getting the container created and the helm chart ready.
Comment Actions
Change 641937 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] lvs: Switch old nontls recommendation-api to lvs_setup
Comment Actions
Change 641938 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] recommendation-api: Switch to service_setup
Comment Actions
Change 641939 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] recommendation-api: Cleanups
Comment Actions
Change 641937 merged by Alexandros Kosiaris:
[operations/puppet@production] lvs: Switch old nontls recommendation-api to lvs_setup
Comment Actions
Change 641938 merged by Alexandros Kosiaris:
[operations/puppet@production] recommendation-api: Switch to service_setup
Comment Actions
Change 641939 merged by Alexandros Kosiaris:
[operations/puppet@production] recommendation-api: Cleanups
Comment Actions
Changes associated with this patch (mostly 27152427487bed18321d582a7f77a301ea114968) have left the following VMs unpuppetized:
- deployment-sca01.deployment-prep.eqiad1.wikimedia.cloud
- deployment-sca02.deployment-prep.eqiad1.wikimedia.cloud
Should those VMs be deleted?
Comment Actions
I 'd say so. There is no way that role/profile is coming back so if they stayed around they would keep on being unpuppetized forever. That could potentially break some parts of restbase in beta, not sure if they are exercised or not though.