Page MenuHomePhabricator
Create Task
Maniphest T255672

Migrate apertium to the deployment pipeline
Closed, ResolvedPublic
Actions

Description

Intro

Apertium is soon going to be the last service residing on the scb cluster and needs to be migrated off of it as we want to shut down that cluster in both DCs. All services from that cluster have been or are at the last stages of being migrated to the kubernetes powered deployment pipeline.

After some discussions[1] and exploring a variety of options , with @KartikMistry, we 've come up to the following approach that should move this forward without too many unknowns.

Move to the pipeline, using upstream Debian packages for apertium

Essentially just rely on the work that @KartikMistry has already been doing upstream (Debian) and create a pipeline image using those debian packages. That means the following:

Pros

  • Use of the pipeline
  • easy and orchestrated deployments/rollbacks
  • No longer relying on puppet and SREs (at least to some degree) for new functionality/languages etc. When new versions of Debian packages are required, debian-backports could (and should!) be used. That decouples the work language team does from SRE.
  • No dedicated hardware/VMs

Cons/Unknown area

  • Kartik still need to upload packages to {Debian Release}-backports if Language team wants latest language pair updates. This is little time taking as backport uploads has different policies.
  • Will need to figure out if the pipeline supports this and what changes might be required.

[1] https://etherpad.wikimedia.org/p/cxserver-2020

Progress

Details

SubjectRepoBranchLines +/-
apertium: Add a TLS enabled releaseoperations/deployment-chartsmaster+17 -4
Add apertium namespaceoperations/deployment-chartsmaster+20 -1
Add tokens and users for 3 new k8s servicesoperations/puppetproduction+36 -0
Add tokens and users for 3 new k8s serviceslabs/privatemaster+24 -0
Add apertium helmfile.doperations/deployment-chartsmaster+89 -0
Add apertium helm chartoperations/deployment-chartsmaster+361 -0
Add blubber.yamlmediawiki/services/apertiummaster+59 -0
apertium: Add it to the pipelineintegration/configmaster+6 -0
Customize query in gerrit

Related Objects
Search...

Event Timeline

akosiaris created this task.Jun 17 2020, 1:18 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJun 17 2020, 1:18 PM
akosiaris triaged this task as Medium priority.Jun 17 2020, 1:18 PM
akosiaris mentioned this in T198901: Migrate production services to kubernetes using the pipeline.
akosiaris added a parent task: T198901: Migrate production services to kubernetes using the pipeline.
Jdforrester-WMF awarded a token.Jun 17 2020, 1:37 PM
Jdforrester-WMF subscribed.
KartikMistry added a project: Language-Team (Language-2020-Focus-Sprint).Jun 18 2020, 1:10 PM
KartikMistry updated the task description. (Show Details)Jun 22 2020, 5:41 AM
KartikMistry added a comment.Jul 8 2020, 6:53 AM

Some relevant updates:

I've fixed Apertium packages for few bugs in Debian with coordination of upstream and packages are stable condition now.

I'll push packages to Debian Backport repository as needed, when we need release of new language pair for WMF.

KartikMistry moved this task from Priority Backlog to In Progress on the Language-Team (Language-2020-Focus-Sprint) board.Jul 9 2020, 1:04 PM
• Rileych assigned this task to KartikMistry.Jul 16 2020, 1:11 PM
• Rileych moved this task from Backlog to Machine Translation on the CX-cxserver board.Jul 20 2020, 2:36 PM
Pginer-WMF edited projects, added Language-Team (Language-2020-July-September); removed Language-Team (Language-2020-Focus-Sprint).Aug 3 2020, 9:32 AM
Pginer-WMF moved this task from Quarter Backlog to In Progress on the Language-Team (Language-2020-July-September) board.
jijiki moved this task from Incoming 🐫 to 🔦Unused2 on the serviceops board.Aug 17 2020, 11:45 PM
KartikMistry moved this task from In Progress to Quarter Backlog on the Language-Team (Language-2020-July-September) board.Aug 25 2020, 8:32 AM
Pginer-WMF edited projects, added Language-Team (Language-2020-October-December); removed Language-Team (Language-2020-July-September).Oct 2 2020, 3:26 PM
Pginer-WMF moved this task from Quarter Backlog to Priority Backlog on the Language-Team (Language-2020-October-December) board.Oct 6 2020, 9:13 AM
KartikMistry moved this task from Priority Backlog to In Progress on the Language-Team (Language-2020-October-December) board.Oct 13 2020, 1:10 PM
KartikMistry added a comment.Oct 13 2020, 1:27 PM

Update: We've https://gerrit.wikimedia.org/r/admin/repos/mediawiki/services/apertium repository.

gerritbot added a comment.Nov 4 2020, 2:35 PM

Change 639161 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[mediawiki/services/apertium@master] WIP: Add blubber.yaml

https://gerrit.wikimedia.org/r/639161

gerritbot added a project: Patch-For-Review.Nov 4 2020, 2:35 PM
gerritbot added a comment.Dec 1 2020, 10:56 AM

Change 644479 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[integration/config@master] apertium: Add it to the pipeline

https://gerrit.wikimedia.org/r/644479

gerritbot added a comment.Dec 1 2020, 10:59 AM

Change 644479 merged by jenkins-bot:
[integration/config@master] apertium: Add it to the pipeline

https://gerrit.wikimedia.org/r/644479

gerritbot added a comment.Dec 1 2020, 2:01 PM

Change 639161 merged by jenkins-bot:
[mediawiki/services/apertium@master] Add blubber.yaml

https://gerrit.wikimedia.org/r/639161

Maintenance_bot removed a project: Patch-For-Review.Dec 1 2020, 2:11 PM
gerritbot added a comment.Dec 1 2020, 2:27 PM

Change 644530 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Add apertium namespace

https://gerrit.wikimedia.org/r/644530

gerritbot added a project: Patch-For-Review.Dec 1 2020, 2:27 PM
gerritbot added a comment.Dec 1 2020, 2:29 PM

Change 644531 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[operations/deployment-charts@master] WIP: Add apertium helm chart

https://gerrit.wikimedia.org/r/644531

KartikMistry updated the task description. (Show Details)Dec 2 2020, 3:50 AM
KartikMistry updated the task description. (Show Details)Dec 2 2020, 3:58 AM
gerritbot added a comment.Dec 3 2020, 12:07 PM

Change 644531 merged by jenkins-bot:
[operations/deployment-charts@master] Add apertium helm chart

https://gerrit.wikimedia.org/r/644531

gerritbot added a comment.Dec 3 2020, 12:19 PM

Change 645064 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Add tokens and users for 3 new k8s services

https://gerrit.wikimedia.org/r/645064

KartikMistry updated the task description. (Show Details)Dec 3 2020, 12:27 PM
KartikMistry updated the task description. (Show Details)Dec 3 2020, 12:30 PM
KartikMistry updated the task description. (Show Details)Dec 3 2020, 1:10 PM
KartikMistry updated the task description. (Show Details)Dec 3 2020, 1:25 PM
akosiaris updated the task description. (Show Details)Dec 3 2020, 1:26 PM
KartikMistry updated the task description. (Show Details)Dec 3 2020, 1:56 PM
gerritbot added a comment.Dec 3 2020, 1:57 PM

Change 645076 had a related patch set uploaded (by KartikMistry; owner: KartikMistry):
[operations/deployment-charts@master] Add apertium helmfile.d

https://gerrit.wikimedia.org/r/645076

KartikMistry updated the task description. (Show Details)Dec 3 2020, 1:58 PM
gerritbot added a comment.Dec 3 2020, 3:02 PM

Change 645076 merged by jenkins-bot:
[operations/deployment-charts@master] Add apertium helmfile.d

https://gerrit.wikimedia.org/r/645076

KartikMistry updated the task description. (Show Details)Dec 3 2020, 5:03 PM
gerritbot added a comment.Dec 4 2020, 11:51 AM

Change 645323 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[labs/private@master] Add tokens and users for 3 new k8s services

https://gerrit.wikimedia.org/r/645323

gerritbot added a comment.Dec 4 2020, 11:52 AM

Change 645323 merged by Alexandros Kosiaris:
[labs/private@master] Add tokens and users for 3 new k8s services

https://gerrit.wikimedia.org/r/645323

akosiaris mentioned this in rLPRIe7036a5756ee: Add tokens and users for 3 new k8s services.Dec 4 2020, 11:56 AM
gerritbot added a comment.Dec 4 2020, 12:00 PM

Change 645064 merged by Alexandros Kosiaris:
[operations/puppet@production] Add tokens and users for 3 new k8s services

https://gerrit.wikimedia.org/r/645064

gerritbot added a comment.Dec 4 2020, 12:48 PM

Change 644530 merged by jenkins-bot:
[operations/deployment-charts@master] Add apertium namespace

https://gerrit.wikimedia.org/r/644530

Stashbot added a comment.Dec 4 2020, 1:07 PM

Mentioned in SAL (#wikimedia-operations) [2020-12-04T13:07:37Z] <akosiaris> create apertium namespace on k8s clusters. T255672

KartikMistry updated the task description. (Show Details)Dec 6 2020, 12:32 PM
KartikMistry updated the task description. (Show Details)Dec 7 2020, 11:58 AM
Stashbot added a comment.Dec 7 2020, 1:38 PM

Mentioned in SAL (#wikimedia-operations) [2020-12-07T13:38:24Z] <kart_> Deployed apertium service to eqiad and codfw (T255672)

KartikMistry updated the task description. (Show Details)Dec 7 2020, 1:39 PM
gerritbot added a comment.Dec 7 2020, 2:38 PM

Change 646688 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] apertium: Add a TLS enabled release

https://gerrit.wikimedia.org/r/646688

gerritbot added a comment.Dec 8 2020, 11:37 AM

Change 646688 merged by jenkins-bot:
[operations/deployment-charts@master] apertium: Add a TLS enabled release

https://gerrit.wikimedia.org/r/646688

KartikMistry updated the task description. (Show Details)Dec 8 2020, 12:13 PM
KartikMistry updated the task description. (Show Details)Dec 8 2020, 2:23 PM
KartikMistry moved this task from In Progress to In Review on the Language-Team (Language-2020-October-December) board.Dec 9 2020, 6:33 AM
KartikMistry updated the task description. (Show Details)Dec 9 2020, 2:35 PM

All done except announcement/notes to the team!

KartikMistry updated the task description. (Show Details)Dec 10 2020, 5:30 AM
KartikMistry moved this task from In Review to Done on the Language-Team (Language-2020-October-December) board.
Pginer-WMF closed this task as Resolved.Dec 18 2020, 11:17 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL