Page MenuHomePhabricator
Create Task
Maniphest T225680

Migrate Proton to k8s
Closed, ResolvedPublic
Actions

Description

  • Chart created, reviewed and merged
  • proton namespaces on k8s created
  • proton tokens for k8s created
  • calico rules created, if applicable
  • helmfile.d stanzas created
  • deploy
  • Switchover traffic from proton hosts to kubernetes

Details

Related Changes in Gerrit:
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

Jdforrester-WMF created this task.Jun 12 2019, 11:03 PM
Restricted Application added a project: Product-Infrastructure-Team-Backlog-Deprecated. · View Herald TranscriptJun 12 2019, 11:03 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
gerritbot added a comment.Jun 12 2019, 11:03 PM

Change 516710 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[mediawiki/services/chromium-render@master] build: Create initial pipeline configuration

https://gerrit.wikimedia.org/r/516710

gerritbot added a project: Patch-For-Review.Jun 12 2019, 11:03 PM
Jdforrester-WMF mentioned this in rMSCRc14056a033a1: build: Create initial pipeline configuration.Jun 13 2019, 3:08 AM
• mobrovac renamed this task from Migrate chromium-render to k8s to Migrate Proton to k8s.Jun 13 2019, 7:23 AM
• mobrovac added projects: Services (next), Platform Team Legacy (Next).
LGoto triaged this task as Medium priority.Jun 20 2019, 3:47 PM
LGoto moved this task from Needs triage to Upcoming on the Product-Infrastructure-Team-Backlog-Deprecated board.
LGoto added subscribers: • Mholloway, LGoto.

@Mholloway Can you update this ticket with a description?

Jdforrester-WMF mentioned this in rMSCR8a14e7be0316: build: Create initial pipeline configuration.Jul 2 2019, 7:52 PM
gerritbot added a comment.Jul 2 2019, 8:23 PM

Change 516710 merged by jenkins-bot:
[mediawiki/services/chromium-render@master] build: Create initial pipeline configuration

https://gerrit.wikimedia.org/r/516710

gerritbot added a comment.Jul 2 2019, 8:45 PM

Change 520317 had a related patch set uploaded (by Jforrester; owner: Jforrester):
[integration/config@master] layout: [chromium-render] Enable pipeline tests

https://gerrit.wikimedia.org/r/520317

• Pchelolo edited projects, added Platform Team Legacy (Watching / External); removed Platform Team Legacy (Next), Services (next).Jul 16 2019, 8:25 PM
MSantos moved this task from Upcoming to Tracking on the Product-Infrastructure-Team-Backlog-Deprecated board.Aug 13 2019, 7:31 PM
thcipriani moved this task from Backlog to Migration on the Release Pipeline board.Aug 20 2019, 1:28 PM
gerritbot added a comment.Sep 3 2019, 9:01 PM

Change 520317 abandoned by Jforrester:
layout: [chromium-render] Enable pipeline tests

Reason:
Done elsewise.

https://gerrit.wikimedia.org/r/520317

Maintenance_bot removed a project: Patch-For-Review.Sep 3 2019, 9:11 PM
MSantos moved this task from Tracking to Kanban on the Product-Infrastructure-Team-Backlog-Deprecated board.Nov 21 2019, 2:08 PM
MSantos edited projects, added Product-Infrastructure-Team-Backlog-Deprecated (Kanban); removed Product-Infrastructure-Team-Backlog-Deprecated.
MSantos raised the priority of this task from Medium to High.Nov 21 2019, 2:17 PM
MSantos mentioned this in T238830: Profile proton memory usage for Helm chart.
MSantos added a subtask: T238830: Profile proton memory usage for Helm chart.Nov 21 2019, 2:25 PM
MSantos added a parent task: Restricted Task.Dec 12 2019, 5:39 PM
• Mholloway assigned this task to MSantos.Feb 5 2020, 3:52 PM
LGoto edited projects, added Product-Infrastructure-Team-Backlog-Deprecated, Epic; removed Product-Infrastructure-Team-Backlog-Deprecated (Kanban).Feb 5 2020, 4:51 PM
LGoto moved this task from Needs triage to Epics on the Product-Infrastructure-Team-Backlog-Deprecated board.
akosiaris subscribed.Mar 5 2020, 4:59 PM
This comment was removed by akosiaris.
akosiaris closed subtask T238830: Profile proton memory usage for Helm chart as Resolved.Mar 6 2020, 12:22 PM
akosiaris claimed this task.Mar 6 2020, 12:25 PM
akosiaris updated the task description. (Show Details)
akosiaris added a subscriber: MSantos.
gerritbot added a comment.Mar 17 2020, 10:24 AM

Change 580294 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[labs/private@master] Add k8s dummy tokens for 3 new services.

https://gerrit.wikimedia.org/r/580294

gerritbot added a project: Patch-For-Review.Mar 17 2020, 10:24 AM
gerritbot added a comment.Mar 17 2020, 10:30 AM

Change 580295 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] Kubernetes: Create token stanzas for some new services

https://gerrit.wikimedia.org/r/580295

gerritbot added a comment.May 29 2020, 12:10 PM

Change 599812 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] Create namespaces/calico rules for new services

https://gerrit.wikimedia.org/r/599812

gerritbot added a comment.May 29 2020, 1:59 PM

Change 580294 merged by Alexandros Kosiaris:
[labs/private@master] Add k8s dummy tokens for 3 new services.

https://gerrit.wikimedia.org/r/580294

akosiaris mentioned this in rLPRI2078705f3c40: Add k8s dummy tokens for 3 new services..May 29 2020, 2:01 PM
gerritbot added a comment.May 29 2020, 2:09 PM

Change 580295 merged by Alexandros Kosiaris:
[operations/puppet@production] Kubernetes: Create token stanzas for some new services

https://gerrit.wikimedia.org/r/580295

gerritbot added a comment.May 29 2020, 2:20 PM

Change 599812 merged by jenkins-bot:
[operations/deployment-charts@master] Create namespaces/calico rules for new services

https://gerrit.wikimedia.org/r/599812

Maintenance_bot removed a project: Patch-For-Review.May 29 2020, 3:10 PM
akosiaris updated the task description. (Show Details)Jun 2 2020, 12:06 PM

@Mholloway, namespaces, rules, tokens have been created. Chart has been merged and publish. You are free to deploy. You will require a change like 968132909b4d24192b2f69a657c14bb30acd7a42 in order to instantiate the first deploy, feel free to add me as a reviewer.

gerritbot added a comment.Jun 3 2020, 9:07 PM

Change 602164 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/deployment-charts@master] Chromium-render: Add initial helmfile stanzas

https://gerrit.wikimedia.org/r/602164

gerritbot added a project: Patch-For-Review.Jun 3 2020, 9:07 PM
gerritbot added a comment.Jun 8 2020, 8:32 PM

Change 602164 merged by jenkins-bot:
[operations/deployment-charts@master] Chromium-render: Add initial helmfile stanzas

https://gerrit.wikimedia.org/r/602164

• Mholloway added a comment.EditedJun 8 2020, 9:08 PM

Hmm, I've merged https://gerrit.wikimedia.org/r/602164 in preparation to deploy the service, but it doesn't look like Puppet is going to create the .hfenv files and private/ subdirectories in the per-cluster service directories as I would expect. Something must be wrong.

Maintenance_bot removed a project: Patch-For-Review.Jun 8 2020, 9:10 PM
• Mholloway updated the task description. (Show Details)Jun 9 2020, 2:02 PM
gerritbot added a comment.Jun 9 2020, 3:12 PM

Change 604046 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[operations/deployment-charts@master] Proton: Fix helmfile paths and namespace refs

https://gerrit.wikimedia.org/r/604046

gerritbot added a project: Patch-For-Review.Jun 9 2020, 3:12 PM
gerritbot added a comment.Jun 9 2020, 3:14 PM

Change 604046 merged by jenkins-bot:
[operations/deployment-charts@master] Proton: Fix helmfile paths and namespace refs

https://gerrit.wikimedia.org/r/604046

Maintenance_bot removed a project: Patch-For-Review.Jun 9 2020, 4:11 PM
• Mholloway added a comment.Jun 9 2020, 5:14 PM

@akosiaris I'm still running into a problem deploying this. It seems that private/secrets.yaml is still missing:

mholloway-shell@deploy1001:/srv/deployment-charts/helmfile.d/services/staging/proton$ source .hfenv; helmfile diff
Adding repo stable https://releases.wikimedia.org/charts/
"stable" has been added to your repositories

Updating repo
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.

skipping missing values file matching "private/secrets.yaml"
Comparing release=production, chart=stable/chromium-render
in ./helmfile.yaml: helm exited with status 1:
  Error: Get http://localhost:8080/api/v1/namespaces/proton/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: dial tcp [::1]:8080: connect: connection refused

The same thing happens for mobileapps.

akosiaris added a comment.Jun 10 2020, 10:38 AM
In T225680#6207008, @Mholloway wrote:

@akosiaris I'm still running into a problem deploying this. It seems that private/secrets.yaml is still missing:

That's actually a warning and not a failure. We can silence it of course, but it's not the reason for the issue below

mholloway-shell@deploy1001:/srv/deployment-charts/helmfile.d/services/staging/proton$ source .hfenv; helmfile diff
Adding repo stable https://releases.wikimedia.org/charts/
"stable" has been added to your repositories

Updating repo
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "stable" chart repository
Update Complete.

skipping missing values file matching "private/secrets.yaml"
Comparing release=production, chart=stable/chromium-render
in ./helmfile.yaml: helm exited with status 1:
  Error: Get http://localhost:8080/api/v1/namespaces/proton/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: dial tcp [::1]:8080: connect: connection refused

Now that's my mistake. Fixed in the puppet/private repo.

The same thing happens for mobileapps.

Same puppet/private repo fixed that as well.

• Mholloway updated the task description. (Show Details)Jun 11 2020, 6:44 PM
gerritbot added a comment.Jun 24 2020, 3:00 PM

Change 607531 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] lvs: Add new proton TLS service

https://gerrit.wikimedia.org/r/607531

gerritbot added a project: Patch-For-Review.Jun 24 2020, 3:00 PM

Change 607532 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] lvs: Switch proton to lvs_setup

https://gerrit.wikimedia.org/r/607532

gerritbot added a comment.Jun 24 2020, 3:00 PM

Change 607533 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] lvs: Switch proton to monitoring_setup

https://gerrit.wikimedia.org/r/607533

gerritbot added a comment.Jun 24 2020, 3:00 PM

Change 607534 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] lvs: Switch proton to production

https://gerrit.wikimedia.org/r/607534

gerritbot added a comment.Jun 24 2020, 3:00 PM

Change 607535 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] proton: Switch dev restbase to talk to TLS proton

https://gerrit.wikimedia.org/r/607535

gerritbot added a comment.Jun 24 2020, 3:00 PM

Change 607536 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] proton: Switch restbase production to TLS

https://gerrit.wikimedia.org/r/607536

gerritbot added a comment.Jun 29 2020, 2:32 PM

Change 607531 merged by Alexandros Kosiaris:
[operations/puppet@production] lvs: Add new proton TLS service

https://gerrit.wikimedia.org/r/c/operations/puppet/ /607531

gerritbot added a comment.Jun 30 2020, 11:38 AM

Change 607532 merged by Alexandros Kosiaris:
[operations/puppet@production] lvs: Switch proton to lvs_setup

https://gerrit.wikimedia.org/r/c/operations/puppet/ /607532

gerritbot added a comment.Jun 30 2020, 12:09 PM

Change 607533 merged by Alexandros Kosiaris:
[operations/puppet@production] lvs: Switch proton to monitoring_setup

https://gerrit.wikimedia.org/r/c/operations/puppet/ /607533

gerritbot added a comment.Jun 30 2020, 12:49 PM

Change 607534 merged by Alexandros Kosiaris:
[operations/puppet@production] lvs: Switch proton to production

https://gerrit.wikimedia.org/r/c/operations/puppet/ /607534

gerritbot added a comment.Jul 1 2020, 10:09 AM

Change 607535 merged by Alexandros Kosiaris:
[operations/puppet@production] proton: Switch dev restbase to talk to TLS proton

https://gerrit.wikimedia.org/r/c/operations/puppet/ /607535

JMeybohm mentioned this in T255877: Move proton to use TLS only.Jul 7 2020, 7:36 AM
gerritbot added a comment.Jul 9 2020, 9:18 AM

Change 607536 abandoned by Alexandros Kosiaris:
[operations/puppet@production] proton: Switch restbase production to TLS

Reason:
Done differently in https://gerrit.wikimedia.org/r/c/operations/puppet/ /610720

https://gerrit.wikimedia.org/r/607536

gerritbot added a comment.Jul 9 2020, 12:23 PM

Change 610789 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/puppet@production] proton: Set LVS level OpenAPI checks on TLS

https://gerrit.wikimedia.org/r/610789

gerritbot added a comment.Jul 9 2020, 12:27 PM

Change 610789 merged by Alexandros Kosiaris:
[operations/puppet@production] proton: Set LVS level OpenAPI checks on TLS

https://gerrit.wikimedia.org/r/610789

gerritbot added a comment.Jul 9 2020, 3:24 PM

Change 610855 had a related patch set uploaded (by Alexandros Kosiaris; owner: Alexandros Kosiaris):
[operations/deployment-charts@master] proton: Amend prometheus-statsd config

https://gerrit.wikimedia.org/r/610855

gerritbot added a comment.Jul 10 2020, 7:09 AM

Change 610855 merged by jenkins-bot:
[operations/deployment-charts@master] proton: Amend prometheus-statsd config

https://gerrit.wikimedia.org/r/610855

akosiaris closed this task as Resolved.Jul 10 2020, 7:25 AM
akosiaris updated the task description. (Show Details)

Resolving. This has been completed quite successfully. New dashboard is at https://grafana.wikimedia.org/d/llIEd7MMz/proton. I 'll archive the old one. \o/

MSantos awarded a token.Jul 13 2020, 12:47 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits