Page MenuHomePhabricator

Deploy push-notifications service to Kubernetes
Closed, ResolvedPublic

Description

Checklist to run push-notifications service under Kubernetes

Details

ProjectBranchLines +/-Subject
operations/deployment-chartsmaster+1 -1
operations/deployment-chartsmaster+1 -1
operations/puppetproduction+4 -0
operations/dnsmaster+2 -0
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -1
operations/puppetproduction+43 -0
operations/dnsmaster+4 -2
operations/deployment-chartsmaster+2 -2
operations/deployment-chartsmaster+1 -1
operations/deployment-chartsmaster+7 -1
operations/puppetproduction+1 -1
operations/puppetproduction+1 -0
operations/deployment-chartsmaster+1 -1
operations/deployment-chartsmaster+2 -2
operations/deployment-chartsmaster+76 -0
operations/puppetproduction+17 -0
operations/deployment-chartsmaster+18 -0
labs/privatemaster+8 -1
Show related patches Customize query in gerrit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
akosiaris triaged this task as Medium priority.
akosiaris updated the task description. (Show Details)
jijiki updated the task description. (Show Details)
akosiaris updated the task description. (Show Details)

Change 613097 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] Create namespaces/calico rules for push-notifications

https://gerrit.wikimedia.org/r/613097

@MSantos Is there an internal service/database push-notifications will be communicating with? If so, let us know so we can add the relevant calico rules

Change 613101 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[labs/private@master] Add k8s dummy tokens for push-notifications

https://gerrit.wikimedia.org/r/613101

Change 613104 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Kubernetes: Create token stanzas for push-notifications

https://gerrit.wikimedia.org/r/613104

@MSantos Is there an internal service/database push-notifications will be communicating with? If so, let us know so we can add the relevant calico rules

@jijiki thanks for pushing this forward. Push Notifications as it is planned for this version has no communication with other internal services/databases.

Change 613101 merged by Effie Mouzeli:
[labs/private@master] Add k8s dummy tokens for push-notifications

https://gerrit.wikimedia.org/r/613101

Change 613104 merged by Effie Mouzeli:
[operations/puppet@production] Kubernetes: Create token stanzas for push-notifications

https://gerrit.wikimedia.org/r/613104

Change 613097 merged by Effie Mouzeli:
[operations/deployment-charts@master] Create namespaces rules for push-notifications

https://gerrit.wikimedia.org/r/613097

Change 621605 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] helmfile: add values for staging environment

https://gerrit.wikimedia.org/r/621605

Change 621605 merged by jenkins-bot:
[operations/deployment-charts@master] helmfile: add values for staging environment

https://gerrit.wikimedia.org/r/621605

Change 622330 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: enable TLS for all environments

https://gerrit.wikimedia.org/r/622330

Change 622330 merged by jenkins-bot:
[operations/deployment-charts@master] push-notifications: enable TLS for all environments

https://gerrit.wikimedia.org/r/622330

Change 622333 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: Bump chart version

https://gerrit.wikimedia.org/r/622333

Change 622333 merged by JMeybohm:
[operations/deployment-charts@master] push-notifications: Bump chart version

https://gerrit.wikimedia.org/r/622333

Push-notifications is up and running in staging. Our next step is to perform the LVS steps and expose the application.

@JMeybohm and I are planning to do so this week, so it will be ready for you when you deploy to production.

Change 623541 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/dns@master] Add entries for push-notifications

https://gerrit.wikimedia.org/r/623541

Change 623544 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/dns@master] Add discovery records for push-notifications

https://gerrit.wikimedia.org/r/623544

Change 623631 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] lvs::configuration: add push-notifications patch 1/3

https://gerrit.wikimedia.org/r/623631

Change 623632 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] lvs::configuration: add push-notifications patch 2/3

https://gerrit.wikimedia.org/r/623632

Change 623634 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] lvs::configuration: add push-notifications patch 3/3

https://gerrit.wikimedia.org/r/623634

Change 623773 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] lvs::configuration: add push-notifications patch 3/4

https://gerrit.wikimedia.org/r/623773

Change 623774 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] conftool: add data for dns discovery for push-notifications

https://gerrit.wikimedia.org/r/623774

Change 623790 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] services_proxy: add push-notifications

https://gerrit.wikimedia.org/r/623790

Change 623774 abandoned by Effie Mouzeli:
[operations/puppet@production] conftool: add data for dns discovery for push-notifications

Reason:
Included in 623631

https://gerrit.wikimedia.org/r/623774

Change 624014 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] lvs::configuration: add push-notifications patch 3/4

https://gerrit.wikimedia.org/r/624014

Change 623634 abandoned by Effie Mouzeli:
[operations/puppet@production] lvs::configuration: add push-notifications patch 3/4

Reason:
rebase hell as always

https://gerrit.wikimedia.org/r/623634

Change 625709 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: add proxy settings

https://gerrit.wikimedia.org/r/625709

Change 625709 merged by jenkins-bot:
[operations/deployment-charts@master] push-notifications: add proxy settings

https://gerrit.wikimedia.org/r/625709

@MSantos please let us know when you are ready to go on production, so we can perform the final steps. Thank you!

MSantos closed subtask Restricted Task as Resolved.Sep 15 2020, 5:12 PM

Change 627846 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: enable mmonitoring

https://gerrit.wikimedia.org/r/627846

Change 627846 merged by jenkins-bot:
[operations/deployment-charts@master] push-notifications: enable monitoring

https://gerrit.wikimedia.org/r/627846

Change 628304 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: deploy to production environment

https://gerrit.wikimedia.org/r/628304

Change 628304 merged by jenkins-bot:
[operations/deployment-charts@master] push-notifications: deploy to production environment

https://gerrit.wikimedia.org/r/628304

What will be the internal URL for this service? I am guessing https://push-notifications.discovery.wmnet:4104 (port 4104 as listed at https://wikitech.wikimedia.org/wiki/Service_ports)? I'm getting a MediaWiki config patch ready for Monday and need to put this in a config variable.

Change 628336 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: enable egress

https://gerrit.wikimedia.org/r/628336

@Mholloway it will be accessible on Monday after we deploy the LVS/DNS patches. Meanwhile you could try something like curl -vk https://kubernetes1004.eqiad.wmnet:4104 (a kubernetes host+port+tls)

Change 628340 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/deployment-charts@master] push-notifications: change version tag to -production

https://gerrit.wikimedia.org/r/628340

On Monday (EU) morning, @JMeybohm and I will push the LVS/DNS patches, so everything will be ready for the mediawiki-config change. There are a couple of things we may need to tune after deployment:

  • number of replicas
  • cpu/memory of each

Please join #wikimedia-serviceops or #wikimedia-sre when you roll out to group0 so someone from serviceops can assist if needed during or after 🛫:)

Sounds good, @jijiki. I'm in both channels. Thank you!

Change 623541 merged by Effie Mouzeli:
[operations/dns@master] Add entries for push-notifications

https://gerrit.wikimedia.org/r/623541

Change 623631 merged by Effie Mouzeli:
[operations/puppet@production] lvs::configuration: add push-notifications patch 1/4

https://gerrit.wikimedia.org/r/623631

Change 623632 merged by Effie Mouzeli:
[operations/puppet@production] lvs::configuration: add push-notifications patch 2/4

https://gerrit.wikimedia.org/r/623632

Mentioned in SAL (#wikimedia-operations) [2020-09-21T11:02:35Z] <effie> restart pybal on lvs2010 and lvs1016 - T256973

Mentioned in SAL (#wikimedia-operations) [2020-09-21T11:22:29Z] <effie> restart pybal on lvs2010 and lvs1016 - T256973

Mentioned in SAL (#wikimedia-operations) [2020-09-21T11:38:40Z] <effie> restart pybal on lvs2009 and lvs1015 - T256973

Change 624014 merged by Effie Mouzeli:
[operations/puppet@production] lvs::configuration: add push-notifications patch 3/4

https://gerrit.wikimedia.org/r/624014

Change 623773 merged by Effie Mouzeli:
[operations/puppet@production] lvs::configuration: add push-notifications patch 4/4

https://gerrit.wikimedia.org/r/623773

Change 623544 merged by Effie Mouzeli:
[operations/dns@master] Add discovery records for push-notifications

https://gerrit.wikimedia.org/r/623544

@JMeybohm and I finished up, you can reach the production environments by using curl -k https://push-notifications.svc.eqiad.wmnet:4104 or curl -k https://push-notifications.svc.codfw.wmnet:4104

Good luck! 🎉

Looking at other service definitions in mediawiki-config ProductionServices.php, it looks like we generally want to be referencing services from MediaWiki in a DC-agnostic way. Do we need a service proxy set up for push-notifications?

Looking at other service definitions in mediawiki-config ProductionServices.php, it looks like we generally want to be referencing services from MediaWiki in a DC-agnostic way. Do we need a service proxy set up for push-notifications?

yes, I guess it will be added by us soon.

Change 623790 merged by Effie Mouzeli:
[operations/puppet@production] services_proxy: add push-notifications

https://gerrit.wikimedia.org/r/623790

Looking at other service definitions in mediawiki-config ProductionServices.php, it looks like we generally want to be referencing services from MediaWiki in a DC-agnostic way. Do we need a service proxy set up for push-notifications?

"http://localhost:6012" should become available now on the appservers (with next puppet run).

jijiki updated the task description. (Show Details)

@Mholloway I am marking this as resolved unless we believe there is a reason not to.

Change 628336 abandoned by Effie Mouzeli:
[operations/deployment-charts@master] push-notifications: enable egress

Reason:
It appears that this is not needed for now

https://gerrit.wikimedia.org/r/628336

Change 656129 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add deployment config for a new k8s service - eventstreams-internal

https://gerrit.wikimedia.org/r/656129