Page MenuHomePhabricator
Create Task
Maniphest T256973

Deploy push-notifications service to Kubernetes
Closed, ResolvedPublic
Actions

Description

Checklist to run push-notifications service under Kubernetes

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
push-notifications: change version tag to -productionoperations/deployment-chartsmaster+1 -1
push-notifications: enable egressoperations/deployment-chartsmaster+1 -1
services_proxy: add push-notificationsoperations/puppetproduction+4 -0
Add discovery records for push-notificationsoperations/dnsmaster+2 -0
lvs::configuration: add push-notifications patch 4/4operations/puppetproduction+1 -1
lvs::configuration: add push-notifications patch 3/4operations/puppetproduction+1 -1
lvs::configuration: add push-notifications patch 2/4operations/puppetproduction+1 -1
lvs::configuration: add push-notifications patch 1/4operations/puppetproduction+43 -0
Add entries for push-notificationsoperations/dnsmaster+4 -2
push-notifications: deploy to production environmentoperations/deployment-chartsmaster+2 -2
push-notifications: enable monitoringoperations/deployment-chartsmaster+1 -1
push-notifications: add proxy settingsoperations/deployment-chartsmaster+7 -1
lvs::configuration: add push-notifications patch 3/4operations/puppetproduction+1 -1
conftool: add data for dns discovery for push-notificationsoperations/puppetproduction+1 -0
push-notifications: Bump chart versionoperations/deployment-chartsmaster+1 -1
push-notifications: enable TLS for all environmentsoperations/deployment-chartsmaster+2 -2
helmfile: add values for staging environmentoperations/deployment-chartsmaster+76 -0
Kubernetes: Create token stanzas for push-notificationsoperations/puppetproduction+17 -0
Create namespaces rules for push-notificationsoperations/deployment-chartsmaster+18 -0
Add k8s dummy tokens for push-notificationslabs/privatemaster+8 -1
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
akosiaris awarded a token.Jul 2 2020, 3:43 PM
akosiaris assigned this task to jijiki.Jul 7 2020, 8:43 AM
akosiaris triaged this task as Medium priority.
akosiaris updated the task description. (Show Details)
akosiaris updated the task description. (Show Details)Jul 7 2020, 12:51 PM
MSantos added a parent task: T250452: New Service Request: Wikimedia push notification service.Jul 8 2020, 3:03 PM
MSantos mentioned this in T250452: New Service Request: Wikimedia push notification service.Jul 8 2020, 3:10 PM
• Mholloway subscribed.Jul 8 2020, 3:39 PM
LGoto moved this task from Needs triage to Tracking on the Product-Infrastructure-Team-Backlog-Deprecated board.Jul 8 2020, 3:39 PM
jijiki added a subtask: Restricted Task.Jul 15 2020, 9:26 AM
jijiki updated the task description. (Show Details)
akosiaris updated the task description. (Show Details)Jul 15 2020, 9:30 AM
akosiaris updated the task description. (Show Details)
gerritbot added a comment.Jul 16 2020, 9:02 AM

Change 613097 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] Create namespaces/calico rules for push-notifications

https://gerrit.wikimedia.org/r/613097

gerritbot added a project: Patch-For-Review.Jul 16 2020, 9:02 AM
jijiki added a comment.Jul 16 2020, 9:03 AM

@MSantos Is there an internal service/database push-notifications will be communicating with? If so, let us know so we can add the relevant calico rules

gerritbot added a comment.Jul 16 2020, 9:19 AM

Change 613101 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[labs/private@master] Add k8s dummy tokens for push-notifications

https://gerrit.wikimedia.org/r/613101

gerritbot added a comment.Jul 16 2020, 9:33 AM

Change 613104 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] Kubernetes: Create token stanzas for push-notifications

https://gerrit.wikimedia.org/r/613104

jijiki added a subtask: T250493: Benchmark CPU and memory usage of push notifications service.Jul 16 2020, 9:37 AM
MSantos added a comment.Jul 16 2020, 10:39 AM
In T256973#6311423, @jijiki wrote:

@MSantos Is there an internal service/database push-notifications will be communicating with? If so, let us know so we can add the relevant calico rules

@jijiki thanks for pushing this forward. Push Notifications as it is planned for this version has no communication with other internal services/databases.

gerritbot added a comment.Jul 22 2020, 10:01 AM

Change 613101 merged by Effie Mouzeli:
[labs/private@master] Add k8s dummy tokens for push-notifications

https://gerrit.wikimedia.org/r/613101

jijiki mentioned this in rLPRIbd44cb8505b3: Add k8s dummy tokens for push-notifications.Jul 22 2020, 10:04 AM
gerritbot added a comment.Jul 22 2020, 12:14 PM

Change 613104 merged by Effie Mouzeli:
[operations/puppet@production] Kubernetes: Create token stanzas for push-notifications

https://gerrit.wikimedia.org/r/613104

gerritbot added a comment.Jul 22 2020, 12:26 PM

Change 613097 merged by Effie Mouzeli:
[operations/deployment-charts@master] Create namespaces rules for push-notifications

https://gerrit.wikimedia.org/r/613097

Maintenance_bot removed a project: Patch-For-Review.Jul 22 2020, 1:10 PM
jijiki added a subtask: T250491: Create Helm chart for push notifications service.Jul 23 2020, 10:56 AM
akosiaris updated the task description. (Show Details)Jul 28 2020, 2:39 PM
Dzahn added a subtask: T259152: puppet errors on contint servers related to helmfiles for push-notifications.Jul 29 2020, 4:42 PM
Dzahn removed a subtask: T259152: puppet errors on contint servers related to helmfiles for push-notifications.
Dzahn mentioned this in T259152: puppet errors on contint servers related to helmfiles for push-notifications.Jul 29 2020, 4:56 PM
jijiki moved this task from Incoming 🐫 to cross-team active on the serviceops-deprecated board.Aug 17 2020, 11:45 PM
MSantos closed subtask T250491: Create Helm chart for push notifications service as Resolved.Aug 19 2020, 12:53 PM
jijiki updated the task description. (Show Details)Aug 20 2020, 9:22 PM
gerritbot added a comment.Aug 20 2020, 10:17 PM

Change 621605 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] helmfile: add values for staging environment

https://gerrit.wikimedia.org/r/621605

gerritbot added a project: Patch-For-Review.Aug 20 2020, 10:17 PM
gerritbot added a comment.Aug 25 2020, 11:14 AM

Change 621605 merged by jenkins-bot:
[operations/deployment-charts@master] helmfile: add values for staging environment

https://gerrit.wikimedia.org/r/621605

gerritbot added a comment.Aug 25 2020, 11:23 AM

Change 622330 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: enable TLS for all environments

https://gerrit.wikimedia.org/r/622330

gerritbot added a comment.Aug 25 2020, 11:28 AM

Change 622330 merged by jenkins-bot:
[operations/deployment-charts@master] push-notifications: enable TLS for all environments

https://gerrit.wikimedia.org/r/622330

gerritbot added a comment.Aug 25 2020, 11:45 AM

Change 622333 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: Bump chart version

https://gerrit.wikimedia.org/r/622333

gerritbot added a comment.Aug 25 2020, 11:56 AM

Change 622333 merged by JMeybohm:
[operations/deployment-charts@master] push-notifications: Bump chart version

https://gerrit.wikimedia.org/r/622333

Maintenance_bot removed a project: Patch-For-Review.Aug 25 2020, 12:11 PM
jijiki updated the task description. (Show Details)Aug 25 2020, 12:20 PM
jijiki added a comment.Aug 25 2020, 12:26 PM

Push-notifications is up and running in staging. Our next step is to perform the LVS steps and expose the application.

@JMeybohm and I are planning to do so this week, so it will be ready for you when you deploy to production.

gerritbot added a comment.Sep 1 2020, 9:05 AM

Change 623541 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/dns@master] Add entries for push-notifications

https://gerrit.wikimedia.org/r/623541

gerritbot added a project: Patch-For-Review.Sep 1 2020, 9:05 AM
gerritbot added a comment.Sep 1 2020, 9:24 AM

Change 623544 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/dns@master] Add discovery records for push-notifications

https://gerrit.wikimedia.org/r/623544

gerritbot added a comment.Sep 1 2020, 6:12 PM

Change 623631 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] lvs::configuration: add push-notifications patch 1/3

https://gerrit.wikimedia.org/r/623631

gerritbot added a comment.Sep 1 2020, 6:13 PM

Change 623632 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] lvs::configuration: add push-notifications patch 2/3

https://gerrit.wikimedia.org/r/623632

gerritbot added a comment.Sep 1 2020, 6:15 PM

Change 623634 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] lvs::configuration: add push-notifications patch 3/3

https://gerrit.wikimedia.org/r/623634

gerritbot added a comment.Sep 2 2020, 11:02 AM

Change 623773 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] lvs::configuration: add push-notifications patch 3/4

https://gerrit.wikimedia.org/r/623773

gerritbot added a comment.Sep 2 2020, 11:07 AM

Change 623774 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] conftool: add data for dns discovery for push-notifications

https://gerrit.wikimedia.org/r/623774

gerritbot added a comment.Sep 2 2020, 1:08 PM

Change 623790 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] services_proxy: add push-notifications

https://gerrit.wikimedia.org/r/623790

gerritbot added a comment.Sep 2 2020, 1:44 PM

Change 623774 abandoned by Effie Mouzeli:
[operations/puppet@production] conftool: add data for dns discovery for push-notifications

Reason:
Included in 623631

https://gerrit.wikimedia.org/r/623774

gerritbot added a comment.Sep 3 2020, 10:42 AM

Change 624014 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] lvs::configuration: add push-notifications patch 3/4

https://gerrit.wikimedia.org/r/624014

gerritbot added a comment.Sep 3 2020, 10:55 AM

Change 623634 abandoned by Effie Mouzeli:
[operations/puppet@production] lvs::configuration: add push-notifications patch 3/4

Reason:
rebase hell as always

https://gerrit.wikimedia.org/r/623634

gerritbot added a comment.Sep 7 2020, 4:08 PM

Change 625709 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: add proxy settings

https://gerrit.wikimedia.org/r/625709

jijiki added a project: User-jijiki.Sep 8 2020, 10:13 AM
jijiki moved this task from Incoming🐅 to In Progress 🏋️‍♀️ on the User-jijiki board.Sep 8 2020, 10:25 AM
gerritbot added a comment.Sep 9 2020, 11:03 AM

Change 625709 merged by jenkins-bot:
[operations/deployment-charts@master] push-notifications: add proxy settings

https://gerrit.wikimedia.org/r/625709

jijiki added a comment.Sep 9 2020, 11:09 AM

@MSantos please let us know when you are ready to go on production, so we can perform the final steps. Thank you!

jijiki updated the task description. (Show Details)Sep 10 2020, 9:43 AM
MSantos closed subtask Restricted Task as Resolved.Sep 15 2020, 5:12 PM
gerritbot added a comment.Sep 16 2020, 2:22 PM

Change 627846 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: enable mmonitoring

https://gerrit.wikimedia.org/r/627846

gerritbot added a comment.Sep 16 2020, 2:31 PM

Change 627846 merged by jenkins-bot:
[operations/deployment-charts@master] push-notifications: enable monitoring

https://gerrit.wikimedia.org/r/627846

jijiki added a subtask: T261635: Test push-notifications service in the staging environment.Sep 16 2020, 4:29 PM
gerritbot added a comment.Sep 18 2020, 9:45 AM

Change 628304 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: deploy to production environment

https://gerrit.wikimedia.org/r/628304

gerritbot added a comment.Sep 18 2020, 10:33 AM

Change 628304 merged by jenkins-bot:
[operations/deployment-charts@master] push-notifications: deploy to production environment

https://gerrit.wikimedia.org/r/628304

• Mholloway added a comment.Sep 18 2020, 1:51 PM

What will be the internal URL for this service? I am guessing https://push-notifications.discovery.wmnet:4104 (port 4104 as listed at https://wikitech.wikimedia.org/wiki/Service_ports)? I'm getting a MediaWiki config patch ready for Monday and need to put this in a config variable.

gerritbot added a comment.Sep 18 2020, 2:01 PM

Change 628336 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: enable egress

https://gerrit.wikimedia.org/r/628336

jijiki added a comment.EditedSep 18 2020, 2:13 PM

@Mholloway it will be accessible on Monday after we deploy the LVS/DNS patches. Meanwhile you could try something like curl -vk https://kubernetes1004.eqiad.wmnet:4104 (a kubernetes host+port+tls)

MSantos closed subtask T261635: Test push-notifications service in the staging environment as Resolved.Sep 18 2020, 2:19 PM
gerritbot added a comment.Sep 18 2020, 2:19 PM

Change 628340 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/deployment-charts@master] push-notifications: change version tag to -production

https://gerrit.wikimedia.org/r/628340

jijiki added a comment.EditedSep 18 2020, 2:26 PM

On Monday (EU) morning, @JMeybohm and I will push the LVS/DNS patches, so everything will be ready for the mediawiki-config change. There are a couple of things we may need to tune after deployment:

  • number of replicas
  • cpu/memory of each

Please join #wikimedia-serviceops or #wikimedia-sre when you roll out to group0 so someone from serviceops can assist if needed during or after 🛫:)

• Mholloway added a comment.Sep 18 2020, 4:01 PM

Sounds good, @jijiki. I'm in both channels. Thank you!

gerritbot added a comment.Sep 21 2020, 10:23 AM

Change 623541 merged by Effie Mouzeli:
[operations/dns@master] Add entries for push-notifications

https://gerrit.wikimedia.org/r/623541

gerritbot added a comment.Sep 21 2020, 10:49 AM

Change 623631 merged by Effie Mouzeli:
[operations/puppet@production] lvs::configuration: add push-notifications patch 1/4

https://gerrit.wikimedia.org/r/623631

gerritbot added a comment.Sep 21 2020, 10:57 AM

Change 623632 merged by Effie Mouzeli:
[operations/puppet@production] lvs::configuration: add push-notifications patch 2/4

https://gerrit.wikimedia.org/r/623632

Stashbot added a comment.Sep 21 2020, 11:02 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-21T11:02:35Z] <effie> restart pybal on lvs2010 and lvs1016 - T256973

Stashbot added a comment.Sep 21 2020, 11:22 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-21T11:22:29Z] <effie> restart pybal on lvs2010 and lvs1016 - T256973

Stashbot added a comment.Sep 21 2020, 11:38 AM

Mentioned in SAL (#wikimedia-operations) [2020-09-21T11:38:40Z] <effie> restart pybal on lvs2009 and lvs1015 - T256973

gerritbot added a comment.Sep 21 2020, 12:22 PM

Change 624014 merged by Effie Mouzeli:
[operations/puppet@production] lvs::configuration: add push-notifications patch 3/4

https://gerrit.wikimedia.org/r/624014

gerritbot added a comment.Sep 21 2020, 12:42 PM

Change 623773 merged by Effie Mouzeli:
[operations/puppet@production] lvs::configuration: add push-notifications patch 4/4

https://gerrit.wikimedia.org/r/623773

gerritbot added a comment.Sep 21 2020, 12:50 PM

Change 623544 merged by Effie Mouzeli:
[operations/dns@master] Add discovery records for push-notifications

https://gerrit.wikimedia.org/r/623544

jijiki added a comment.Sep 21 2020, 1:05 PM

@JMeybohm and I finished up, you can reach the production environments by using curl -k https://push-notifications.svc.eqiad.wmnet:4104 or curl -k https://push-notifications.svc.codfw.wmnet:4104

Good luck! 🎉

• Mholloway added a comment.Sep 21 2020, 1:06 PM

Thank you, @jijiki!

• Mholloway added a comment.Sep 21 2020, 3:10 PM

Looking at other service definitions in mediawiki-config ProductionServices.php, it looks like we generally want to be referencing services from MediaWiki in a DC-agnostic way. Do we need a service proxy set up for push-notifications?

Joe subscribed.Sep 21 2020, 3:29 PM
In T256973#6480180, @Mholloway wrote:

Looking at other service definitions in mediawiki-config ProductionServices.php, it looks like we generally want to be referencing services from MediaWiki in a DC-agnostic way. Do we need a service proxy set up for push-notifications?

yes, I guess it will be added by us soon.

gerritbot added a comment.Sep 21 2020, 5:15 PM

Change 623790 merged by Effie Mouzeli:
[operations/puppet@production] services_proxy: add push-notifications

https://gerrit.wikimedia.org/r/623790

JMeybohm added a comment.Sep 21 2020, 5:47 PM
In T256973#6480180, @Mholloway wrote:

Looking at other service definitions in mediawiki-config ProductionServices.php, it looks like we generally want to be referencing services from MediaWiki in a DC-agnostic way. Do we need a service proxy set up for push-notifications?

"http://localhost:6012" should become available now on the appservers (with next puppet run).

• Mholloway added a comment.Sep 21 2020, 5:52 PM

Excellent! Thank you!

jijiki closed this task as Resolved.Sep 22 2020, 11:16 AM
jijiki updated the task description. (Show Details)

@Mholloway I am marking this as resolved unless we believe there is a reason not to.

gerritbot added a comment.Oct 22 2020, 7:58 PM

Change 628336 abandoned by Effie Mouzeli:
[operations/deployment-charts@master] push-notifications: enable egress

Reason:
It appears that this is not needed for now

https://gerrit.wikimedia.org/r/628336

gerritbot added a comment.Jan 14 2021, 11:16 AM

Change 656129 had a related patch set uploaded (by Elukey; owner: Elukey):
[operations/puppet@production] Add deployment config for a new k8s service - eventstreams-internal

https://gerrit.wikimedia.org/r/656129

Dzahn mentioned this in T288546: Rotate APNS key before deploying Push Notifications to Production.Apr 26 2022, 9:43 PM
gerritbot added a comment.Jun 4 2025, 7:35 AM

Change #628340 abandoned by Effie Mouzeli:

[operations/deployment-charts@master] push-notifications: change version tag to -production

https://gerrit.wikimedia.org/r/628340

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits