Push notification service should make deletion requests to MediaWiki for invalid or expired subscriptions
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Mholloway
	Aug 12 2020, 2:44 PM

Description

Follow up from T259148.

When a push notification provider API provides a response indicating that a token or subscription is invalid or expired, the service should make a request to the MediaWiki API to delete that push subscription.

See the API docs at api.php?action=help&modules=echopushsubscriptions%2Bdelete as updated once https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Echo/+/619570 lands. (That module should probably also be updated to accept batched deletion requests.)

AC

The bot secrets are added to production environment

Details

Subject	Repo	Branch	Lines +/-
WIP: push-notif: add stanzas for requests to MWAPI	operations/deployment-charts	master	+17 -0
push-notifications: enable service proxy	operations/deployment-charts	master	+17 -0
Clean up invalid subscriptions	mediawiki/services/push-notifications	master	+480 -270
Add MW API login support	mediawiki/services/push-notifications	master	+141 -0
Restore MW API request support code	mediawiki/services/push-notifications	master	+125 -123

Customize query in gerrit

Related Objects
Search...

Status	Assigned	Task
Resolved	Jgiannelos	T260247 Push notification service should make deletion requests to MediaWiki for invalid or expired subscriptions
Resolved	jijiki	T262957 Set up secrets for Token clean-up
Resolved	MSantos	T264101 Find a way for the push service to authenticate to MediaWiki in beta and production

Event Timeline

• Mholloway created this task.Aug 12 2020, 2:44 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptAug 12 2020, 2:44 PM

LGoto triaged this task as High priority.Aug 12 2020, 3:34 PM

LGoto edited projects, added Product-Infrastructure-Team-Backlog-Deprecated (Kanban); removed Product-Infrastructure-Team-Backlog-Deprecated.

• Mholloway claimed this task.Aug 17 2020, 7:01 PM

• Mholloway updated the task description. (Show Details)

• Mholloway moved this task from To Do to Doing on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.

Change 620770 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/push-notifications@master] Restore MW API request support code

https://gerrit.wikimedia.org/r/620770

gerritbot added a project: Patch-For-Review.Aug 17 2020, 7:20 PM

Change 620800 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/push-notifications@master] Add MW API login support

https://gerrit.wikimedia.org/r/620800

• Mholloway mentioned this in T259148: [Spike] Token expiration handling.Aug 17 2020, 11:59 PM

Blocking this on discussion with CPT (Petr).

Lowering the priority since we're not considering this a release blocker.

• Mholloway moved this task from Blocked to Doing on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Aug 20 2020, 6:16 PM

MSantos updated the task description. (Show Details)Aug 21 2020, 11:36 AM

MSantos added a subscriber: jijiki.

Change 620770 merged by jenkins-bot:
[mediawiki/services/push-notifications@master] Restore MW API request support code

https://gerrit.wikimedia.org/r/620770

Change 620800 merged by jenkins-bot:
[mediawiki/services/push-notifications@master] Add MW API login support

https://gerrit.wikimedia.org/r/620800

Change 622376 had a related patch set uploaded (by Mholloway; owner: Michael Holloway):
[mediawiki/services/push-notifications@master] WIP: Clean up invalid subscriptions

https://gerrit.wikimedia.org/r/622376

• Mholloway moved this task from Doing to Code Review on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Aug 26 2020, 8:58 PM

Change 622376 merged by jenkins-bot:
[mediawiki/services/push-notifications@master] Clean up invalid subscriptions

https://gerrit.wikimedia.org/r/622376

• Mholloway moved this task from Code Review to To Deploy on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Aug 28 2020, 6:58 PM

MarcoAurelio mentioned this in T261625: Limit the new "push subscription managers" user group to metawiki.Aug 31 2020, 10:16 AM

Change 624043 had a related patch set uploaded (by MSantos; owner: MSantos):
[operations/deployment-charts@master] WIP: push-notif: add stanzas for requests to MWAPI

https://gerrit.wikimedia.org/r/624043

@Mholloway what should be done here to move this forward to production?

MSantos mentioned this in T262957: Set up secrets for Token clean-up.Sep 15 2020, 5:38 PM

• Mholloway added a subtask: T262957: Set up secrets for Token clean-up.Sep 15 2020, 5:59 PM

• Mholloway moved this task from To Deploy to Sign off on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Sep 23 2020, 11:05 PM

• Mholloway moved this task from Sign off to Doing on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.Sep 23 2020, 11:59 PM

I'm seeing the following error when the service tries to connect to the MediaWiki API. It looks like a (cluster-internal) TLS configuration issue.

{
  "headers": {
    "content-type": "application/problem+json"
  },
  "stack": "HTTPError: unable to verify the first certificate\n    at request.then (/srv/service/node_modules/preq/index.js:246:19)\n    at tryCatcher (/srv/service/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/srv/service/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/srv/service/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/srv/service/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/srv/service/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/srv/service/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/srv/service/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/srv/service/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues [as _onImmediate] (/srv/service/node_modules/bluebird/js/release/async.js:15:14)\n    at runCallback (timers.js:705:18)\n    at tryOnImmediate (timers.js:676:5)\n    at processImmediate (timers.js:658:5)",
  "name": "push-notifications",
  "message": "unable to verify the first certificate",
  "body": {
    "internalErr": "unable to verify the first certificate",
    "internalURI": "https://api-rw.discovery.wmnet/w/api.php",
    "internalMethod": "post",
    "detail": "unable to verify the first certificate",
    "type": "internal_http_error",
    "internalStack": "Error: unable to verify the first certificate\n    at TLSSocket.onConnectSecure (_tls_wrap.js:1055:34)\n    at TLSSocket.emit (events.js:189:13)\n    at TLSSocket._finishInit (_tls_wrap.js:633:8)"
  },
  "status": 504,
  "levelPath": "debug"
}

https://logstash.wikimedia.org/goto/50e805f06bdb1ac27ca8eedc732df8ab

Looks like we didn't tell nodejs to use the puppet CA.

Or even better: it should use the service proxy directly (which automagically sets up https for you)

Change 629656 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/deployment-charts@master] push-notifications: enable service proxy

https://gerrit.wikimedia.org/r/629656

• sdkim added subscribers: • Charlotte, Dbrant.Sep 24 2020, 2:02 PM

Change 629656 merged by jenkins-bot:
[operations/deployment-charts@master] push-notifications: enable service proxy

https://gerrit.wikimedia.org/r/629656

• Mholloway added a subtask: T264101: Find a way for the push service to authenticate to MediaWiki in beta and production.Sep 29 2020, 3:44 PM

• Mholloway removed • Mholloway as the assignee of this task.Sep 29 2020, 3:46 PM

• Mholloway moved this task from Doing to To Do on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.

• Mholloway renamed this task from Push notification service should make deletion requests to the MW API for invalid or expired subscriptions to Push notification service should make deletion requests to MediaWiki for invalid or expired subscriptions.Sep 29 2020, 5:28 PM

MSantos merged a task: T251463: Dead subscription cleanup job.Oct 1 2020, 10:50 AM

MSantos added a subscriber: dr0ptp4kt.

The TLS issue should be gone with the introduction of the service proxy for wikifeeds. @Mholloway can you confirm that's the case?

In T260247#6508622, @Joe wrote:

The TLS issue should be gone with the introduction of the service proxy for wikifeeds. @Mholloway can you confirm that's the case?

Yes, confirmed.

Joe unsubscribed.Oct 5 2020, 6:20 AM

LGoto edited projects, added Product-Infrastructure-Team-Backlog-Deprecated; removed Product-Infrastructure-Team-Backlog-Deprecated (Kanban).Oct 28 2020, 3:34 PM

LGoto moved this task from Needs triage to Upcoming on the Product-Infrastructure-Team-Backlog-Deprecated board.

MSantos moved this task from Upcoming to Kanban on the Product-Infrastructure-Team-Backlog-Deprecated board.Nov 24 2020, 6:22 PM

MSantos edited projects, added Product-Infrastructure-Team-Backlog-Deprecated (Kanban); removed Product-Infrastructure-Team-Backlog-Deprecated.

MSantos moved this task from To Do to Sign off on the Product-Infrastructure-Team-Backlog-Deprecated (Kanban) board.

MSantos closed subtask T264101: Find a way for the push service to authenticate to MediaWiki in beta and production as Resolved.Dec 1 2020, 4:46 PM

Change 624043 abandoned by JMeybohm:
[operations/deployment-charts@master] WIP: push-notif: add stanzas for requests to MWAPI

Reason:
Already mergen as part of I66a31dbf0767eaff779358d5e37c9ff93091edda

https://gerrit.wikimedia.org/r/624043

jijiki closed subtask T262957: Set up secrets for Token clean-up as Resolved.Feb 5 2021, 7:55 AM