* Affected components:
** Echo extension for MediaWiki (producer, and consumer for web).
** Mobile apps (consumer).
* Engineer for initial implementation: Michael Holloway, Mateus Santos, Bernd Sitzmann
* Code steward: Product Infrastructure
### Motivation
The Wikimedia product teams have a long-standing desire to leverage push notification technology to drive user engagement and retention.
#### Initial use cases
- ** Edit reminders**: Push notification support would allow product teams to re-engage new editors by thanking them for their contributions across platforms and offering them encouragement to continue contributing to the projects.
- **New contributor onboarding**: Push notification support would allow Wikimedia product teams to engage new contributors by informing them about new features, settings, etc., across the product platforms.
- **User-specific notifications about on-wiki events**: Push notification support would allow informing users in near-real time about on-wiki actions that affect them, regardless of whether they are currently logged in. This is essentially another notification mechanism for the kinds of events handled by Echo.
#### Requirements
* Maintain a database of push subscriber information
* Expose a public API allowing subscribers to create, view, update, or delete their push subscriptions
* Expose a private (cluster-internal) API for requesting that a notification be pushed to one or more subscribers
* Map received notification requests to push subscriptions and forward notification requests to applicable push vendors
* Process notification requests in near-real time
-------
## Exploration
### Status quo
The [[ https://www.mediawiki.org/wiki/Extension:Echo | Echo ]] extension to MediaWiki provides a system for defining events and associated notifications pertaining to on-wiki actions. At present, users can be notified via the Echo notifications UI on the web site, or by e-mail. Both are limited to logged-in users.
### Prior art
##### Echo Web Push
A long-standing [[ https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/Echo/+/371615/ | WIP patch ]] from 2017 adds Web Push support to Echo. This RFC goes beyond the scope of that patch in the following ways:
- It will support pushing notification requests to mobile app push notification providers in addition to just web push.
- It can support types of notifications that are out of scope for Echo because they are not specific to a logged-in user.
- It will allow sending notifications to users/devices regardless of their account registration or current login state.
For this RFC, a new Echo notifier type would need to be added to support sending notifications about Echo events through the new push notification backend
##### 2017 push notifications technical plan
A [[ https://docs.google.com/document/d/1nVW3XC7PO3XDKRz72fCFDoZ-CA_lrhzNekUldPn5wFA | technical plan ]] document for a proposed push notifications service was created in 2017, but the project never reached implementation. This RFC is heavily informed by that plan, but the primary use cases of interest have changed.
The highest product priority behind this RFC is providing notifications about on-wiki events to support new contributors. Connecting Echo to the planned push notification service is therefore in scope for this RFC. On the other hand, subscribing to page edits and topics, which were prioritized in the 2017 plan, are now out of scope for the initial investment on this RFC.
-------
## Proposal
A system for receiving and managing push notification subscriptions and transmitting notification requests to push providers.
### System components
The push notifications architecture will consist of the following components:
# The primary component will be an external service for managing push subscriptions and receiving notification requests for submission to vendor push providers, including Mozilla Push Service (MPS), Apple Push Notification Service (APNS), and Google’s Firebase Cloud Messaging (FCM).
# A new push notifier type will be added to Echo to provide support for sending Echo events to the push service.
{F31761721, height=350, layout=center}
### Subscription management
The system will be required to maintain subscription tokens for the appropriate push vendor for all subscribers. A public API will be created for clients to manage subscriptions. The specific endpoints available will be similar to those described in the 2017 technical plan.
We propose storing push subscription data in a MySQL table managed by the external service. Our priorities for subscription storage are to ensure low-latency reads when processing notification requests, and to allow for easy horizontal scaling should the service’s resource requirements increase in the future.
A draft schema for subscription storage is [[ https://www.mediawiki.org/wiki/User:MHolloway_(WMF)/Drafts/Push_DB_Schema | here ]].
### Notification request processing
The system will expose a private (cluster-internal) HTTP REST API for receiving notification requests for processing. The specific endpoints available for notification request submission will be similar to those described in the 2017 technical plan.
Current [[ https://phabricator.wikimedia.org/T247438 | estimates ]] project a modest level of expected incoming notification request traffic (<1 req/s), though the rate of incoming requests will likely vary by time of day, month of the year (school in or out of session), and, given the product focus on new contributors, any campaigns in effect targeting new contributors.
### Notification limits per event type
The product teams require configurable limits on the number of push notifications received within a specific period of time. Further, these limits should be configurable by event type. More specific requirements of this notification limit functionality will be entered in a Phabricator task after further product manager discussion.
### Metrics
The operational health of the push notifications infrastructure will be tracked by way of metrics including:
- Subscription request rate (req/s)*
- Subscription deletion request rate (per platform) (req/s)*
- Total subscription count*
- Subscription request processing time (ms)**
- Notification request rate (req/s)*
- Notification request processing time (ms)**
- Error rates per endpoint
%%%* Tracked per platform (web, Android, iOS)
** At p50, p90, p95, p99%%%
Alerts will be configured for request spikes or when error rates pass a reasonable threshold.
### Sunset/Rollback
Push notifications are not critical to the operation of MediaWiki, and disabling them should not negatively affect any other software component running in Wikimedia production. The push notifications infrastructure will be largely self-contained to permit easy shutdown in the event of emergency or if push notifications are no longer needed by the Wikimedia products.
### Why an external service?
The Wikimedia developer community has adopted a set of [[ https://www.mediawiki.org/wiki/Requests_for_comment/Standards_for_external_services | criteria ]] for assessing whether a feature may be implemented as a service external to MediaWiki. According to these criteria, the proposed functionality is suitable for implementation as an external service. The functionality is self-contained and does not depend on having a consistent and current view of MediaWiki state. It does not require direct access to the MediaWiki database, and does not require features or functionality provided by MediaWiki or its extensions. Furthermore, a push notification service will likely involve resource usage spikes that make the functionality more suitable for running in a separate, dedicated environment.
Several open-source push notification server projects exist on the web. Building in an external service will allow us to scale independently of MediaWiki, and working from an existing open-source project will allow us to build on the lessons learned by prior push notification service implementers so that we can focus the greater part of our efforts on implementing any specific custom functionality required by Wikimedia Product and ensuring that the service meets the requirements of Wikimedia’s production environment.
The push service will be written in Node.js, which we have substantial experience working with and running in production. We [[ https://docs.google.com/spreadsheets/d/1MURj6jpwOlbI8gihoP8IGGxKjZnh-vmkOjNl9ijJZF0 | reviewed ]] several existing open-source push service projects, and found that any of them would require significant updating in order to meet our product and operational requirements. None are suitable out of the box; indeed, most of the projects have been unmaintained for years. This being the case, we plan to build a new Node.js service based on the Wikimedia node service template. Implementation details on request handling and interactions with push vendors will be informed substantially by DailyMotion’s [[ https://github.com/rs/pushd | pushd ]] project.
### Scope of work (Q4 2019-2020)
The Product Infrastructure team will work in Q4 to launch the baseline push notification infrastructure. This effort will include:
# Building and launching the external service to manage push subscriptions and notification requests;
# Adding a new Echo notifier type to allow for submitting events to the external push notification system; and
# Adding a table to Echo to maintain a mapping of Wikimedia central user IDs to push service subscription IDs.
Our initial focus will be on supporting push notifications in the apps. The Wikipedia apps have long been missing native push notifications, a basic feature of most mobile apps today. Supporting the apps first will also provide us with the opportunity to validate the push notification infrastructure and to work out any issues before exposing it to traffic at Wikipedia’s web scale. Support for web push will be a reach goal for initial rollout.
We anticipate that new Echo event types will be required in the future to support envisioned Product use cases for push notifications. However, this is fundamentally an infrastructure project and not a feature project, and no new Echo event types will be created as part of the initial release.
### See also
#### Platform-specific push API docs
Web:
* https://developer.mozilla.org/en-US/docs/Web/API/Push_API
* https://developer.mozilla.org/en-US/docs/Web/API/Notifications_API
* https://developers.google.com/web/fundamentals/push-notifications
iOS:
* https://developer.apple.com/library/archive/documentation/NetworkingInternet/Conceptual/RemoteNotificationsPG/APNSOverview.html
Android:
* https://firebase.google.com/docs/cloud-messaging