Page MenuHomePhabricator

New Service Request - Change Propagation
Closed, ResolvedPublic

Description

About

This is a request for deploying the change propagation service on SCB as part of the Event-Platform project. The service will be used to track events emitted to the Kafka cluster and, according to the configured rules, will dispatch requests to various (internal) services. In the first iteration, RESTBase will be the only recipient (HTML changes, MobileApps ones, etc), but the plan is to eventually have other usages, like Varnish cache purging.

Request Flow

Events are produced by MediaWiki and RESTBase; the former emits events to the Kafka cluster when new revisions are created, while the latter produces events when storage contents changes (HTML, MobileApps, summary, ...). These events are sent to the HTTP Proxy service, collocated with the Kafka cluster on kafka100[12].eqiad.wmnet. Once validated, they enter Kafka's pipeline and get consumed by the change propagation service. For each message matching one of the predefined rules a request is dispatched. Currently, all of the requests will be made to RESTBase. Based on the event, RESTBase may respond with new events that get enqueued in Kafka.

The most common workflow will be the following:

  1. An edit is made to a MW page
  2. MW emits the revision_create event
  3. The proxy service validates it and enqueues it in Kafka
  4. The change propagation serivce consumes it by sending a request to RESTBase to render the new version of the page
  5. Once the new render has been stored, RESTBase emits a new event signalling a new HTML render is available
  6. The change propagation service reacts to it by sending another request to RESTBase, this time asking for a re-render of the MobileApps response
  7. RESTBase requests the new version from MobileApps, stores it and emits a second event signalling the MobileApps content for the given title has changed

Service Info

Owner@mobrovac / Services
Contact person@mobrovac
TimelineEoQ
Source codeGitHub repo
Target clusterSCB

See Also

Event Timeline

mobrovac created this task.Mar 1 2016, 1:54 PM
Restricted Application added subscribers: StudiesWorld, Aklapper. · View Herald TranscriptMar 1 2016, 1:54 PM
mobrovac updated the task description. (Show Details)
mobrovac added a parent task: T114443: EventBus MVP.
mobrovac removed subscribers: Aklapper, StudiesWorld.

Change 275772 had a related patch set uploaded (by Mobrovac):
Introducing changeprop role and puppet module

https://gerrit.wikimedia.org/r/275772

Change 275891 had a related patch set uploaded (by Mobrovac):
Assign changeprop service to scb cluster

https://gerrit.wikimedia.org/r/275891

Change 278712 had a related patch set uploaded (by Mobrovac):
Add the Scap3 configuration

https://gerrit.wikimedia.org/r/278712

Change 278712 merged by Mobrovac:
Add the Scap3 configuration

https://gerrit.wikimedia.org/r/278712

Change 278714 had a related patch set uploaded (by Mobrovac):
Initial import of change-propagation

https://gerrit.wikimedia.org/r/278714

Change 278714 merged by Mobrovac:
Initial import of change-propagation

https://gerrit.wikimedia.org/r/278714

mobrovac triaged this task as High priority.

Change 275772 merged by Alexandros Kosiaris:
Introducing changeprop role and puppet module

https://gerrit.wikimedia.org/r/275772

Change 279594 had a related patch set uploaded (by Alexandros Kosiaris):
Introduce changeprop.svc.${::site}.wmnet

https://gerrit.wikimedia.org/r/279594

Change 279594 merged by Alexandros Kosiaris:
Introduce changeprop.svc.${::site}.wmnet

https://gerrit.wikimedia.org/r/279594

Change 275891 merged by Alexandros Kosiaris:
Assign changeprop service to scb cluster

https://gerrit.wikimedia.org/r/275891

The service is now operational in production, but we are not able to reliably redeploy it due to T130948: Scap3 promote stage not working. This can be considered resolved once the blocker has been removed.

mobrovac claimed this task.Mar 29 2016, 3:08 PM
mobrovac removed a subscriber: gerritbot.
mobrovac closed this task as Resolved.Apr 1 2016, 10:06 AM
mobrovac moved this task from Inbox to In progress on the Service-deployment-requests board.