Page MenuHomePhabricator

Add kafka support for tile-pregeneration events
Open, Needs TriagePublic

Description

Context

In our new architecture we decided to move from using tilerator as a map tile pregeneration queuing system backed by Redis to native tegola functionality backed by Kafka.
One of the suggested ways to publish messages to the kafka deployment on wikimedia infra is EventGate, an HTTP API where event producers can POST messages to a predefined stream based on a predefined schema.

Here is the documentation around setting up instrumentation for events: https://wikitech.wikimedia.org/wiki/Event_Platform/Instrumentation_How_To
For more information about the Event Platform here is the project documentation: https://wikitech.wikimedia.org/wiki/Event_Platform

Action items
  • Create schema for map tile invalidation events
  • Add configuration to register the event stream

Event Timeline

Change 715011 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[schemas/event/secondary@master] WIP: Map tile expiration event schema

https://gerrit.wikimedia.org/r/715011

Change 715028 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[operations/mediawiki-config@master] Configure event stream for map tile expiration

https://gerrit.wikimedia.org/r/715028

Hey @Jgiannelos, your proposed event schema looks good to me. That said, I think it belongs in the primary event schema repository as an event that (quoting the README) "directly affect[s] user-facing features."

What will be producing these events to Kafka, and what will be consuming them from Kafka?

The OSM import pipeline is going to produce the events on each run and then our tile server is going to consume them periodically and invalidate/pre-generate stale tiles.

Change 715011 abandoned by Jgiannelos:

[schemas/event/secondary@master] Map tile expiration event schema

Reason:

Primary schema repository is probably a better fit for this event since map tile invalidation is a user facing change.

https://gerrit.wikimedia.org/r/715011

Change 716219 had a related patch set uploaded (by Jgiannelos; author: Jgiannelos):

[schemas/event/primary@master] Map tile expiration event schema

https://gerrit.wikimedia.org/r/716219

Change 716219 merged by Ottomata:

[schemas/event/primary@master] Map tile state change event schema

https://gerrit.wikimedia.org/r/716219