Page MenuHomePhabricator

[gitlab-webhooks] Provide a server-sent events API for rebroadcast of GitLab webhook data
Closed, ResolvedPublicFeature

Description

I talked with @dancy on irc and have decided to try this variant:

  • Implement a new type of sink in gitlab-webhooks that allows arbitrary client subscriptions to the event stream. This could be a pull system that buffers events and replays them when an endpoint is called (like Phorge's feed.query endpoint), or a push system using Server Sent Events (SSE) or similar technology (like EventStreams and the work-in-progress on T361518). A possible drawback of this design is the currently synchronous nature of gitlab-webhooks' internal event handling. Implementing threading or asyncio there might be required to ensure that adding subscribers does not starve the other sinks of processing time.

Specifically I will submit an MR (or stack of MRs) that will:

  • Update gitlab-webhooks to use asyncio via hypercorn and the quart framework
  • Add an SSE API for clients like wikibugs to use to subscribe to the webhook data stream
  • Introduce additional changes needed to make the resulting tool run well from a build service managed container

The conversion to build service containers is needed to support ASGI with hypercorn. The legacy python container support in Toolforge is all uwsgi based. At the moment I think the only thing that will need changing to work well is converting the configuration of the app to use envvars for secrets rather than loading them from a yaml file on NFS. The latter is still technically possible with a custom container, but reducing NFS dependencies is more optimal.

Event Timeline

bd808 changed the task status from Open to In Progress.Apr 10 2024, 9:55 PM
bd808 claimed this task.
bd808 triaged this task as Medium priority.
bd808 moved this task from To Do to In Dev/Progress on the User-bd808 board.

bd808 merged https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/28

Provide a server-sent events API for rebroadcast of GitLab webhook data

Mentioned in SAL (#wikimedia-cloud) [2024-04-18T20:59:53Z] <wmbot~bd808@tools-bastion-12> Built image for 4ca07721 (T362288)

Mentioned in SAL (#wikimedia-cloud) [2024-04-18T21:03:24Z] <wmbot~bd808@tools-bastion-12> Stop & start to switch to buildservice managed image (T362288)

Play around with it via curl --no-buffer -H 'Accept: text/event-stream' 'https://gitlab-webhooks.toolforge.org/sse/'