Page MenuHomePhabricator

Is the Phabricator webhook sink broken?
Closed, ResolvedPublicBUG REPORT


Did the implementation for T362288: [gitlab-webhooks] Provide a server-sent events API for rebroadcast of GitLab webhook data break other sinks?

April 21, 2024
[12:35]  <    taavi> is the gitlab MR to phabricator comment webhook broken? did not leave a comment, nor did
[22:15]  < wikibugs> GitLab (Integrations), Wikibugs: Automate setup of comment, pipeline, and job webhooks for all GitLab projects - (bd808) >>! In T362940#9728392, @brennen wrote: > I guess my other thought about a home for this is that it could live in [[ https://gitlab.wi...
April 22, 2024
[16:58]  <    bd808> taavi: it is possible that I did break the gitlab hooks with I will open a bug report and investigate.


TitleReferenceAuthorSource BranchDest Branch
[DO NOT MERGE] Just something to allow a MRrepos/releng/gitlab-webhooks!30bd808work/bd808/T363114-verifymain
phab: Add missing await instructionsrepos/releng/gitlab-webhooks!29bd808work/bd808/T363114-broken-sinksmain
Customize query in GitLab

Event Timeline

bd808 triaged this task as High priority.
bd808 added a subscriber: taavi.

I can assert that webhook inputs were received by the running tool for both and

$ grep '' events.log | jq .object_attributes.action
$ grep '' events.log | jq .object_attributes.action

This is good in that it means we can replay the events once we figure out what else is wrong with the tool.

Mentioned in SAL (#wikimedia-cloud) [2024-04-23T00:17:19Z] <wmbot~bd808@tools-bastion-12> Built new container image from 0e8b79e4 (T363114)

Mentioned in SAL (#wikimedia-cloud) [2024-04-23T00:17:59Z] <wmbot~bd808@tools-bastion-12> Restarted to pick up proposed fixes for T363114 looks to have fixed the unintended regression.

Now the question is if we should attempt to re-process the 775 events that were logged between the bad code was introduced in and the fix in MR!29. These events are logged in /data/project/gitlab-webhooks/logs/events.log.2024-04-22. A custom runner will need to be devised to reprocess them. It should be possible to do that processing via python3 in a webservice shell container.
import logging

from sinks.phabricator import PhabricatorSink
sink = PhabricatorSink(None)

import asyncio
import json
import time

loop = asyncio.get_event_loop()
seen = 0
with open("/data/project/gitlab-webhooks/logs/events.log.2024-04-22") as f:
  for line in f:
    event = json.loads(line)
    if event.get("object_kind") == "merge_request":
      seen += 1
print("processed %s events" % seen)
$ launcher python3
... lots of log output...
INFO:sinks.phabricator:Added a comment on task T363114
processed 193 events

Sorry for the mess @taavi. Thank you for noticing the problem and reporting it.