Is the Phabricator webhook sink broken?
Closed, ResolvedPublicBUG REPORT
Actions

Assigned To

Authored By

	bd808
	Mon, Apr 22, 5:01 PM

Description

Did the implementation for T362288: [gitlab-webhooks] Provide a server-sent events API for rebroadcast of GitLab webhook data break other sinks?

April 21, 2024
[12:35]  <    taavi> is the gitlab MR to phabricator comment webhook broken? https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/10 did not leave a comment, nor did https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/37
[22:15]  < wikibugs> GitLab (Integrations), Wikibugs: Automate setup of comment, pipeline, and job webhooks for all GitLab projects - https://phabricator.wikimedia.org/T362940#9732453 (bd808) >>! In T362940#9728392, @brennen wrote: > I guess my other thought about a home for this is that it could live in [[ https://gitlab.wi...
April 22, 2024
[16:58]  <    bd808> taavi: it is possible that I did break the gitlab hooks with https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/28. I will open a bug report and investigate.

Details

	Title	Reference	Author	Source Branch	Dest Branch
	[DO NOT MERGE] Just something to allow a MR	repos/releng/gitlab-webhooks!30	bd808	work/bd808/T363114-verify	main
	phab: Add missing await instructions	repos/releng/gitlab-webhooks!29	bd808	work/bd808/T363114-broken-sinks	main

Customize query in GitLab

Related Objects

Mentioned Here: T362288: [gitlab-webhooks] Provide a server-sent events API for rebroadcast of GitLab webhook data

Event Timeline

bd808 created this task.Mon, Apr 22, 5:01 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMon, Apr 22, 5:01 PM

bd808 claimed this task.Mon, Apr 22, 5:01 PM

bd808 triaged this task as High priority.

bd808 added a subscriber: taavi.

Restricted Application added a project: User-bd808. · View Herald TranscriptMon, Apr 22, 5:01 PM

I can assert that webhook inputs were received by the running tool for both https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/10 and https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/37

$ grep 'https://gitlab.wikimedia.org/repos/ci-tools/banana-checker/-/merge_requests/10' events.log | jq .object_attributes.action
"open"
"update"
"update"
"update"
"update"
"update"
$ grep 'https://gitlab.wikimedia.org/repos/ci-tools/libup/-/merge_requests/37' events.log | jq .object_attributes.action
"open"
"merge"

This is good in that it means we can replay the events once we figure out what else is wrong with the tool.

Mentioned in SAL (#wikimedia-cloud) [2024-04-23T00:17:19Z] <wmbot~bd808@tools-bastion-12> Built new container image from 0e8b79e4 (T363114)

Mentioned in SAL (#wikimedia-cloud) [2024-04-23T00:17:59Z] <wmbot~bd808@tools-bastion-12> Restarted to pick up proposed fixes for T363114

bd808 opened https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/30

[DO NOT MERGE] Just something to allow a MR

https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/29 looks to have fixed the unintended regression.

Now the question is if we should attempt to re-process the 775 events that were logged between the bad code was introduced in https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/28 and the fix in MR!29. These events are logged in /data/project/gitlab-webhooks/logs/events.log.2024-04-22. A custom runner will need to be devised to reprocess them. It should be possible to do that processing via python3 in a webservice shell container.

bd808 opened https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/29

phab: Add missing await instructions

bd808 merged https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/29

phab: Add missing await instructions

reprocess.py

import logging
logging.basicConfig(level=logging.DEBUG)
logging.captureWarnings(True)

from sinks.phabricator import PhabricatorSink
sink = PhabricatorSink(None)

import asyncio
import json
import time

loop = asyncio.get_event_loop()
seen = 0
with open("/data/project/gitlab-webhooks/logs/events.log.2024-04-22") as f:
  for line in f:
    event = json.loads(line)
    if event.get("object_kind") == "merge_request":
      seen += 1
      loop.run_until_complete(sink.notify(event))
      time.sleep(3)
print("processed %s events" % seen)

$ launcher python3 reprocess.py
... lots of log output...
INFO:sinks.phabricator:Added a comment on task T363114
processed 193 events

Sorry for the mess @taavi. Thank you for noticing the problem and reporting it.

bd808 closed https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/30