Page MenuHomePhabricator

gitlab-webhooks: toolforge webservice logs -f choking on a log message
Closed, ResolvedPublic

Description

For the gitlab-webhooks tool, when I run toolforge webservice logs -f, it seems to get stuck on a particular message. I saved the tail end of the output in P93297 (WMF-NDA).

I know the service is still active (evidenced by updates to an events.log file that it generates).

While investigating I redirected stdout to a file and saw this in the terminal:

$ toolforge webservice logs -f > /tmp/output
Exception in thread Thread-1 (_queue_log_entries):
Traceback (most recent call last):
  File "/usr/lib/python3.13/threading.py", line 1043, in _bootstrap_inner
    self.run()
    ~~~~~~~~^^
  File "/usr/lib/python3.13/threading.py", line 994, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/toolforge_weld/logs/kubernetes.py", line 52, in _queue_log_entries
    for entry in self._get_pod_logs(*args, **kwargs):
                 ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/toolforge_weld/logs/kubernetes.py", line 47, in _get_pod_logs
    datetime=parse_date(datetime),
             ~~~~~~~~~~^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 1365, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/dateutil/parser/_parser.py", line 643, in parse
    raise ParserError("String does not contain a date: %s", timestr)
dateutil.parser._parser.ParserError: String does not contain a date:

I hope that reveals something to someone.

Details

Related Changes in GitLab:
TitleReferenceAuthorSource BranchDest Branch
toolforge_weld/logs/kubernetes.py: Handle continuation log recordsdancy/toolforge-weld!1dancymain-I4e92c7d6a6c69ddae62edb999f9f467438c0ccb7main
Customize query in GitLab

Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Noting that it doesn't always stop at the same message. For example, I ran again recently and it stopped at an earlier timestamp of 2026-05-27T17:04:40+00:00

It looks like stream of container log records being returned from Kubernetes is getting mangled, possibly due to weird Unicode characters?

P93394 The event that breaks gitlab-webhooks container log stream
P93395 The lines that KubernetesSource._get_pod_logs receives

dancy claimed this task.

Looks good now.