Page MenuHomePhabricator

wb2-phab having issues
Closed, ResolvedPublicBUG REPORT

Description

Leaving a note that when restarting wikibugs today using the instructions found at https://www.mediawiki.org/wiki/Wikibugs the job wb2-phab keeps logging errors at wb2-phab.err. @valhallasw said he'll be able to look at it tomorrow (his) morning. Tracking task for general awareness as well.

Event Timeline

MarcoAurelio changed the subtype of this task from "Task" to "Bug Report".Mar 18 2019, 10:07 PM

Looks like it is working again. Keeping it open however so it can be investigated.

Looking at the logs, my hypothesis is the following:

  • the json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) are probably from when Phabricator was offline and throwing 503's. I should add a test for that situation.
  • there is a bug where the bot can't handle transactionPHIDs not being available in the JSON -- might be happening in the case of security tasks. Typical backtrace:
[2019-18-03T20:35] /mnt/nfs/labstore-secondary-tools-project/wikibugs/py35-stretch/bin/python3 exited with code 1. Respawning...
Traceback (most recent call last):
  File "/data/project/wikibugs/wikibugs2/wikibugs.py", line 325, in <module>
    time.sleep(1)
KeyboardInterrupt
Traceback (most recent call last):
  File "/data/project/wikibugs/wikibugs2/wikibugs.py", line 325, in <module>
    time.sleep(1)
KeyboardInterrupt
Traceback (most recent call last):
  File "/data/project/wikibugs/wikibugs2/wikibugs.py", line 223, in process_event
    task_page, event_info['data']['transactionPHIDs']
KeyError: 'transactionPHIDs'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/project/wikibugs/wikibugs2/wikibugs.py", line 324, in <module>
    bugs.poll()
  File "/data/project/wikibugs/wikibugs2/wikibugs.py", line 84, in poll
    self.process_event(event)
  File "/data/project/wikibugs/wikibugs2/wikibugs.py", line 226, in process_event
    logger.exception("Could not retrieve anchor for %s" % event_info['data']['transactionPHIDs'])
KeyError: 'transactionPHIDs'

I need to check whether those events are logged before the crash.

In all cases, the bot restarts, so the effect is limited.

Change 497861 had a related patch set uploaded (by Merlijn van Deen; owner: Merlijn van Deen):
[labs/tools/wikibugs2@master] Do not crash when tokens are given

https://gerrit.wikimedia.org/r/497861

It's always great when you fix a bug twice, merge once, then come back yet another year later to merge the first one.

https://gerrit.wikimedia.org/r/c/labs/tools/wikibugs2/+/561346/ (Jan '20) was merged first and solved the issue. I'm merging 497861 (Mar '19) as it adds a few bonus tests.

valhallasw claimed this task.

Change 497861 merged by jenkins-bot:

[labs/tools/wikibugs2@master] Do not crash when tokens are given

https://gerrit.wikimedia.org/r/497861