Page MenuHomePhabricator

TestMakeDist fails on over-specified assertion
Closed, ResolvedPublicBUG REPORT

Description

Steps to replicate the issue (include links if applicable):

  • MacOS 12.6.1
  • Python 3.9.13
  • checkout pywikibot/core.git to 77482f4bab1822c7b604f7d1a4787ad2742ec991
  • pytest tests/make_dist_tests.py

What happens?:

TestMakeDist.test_handle_args and TestMakeDist.test_handle_args_empty both fail with similar errors:

tests/make_dist_tests.py:26: in _test_argv
    self.assertEqual(__file__, sys.argv[0])
E   AssertionError: '/Users/roy/dev/pywikibot/pywikibot-git/tests/make_dist_tests.py' != '/Users/roy/dev/pywikibot/venv/bin/pytest'
E   - /Users/roy/dev/pywikibot/pywikibot-git/tests/make_dist_tests.py
E   + /Users/roy/dev/pywikibot/venv/bin/pytest

The failure happens in _test_argv():

if os.environ.get('PYWIKIBOT_TEST_RUNNING', '0') != '1':
    self.assertEqual(__file__, sys.argv[0])

Note the exact value of sys.argv[0] varies depending on how I run the test. The above is when I run it from the command line. Inside vscode, I get:

/Users/roy/dev/pywikibot/venv/lib/python3.9/site-packages/pytest/__main__.py

and in vscode test debug mode, it's:

/Users/roy/.vscode/extensions/ms-python.python-2023.8.0/pythonFiles/testlauncher.py

This seems like a pointless test, as it's asserting a specific behavior for something which is environment dependent. I suggest deleting the argv[0] test completely.

Event Timeline

Change 919243 had a related patch set uploaded (by JJMC89; author: JJMC89):

[pywikibot/core@master] [FIX] do not test sys.argv[0]

https://gerrit.wikimedia.org/r/919243

JJMC89 changed the task status from Open to In Progress.May 12 2023, 5:17 PM
JJMC89 claimed this task.
JJMC89 added a project: Pywikibot.

Change 919243 merged by jenkins-bot:

[pywikibot/core@master] [FIX] do not test sys.argv[0]

https://gerrit.wikimedia.org/r/919243

JJMC89 triaged this task as Medium priority.
JJMC89 moved this task from Backlog to Test failures on the Pywikibot-tests board.
JJMC89 removed a project: Patch-For-Review.

What is the reason to run the test with the PYWIKIBOT_TEST_RUNNING flag set which is to be used for CI test only. Without this flag the test would pass in your environment.

I think you've got that backwards, the flag is not set, so it defaults to '0', which is != '1', so the assertion is run. Which leads to a couple of additional observations:

If the intent is that this is only run when PYWIKIBOT_TEST_RUNNING == 1, the comparison is backwards. But, deeper than that, why are there tests that are only supposed to run in CI and not otherwise? And looking at the description in pywikibot-git/tests/README.rst, the usage here is contrary to the documented use.

You are right, my fault. And indeed this test is senseless.