Page MenuHomePhabricator

test_https_cert_error is failing with AssertionError because it detects more warnings than expected
Closed, ResolvedPublic

Description

https://travis-ci.org/wikimedia/pywikibot-core/jobs/214732977#L4224

________________ HttpsCertificateTestCase.test_https_cert_error ________________

self = <tests.http_tests.HttpsCertificateTestCase testMethod=test_https_cert_error>

    def test_https_cert_error(self):

        """Test if http.fetch respects disable_ssl_certificate_validation."""

        self.assertRaisesRegex(pywikibot.FatalServerError, self.CERT_VERIFY_FAILED_RE,

                               http.fetch,

                               uri='https://testssl-expire-r2i2.disig.sk/index.en.html')

        http.session.close()  # clear the connection

    

        with warnings.catch_warnings(record=True) as warning_log:

            response = http.fetch(

                uri='https://testssl-expire-r2i2.disig.sk/index.en.html',

                disable_ssl_certificate_validation=True)

        r = response.content

        self.assertIsInstance(r, unicode)

        self.assertTrue(re.search(r'<title>.*</title>', r))

        http.session.close()  # clear the connection

    

        # Verify that it now fails again

        self.assertRaisesRegex(pywikibot.FatalServerError, self.CERT_VERIFY_FAILED_RE,

                               http.fetch,

                               uri='https://testssl-expire-r2i2.disig.sk/index.en.html')

        http.session.close()  # clear the connection

    

        # Verify that the warning occurred

>       self.assertEqual(len(warning_log), 1)

E       AssertionError: 2 != 1

According to L1566 the warnings are:

tests/http_tests.py::HttpsCertificateTestCase::test_https_cert_error WARNING: /home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/_code/source.py:343: ResourceWarning: unclosed <socket.socket fd=7, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.17.0.4', 47346), raddr=('195.28.75.37', 443)>

  astnode = compile(content, "source", "exec", 1024)  # 1024 for AST

FAILED

We had a similar issue in T151248: test_https_cert_error failed on Travis CI

Event Timeline

Dalba renamed this task from test_https_cert_error is failing with AssertionError because it detect more warnings than expected to test_https_cert_error is failing with AssertionError because it detects more warnings than expected.Mar 25 2017, 6:01 AM

I could not reproduce the issue on my PC. I tried the same python version (3.4.2) and the same package versions [I may have missed some packages though]. Nothing worked, the test always passed successfully.

My current guess is that the issue can only be reproduced on Ubuntu precise. Using dist: trusty in .travis.yml file, made the test pass. Unfortunately I don't have direct access to any precise machine.

Using print([wl.__dict__ for wl in warning_log]) before the assert shows that the unexpected warning is:

{'_category_name': 'ResourceWarning',

  'category': <class 'ResourceWarning'>,

  'file': None,

  'filename': '/home/travis/virtualenv/python3.4.2/lib/python3.4/_weakrefset.py',

  'line': None,

  'lineno': 60,

  'message': ResourceWarning("unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.17.0.6', 53280), raddr=('195.28.75.37', 443)>",)}

I monkeypatched the warnings.warn using the following code:

import traceback
import warnings
import sys

def warn_with_traceback(message, category, filename, lineno, file=None,
                        line=None):

    log = file if hasattr(file, 'write') else sys.stderr
    traceback.print_stack(file=log)
    log.write(
        warnings.formatwarning(message, category, filename, lineno,
                               line))

warnings.showwarning = warn_with_traceback

The result was:

File "setup.py", line 224, in <module>

  use_2to3=False

File "/opt/python/3.4.2/lib/python3.4/distutils/core.py", line 148, in setup

  dist.run_commands()

File "/opt/python/3.4.2/lib/python3.4/distutils/dist.py", line 955, in run_commands

  self.run_command(cmd)

File "/opt/python/3.4.2/lib/python3.4/distutils/dist.py", line 974, in run_command

  cmd_obj.run()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/ptr.py", line 150, in run

  self._super_run()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/ptr.py", line 142, in _super_run

  self.with_project_on_sys_path(self.run_tests)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/setuptools/command/test.py", line 122, in with_project_on_sys_path

  func()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/ptr.py", line 199, in run_tests

  self.result_code = __import__('pytest').main()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/config.py", line 57, in main

  return config.hook.pytest_cmdline_main(config=config)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__

  return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec

  return self._inner_hookexec(hook, methods, kwargs)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>

  _MultiCall(methods, kwargs, hook.spec_opts).execute()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute

  res = hook_impl.function(*args)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/main.py", line 127, in pytest_cmdline_main

  return wrap_session(config, _main)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/main.py", line 98, in wrap_session

  session.exitstatus = doit(config, session) or 0

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/main.py", line 133, in _main

  config.hook.pytest_runtestloop(session=session)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__

  return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec

  return self._inner_hookexec(hook, methods, kwargs)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>

  _MultiCall(methods, kwargs, hook.spec_opts).execute()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute

  return _wrapped_call(hook_impl.function(*args), self.execute)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call

  call_outcome = _CallOutcome(func)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__

  self.result = func()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute

  res = hook_impl.function(*args)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/main.py", line 154, in pytest_runtestloop

  item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__

  return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec

  return self._inner_hookexec(hook, methods, kwargs)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>

  _MultiCall(methods, kwargs, hook.spec_opts).execute()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute

  return _wrapped_call(hook_impl.function(*args), self.execute)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call

  call_outcome = _CallOutcome(func)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__

  self.result = func()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute

  return _wrapped_call(hook_impl.function(*args), self.execute)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call

  call_outcome = _CallOutcome(func)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__

  self.result = func()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute

  res = hook_impl.function(*args)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/runner.py", line 66, in pytest_runtest_protocol

  runtestprotocol(item, nextitem=nextitem)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/runner.py", line 79, in runtestprotocol

  reports.append(call_and_report(item, "call", log))

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/runner.py", line 135, in call_and_report

  report = hook.pytest_runtest_makereport(item=item, call=call)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__

  return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec

  return self._inner_hookexec(hook, methods, kwargs)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>

  _MultiCall(methods, kwargs, hook.spec_opts).execute()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 613, in execute

  return _wrapped_call(hook_impl.function(*args), self.execute)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 248, in _wrapped_call

  call_outcome = _CallOutcome(func)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 265, in __init__

  self.result = func()

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute

  res = hook_impl.function(*args)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/runner.py", line 272, in pytest_runtest_makereport

  longrepr = item.repr_failure(excinfo)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/python.py", line 600, in repr_failure

  return self._repr_failure_py(excinfo, style=style)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/python.py", line 593, in _repr_failure_py

  style=style)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/main.py", line 449, in _repr_failure_py

  style=style, tbfilter=tbfilter)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/_code/code.py", line 427, in getrepr

  return fmt.repr_excinfo(self)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/_code/code.py", line 631, in repr_excinfo

  reprtraceback = self.repr_traceback(excinfo)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/_code/code.py", line 611, in repr_traceback

  reprentry = self.repr_traceback_entry(entry, einfo)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/_code/code.py", line 558, in repr_traceback_entry

  source = self._getentrysource(entry)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/_code/code.py", line 483, in _getentrysource

  source = entry.getsource(self.astcache)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/_code/code.py", line 195, in getsource

  source = self.frame.code.fullsource

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/_code/code.py", line 59, in fullsource

  full, _ = source.findsource(self.raw)

File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/_pytest/_code/source.py", line 250, in findsource

  sourcelines, lineno = py.std.inspect.findsource(obj)

File "/opt/python/3.4.2/lib/python3.4/inspect.py", line 661, in findsource

  module = getmodule(object, file)

File "/opt/python/3.4.2/lib/python3.4/inspect.py", line 619, in getmodule

  for modname, module in list(sys.modules.items()):

File "/home/travis/build/myusername/pywikibot-core/tests/http_tests.py", line 59, in warn_with_traceback

  traceback.print_stack(file=log)
 /opt/python/3.4.2/lib/python3.4/inspect.py:619: ResourceWarning: unclosed <socket.socket fd=5, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('172.17.0.11', 34900), raddr=('195.28.75.37', 443)>

for modname, module in list(sys.modules.items()):

I could not understand what's going on there.

Looking back at T151248 and its solution, the only other related place I could find that uses http.fetch was inside aspects.py:CheckHostnameMixin.setUpClass.
Closing http.session there didn't solve the issue either.

At this point I'm going to propose to ignore the mysterious ResourceWarning. The main purpose of that test is to make sure the warning is raised, not that other warnings are not raised, therefore it should be OK to ignore it until someone finds a solution.

Patch is coming...

Change 345172 had a related patch set uploaded (by Dalba):
[pywikibot/core@master] test_https_cert_error: Ignore other warnings

https://gerrit.wikimedia.org/r/345172

Change 345172 merged by jenkins-bot:
[pywikibot/core@master] test_https_cert_error: Ignore other warnings

https://gerrit.wikimedia.org/r/345172