I think when a dry test fails to set up that subsequent test also fail when they create a site instance and you get errors like:
====================================================================== ERROR: setUpClass (tests.deletionbot_tests.TestDeletionBotWrite) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/xzise/Programms/pywikibot/core/tests/aspects.py", line 578, in setUpClass super(RequireUserMixin, cls).setUpClass() File "/home/xzise/Programms/pywikibot/core/tests/aspects.py", line 531, in setUpClass super(SiteWriteMixin, cls).setUpClass() File "/home/xzise/Programms/pywikibot/core/tests/aspects.py", line 456, in setUpClass super(CheckHostnameMixin, cls).setUpClass() File "/home/xzise/Programms/pywikibot/core/tests/aspects.py", line 830, in setUpClass interface=interface) File "/home/xzise/Programms/pywikibot/core/pywikibot/__init__.py", line 604, in Site _sites[key] = interface(code=code, fam=fam, user=user, sysop=sysop) File "/home/xzise/Programms/pywikibot/core/tests/aspects.py", line 377, in __init__ % (fam, code)) pywikibot.exceptions.SiteDefinitionError: Loading site test:test during dry test not permitted
I need to further study that but when I did one test manually it worked fine but as soon as I did it in the test suite it errored with a similar message as above. Now that I've fixed that one place (tests.dry_api_tests.DryCachedRequestTests.setUp to be exact) the tests don't fail anymore.