If I read the output correctly, pyflakes is parsing the entire python standard library, which of course does not follow pyflakes standards....

It happened a few times before the big crash, and then didnt happen again for a few days
It should be skipping hidden files in`.tox` by default.

T117570 is unrelated; it is about improving the results of the flake8 rule

I havent been able to reproduce this locally.

The logs on https://gerrit.wikimedia.org/r/#/c/294907/ show it is definitely entering the .tox directory.

I tested ffind with tox 1.9.2 at https://travis-ci.org/jayvdb/ffind/builds/138550696 , and ffind 1.0.2 with tox on pywikibot-core at
https://travis-ci.org/jayvdb/pywikibot-core/builds/138559286

Only the desired files, and not .tox files, are shown, and the pyflakes command succeeds.

So I suspect there is some obscure change to the CI nodes on June 1-2.

If it cant be fixed soon, we test revert to using flake8 instead of ffind to do the pyflakes checks on multiple environments.

We just got lucky.
https://bitbucket.org/hpk42/tox/commits/b609deef30f3 allows find to be a replacement for ffind. It isnt cross platform by default, but gnu find does exist for windows.

So alternatives on https://gerrit.wikimedia.org/r/#/c/260909/ were two incomplete/unusable solutions:

• pyflakes . (PS1..) : included .tox
• pyflakes [a-z]* (PS5) - depends on shell globbing, which tox and pyflakes does not do.

And two usable solutions:

• find . -name '*.py' -and -not -name '.*' | xargs pyflakes (PS6)
• ffind .. (PS7)

Using find and xargs invokes non-Windows GNU tools, but there should be python equivalents, and that is how I found ffind which worked efficiently after a bit of improvement.

fwiw, the find command should be find . -name '*.py' -a '!' -path '*/.*' -a '!' -name 'user-config.py'

Other solutions on the xargs area would be

• https://pypi.python.org/pypi/disttask , but that doesnt re-use existing pyflakes runtimes, so is very quick but not very efficient.
• https://code.google.com/archive/p/ypipe/ is the opposite: re-using processes for multiple files, but not using concurrency of processes. (and thus is similar to ffind)

https://github.com/drmikehenry/findx is a very nice wrapper of find and xargs , and doesnt require {} , which currently tox mishandles (but will be fixed in the next version). findx is very likely to be Unix and even GNU specific though. But it works (see PS17 of https://gerrit.wikimedia.org/r/#/c/294907 ), with a little modification, and gives us a quick fix for this bug.

Change 294907 had a related patch set uploaded (by John Vandenberg):
Replace ffind with findx

https://gerrit.wikimedia.org/r/294907

The findx customisation has been PR'd Upstream : https://github.com/drmikehenry/findx/pull/3

In T137628#2395939, @jayvdb wrote:

The findx customisation has been PR'd Upstream : https://github.com/drmikehenry/findx/pull/3

Note the maintainer is active and helpful. Seems like waiting for a brief period will bring a better long term solution.

Change 294907 merged by jenkins-bot:
Replace ffind with findx

https://gerrit.wikimedia.org/r/294907

Change 296190 had a related patch set uploaded (by John Vandenberg):
Use released findx 0.9.9

https://gerrit.wikimedia.org/r/296190

Change 296190 merged by jenkins-bot:
Use released findx 0.9.9

https://gerrit.wikimedia.org/r/296190