Page MenuHomePhabricator

Flake8 fails on labs-tools-heritage, complaining about py3 stuff
Closed, ResolvedPublic

Description

This was first noticed in July 2018.

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/flake8/checker.py", line 666, in _run_checks_from_queue
    checker.run_checks(results_queue, statistics_queue)
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/flake8/checker.py", line 606, in run_checks
    self.run_ast_checks()
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/flake8/checker.py", line 517, in run_ast_checks
    for (line_number, offset, text, check) in runner:
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/unicode_string_literal.py", line 38, in visit_tree
    for error in self.visit_tree(child):
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/unicode_string_literal.py", line 38, in visit_tree
    for error in self.visit_tree(child):
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/unicode_string_literal.py", line 38, in visit_tree
    for error in self.visit_tree(child):
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/unicode_string_literal.py", line 35, in visit_tree
    for error in self.visit_node(node):
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/unicode_string_literal.py", line 51, in visit_node
    if not isinstance(node.func.value.s, unicode):
NameError: name 'unicode' is not defined

But also:

./tests/test_monuments_config_validation.py:83:30: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:84:38: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:85:31: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:86:34: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:88:32: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:91:34: F821 undefined name 'unicode'
./bin/deploy_message_from_git_log.py:57:31: E999 SyntaxError: invalid syntax
./erfgoedbot/populate_image_table.py:107:20: F821 undefined name 'unicode'
./erfgoedbot/populate_image_table.py:132:22: F821 undefined name 'unicode'
./erfgoedbot/common.py:114:21: F821 undefined name 'unicode'
./erfgoedbot/common.py:161:24: F821 undefined name 'unicode'
./erfgoedbot/common.py:162:26: F821 undefined name 'unicode'
./erfgoedbot/common.py:185:33: F821 undefined name 'unicode'
./erfgoedbot/common.py:186:20: F821 undefined name 'unicode'
./erfgoedbot/add_object_location_monuments.py:67:51: F821 undefined name 'unicode'
./erfgoedbot/add_object_location_monuments.py:69:30: F821 undefined name 'unicode'
./erfgoedbot/monument_tables.py:112:68: F821 undefined name 'unicode'
./erfgoedbot/add_coord_to_articles.py:203:20: E999 SyntaxError: invalid syntax
./erfgoedbot/statistics_table.py:75:43: F821 undefined name 'unicode'
./erfgoedbot/top_streets.py:120:22: E999 SyntaxError: invalid syntax
./erfgoedbot/update_database.py:674:25: E999 SyntaxError: invalid syntax

Related Objects

Mentioned In
T216364: Upgrade heritage to Strech
Mentioned Here
T216364: Upgrade heritage to Strech
rTHER0aac903a031e: Make cursor.execute format string
rTHER1b3db24e7f1b: Localisation updates from https://translatewiki.net.
rTHERcca097802093: Capture task id even if first line
rTHER88a754a69e6e: Rename methods for clarity in database_statistics
rTHERd93b940702f9: Localisation updates from https://translatewiki.net.
rTHERa571c0390119: Localisation updates from https://translatewiki.net.
rTHER1839191862a4: Localisation updates from https://translatewiki.net.
rTHER0591ddce39ee: Localisation updates from https://translatewiki.net.
rTHER672896143ac8: Localisation updates from https://translatewiki.net.
rTHERf1e82c5aaf4e: Localisation updates from https://translatewiki.net.
rTHER222780b7ae39: Localisation updates from https://translatewiki.net.
rTHER30813e8e0183: Localisation updates from https://translatewiki.net.
rTHERe1a242293f91: Localisation updates from https://translatewiki.net.
rTHER94ddd2d0accc: Localisation updates from https://translatewiki.net.
rTHERc6f7694e5726: Localisation updates from https://translatewiki.net.
rTHERe019c8add8ab: Localisation updates from https://translatewiki.net.
rTHER089ff8788dc2: Localisation updates from https://translatewiki.net.
rTHER3efdc7e11c28: Localisation updates from https://translatewiki.net.
rTHER033ff2757809: Localisation updates from https://translatewiki.net.
rTHER2e35e271ee5a: Localisation updates from https://translatewiki.net.
rTHERba68f7b1de07: Localisation updates from https://translatewiki.net.
rTHER975f530b7b12: Localisation updates from https://translatewiki.net.
rTHER58c8f34c08fe: Localisation updates from https://translatewiki.net.
rTHER8f30ee789c76: Set basepython to Python 2.7 in tox.ini
rTHERa39179fbbd3f: Add Wikidata-Item (as WD-Item) for at_de
rTHER00d50e03fee1: Base Python bot Dockerfile on debian:jessie instead of `ubuntu`
rTHER0c0108e4071d: Remove the `security` extra for `requests` and `cryptography`
rTHERafb11ea75917: Bump php-sqllint in composer.json
rTHERec66b4f02552: Remove redundant dev-dependencies from Pipfile
rTHER8d23f1e0ffc4: Extract more constants from bin script into defaults.sh
T222512: CI: upgrade tox or allow to override its version per-project
T224405: Migrate heritage to py3

Event Timeline

JeanFred created this task.

Prevents merges

Change 532136 had a related patch set uploaded (by Jean-Frédéric; owner: Jean-Frédéric):
[labs/tools/heritage@master] Remove flake8 extension unicode-string-literal from tox file

https://gerrit.wikimedia.org/r/532136

JeanFred renamed this task from Flake8 fails on labs-tools-heritage, complaining about NameError: name 'unicode' is not defined to Flake8 fails on labs-tools-heritage, complaining about py3 stuff.Aug 25 2019, 8:27 AM
JeanFred updated the task description. (Show Details)

The unicode-string-litteral issue was the top of the iceberg: now that flake8 runs, it complains about a lot of violations:

./tests/test_monuments_config_validation.py:83:30: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:84:38: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:85:31: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:86:34: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:88:32: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:91:34: F821 undefined name 'unicode'
./bin/deploy_message_from_git_log.py:57:31: E999 SyntaxError: invalid syntax
./erfgoedbot/populate_image_table.py:107:20: F821 undefined name 'unicode'
./erfgoedbot/populate_image_table.py:132:22: F821 undefined name 'unicode'
./erfgoedbot/common.py:114:21: F821 undefined name 'unicode'
./erfgoedbot/common.py:161:24: F821 undefined name 'unicode'
./erfgoedbot/common.py:162:26: F821 undefined name 'unicode'
./erfgoedbot/common.py:185:33: F821 undefined name 'unicode'
./erfgoedbot/common.py:186:20: F821 undefined name 'unicode'
./erfgoedbot/add_object_location_monuments.py:67:51: F821 undefined name 'unicode'
./erfgoedbot/add_object_location_monuments.py:69:30: F821 undefined name 'unicode'
./erfgoedbot/monument_tables.py:112:68: F821 undefined name 'unicode'
./erfgoedbot/add_coord_to_articles.py:203:20: E999 SyntaxError: invalid syntax
./erfgoedbot/statistics_table.py:75:43: F821 undefined name 'unicode'
./erfgoedbot/top_streets.py:120:22: E999 SyntaxError: invalid syntax
./erfgoedbot/update_database.py:674:25: E999 SyntaxError: invalid syntax

I dived through the logs of rCICF integration-config and did not find anything obvious along the lines of “Switch to Py3” − maybe this is related to T222512: CI: upgrade tox or allow to override its version per-project?

Based on the above task we should be able to just use an older tox image?

Although:

$ docker run --rm --entrypoint=bash -it docker-registry.wikimedia.org/releng/tox:0.4.0

$ nobody@676488683da1:/src$ tox --version
3.10.0 imported from /usr/local/lib/python3.5/dist-packages/tox/__init__.py

$ nobody@676488683da1:/src$ python --version
Python 2.7.13

So even though 2.7 is still the default Python, tox uses Py3.
Same goes for releng:0.3.1:

$ nobody@4350aafb13ba:/src$ tox --version
2.9.1 imported from /usr/local/lib/python3.5/dist-packages/tox/__init__.py

So, sure, we should T224405: Migrate heritage to py3 but I’d rather not have to do that right now. Would there be a way to go back to a py2-based tox?

cc @hashar

Change 532245 had a related patch set uploaded (by Jean-Frédéric; owner: Jean-Frédéric):
[labs/tools/heritage@master] Set basepython to Python 2.7 in tox.ini

https://gerrit.wikimedia.org/r/532245

Ah, setting basepython is good enough as it seems :)

Change 532245 merged by jenkins-bot:
[labs/tools/heritage@master] Set basepython to Python 2.7 in tox.ini

https://gerrit.wikimedia.org/r/532245

Change 532136 abandoned by Jean-Frédéric:
Remove flake8 extension unicode-string-literal from tox file

Reason:
Abandoned in favor of https://gerrit.wikimedia.org/r/#/c/labs/tools/heritage/ /532245/

https://gerrit.wikimedia.org/r/532136