Page MenuHomePhabricator
Create Task
Maniphest T231152

Flake8 fails on labs-tools-heritage, complaining about py3 stuff
Closed, ResolvedPublic
Actions

Description

This was first noticed in July 2018.

Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/flake8/checker.py", line 666, in _run_checks_from_queue
    checker.run_checks(results_queue, statistics_queue)
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/flake8/checker.py", line 606, in run_checks
    self.run_ast_checks()
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/flake8/checker.py", line 517, in run_ast_checks
    for (line_number, offset, text, check) in runner:
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/unicode_string_literal.py", line 38, in visit_tree
    for error in self.visit_tree(child):
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/unicode_string_literal.py", line 38, in visit_tree
    for error in self.visit_tree(child):
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/unicode_string_literal.py", line 38, in visit_tree
    for error in self.visit_tree(child):
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/unicode_string_literal.py", line 35, in visit_tree
    for error in self.visit_node(node):
  File "labs-tools-heritage/.tox/flake8/lib/python3.6/site-packages/unicode_string_literal.py", line 51, in visit_node
    if not isinstance(node.func.value.s, unicode):
NameError: name 'unicode' is not defined

But also:

./tests/test_monuments_config_validation.py:83:30: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:84:38: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:85:31: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:86:34: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:88:32: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:91:34: F821 undefined name 'unicode'
./bin/deploy_message_from_git_log.py:57:31: E999 SyntaxError: invalid syntax
./erfgoedbot/populate_image_table.py:107:20: F821 undefined name 'unicode'
./erfgoedbot/populate_image_table.py:132:22: F821 undefined name 'unicode'
./erfgoedbot/common.py:114:21: F821 undefined name 'unicode'
./erfgoedbot/common.py:161:24: F821 undefined name 'unicode'
./erfgoedbot/common.py:162:26: F821 undefined name 'unicode'
./erfgoedbot/common.py:185:33: F821 undefined name 'unicode'
./erfgoedbot/common.py:186:20: F821 undefined name 'unicode'
./erfgoedbot/add_object_location_monuments.py:67:51: F821 undefined name 'unicode'
./erfgoedbot/add_object_location_monuments.py:69:30: F821 undefined name 'unicode'
./erfgoedbot/monument_tables.py:112:68: F821 undefined name 'unicode'
./erfgoedbot/add_coord_to_articles.py:203:20: E999 SyntaxError: invalid syntax
./erfgoedbot/statistics_table.py:75:43: F821 undefined name 'unicode'
./erfgoedbot/top_streets.py:120:22: E999 SyntaxError: invalid syntax
./erfgoedbot/update_database.py:674:25: E999 SyntaxError: invalid syntax

Details

Related Changes in Gerrit:
SubjectRepoBranchLines +/-
Remove flake8 extension unicode-string-literal from tox filelabs/tools/heritagemaster+1 -3
Set basepython to Python 2.7 in tox.inilabs/tools/heritagemaster+1 -0
Customize query in gerrit

Related Objects

Mentioned In
T216364: Upgrade heritage to Strech
Mentioned Here
T216364: Upgrade heritage to Strech
rTHER0aac903a031e: Make cursor.execute format string
rTHER1b3db24e7f1b: Localisation updates from https://translatewiki.net.
rTHERcca097802093: Capture task id even if first line
rTHER88a754a69e6e: Rename methods for clarity in database_statistics
rTHERd93b940702f9: Localisation updates from https://translatewiki.net.
rTHERa571c0390119: Localisation updates from https://translatewiki.net.
rTHER1839191862a4: Localisation updates from https://translatewiki.net.
rTHER0591ddce39ee: Localisation updates from https://translatewiki.net.
rTHER672896143ac8: Localisation updates from https://translatewiki.net.
rTHERf1e82c5aaf4e: Localisation updates from https://translatewiki.net.
rTHER222780b7ae39: Localisation updates from https://translatewiki.net.
rTHER30813e8e0183: Localisation updates from https://translatewiki.net.
rTHERe1a242293f91: Localisation updates from https://translatewiki.net.
rTHER94ddd2d0accc: Localisation updates from https://translatewiki.net.
rTHERc6f7694e5726: Localisation updates from https://translatewiki.net.
rTHERe019c8add8ab: Localisation updates from https://translatewiki.net.
rTHER089ff8788dc2: Localisation updates from https://translatewiki.net.
rTHER3efdc7e11c28: Localisation updates from https://translatewiki.net.
rTHER033ff2757809: Localisation updates from https://translatewiki.net.
rTHER2e35e271ee5a: Localisation updates from https://translatewiki.net.
rTHERba68f7b1de07: Localisation updates from https://translatewiki.net.
rTHER975f530b7b12: Localisation updates from https://translatewiki.net.
rTHER58c8f34c08fe: Localisation updates from https://translatewiki.net.
rTHER8f30ee789c76: Set basepython to Python 2.7 in tox.ini
rTHERa39179fbbd3f: Add Wikidata-Item (as WD-Item) for at_de
rTHER00d50e03fee1: Base Python bot Dockerfile on debian:jessie instead of `ubuntu`
rTHER0c0108e4071d: Remove the `security` extra for `requests` and `cryptography`
rTHERafb11ea75917: Bump php-sqllint in composer.json
rTHERec66b4f02552: Remove redundant dev-dependencies from Pipfile
rTHER8d23f1e0ffc4: Extract more constants from bin script into defaults.sh
T222512: CI: upgrade tox or allow to override its version per-project
T224405: Migrate heritage to py3

Event Timeline

JeanFred created this task.Aug 25 2019, 8:06 AM
JeanFred triaged this task as High priority.

Prevents merges

gerritbot added a comment.Aug 25 2019, 8:12 AM

Change 532136 had a related patch set uploaded (by Jean-Frédéric; owner: Jean-Frédéric):
[labs/tools/heritage@master] Remove flake8 extension unicode-string-literal from tox file

https://gerrit.wikimedia.org/r/532136

gerritbot added a project: Patch-For-Review.Aug 25 2019, 8:12 AM
JeanFred renamed this task from Flake8 fails on labs-tools-heritage, complaining about NameError: name 'unicode' is not defined to Flake8 fails on labs-tools-heritage, complaining about py3 stuff.Aug 25 2019, 8:27 AM
JeanFred updated the task description. (Show Details)
JeanFred added a subscriber: hashar.Aug 25 2019, 8:42 AM

The unicode-string-litteral issue was the top of the iceberg: now that flake8 runs, it complains about a lot of violations:

./tests/test_monuments_config_validation.py:83:30: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:84:38: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:85:31: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:86:34: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:88:32: F821 undefined name 'unicode'
./tests/test_monuments_config_validation.py:91:34: F821 undefined name 'unicode'
./bin/deploy_message_from_git_log.py:57:31: E999 SyntaxError: invalid syntax
./erfgoedbot/populate_image_table.py:107:20: F821 undefined name 'unicode'
./erfgoedbot/populate_image_table.py:132:22: F821 undefined name 'unicode'
./erfgoedbot/common.py:114:21: F821 undefined name 'unicode'
./erfgoedbot/common.py:161:24: F821 undefined name 'unicode'
./erfgoedbot/common.py:162:26: F821 undefined name 'unicode'
./erfgoedbot/common.py:185:33: F821 undefined name 'unicode'
./erfgoedbot/common.py:186:20: F821 undefined name 'unicode'
./erfgoedbot/add_object_location_monuments.py:67:51: F821 undefined name 'unicode'
./erfgoedbot/add_object_location_monuments.py:69:30: F821 undefined name 'unicode'
./erfgoedbot/monument_tables.py:112:68: F821 undefined name 'unicode'
./erfgoedbot/add_coord_to_articles.py:203:20: E999 SyntaxError: invalid syntax
./erfgoedbot/statistics_table.py:75:43: F821 undefined name 'unicode'
./erfgoedbot/top_streets.py:120:22: E999 SyntaxError: invalid syntax
./erfgoedbot/update_database.py:674:25: E999 SyntaxError: invalid syntax

I dived through the logs of rCICF integration-config and did not find anything obvious along the lines of “Switch to Py3” − maybe this is related to T222512: CI: upgrade tox or allow to override its version per-project?

Based on the above task we should be able to just use an older tox image?

Although:

$ docker run --rm --entrypoint=bash -it docker-registry.wikimedia.org/releng/tox:0.4.0

$ nobody@676488683da1:/src$ tox --version
3.10.0 imported from /usr/local/lib/python3.5/dist-packages/tox/__init__.py

$ nobody@676488683da1:/src$ python --version
Python 2.7.13

So even though 2.7 is still the default Python, tox uses Py3.
Same goes for releng:0.3.1:

$ nobody@4350aafb13ba:/src$ tox --version
2.9.1 imported from /usr/local/lib/python3.5/dist-packages/tox/__init__.py

So, sure, we should T224405: Migrate heritage to py3 but I’d rather not have to do that right now. Would there be a way to go back to a py2-based tox?

cc @hashar

gerritbot added a comment.Aug 25 2019, 3:11 PM

Change 532245 had a related patch set uploaded (by Jean-Frédéric; owner: Jean-Frédéric):
[labs/tools/heritage@master] Set basepython to Python 2.7 in tox.ini

https://gerrit.wikimedia.org/r/532245

JeanFred added a comment.Aug 25 2019, 3:20 PM

Ah, setting basepython is good enough as it seems :)

gerritbot added a comment.Aug 25 2019, 4:46 PM

Change 532245 merged by jenkins-bot:
[labs/tools/heritage@master] Set basepython to Python 2.7 in tox.ini

https://gerrit.wikimedia.org/r/532245

gerritbot added a comment.Aug 25 2019, 4:49 PM

Change 532136 abandoned by Jean-Frédéric:
Remove flake8 extension unicode-string-literal from tox file

Reason:
Abandoned in favor of https://gerrit.wikimedia.org/r/#/c/labs/tools/heritage/ /532245/

https://gerrit.wikimedia.org/r/532136

Maintenance_bot removed a project: Patch-For-Review.Aug 25 2019, 5:10 PM
JeanFred closed this task as Resolved.Aug 25 2019, 5:24 PM
Stashbot added a comment.Aug 26 2019, 9:09 PM

Mentioned in SAL (#wikimedia-cloud) [2019-08-26T21:09:25Z] <JeanFred> Deploy latest from Git master: 0aac903, 1b3db24, cca0978, 88a754a, d93b940, a571c03, 1839191, 0591ddc, 6728961, f1e82c5, 222780b, 30813e8, e1a2422, 94ddd2d, c6f7694, e019c8a, 089ff87, 3efdc7e, 033ff27, f0fdecb, 2e35e27, ba68f7b, 975f530, 58c8f34, 8f30ee7 (T231152), a39179f, 00d50e0 (T216364), 0c0108e, afb11ea, ec66b4f, 8d23f1e,

Stashbot mentioned this in T216364: Upgrade heritage to Strech.Aug 26 2019, 9:09 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits