Page MenuHomePhabricator

Error while checking binary files for python shebang
Closed, ResolvedPublic

Description

In https://gerrit.wikimedia.org/r/c/operations/puppet/+/516752 I was updating a binary file and CI failed with:

11:11:46 Removing json (2.1.0)
11:11:46 Removing puppet-syntax (2.4.1)
11:11:46 Bundle updated!
11:11:46 + export TOX_TESTENV_PASSENV=PY_COLORS
11:11:46 + TOX_TESTENV_PASSENV=PY_COLORS
11:11:46 + export PY_COLORS=1
11:11:46 + PY_COLORS=1
11:11:46 + export SPEC_OPTS=--tty
11:11:46 + SPEC_OPTS=--tty
11:11:46 + bundle exec rake test
11:11:46 + tee /srv/workspace/log/rake.log
11:11:47 sh: 1: file: not found
11:11:47 rake aborted!
11:11:47 ArgumentError: invalid byte sequence in UTF-8
11:11:47 /srv/workspace/puppet/rake_modules/taskgen.rb:279:in `block (2 levels) in setup_python_extensions'
11:11:47 /srv/workspace/puppet/rake_modules/taskgen.rb:272:in `block in setup_python_extensions'
11:11:47 Tasks: TOP => test => parallel => python_extensions
11:11:47 (See full trace by running task with --trace)

My hunch is that attempting to decode the binary file as utf8 fails, namely when searching for shebang: if shebang =~ /^#!.*python/ || mime_type == 'text/x-python'

Event Timeline

ArielGlenn triaged this task as Medium priority.Jun 14 2019, 7:12 AM

The code should test whether the first line read from the file is valid_encoding?, else its binary I guess?

The container lacks the file utility: sh: 1: file: not found :-\

Change 517007 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] docker: add file to operations-puppet container

https://gerrit.wikimedia.org/r/517007

Change 517007 merged by jenkins-bot:
[integration/config@master] docker: add file to operations-puppet container

https://gerrit.wikimedia.org/r/517007

Change 517009 had a related patch set uploaded (by Hashar; owner: Hashar):
[integration/config@master] Bump operations/puppet container

https://gerrit.wikimedia.org/r/517009

Change 517009 merged by jenkins-bot:
[integration/config@master] Bump operations/puppet container

https://gerrit.wikimedia.org/r/517009

Change 517390 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] python CI: skips tests for binary files

https://gerrit.wikimedia.org/r/517390

Change 517390 had a related patch set uploaded (by Jbond; owner: John Bond):
[operations/puppet@production] python CI: skips tests for binary files

https://gerrit.wikimedia.org/r/517390

Change 517390 merged by Jbond:
[operations/puppet@production] python CI: skips tests for binary files

https://gerrit.wikimedia.org/r/517390

jbond claimed this task.

I think that with my latest change and the one from Antonie this should be fixed, please reopen if you still see issues