Page MenuHomePhabricator
Create Task
Maniphest T185598

property-suggester-scripts master can't be installed on tool forge
Closed, ResolvedPublic
Actions

Description

The latest version of property-suggester-scripts can no longer be installed on tool forge:

hoo@tools-bastion-03:~/PropertySuggester-Python$ python setup.py install --user 
…
Running setuptools-38.4.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-mBWyEZ/setuptools-38.4.0/egg-dist-tmp-hP75zU
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'long_description_content_type'
  warnings.warn(msg)
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'project_urls'
  warnings.warn(msg)
Traceback (most recent call last):
  File "setup.py", line 32, in <module>
    "Operating System :: OS Independent"
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 73, in run
    self.do_egg_install()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/install.py", line 96, in do_egg_install
    cmd.run()
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 381, in run
    self.easy_install(spec, not self.no_deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 597, in easy_install
    return self.install_item(None, spec, tmpdir, deps, True)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 648, in install_item
    self.process_distribution(spec, dist, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 694, in process_distribution
    [requirement], self.local_index, self.easy_install
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 620, in resolve
    dist = best[req.key] = env.best_match(req, ws, installer)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 858, in best_match
    return self.obtain(req, installer) # try and download/install
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 870, in obtain
    return installer(requirement)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 616, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 646, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 834, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1040, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/command/easy_install.py", line 1025, in run_setup
    run_setup(setup_script, args)
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 50, in run_setup
    lambda: execfile(
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 100, in run
    return func()
  File "/usr/lib/python2.7/dist-packages/setuptools/sandbox.py", line 52, in <lambda>
    {'__file__':setup_script, '__name__':'__main__'}
  File "setup.py", line 190, in <module>
    
  File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
    _setup_distribution = dist = klass(attrs)
  File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 243, in __init__
    _Distribution.__init__(self,attrs)
  File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
    self.finalize_options()
  File "/usr/lib/python2.7/dist-packages/setuptools/dist.py", line 277, in finalize_options
    ep.load()(self, ep.name, value)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2093, in load
    raise ImportError("%r has no %r attribute" % (entry,attr))
ImportError: <module 'setuptools.dist' from '/usr/lib/python2.7/dist-packages/setuptools/dist.pyc'> has no 'check_specifier' attribute

(it similarly fails with python3)

Due to this I'm doing the run on 80b3523778e7c589be63edf6f400c3d049a114fc now.

In order to fix this, we need to either remove the build step (can we?) or get a custom VM with a more up to date Python environment.

Details

SubjectRepoBranchLines +/-
Include analyzer.ini when installing the librarywikibase/property-suggester-scriptsmaster+2 -0
Customize query in gerrit

Related Objects

Event Timeline

hoo created this task.Jan 23 2018, 7:54 PM
Restricted Application added a project: User-Ladsgroup. · View Herald TranscriptJan 23 2018, 7:54 PM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
Ladsgroup added a comment.Jan 23 2018, 7:56 PM

I haven't changed anything recently, have you tried virtualenv?

Ladsgroup added a comment.Jan 23 2018, 7:57 PM

let me try virtualenv and let you know

hoo added a comment.Jan 23 2018, 7:58 PM
In T185598#3920782, @Ladsgroup wrote:

I haven't changed anything recently, have you tried virtualenv?

I haven't… probably I should.

This was updated on Fri Nov 17 12:33:06 2017 +0100, apparently I didn't update my tool forge checkout of this since.

Ladsgroup added a comment.Jan 23 2018, 8:04 PM

pip version in toolforge is old and that causes this problem: https://stackoverflow.com/questions/41141657/importerror-module-setuptools-dist-has-no-check-specifier-attribute
That can be easily bypassed using virtualenv: P6627

Ladsgroup added a comment.Jan 23 2018, 8:07 PM

Overall, I'm a huge fan of getting the whole thing more productionized, like getting a dedicated vm, puppetize it, etc.

hoo added a comment.Feb 21 2018, 4:17 PM

I installed it using a virtualenv now… but I then manually had to: cp propertysuggester/analyzer/analyzer.ini lib/python3.4/site-packages/propertysuggester-3.0.0-py3.4.egg/propertysuggester/analyzer/ :/

Ladsgroup added a comment.Feb 21 2018, 6:10 PM

Oh, That's easily fixable. Let me get that worked out.

gerritbot subscribed.Feb 21 2018, 9:56 PM

Change 413260 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[wikibase/property-suggester-scripts@master] Include analyzer.ini when installing the library

https://gerrit.wikimedia.org/r/413260

gerritbot added a project: Patch-For-Review.Feb 21 2018, 9:56 PM
gerritbot added a comment.Feb 22 2018, 3:19 PM

Change 413260 merged by jenkins-bot:
[wikibase/property-suggester-scripts@master] Include analyzer.ini when installing the library

https://gerrit.wikimedia.org/r/413260

hoo added a comment.Feb 23 2018, 2:06 PM

Now I encountered:

+ python3 ./build/lib/scripts/analyzer.py /home/hoo/dumpconvert.csv /home/hoo/analyzed-out
INFO:root:computing table
INFO:root:processed 100000 entities
…
INFO:root:processed 42200000 entities
INFO:root:writing csv
Traceback (most recent call last):
  File "./build/lib/scripts/analyzer.py", line 28, in <module>
    CsvGenerator.create_pair_csv(rules, args.output)
  File "/mnt/nfs/labstore-secondary-tools-home/hoo/wikibase-property-suggester-scripts/lib/python3.4/site-packages/propertysuggester-3.0.0-py3.4.egg/propertysuggester/analyzer/CsvGenerator.py", line 15, in create_pair_csv
    "context"))
TypeError: 'str' does not support the buffer interface
hoo added a comment.Feb 23 2018, 2:53 PM

I tried addressing this by changing the output in analyzer.py to:

parser.add_argument("output",
                    help="The CSV output file (default=sys.stdout)",
                    default=sys.stdout, nargs='?',
                    type=CompressedFileType('wt'))
hoo closed this task as Resolved.Apr 10 2018, 11:52 AM
hoo removed a project: Patch-For-Review.
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL