Page MenuHomePhabricator

No module named 'pymysql' -- python3-mysql installed on toolforge bastions but not on exec nodes
Closed, ResolvedPublic

Description

Hello, i get ImportError No module named 'pymysql', I get this error just in crontab, but when i run the code runtime the code work well. this problem in Toolserver.

Event Timeline

Andrew renamed this task from No module named 'pymysql' to No module named 'pymysql' -- python3-mysql installed on toolforge bastions but not on exec nodes.Aug 29 2017, 2:01 PM

The preferred solution for Python 2/3 packages is for the tool to use a virtual environment, but if this is installed on bastions via apt-get and not on exec nodes that is a problem we should fix.

$ clush -b -w @tools-bastion -- 'dpkg -l|grep python3-mysql.connector'
---------------
tools-bastion-[02-03,05].tools.eqiad.wmflabs (3)
---------------
ii  python3-mysql.connector                                     1.1.6-1                                    all          pure Python implementation of MySQL Client/Server protocol (Python3)
$ clush -b -w @tools-exec -- 'dpkg -l|grep python3-mysql.connector'
---------------
tools-exec-[1401-1442].tools.eqiad.wmflabs,tools-exec-gift-trusty-01.tools.eqiad.wmflabs (43)
---------------
ii  python3-mysql.connector                                     1.1.6-1                                    all          pure Python implementation of MySQL Client/Server protocol (Python3)

It seems to be installed on all of the bastions and the exec nodes in the job grid.


@Jar can you provide the exact crontab entry you are using and tell us which tool is having this problem?

@bd808

Traceback (most recent call last):
  File "core/pwb.py", line 262, in <module>
    if not main():
  File "core/pwb.py", line 255, in main
    run_python_file(filename, [filename] + args, argvu, file_package)
  File "core/pwb.py", line 121, in run_python_file
    main_mod.__dict__)
  File "core/scripts/x.py", line 5, in <module>
    import pymysql as mysqldb
ImportError: No module named 'pymysql'
CRITICAL: Closing network session.

i use PuTTY.

Ok. I checked for the wrong Python mysql library the first time. Here are the results for the python3-pymsql library:

$ clush -b -w @tools-bastion -- 'dpkg -l|grep python3-pymysql'
---------------
tools-bastion-03.tools.eqiad.wmflabs
---------------
ii  python3-pymysql                                             0.7.2-1                                    all          Pure-Python MySQL Driver - Python 3.x
clush: tools-bastion-[02,05].tools.eqiad.wmflabs (2): exited with exit code 1
$ clush -b -w @tools-exec -- 'dpkg -l|grep python3-pymysql'
clush: tools-exec-[1401-1442].tools.eqiad.wmflabs,tools-exec-gift-trusty-01.tools.eqiad.wmflabs (43): exited with exit code 1

This shows that the python3-pymsql apt package is installed only on tools-bastion-03.tools.eqiad.wmflabs (also known as tools-login.wmflabs.org). I can not find any mention of the python3-pymsql library in our Puppet config. I also can't find any dependency chain that would have caused it to be installed as a dependency of another package:

$ apt-cache rdepends --recurse python3-pymysql
python3-pymysql
Reverse Depends:

@Jar I think the best thing for you to do would be to switch from using pymsql which seems to only accidentally be installed on this one server to using mysql.connector which is installed everywhere in the job grid. See https://dev.mysql.com/doc/connector-python/en/connector-python-examples.html for usage instructions for the available library.

@bd808 I now using mysql.connector, Thank you.

valhallasw claimed this task.

I removed the package from tools-bastion-03 to prevent future confusion.

$ sudo apt-get purge python3-pymysql
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-image-3.13.0-117-generic linux-image-3.13.0-119-generic
  linux-image-3.13.0-121-generic linux-image-3.13.0-123-generic
  linux-image-3.13.0-125-generic linux-image-3.13.0-126-generic
  linux-tools-3.13.0-123 linux-tools-3.13.0-123-generic linux-tools-3.13.0-125
  linux-tools-3.13.0-125-generic linux-tools-3.13.0-126
  linux-tools-3.13.0-126-generic
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  python3-pymysql*
0 upgraded, 0 newly installed, 1 to remove and 53 not upgraded.
After this operation, 302 kB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 194900 files and directories currently installed.)
Removing python3-pymysql (0.7.2-1) ...