Page MenuHomePhabricator

Stretch grid problem: python 'oursql' package fails to compile with MariaDB client libraries
Closed, ResolvedPublic

Description

Documentation at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Python updated to remove recommendation of using oursql, added a note about compilation problems, and link to this task. I also added a FAQ for the migration from Trusty to Stretch at https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation#Python_'oursql'_package_fails_to_compile.


While migrating spbot to the Stretch job grid I ran into a problem while installing oursql with pip (for python 2.7).

.-(~)-------------------------------------------------------------(tools.spbot@tools-sgebastion-06)-
`--> virtualenv venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /mnt/nfs/labstore-secondary-tools-project/spbot/venv/bin/python2
Also creating executable in /mnt/nfs/labstore-secondary-tools-project/spbot/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
.-(~)-------------------------------------------------------------(tools.spbot@tools-sgebastion-06)-
`--> source venv/bin/activate
.-(~)-------------------------------------------------------------(tools.spbot@tools-sgebastion-06)-
`--> pip install --upgrade pip
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Requirement already up-to-date: pip in ./venv/lib/python2.7/site-packages (19.0.2)
.-(~)-------------------------------------------------------------(tools.spbot@tools-sgebastion-06)-
`--> pip install oursql
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting oursql
  Using cached https://files.pythonhosted.org/packages/8c/88/9f53a314a2af6f56c0a1249c5673ee384b85dc791bac5c1228772ced3502/oursql-0.9.3.2.tar.gz
Building wheels for collected packages: oursql
  Building wheel for oursql (setup.py) ... error
  Complete output from command /mnt/nfs/labstore-secondary-tools-project/spbot/venv/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-reczzD/oursql/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-73iHZ5 --python-tag cp27:
  cython not found, using previously-cython'd .c file.
  running bdist_wheel
  running build
  running build_ext
  building 'oursql' extension
  mysql_config --cflags
  creating build
  creating build/temp.linux-x86_64-2.7
  creating build/temp.linux-x86_64-2.7/oursqlx
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c oursqlx/oursql.c -o build/temp.linux-x86_64-2.7/oursqlx/oursql.o -I/usr/include/mysql
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c oursqlx/compat.c -o build/temp.linux-x86_64-2.7/oursqlx/compat.o -I/usr/include/mysql
  In file included from oursqlx/compat.c:55:0:
  oursqlx/_exceptions.c: In function ‘_oursqlx_exc_from_errno’:
  oursqlx/_exceptions.c:621:50: error: ‘errmsg_section_start’ undeclared (first use in this function)
                 for(unsigned int i = 0; i < sizeof(errmsg_section_start)/sizeof(int); ++i) {
                                                    ^~~~~~~~~~~~~~~~~~~~
  oursqlx/_exceptions.c:621:50: note: each undeclared identifier is reported only once for each function it appears in
  oursqlx/_exceptions.c:623:53: error: ‘errmsg_section_size’ undeclared (first use in this function)
                   int max = errmsg_section_start[i] + errmsg_section_size[i] - 1;
                                                       ^~~~~~~~~~~~~~~~~~~
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for oursql
  Running setup.py clean for oursql
Failed to build oursql
Installing collected packages: oursql
  Running setup.py install for oursql ... error
    Complete output from command /mnt/nfs/labstore-secondary-tools-project/spbot/venv/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-reczzD/oursql/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-lOJZZC/install-record.txt --single-version-externally-managed --compile --install-headers /mnt/nfs/labstore-secondary-tools-project/spbot/venv/include/site/python2.7/oursql:
    cython not found, using previously-cython'd .c file.
    running install
    running build
    running build_ext
    building 'oursql' extension
    mysql_config --cflags
    creating build
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/oursqlx
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c oursqlx/oursql.c -o build/temp.linux-x86_64-2.7/oursqlx/oursql.o -I/usr/include/mysql
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-2.7.13=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c oursqlx/compat.c -o build/temp.linux-x86_64-2.7/oursqlx/compat.o -I/usr/include/mysql
    In file included from oursqlx/compat.c:55:0:
    oursqlx/_exceptions.c: In function ‘_oursqlx_exc_from_errno’:
    oursqlx/_exceptions.c:621:50: error: ‘errmsg_section_start’ undeclared (first use in this function)
                   for(unsigned int i = 0; i < sizeof(errmsg_section_start)/sizeof(int); ++i) {
                                                      ^~~~~~~~~~~~~~~~~~~~
    oursqlx/_exceptions.c:621:50: note: each undeclared identifier is reported only once for each function it appears in
    oursqlx/_exceptions.c:623:53: error: ‘errmsg_section_size’ undeclared (first use in this function)
                     int max = errmsg_section_start[i] + errmsg_section_size[i] - 1;
                                                         ^~~~~~~~~~~~~~~~~~~
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/mnt/nfs/labstore-secondary-tools-project/spbot/venv/bin/python2 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-reczzD/oursql/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-lOJZZC/install-record.txt --single-version-externally-managed --compile --install-headers /mnt/nfs/labstore-secondary-tools-project/spbot/venv/include/site/python2.7/oursql" failed with error code 1 in /tmp/pip-install-reczzD/oursql/

Sorry for the full stack trace, but I'm not really familiar with venv and only following these steps.

Event Timeline

bd808 added a subscriber: bd808.

This seems to be https://github.com/python-oursql/oursql/issues/5 "Does not compile with MariaDB client libraries".

The fork at https://github.com/sqlobject/oursql has the patch needed to compile against the MariaDB client libraries:

$ virtualenv T215963
$ ./T215963/bin/pip install git+git://github.com/sqlobject/oursql.git@7a2970671d36cb8d79906cc13e8905e95bf5eba8#egg=oursql
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
Collecting oursql from git+git://github.com/sqlobject/oursql.git@7a2970671d36cb8d79906cc13e8905e95bf5eba8#egg=oursql
  Cloning git://github.com/sqlobject/oursql.git (to revision 7a2970671d36cb8d79906cc13e8905e95bf5eba8) to /tmp/pip-install-xqbCbm/oursql
Building wheels for collected packages: oursql
  Building wheel for oursql (setup.py) ... done
  Stored in directory: /tmp/pip-ephem-wheel-cache-Y2AT4f/wheels/d3/8d/09/1b0aa2514d5c159735dfaa711393de224e919b5f20d26e842c
Successfully built oursql
Installing collected packages: oursql
Successfully installed oursql-0.9.3.2
$ ./T215963/bin/pip freeze
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
oursql==0.9.3.2
pkg-resources==0.0.0

With no new releases since May 2016 and no merge activity in the repo since moving from launchpad to github it looks like oursql is an abandoned project. Migrating your code to pymysql might really be the best option in the longer term.

bd808 renamed this task from Stretch grid problem: pip install oursql fail to Stretch grid problem: python 'oursql' package fails to compile with MariaDB client libraries.Feb 20 2019, 10:17 PM
bd808 triaged this task as Medium priority.Feb 20 2019, 10:20 PM
bd808 changed the task status from Open to Stalled.Feb 20 2019, 10:40 PM
bd808 lowered the priority of this task from Medium to Lowest.

Documentation at https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Python updated to remove recommendation of using oursql, add a note about compilation problems, and link to this task. I also added a FAQ for the migration from Trusty to Stretch at https://wikitech.wikimedia.org/wiki/News/Toolforge_Trusty_deprecation#Python_'oursql'_package_fails_to_compile.

Legoktm assigned this task to bd808.
Legoktm added a subscriber: Legoktm.

I'm going to close this as resolved because the solution (using an oursql fork or switching to pymysql) is documented in the migration notes - there's not much more we can do on the Toolforge side unfortunately to support an abandoned library.