Page MenuHomePhabricator

Add libmysqlclient-dev to Toolforge Kubernetes Python3 runtime image
Closed, ResolvedPublic

Description

Subtask of T140110 :

libmysqlclient-dev is included in the Python2 runtime image but not in the Python3 image:

It would be great if that could be available in the Python3 image too.

Current workaround: use a pure python replacement: pip install pymysql and monkey-patch to emulate the official client:

import pymysql
pymysql.install_as_MySQLdb()

but that probably comes at a performance cost (and it would be cleaner without that).

Event Timeline

Pintoch created this task.Mar 21 2018, 12:08 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 21 2018, 12:08 PM

Which python3 package requires libmysqlclient-dev? MySQL-python don't support python3

Hmmm… I am not sure I understand your reaction… Are you opposing this addition, then? Should I keep monkey-patching my libraries to use the pure python alternative? In that case, why is libmysqlclient-dev included in the Python 2 docker image in the first place?

Are you opposing this addition, then?

No. What I am expressing is the frustration of seeing another wheel being reinvented. So far the majority of python3 tools are quite happy about using pymysql as the driver; but django don't like it, for some reason. And (probably naive question) why doesn't mysqlclient make pre-built binary wheels?

I'm not sure whether supporting mysqlclient by adding libmysqlclient-dev is a great choice, hence adding cloud-services-team for someone else to evaluate this.

Should I keep monkey-patching my libraries to use the pure python alternative?

That's one way, if the outcome of this task is declined.

In that case, why is libmysqlclient-dev included in the Python 2 docker image in the first place?

Because MySQL-python was the de-facto standard python2 mysql driver, and too many tools use that.

zhuyifei1999 renamed this task from Add libmysqlclient-dev to Toollabs Python3 runtime image to Add libmysqlclient-dev to Toolforge Kubernetes Python3 runtime image.Mar 22 2018, 6:36 PM
Chicocvenancio triaged this task as Normal priority.EditedMar 22 2018, 6:49 PM

I share @zhuyifei1999's frustration here. Adding a system package is possible, but will add to image build time and size, and this is something that should not be needed.

That being said, Django is a popular enough framework that I do think it should be supported "as recommended" or as close to that as possible. If no one strongly objects I will send a patch by tomorrow and build the new images once/if that is approved.

Okay - I don't really know much about the MySQL ecosystem to be honest so I cannot really judge (I use postgres when I can). I haven't run into performance issues with pymysql yet - if you are worried about the impact of this maybe we can wait and see if my app (https://tools.wmflabs.org/editgroups/) scales fine in this state first.

I must say that for a newcomer to tool forge it's very confusing to discover that django's default database driver can't be installed while following the documentation where django is even given as an example. If mysqlclient-dev can't be made available, please update the wiki page with a clear notice of how to get django running in these conditions.

@Chicocvenancio I agree with Yury - it makes it significantly harder to deploy Django projects.

It would be great to have a definitive decision on this issue:

  • either the library is accepted in the Docker image;
  • or it is refused - in that case we need to document how to monkey-patch Django. I can help with that, but I will only spend time on this if I can be sure that this is the way forward…

I have added a note to the documentation about this limitation. The django oauth tool tutorial probably needs updating, although it focuses on sqlite.

Yury_Bulka added a comment.EditedJul 24 2018, 12:38 PM

I share @zhuyifei1999's frustration here. Adding a system package is possible, but will add to image build time and size, and this is something that should not be needed.

That being said, Django is a popular enough framework that I do think it should be supported "as recommended" or as close to that as possible. If no one strongly objects I will send a patch by tomorrow and build the new images once/if that is approved.

I don't think that "image build time and size" is more expensive than the human-time the users need to spend to 1) discover this issue and 2) patch their code manually to workaround this problem.

Is adding a single dev package to the Dockerfile alongside the other ones really such a big and problematic decision that requiring each user to patch their code is a better alternative?

Change 447622 had a related patch set uploaded (by Zhuyifei1999; owner: Zhuyifei1999):
[operations/docker-images/toollabs-images@master] Add libmysqlclient-dev to python 3 base docker image

https://gerrit.wikimedia.org/r/447622

Change 447622 merged by jenkins-bot:
[operations/docker-images/toollabs-images@master] Add libmysqlclient-dev to python 3 base docker image

https://gerrit.wikimedia.org/r/447622

Mentioned in SAL (#wikimedia-cloud) [2018-07-27T04:52:41Z] <zhuyifei1999_> rebuilding python/base docker container T190274

zhuyifei1999 closed this task as Resolved.Jul 27 2018, 4:57 AM
zhuyifei1999 claimed this task.
tools.zhuyifei1999-test@tools-bastion-02:~$ webservice --backend kubernetes python shell
If you don't see a command prompt, try pressing enter.
tools.zhuyifei1999-test@interactive:~$ 
tools.zhuyifei1999-test@interactive:~$ mysql_config
bash: mysql_config: command not found
tools.zhuyifei1999-test@interactive:~$ logout
Session ended, resume using 'kubectl attach interactive -c interactive -i -t' command when the pod is running
Pod stopped. Session cannot be resumed.

[BUILD]

tools.zhuyifei1999-test@tools-bastion-02:~$ webservice --backend kubernetes python shell
If you don't see a command prompt, try pressing enter.
tools.zhuyifei1999-test@interactive:~$ 
tools.zhuyifei1999-test@interactive:~$ mysql_config
Usage: /usr/bin/mysql_config [OPTIONS]
Options:
        --cflags         [-I/usr/include/mysql -DBIG_JOINS=1  -fno-strict-aliasing    -g -DNDEBUG]
        --include        [-I/usr/include/mysql]
        --libs           [-L/usr/lib/x86_64-linux-gnu -lmysqlclient -lpthread -lz -lm -ldl]
        --libs_r         [-L/usr/lib/x86_64-linux-gnu -lmysqlclient_r -lpthread -lz -lm -ldl]
        --plugindir      [/usr/lib/mysql/plugin]
        --socket         [/var/run/mysqld/mysqld.sock]
        --port           [0]
        --version        [5.5.60]
        --libmysqld-libs [-L/usr/lib/x86_64-linux-gnu -lmysqld -lpthread -lz -lm -lwrap -lcrypt -ldl -laio]
        --variable=VAR   VAR is one of:
                pkgincludedir [/usr/include/mysql]
                pkglibdir     [/usr/lib/x86_64-linux-gnu]
                plugindir     [/usr/lib/mysql/plugin]
tools.zhuyifei1999-test@interactive:~$ logout
Session ended, resume using 'kubectl attach interactive -c interactive -i -t' command when the pod is running
Pod stopped. Session cannot be resumed.