Page MenuHomePhabricator
Create Task
Maniphest T220267

Flask-SQLAlchemy doesn't work properly at Kubernetes Python backend
Closed, ResolvedPublic
Actions

Description

See the commands below. I've first reported this to #wikimedia-cloud, and @aborrero recommended me to create a Phab ticket, so here it is.

FYI, venv and venv-bastion have the same packages installed (list is here), the only difference is that venv was created on tools-interactive, while venv-bastion was created at the bastion using virtualenv -p python3.

tools.wikinity@tools-sgebastion-07 ~/wikinity
$ source venv-bastion/bin/activate
(venv-bastion) tools.wikinity@tools-sgebastion-07 ~/wikinity
$ cd src
(venv-bastion) tools.wikinity@tools-sgebastion-07 ~/wikinity/src
$ flask shell
/mnt/nfs/labstore-secondary-tools-project/wikinity/www/python/venv-bastion/lib/python3.5/site-packages/flask_sqlalchemy/__init__.py:794: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True or False to suppress this warning.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
Sending requests with default User-Agent.  Set 'user_agent' on mwoauth.flask.MWOAuth to quiet this message.
Python 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516] on linux
App: app [production]
Instance: /mnt/nfs/labstore-secondary-tools-project/wikinity/www/python/src/instance
>>> from app import StoredMap
>>> StoredMap.query.all()
[]
>>>
(venv-bastion) tools.wikinity@tools-sgebastion-07 ~/wikinity/src
$ deactivate
tools.wikinity@tools-sgebastion-07 ~/wikinity/src
$ webservice --backend=kubernetes python shell
Defaulting container name to interactive.
Use 'kubectl describe pod/interactive -n wikinity' to see all of the containers in this pod.
If you don't see a command prompt, try pressing enter.
$
tools.wikinity@interactive ~
$ cd wikinity
tools.wikinity@interactive /data/project/wikinity/wikinity
$ source venv/bin/activate
(venv) tools.wikinity@interactive /data/project/wikinity/wikinity
$ cd src
(venv) tools.wikinity@interactive /data/project/wikinity/wikinity/src
$ flask shell
/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/flask_sqlalchemy/__init__.py:794: FSADeprecationWarning: SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and will be disabled by default in the future.  Set it to True or False to suppress this warning.
  'SQLALCHEMY_TRACK_MODIFICATIONS adds significant overhead and '
Sending requests with default User-Agent.  Set 'user_agent' on mwoauth.flask.MWOAuth to quiet this message.
Python 3.4.2 (default, Feb  7 2019, 06:08:06)
[GCC 4.9.2] on linux
App: app [production]
Instance: /data/project/wikinity/www/python/src/instance
>>> from app import StoredMap
>>> StoredMap.query.all()
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 3161, in all
    return list(self)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 3317, in __iter__
    return self._execute_and_instances(context)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 3339, in _execute_and_instances
    querycontext, self._connection_from_session, close_with_result=True
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 3354, in _get_bind_args
    mapper=self._bind_mapper(), clause=querycontext.statement, **kw
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/orm/query.py", line 3332, in _connection_from_session
    conn = self.session.connection(**kw)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 1123, in connection
    execution_options=execution_options,
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 1129, in _connection_for_bind
    engine, execution_options
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/orm/session.py", line 430, in _connection_for_bind
    conn = bind._contextual_connect()
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2226, in _contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 2262, in _wrap_pool_connect
    return fn()
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/pool/base.py", line 354, in connect
    return _ConnectionFairy._checkout(self)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/pool/base.py", line 751, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/pool/base.py", line 483, in checkout
    rec = pool._do_get()
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/pool/impl.py", line 138, in _do_get
self._dec_overflow()
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
    raise value
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/pool/impl.py", line 135, in _do_get
    return self._create_connection()
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/pool/base.py", line 299, in _create_connection
    return _ConnectionRecord(self)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/pool/base.py", line 428, in __init__
    self.__connect(first_connect_check=True)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/pool/base.py", line 640, in __connect
    ).exec_once(self.connection, self)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/event/attr.py", line 287, in exec_once
    self(*args, **kw)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/event/attr.py", line 297, in __call__
    fn(*args, **kw)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/util/langhelpers.py", line 1443, in go
    return once_fn(*arg, **kw)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/strategies.py", line 199, in first_connect
    dialect.initialize(c)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/dialects/mysql/base.py", line 2349, in initialize
    self._detect_sql_mode(connection)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/dialects/mysql/base.py", line 2796, in _detect_sql_mode
    connection.execute("SHOW VARIABLES LIKE 'sql_mode'"),
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 982, in execute
    return self._execute_text(object_, multiparams, params)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1155, in _execute_text
    parameters,
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
    e, statement, parameters, cursor, context
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
    raise value.with_traceback(tb)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
    cursor, statement, parameters, context
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
    cursor.execute(statement, parameters)
  File "/data/project/wikinity/www/python/venv/lib/python3.4/site-packages/MySQLdb/cursors.py", line 203, in execute
    raise ProgrammingError(str(m))
sqlalchemy.exc.ProgrammingError: (MySQLdb._exceptions.ProgrammingError) unsupported operand type(s) for %: 'bytes' and 'tuple'
[SQL: SHOW VARIABLES LIKE 'sql_mode']
(Background on this error at: http://sqlalche.me/e/f405)
>>>
(venv) tools.wikinity@interactive /data/project/wikinity/wikinity/src
$ deactivate
tools.wikinity@interactive /data/project/wikinity/wikinity/src
$ logout
Session ended, resume using 'kubectl attach interactive -c interactive -i -t' command when the pod is running
Pod stopped. Session cannot be resumed.

tools.wikinity@tools-sgebastion-07 ~/wikinity/src
$

Related Objects

Event Timeline

Urbanecm created this task.Apr 6 2019, 1:51 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptApr 6 2019, 1:51 PM
Urbanecm updated the task description. (Show Details)Apr 6 2019, 2:16 PM
Urbanecm added a comment.Apr 6 2019, 2:23 PM

FTR, I've temporarily converted wikinity to run using the Gridengine webservice, where everything works as expected. wikinity-test runs the same code, but via kubernetes, you can test there if you want/need to :).

An interesting thing I noticed during installing packages on the tools-interactive host is a lot of messages like ext/_yaml.h:2:18: fatal error: yaml.h: No such file or directory, and then Error compiling module, falling back to pure Python. P8362 contains the full log from instalation. Maybe it can guide you to the cause of this error?

Urbanecm added a project: Wikinity.Apr 6 2019, 2:24 PM
Urbanecm moved this task from Backlog to Blocked/External on the Wikinity board.
bd808 subscribed.Apr 18 2019, 5:25 AM
In T220267#5090418, @Urbanecm wrote:

An interesting thing I noticed during installing packages on the tools-interactive host is a lot of messages like ext/_yaml.h:2:18: fatal error: yaml.h: No such file or directory, and then Error compiling module, falling back to pure Python. P8362 contains the full log from instalation. Maybe it can guide you to the cause of this error?

This is expected. We do not install many development libraries in the python Kubernetes Docker containers. PyYAML is doing the right thing by trying to build the libyaml accelerated code and falling back to the pure Python implementation when that fails.

bd808 added a comment.Apr 18 2019, 5:32 AM

The bug here seems to be in improper python3 string handling inside the mysqlclient library that you are using. It looks like you have the same version (mysqlclient==1.4.2.post1) installed in both environments. I'm not sure why it would work on Python 3.5.3 and fail on Python 3.4.2, but that seems to be the case.

We should have a Python 3.5 Kubernetes container soon (T219091: toolforge: Update python image to Stretch). Waiting for that to be made available and then testing again may be the easiest solution right now.

bd808 updated the task description. (Show Details)Aug 8 2019, 3:53 AM

@Urbanecm Python 3.5 has been available for Kubernetes containers for a while now. You could try rebuilding your test environment using that and see if this mystery bug disappears when using it.

aborrero closed this task as Resolved.Nov 22 2019, 10:19 AM
aborrero claimed this task.
aborrero triaged this task as Medium priority.
aborrero moved this task from Backlog to Tracking on the Toolforge board.

Closing task now. Please reopen if required.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits