Page MenuHomePhabricator

Wikilabels: SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
Closed, ResolvedPublic

Description

I run the following commands to test wikilabels on Ubuntu 20.04:

$ git clone git@github.com:wiki-ai/wikilabels.git
$ docker-compose up -d --build
$ sudo apt install npm
$ docker-compose exec postgres psql wikilabels -U wikilabels -f /wikilabels/wikilabels/database/schema.sql
$ docker-compose exec postgres psql wikilabels -U wikilabels -f /wikilabels/wikilabels/database/schema-testdata.sql
$ docker-compose up

Then I went to http://localhost:8080/, where I've clicked on /ui/ and then clicked on ptwiki. After a few seconds, it shows an error page with this:

ylibmc.ServerDown
pylibmc.ServerDown: error 47 from memcached_set: (0x7fac840ad980) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: 127.0.0.1:11211 -> libmemcached/connect.cc:720

Traceback (most recent call last)
File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2334, in __call__
return self.wsgi_app(environ, start_response)
File "/usr/local/lib/python3.5/site-packages/beaker/middleware.py", line 156, in __call__
return self.wrap_app(environ, session_start_response)
File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2324, in wsgi_app
return response(environ, start_response)
File "/usr/local/lib/python3.5/site-packages/werkzeug/wrappers/base_response.py", line 699, in __call__
start_response(status, headers)
File "/usr/local/lib/python3.5/site-packages/beaker/middleware.py", line 150, in session_start_response
session.persist()
File "/usr/local/lib/python3.5/site-packages/beaker/session.py", line 834, in persist
self._session().save()
File "/usr/local/lib/python3.5/site-packages/beaker/session.py", line 483, in save
self.namespace['session'] = data
File "/usr/local/lib/python3.5/site-packages/beaker/ext/memcached.py", line 205, in __setitem__
self.set_value(key, value)
File "/usr/local/lib/python3.5/site-packages/beaker/ext/memcached.py", line 202, in set_value
mc.set(self._format_key(key), value)
pylibmc.ServerDown: error 47 from memcached_set: (0x7fac840ad980) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY, host: 127.0.0.1:11211 -> libmemcached/connect.cc:720
The debugger caught an exception in your WSGI application. You can now look at the traceback which led to the error.
To switch between the interactive traceback and the plaintext one, you can click on the "Traceback" headline. From the text traceback you can also create a paste of it. For code execution mouse-over the frame you want to debug and click on the console icon on the right side.

You can execute arbitrary Python code in the stack frames and there are some extra helpers available for introspection:

dump() shows all variables in the frame
dump(obj) dumps all that's known about the object

and in the terminal I see this:

$ docker-compose up
wikilabels_postgres_1 is up-to-date
wikilabels_wikilabels_1 is up-to-date
Attaching to wikilabels_postgres_1, wikilabels_wikilabels_1
postgres_1    | The files belonging to this database system will be owned by user "postgres".
postgres_1    | This user must also own the server process.
postgres_1    | 
postgres_1    | The database cluster will be initialized with locale "en_US.utf8".
postgres_1    | The default database encoding has accordingly been set to "UTF8".
postgres_1    | The default text search configuration will be set to "english".
postgres_1    | 
postgres_1    | Data page checksums are disabled.
postgres_1    | 
postgres_1    | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1    | creating subdirectories ... ok
postgres_1    | selecting dynamic shared memory implementation ... posix
postgres_1    | selecting default max_connections ... 100
postgres_1    | selecting default shared_buffers ... 128MB
postgres_1    | selecting default time zone ... UTC
postgres_1    | creating configuration files ... ok
postgres_1    | running bootstrap script ... ok
postgres_1    | sh: locale: not found
postgres_1    | 2020-05-11 18:19:39.115 UTC [30] WARNING:  no usable system locales were found
postgres_1    | performing post-bootstrap initialization ... ok
postgres_1    | syncing data to disk ... ok
postgres_1    | 
postgres_1    | 
postgres_1    | Success. You can now start the database server using:
postgres_1    | 
postgres_1    |     pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1    | 
postgres_1    | initdb: warning: enabling "trust" authentication for local connections
postgres_1    | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1    | --auth-local and --auth-host, the next time you run initdb.
postgres_1    | waiting for server to start....2020-05-11 18:19:39.537 UTC [35] LOG:  starting PostgreSQL 12.2 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.2.0) 9.2.0, 64-bit
postgres_1    | 2020-05-11 18:19:39.538 UTC [35] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1    | 2020-05-11 18:19:39.548 UTC [36] LOG:  database system was shut down at 2020-05-11 18:19:39 UTC
postgres_1    | 2020-05-11 18:19:39.550 UTC [35] LOG:  database system is ready to accept connections
postgres_1    |  done
postgres_1    | server started
postgres_1    | CREATE DATABASE
postgres_1    | 
postgres_1    | 
postgres_1    | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1    | 
postgres_1    | waiting for server to shut down....2020-05-11 18:19:39.691 UTC [35] LOG:  received fast shutdown request
postgres_1    | 2020-05-11 18:19:39.692 UTC [35] LOG:  aborting any active transactions
postgres_1    | 2020-05-11 18:19:39.692 UTC [35] LOG:  background worker "logical replication launcher" (PID 42) exited with exit code 1
postgres_1    | 2020-05-11 18:19:39.693 UTC [37] LOG:  shutting down
postgres_1    | 2020-05-11 18:19:39.699 UTC [35] LOG:  database system is shut down
postgres_1    |  done
postgres_1    | server stopped
postgres_1    | 
postgres_1    | PostgreSQL init process complete; ready for start up.
postgres_1    | 
postgres_1    | 2020-05-11 18:19:39.797 UTC [1] LOG:  starting PostgreSQL 12.2 on x86_64-pc-linux-musl, compiled by gcc (Alpine 9.2.0) 9.2.0, 64-bit
postgres_1    | 2020-05-11 18:19:39.797 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1    | 2020-05-11 18:19:39.797 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1    | 2020-05-11 18:19:39.798 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1    | 2020-05-11 18:19:39.808 UTC [46] LOG:  database system was shut down at 2020-05-11 18:19:39 UTC
postgres_1    | 2020-05-11 18:19:39.810 UTC [1] LOG:  database system is ready to accept connections
wikilabels_1  |  * Serving Flask app "wikilabels" (lazy loading)
wikilabels_1  |  * Environment: production
wikilabels_1  |    WARNING: This is a development server. Do not use it in a production deployment.
wikilabels_1  |    Use a production WSGI server instead.
wikilabels_1  |  * Debug mode: on
wikilabels_1  | /usr/local/lib/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
wikilabels_1  |   """)
wikilabels_1  |  * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
wikilabels_1  |  * Restarting with stat
wikilabels_1  | /usr/local/lib/python3.5/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
wikilabels_1  |   """)
wikilabels_1  |  * Debugger is active!
wikilabels_1  |  * Debugger PIN: 168-099-702
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:01] "GET / HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:01] "GET /oojs-ui-static/oojs-ui-mediawiki.css HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:01] "GET /static/css/wikilabels-site-specific.css HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:01] "GET /static/css/common.css HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:01] "GET /static/lib/wmui-style-guide.min.css HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:01] "GET /static/css/home.css HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:03] "GET /favicon.ico HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:05] "GET /ui/ HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:07] "GET /ui/ptwiki HTTP/1.1" 308 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:07] "GET /ui/ptwiki/ HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:07] "GET /oojs-static/oojs.jquery.js HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:07] "GET /oojs-ui-static/oojs-ui-mediawiki.min.js HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:07] "GET /oojs-ui-static/oojs-ui.min.js HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:07] "GET /static/gadget/packed.js?b49a2993 HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:07] "GET /static/lib/mediaWiki/diffs.css HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:07] "GET /oojs-ui-static/oojs-ui-mediawiki.min.css HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:07] "GET /static/gadget/packed.css?1aef7d66 HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:08] "GET /gadget/WikiLabels.messages.js HTTP/1.1" 200 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:11] "GET /w/load.php?modules=jquery%7Cmediawiki.base&skin=vector&version=1kums HTTP/1.1" 404 -
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:12] "GET /auth/whoami/ HTTP/1.1" 500 -
wikilabels_1  | Traceback (most recent call last):
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2334, in __call__
wikilabels_1  |     return self.wsgi_app(environ, start_response)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/middleware.py", line 156, in __call__
wikilabels_1  |     return self.wrap_app(environ, session_start_response)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2324, in wsgi_app
wikilabels_1  |     return response(environ, start_response)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/werkzeug/wrappers/base_response.py", line 699, in __call__
wikilabels_1  |     start_response(status, headers)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/middleware.py", line 150, in session_start_response
wikilabels_1  |     session.persist()
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/session.py", line 834, in persist
wikilabels_1  |     self._session().save()
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/session.py", line 483, in save
wikilabels_1  |     self.namespace['session'] = data
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/ext/memcached.py", line 205, in __setitem__
wikilabels_1  |     self.set_value(key, value)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/ext/memcached.py", line 202, in set_value
wikilabels_1  |     mc.set(self._format_key(key), value)
wikilabels_1  | pylibmc.ConnectionError: error 3 from memcached_set: (0x7fac7c02cdd0) CONNECTION FAILURE(Connection refused),  host: 127.0.0.1:11211 -> libmemcached/connect.cc:156
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:12] "GET /auth/whoami/ HTTP/1.1" 500 -
wikilabels_1  | Traceback (most recent call last):
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2334, in __call__
wikilabels_1  |     return self.wsgi_app(environ, start_response)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/middleware.py", line 156, in __call__
wikilabels_1  |     return self.wrap_app(environ, session_start_response)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2324, in wsgi_app
wikilabels_1  |     return response(environ, start_response)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/werkzeug/wrappers/base_response.py", line 699, in __call__
wikilabels_1  |     start_response(status, headers)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/middleware.py", line 150, in session_start_response
wikilabels_1  |     session.persist()
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/session.py", line 834, in persist
wikilabels_1  |     self._session().save()
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/session.py", line 483, in save
wikilabels_1  |     self.namespace['session'] = data
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/ext/memcached.py", line 205, in __setitem__
wikilabels_1  |     self.set_value(key, value)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/ext/memcached.py", line 202, in set_value
wikilabels_1  |     mc.set(self._format_key(key), value)
wikilabels_1  | pylibmc.ServerDown: error 47 from memcached_set: (0x7fac7c02cdd0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: 127.0.0.1:11211 -> libmemcached/connect.cc:720
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:14] "GET /auth/initiate/?wiki=ptwiki HTTP/1.1" 500 -
wikilabels_1  | Traceback (most recent call last):
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2334, in __call__
wikilabels_1  |     return self.wsgi_app(environ, start_response)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/middleware.py", line 156, in __call__
wikilabels_1  |     return self.wrap_app(environ, session_start_response)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/flask/app.py", line 2324, in wsgi_app
wikilabels_1  |     return response(environ, start_response)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/werkzeug/wrappers/base_response.py", line 699, in __call__
wikilabels_1  |     start_response(status, headers)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/middleware.py", line 150, in session_start_response
wikilabels_1  |     session.persist()
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/session.py", line 834, in persist
wikilabels_1  |     self._session().save()
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/session.py", line 483, in save
wikilabels_1  |     self.namespace['session'] = data
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/ext/memcached.py", line 205, in __setitem__
wikilabels_1  |     self.set_value(key, value)
wikilabels_1  |   File "/usr/local/lib/python3.5/site-packages/beaker/ext/memcached.py", line 202, in set_value
wikilabels_1  |     mc.set(self._format_key(key), value)
wikilabels_1  | pylibmc.ServerDown: error 47 from memcached_set: (0x7fac7c02cdd0) SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY,  host: 127.0.0.1:11211 -> libmemcached/connect.cc:720
wikilabels_1  | 172.18.0.1 - - [11/May/2020 18:33:14] "GET /auth/initiate/?__debugger__=yes&cmd=resource&f=style.css HTTP/1.1" 200 -

In case it is relevant, here a re some logs from the install:




Event Timeline

Restricted Application added a subscriber: Aklapper. · View Herald Transcript

Woah. Looks like this is something to do with memcached. I haven't seen this error before.

I can't seem to find a place where memcached is installed during the build.

I've tried to do this:

diff --git a/Dockerfile b/Dockerfile
index 5fc2d0d..19f39a5 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -5,7 +5,8 @@ RUN apt-get update && apt-get install -y \
     g++ \
     python3-dev \
     libmemcached-dev \
-    libz-dev
+    libz-dev \
+    memcached
 
 COPY . /wikilabels
 WORKDIR /wikilabels

but the error persisted :(

Hmm. I wonder if Docker will need to open up that port in order for us to be able to talk to it from uwsgi.

Halfak assigned this task to He7d3r.
Halfak moved this task from Parked to Completed on the Machine-Learning-Team (Active Tasks) board.

Thanks for your work on this, @He7d3r